One of the most striking features of the Apple Vision Pro is viewing spatial photos and videos. It feels futuristic and the added depth immerses you in memories or new worlds. There's a problem though - in order to take them you either need to be in the Vision Pro or use an iPhone 15 Pro, which doesn't add much depth nor does it capture spatial photos.

We all have decades of personal photos and videos which would be awesome to view in spatial, which is why I built Spatial Media Toolkit - to convert those images and videos from 2D to 3D spatial. With Spatial Media Toolkit 2.0 on the Mac, this has gotten easier than ever with batch processing. Here's a quick overview of how to do it.

Download Spatial Media Toolkit from the Mac App Store and open the photos and videos you'd like to convert

Download here. Spatial Media Toolkit enables you to select from Photos, your filesystem or even dragging and dropping into the app from Finder or other apps (including websites). How you do it all depends on where you have things stored. If they're in iCloud, Spatial Media Toolkit will automatically download them.

Adjust the amount of depth and preview the conversion

In general, the default value gives you a decent amount of depth without being too extreme, but you can tweak this to your liking. It's hard to visualize on the Mac but Spatial Media Toolkit provides several visualization options:

• Wiggle: jumps back and forth between the views to get a 3D effect

• Crosseyed: you can cross your eyes, like a magic eye image, to get a sense of the depth added

• Parallel: hard to visualize but there are some steps here

Spatial Media Toolkit is also on the Vision Pro and you can get a feel for the depth slider on there with individual photos and videos. Previews in the Vision Pro app show real 3D spatial instantly as soon as you open a photo or video.

You can also change what frame you're previewing in a video by sliding the seek bar on the video preview.

Convert all and wait for them to complete

You can convert everything in the queue by using the convert button on the side bar, or you can convert individual videos by clicking on them on the side bar and clicking convert on the tool bar under the preview.

Photos convert quite quickly and will even already be completed for any you've previewed. Videos can take some time.

Saving and viewing on your Vision Pro

You can save all completed photos and videos using the side bar, or any individual item can be saved or shared from the toolbar under the preview.

Every Apple ID gets 5GB free of iCloud storage. The easiest way to get items to your Vision Pro is via iCloud.

Sync via iCloud Photos

Ensure you have iCloud Photos set up on your Mac and Vision Pro. Apple has an article here detailing how to do that.

In Spatial Media Toolkit, click the Photos button under the save section on the side bar, which will save all completed conversions to your Photos app.

Once Mac Photos app syncs, upon opening the Photos app on your Apple Vision Pro, you should see your new conversions in your Spatial album.

You may need to force the Mac Photos app to sync which you can do by scrolling to the bottom of the Library. There you'll see text regarding the state of your iCloud Photos sync, with an option to Sync Now if it hasn't.

Sync via iCloud Drive

By selecting Files next to the Photos button you clicked before, you can choose to save them all to a folder to sync which will be viewable in the Files app on the Vision Pro.

Send via Mail, Messages or AirDrop

Spatial content can also be transferred to your Apple Vision Pro by saving to Files or Photos and using any other form of file sharing MacOS supports. You can even AirDrop them all from Finder, or text them to friends with AVPs.

Converting for Viewing on Your Meta Quest

Once you've followed the steps above to convert, you can follow the saving steps above to get them to your iPhone or send them to your Android device.

You can transfer to your Quest by opening the Gallery in the Meta Horizons app on your smart phone. There you can select Spatial videos to share with your Quest device.

What Is Spatial Video? A Deep Dive into Immersive Video Technology

Spatial video is revolutionizing how we capture and experience digital memories. This cutting-edge format goes beyond traditional 2D video, offering an immersive 3D experience that brings your memories to life. In this post, we'll explore what spatial video is, how it works, and how you can start creating your own spatial content today.

Understanding Spatial Video

At its core, spatial video is a 3D video format that captures depth information along with visual data. Unlike standard 2D videos, spatial videos create a sense of depth and dimensionality when viewed on compatible devices, such as virtual reality (VR) or augmented reality (AR) headsets.

Technical Implementation

Spatial video relies on several key technologies:

1. Multiview HEVC (MV-HEVC):

- This is the primary codec used for spatial video.

- MV-HEVC encodes multiple views (typically left and right eye views) into a single bitstream.

- It leverages inter-view prediction to efficiently compress the multiple viewpoints.

2. Stereoscopic Capture:

- Spatial videos are typically recorded using two cameras or lenses, simulating human binocular vision.

- These dual feeds capture slightly different perspectives of the same scene.

3. Depth Mapping:

- Some spatial video systems incorporate depth sensors or use AI algorithms to generate depth maps.

- These depth maps provide additional 3D information, enhancing the immersive experience.

4. Metadata:

- Spatial videos include metadata that describes how to display the content in 3D.

- This metadata ensures proper rendering on compatible devices.

Apple's Spatial Video Implementation

Apple has recently introduced spatial video capture with the iPhone 15 Pro and iPhone 15 Pro Max. Here's how their system works:

- Uses both the main and ultra-wide cameras in landscape orientation.

- Records at 1080p and 30 frames per second.

- Employs computational photography to align and process the dual camera feeds.

- Saves videos in HEVC format, with a file size of about 130MB per minute.

- Can be played as regular 2D videos on non-compatible devices.

- Designed for viewing on the upcoming Apple Vision Pro headset.

Viewing Spatial Videos

The true magic of spatial video comes alive on compatible devices like VR headsets. On Apple's Vision Pro, for example:

- Videos can be displayed at life-size scale.

- Viewers can choose between windowed playback or an immersive full-view mode.

- The 3D effect creates a sense of presence, as if looking through a window into the recorded moment.

Creating Spatial Videos with Spatial Media Toolkit

While native spatial video capture is currently limited to high-end devices like the iPhone 15 Pro, there's a powerful alternative available: the Spatial Media Toolkit. This innovative software allows you to convert your existing 2D videos into immersive spatial format.

Key features of the Spatial Media Toolkit:

1. AI-Powered Conversion:

- Utilizes advanced machine learning algorithms to analyze 2D video and generate depth information.

- Creates a convincing 3D effect from standard video input.

2. Compatibility:

- Converts videos for viewing on various VR and AR platforms, not just Apple's ecosystem.

- Supports a wide range of input video formats.

3. Customization:

- Offers adjustable parameters for depth effects.

- Allows fine-tuning to achieve the best results for each video.

4. Preview and Export:

- Provides real-time preview of the 3D effect within the app.

- Exports in formats compatible with Vision Pro and Meta Quest devices

5. Batch Processing:

- Convert entire libraries of 2D videos to spatial format efficiently.

6. High-Quality Output:

- Supports 4K and beyond resolution for both input and output.

7. Cross-Platform Availability:

- Available on iOS, macOS, and visionOS.

- Boasts a 4.8-star rating with over 500 ratings on the Vision Pro App Store.

The Spatial Media Toolkit democratizes spatial video creation, allowing anyone with a collection of 2D videos to enter the world of immersive content. Whether you're looking to breathe new life into old memories or create engaging 3D content for VR platforms, this tool bridges the gap between traditional and spatial video.

The Future of Spatial Video

As technology advances, we can expect improvements in spatial video:

- More devices capable of native spatial video capture.

- Enhanced AI for even more realistic 2D to 3D conversion.

Spatial video represents the next frontier in personal media, offering a more immersive and lifelike way to capture and relive our memories. While native capture options are currently limited, tools like the Spatial Media Toolkit make it possible for anyone to start creating and experiencing spatial content today. As this technology continues to evolve, it's set to transform how we share and experience our digital memories.

Spatial Media Toolkit pioneered 2D to 3D conversion on the Vision Pro. Convert your videos and photos to 3D spatial, right on your AVP as well as your other Apple devices and relive your memories in 3D.

TLDR; I have 2 highly requested features I'd love your feedback on, please join the TestFlight and email me at feedback at strangemesa.com with any bugs/feedback/suggestions:

• Batch video conversion [macOS, visionOS] [iOS coming soon]

  • After selecting a video on the Conversion tab, you can opt to Enqueue the video to render from the Queue tab. Queue up a bunch, set it and forget it.

• Realtime spatial playback (no conversion necessary!) [visionOS]

  • Select a 2D video from the Gallery tab and tap the Spatial button in the upper left corner of the video to play your video in spatial without having to convert it first.

  • Some corners were cut for this to happen in realtime, so the quality is lower than the full conversion.

When on the TestFlight build, any Pro features are free. There will be an Apple payment dialogue but you'll note it says that it is for testing purposes only, and that no money will be charged. If you're already a paid user, this will not effect your subscription/lifetime purchase and switching back to the App Store version will be seamless.

Some next steps:

• Batch video selection

  • Instead of having to enqueue one by one, allow for multi-select

• Batch save

  • I took the fastest possible route to getting batch processing, so this just queues them up. Right now you'll still need to save them one by one after conversion completes.

• Batch photo conversion

  • Photos currently need to be done one by one. Conversion happens within milliseconds so queueing isn't necessary, but would be nice to be able to select a bunch, automatically save them all out in one fell swoop.

• Improved Gallery

  • Add a similar feature to the realtime spatial playback to the Gallery photos viewer where you can just browse everything in your Photos in 3D without having to convert first. Build an experience like the Photos app, but everything is spatial even if it's originally 2D

• Light editing functionality

  • Trim, crop, and compose multiple spatial videos together

Thank you all for the support!

The launch of the Apple Vision Pro has popularized a couple new formats they call Spatial. There are two different formats: Spatial Photos and Spatial Videos. Spatial Photos take advantage of the HEIF standard, and include two different labeled images in one HEIF file representing the left and right eye views. Spatial Videos are extensions to the HEVC encoder stored in mp4 or mov containers called MV-HEVC or multi-view HEVC.

If you have existing VR content and you are a bit tech savvy, you can take your content and a great tool by Mike Swanson called spatial. You can read the documentation to convert images to Spatial.

Alternatively, if you have an iPhone, Mac or Vision Pro, we build an app called Spatial Media Toolkit. Spatial Media Toolkit not only can convert your VR content to Spatial photos and videos, but it can also convert your normal 2D photos and videos to Spatial using AI. You can download it here.

After converting, you can send it to your Apple Vision Pro or Meta Quest to view your photos and videos as 3D Spatial.