Apple Vision Pro

Apple Vision Pro and Spatial Videos: A New Age of Immersive Memories with Spatial Computing

I built my first Augmented Reality (AR) prototype in 2005: a memory album. My early work explored cinema and video at a time when the AR industry was heavily focused on 3D models. I was among the first to introduce the use of 2D videos in AR.

I was captivated by the dream-like quality of a virtual moving image on a physical piece of paper. Low resolution polygons back then left me uninspired, the fidelity just wasn’t there. 2D videos in contrast seemed to heighten the illusion, they were somehow more realistic and magical, more human like, and even organic in comparison to the clunky, usually awkward 3D models in AR. There was something beautiful, romantic even about seeing an older technology (video) play back in a new technology (AR).

And here we are again with Apple Vision Pro and Spatial Videos, almost two decades later. Yes, 3D polygons are infinitely better today, but there’s still something completely spellbinding about immersive video and reliving memories in this way.

Apple Vision Pro Spatial Video

The AR memory album and virtual souvenirs I built back in 2005 presented digital 2D video footage from my travels. (For context, these were pre-iPhone days; the photos and videos I used to document my trips were all captured on a digital camera.) My interest in creating these AR experiences was due in part to a desire to capture brief live moments from my sojourns that were beyond still photographs; video clips temporarily transported me back to these locales to relive those moments. These moving images assisted to evoke and recollect a past vision of a particular location: once again seeing how the waves crashed, how the wind blew, how my body moved in a space which I no longer have physical access to.

I found that unlike the digital photographs that I took and would eventually print and place in an album, these moving images most often remained archived on a disc or on my computer never to be experienced again (remember, no iPhone Photos app to store these memories in your pocket back then). I wanted to create a tactile object where I could hold and view these live moments again, alongside my still photographs, offering an opportunity to move through the images, extending into and beyond their virtual viewing space.

Interestingly, as I look back, none of the moving images I chose to include featured people; they were all pans of landscapes. I viewed this as an opportunity to document the physical places I traversed, as a form of souvenir that would allow me to virtually revisit and enter that space again via a moving image that captured my field of vision in a horizontal pan. The absence of other people in the footage aided in creating an intimate, uninterrupted space, as though that particular moment was for me, undisturbed by anyone else: a private memory, between that place and me. And hence, I’m very excited to view panoramas on Vision Pro, which Apple describes as wrapping around you, “making you feel like you’re standing right where you took them.” A feeling and experience I sought to create from the start when the technology was so nascent. It’s thrilling what can be accomplished with the technology today!

Experiencing a panorama on Apple Vision Pro

Of particular interest to my work in AR in those early days — and reignited with Apple Vision Pro — is how emerging technologies have allowed us not only to see, but to *see again* that which is no longer, to recollect and to remember, and to recreate the past with verisimilitude. After all, the very nature of memories is virtual.

In the book, Memory Trade, Darren Tofts and Murray McKeich refer to philosopher Henri Bergson's, Matter and Memory. Tofts and McKeich write, "The apparent problem of how the past remains with us and yet is in actuality no longer present, is resolved through the identification of recollection as being virtual".

I referenced these passages in my doctoral work several years ago on AR as a new medium drawing parallels to the process within which the virtual videos in my memory albums came into view.

In Bergson's words: “Whenever we are trying to recover a recollection, to call up some period of our history, we become conscious of an act sui generis by which we detach ourselves from the present in order to replace ourselves, first in the past in general, then in a certain region of the past — a work of adjustment, something like the focusing of a camera. But our recollection still remains virtual; we simply prepare ourselves to receive it by adopting the appropriate attitude. Little by little it comes into view like a condensing cloud; from the virtual state it passes into the actual; and as its outlines become more distinct and its surface takes on colour, it tends to imitate perception.”

There’s such wonderment in Bergson’s description. It paints a beautiful image that is still very relevant today and even serves as a metaphor for Spatial Videos. I particularly enjoyed reading CNET, Editor at Large, Scott Stein recount his experience of viewing personal Spatial Videos on Vision Pro (captured on iPhone 15 Pro). “It made me want to climb through that fuzzy-bordered window and join my family again on the other side,” he wrote. (Stein, by the way, joined us as our special guest on RealityPl.us — Watch it on YouTube and listen on Apple Podcasts).

“Apple's choice to frame the spatial videos in a fuzzy sort of frame makes it look more like the videos are almost holographically being cast into the room, and the edges sort of dissolve a bit,” writes Stein. I think that softness is also echoed in the way we will likely gaze at and observe these memories.

“Their 3D, almost ghostly playback quality feels like it's aiming for an immersive memory more than a straight-up video playback,” Stein adds. There’s perhaps even some fragility here intentionally in the design of reliving, in the passing and preservation of time, experiencing a new poetic memory machine of sorts.

I’m reminded of that same ghost like quality the first time I saw my videos in AR in the memory album and how fragile these memories were in their virtual state. At times, the augmented video in my memory albums could ‘break’ if moved, or be temporarily interrupted revealing the background image of the AR marker (which may be thought of as a moment of distraction in recollection or even representative of the difficulty in attempting to place a memory or envision a particular image). These breaks, or fractures, as I described them in later work, also serve as reminders of how fleeting these past moments are and how they are only viewable in their temporary and instable state of (virtual) recollection.

(We see this delicate state of preserved memories in immersive virtual videos and momentary fractures depicted in films like Minority Report. My cohost Avi Bar-Zeev and I discuss the film with Design Fiction guru Julian Bleecker in our first episode of RealityPl.us.)

Snapchat Spectacles 3 with 3D Viewer

We’ve also seen the attempt at capturing and reliving memories through immersive technology previously with Snap Spectacles. Version 3 shipped in 2020 with a 3D viewer (essentially Google Cardboard). I still have mine. “With Spectacles 3, you can step back into a memory — and re-experience it using the included 3D Viewer.” There was also the option to export moments in YouTube’s VR180 format to experience them on any device that is compatible with YouTube VR — such as the included 3D Viewer, Google Cardboard, Oculus, and Vive.

When I began building AR experiences nearly 20 years ago, I was inspired by the magic and history of the moving image, Victorian stereoscopes, and the concept of the ‘Armchair Traveller.’

American Oliver Wendell Holmes’s Stereopticon (advertisement pictured below) dominated the market from the 1880s through the 1940s. In June 1859 Holmes wrote in Atlantic Monthly, “The first effect of looking at a good photograph through the stereoscope is a surprise such as no painting ever produced. The mind feels its way into the very depths of the picture. The scraggy branches of a tree in the foreground run out at us as if they would scratch our eyes out. The elbow of a figure stands forth as to make us almost uncomfortable.”

Late 19th-century advertisement for a stereoscope. New Orleans Museum of Art.

Photographers were being dispatched around the world by large companies such as the Keystone View Company to capture scenes previously only known to the public through illustrations in newspapers and magazines. From ancient wonders like the Great Pyramids of Giza to new engineering marvels like skyscrapers in New York, these amazing sights were now visible to those unable to travel such distances at great monetary expense.

New Orleans Museum of Art Curator Russell Lord explains, “While photography, in general, often provided access to the inaccessible, stereographs provided a palpable experience of far-away places, transforming the ‘there and then’ to the ‘here and now.’”

And here and now we are over a century later with a stereoscopic resurgence in the age of Spatial Computing. But this time it’s personal. I do agree with individuals like Om Malik that Vision Pro will change photography and this will only be heightened by documenting and reliving personal moments. As Stein shared in his experience with Spatial Videos, there’s something extra special that happens when the imagery you see is yours: your family, your friends, your pet, your special moments. I’m ready. Again.

Apple Vision Pro is available in the US starting February 2, 2024.