VR and the Power of Immersive Storytelling — Inside COVID19 with Gary Yost
The impact of COVID-19 across the world has been undoubtedly severe. It’s important to preserve and share people’s stories for the sake of remembrance, and because experiences are transformative. This is obvious, but worth mentioning because as art continues to evolve, new and traditional methods are refined.
For example, virtual reality and other forms of augmented reality. Together, they’ve impacted not only the cinematic world but also industries such as healthcare, video games, tourism, space exploration, and education.
Bringing several of those fields together is filmmaker and software designer Gary Yost, founder of the WisdomVR project dedicated to “creating a living library of immersive cinematic experiences and an open source, global educational platform.”
Through WisdomVR, Gary and his team created Inside COVID19, a virtual reality documentary that allows viewers to experience working on the front lines of the pandemic first-hand, following alongside American doctor Dr. Josiah Child.
Inside COVID19 explores how the coronavirus spreads in the body and through an increasingly connected population, taking a close look at social, political, and medical responses to COVID-19. The documentary was nominated for an Emmy Award for Outstanding Interactive Program — 2021. Below is my interview with Gary.
Andrew Cheek: First, could you share a little background about the Inside COVID19 documentary and how it came into being?
Gary Yost: Inside COVID19 grew out of our mission at the WisdomVR Project, which is to capture personal stories using this amazing medium of presence that has the potential to connect in the deepest way possible. The idea behind all of our work is that after you’ve seen it and take the headset off, you feel more connected to the world than you were before. That’s the core reason why we’re not making regular films… connection.
The story of the pandemic is in many ways a story of isolation, confusion and our search for meaning within the past 18 months of chaos. With traditional 2D filmmaking, viewers watch a story unfold on a screen (that these days is usually very small), giving them lots of opportunities to be distracted by other things happening in their lives. With stereoscopic immersive VR360 filmmaking, viewers are completely surrounded by the subject’s life with no opportunity to look away. This brings an unrivaled intensity to the experience and with Inside COVID19, witnessing Dr. Child’s life is so powerful that some people have told us that it has almost too much impact for them… it’s TOO intense. No other medium can do that and we use that intensity to cut through the confusion.
Did you have a desired effect or goal in mind with Inside COVID19?
VR films have the ability to transport you directly into the subject’s world and we capitalized on how open Dr. Child was, giving us access to his work, home, family and even his private CT scan data. As you sit in his living room with his wife and two children as they directly share his near-death experience, it’s clear that VR provides the highest level of emotional impact.
Many people might not know someone who’s been so deeply affected by COVID-19 but after watching our piece you’ll come away from it with the visceral feeling that you do. In what other medium can you become embodied inside one of your epithelial cells and watch a virus enter and begin replicating right in front of you?
Inside COVID19 is a detailed record of a physician’s experience both treating and suffering through what will be remembered as a pivotal moment in modern human history. By presenting a step-by-step recording of how the pandemic evolved in 2020, we’re helping future generations remember exactly how little we knew when all of this began and how rapidly the global community came together.
We believe the best path into any topic is through a personal story. The story of this pandemic is huge, but the medium of immersive video is intimate. When we have a personal connection to someone who’s suffered, it orients us into the pandemic in a way that no other experience can. That moves people to take steps to keep themselves and their families safer because it’s not an abstraction anymore. Our main hope is that this film will help reinforce public health messaging and drive vaccine acceptance.
What has the response been like?
Of course it felt great to be nominated for an Emmy award and we’ve had a tremendous amount of interest from the press. Anytime that you’re recognized by a group of peers for your creative effort it’s an affirmation that there’s meaning in your work outside of the personal satisfaction it provides. And in the case of the Emmy, we were nominated along with two of our immersive cinematic heroes… Félix Lajeunesse & Paul Raphaël from Felix & Paul Studios. Their groundbreaking work developing new 360 video techniques has been the inspiration for everyone in this field. Their Space Explorers: ISS experience is such a tour de force that it deserved to win the award! For us the nomination itself was the win.
The wide recognition we’ve gotten for Inside COVID19 gives the story more reach, touches more people’s lives and hopefully makes people safer as we enter the next chapter of the pandemic. The response we’ve seen shows us that our WisdomVR stories, both cultural and scientific, are contributing to a strong and healthy society.
What new perspective did you want to share on the pandemic — or anything else, for that matter — with this documentary?
Not very many people personally know someone who’s been physically devastated by the SARS-CoV-2 virus. Our goal has been to make this real, personal and visceral. Inside COVID19 is a detailed record of a physician’s experience both treating and suffering through what will be remembered as a pivotal moment in modern human history. By presenting a step-by-step recording of how the pandemic evolved in 2020, we’re helping future generations remember exactly how little we knew when all of this began and how rapidly the global community came together.
At the start of the pandemic we knew that the original mission of WisdomVR’s project to preserve elders’ memories was more important than ever, but like so many we were at a loss for how to safely continue or what to do next. My directing partner Adam Loften and I were both very concerned about the mysterious news from China and by the time the virus arrived in the U.S. we were compulsively trying to learn as much as possible to stay safe. Beyond the fear, it was also a fascinating moment and once we met Dr. Child, we knew that he could be our guide through this tremendously challenging time.
The global tragedy has filled us with urgency that this is a transformative moment. That urgency itself inspired us and Inside COVID19 is our contribution to this transformation. It is also the only immersive record of the pandemic yet made, and the only experience in which you can both shrink down to the molecular level to watch the SARS-CoV-2 virus replicate in one of your cells and then zoom out into space and look back on our beautiful planet.
What is it about VR as a medium that made you want to use it when making Inside COVID19?
Our mission is to capture personal stories using this amazing medium of presence with the potential to connect in the deepest way possible. The idea behind all of our work is that after you’ve seen it and take the headset off, you feel more connected to the world than you were before. That’s the core reason why we’re not making regular films… connection.
The story of the pandemic is in many ways a story of isolation, confusion and our search for meaning within the past 18 months of chaos. With traditional 2D filmmaking, viewers watch a story unfold on a screen (that these days is usually very small), giving them lots of opportunities to be distracted by other things happening in their lives. With stereoscopic immersive VR360 filmmaking, viewers are completely surrounded by the subject’s life with no opportunity to look away. This brings an unrivaled intensity to the experience and with Inside COVID19, witnessing Dr. Child’s life is so powerful that some people have told us that it has almost too much impact for them… it’s TOO intense. No other medium can do that and we use that intensity to cut through the confusion.
The impact of this experience is impossible to convey with words. When you experience this story about the pandemic in a headset, completely surrounded by hyper-real stereoscopic 360-degree imagery, you become a true voyeur… a being embodied inside the story. VR films transport you directly into the subject’s world and we capitalize on how open Dr. Child was, giving us intimate access to his work, home, family and even his private CT scan data. We used those scans to create 3D animations, flying deep inside his chest to see the massive damage COVID-19 caused in his lungs… can you get more intimate than that?
How do you think VR is revolutionizing filmmaking? And in the same vein, how do you think filmmaking and storytelling will change in the coming decades, whether that’s from VR or things like cryptocurrency and NFTs?
These are big questions! To answer your first one, ever since people began telling stories around the fire in the earliest time in our history, storytelling has always been about creating the most immersive story possible. We used whatever tools we had at our disposal, from making shadows on the wall of a cave, to painting dioramas on the walls of theaters in the 19th century (this is what Louis Daguerre did before he became one of the inventors of photography). Many people don’t realize it, but before his interest in photography, Daguerre was obsessed with the dynamics of light and how it could be used to create immersive panoramic imagery. In the 19th century, panoramic paintings were a popular form and people would line up to view these huge landscapes or battle scenes as if they were theater. Through his groundbreaking works of diorama, Daguerre went beyond the panorama by using light to turn his huge paintings into scenes of movement and life.
What we’re trying to do now is just the same… to fully immerse the viewer into a new world, and although it’s still not a perfect medium, it does that more successfully than in the 16:9 flat rectilinear filmmaking domain.
Our capture system array uses 10 individual camera modules, tightly packed into a radial array, to capture a stereoscopic representation of everything happening around the viewer.
The changes for documentary VR filmmaking in the near future will come from new camera arrays that capture lightfields, allowing for viewers to move around in their seats and see accurate parallax effects as they move. And going way beyond that there will eventually be full 3D scanning techniques that capture real-time events at roomscale in full 3D, allowing for viewers to walk around a captured (non CGI) scene. While the lightfield technique is both practical and doable (and I’ve seen it), the roomscale 3D capture technique is probably nothing I’ll see (outside of maybe a tech demo) in my lifetime.
What are your thoughts on the limitations and challenges of VR and other forms of extended reality, especially in terms of accessibility?
Virtual Reality, as a headset-based medium, is fraught with limitations and challenges, and here are some:
Headsets are expensive and until they are $150 add-ons to your phone (like a pair of wireless earbuds or a smart watch) they will continue to be out of reach of most people.
The nature of putting screens an inch away from your eyeballs is fraught with optical issues and the current methods of using fresnel lenses is far from optimal due to everything outside the sweet spot being blurry.
There is currently no way of capturing even 180° video with full 6 Degree-of-Freedom coverage, which means that, for non CGI sources like documentary films, a viewer must keep their head in the same place, constrained to only rotate 360°, or the world will appear to move with the viewer when they translate their head on the XYZ axes. Fixing this requires light-field capture methods, along with the editing tools to work with them, providing at least a meter of XYZ translation so a viewer, in their seat, can comfortably move and see natural parallax in real-time.
Heat and power consumption are going to continue to be challenges as headset display technology gets closer and closer to retina resolution, with 120hz update frequency.
Headset-based AR challenges go much further and will take even longer to solve than for VR. Until then, camera passthrough methods for VR headsets will allow AR developers to experiment with mixed reality.
How do you think art functions as an instrument of change in the world?
WisdomVR believes in this new art form because it allows unmediated experiences of people and places, preserving them in situ for generations to come. Understanding the highest potential of this storytelling medium is something we’re devoting the rest of my life to.
As millions of new headsets are entering people’s lives around the world, a mass environmental and cultural extinction is simultaneously taking place. VR360 is an opportunity to preserve and share precious people, places and cultures, especially those that are most vulnerable. WisdomVR is dedicated to two things: 1) refining the tools to create the highest-quality impactful VR360 documentaries and 2) sharing everything we learn along the way so that others can preserve their own precious heritage and cultures. There’s a huge gap in the VR space for these types of humanities experiences and we want to change that. The audience of viewers has a hunger for meaning and even if early adopters aren’t conscious of it now, this immersive medium will drive a new wave of humanities and social impact.
Our long-term goal is to create a living library of immersive cinematic documentary experiences where people can sit with amazing people like Dr. Child and have a 1-to-1 communication of wisdom. In addition to creating discussion guides for each experience, we are also developing an open-source VR360 heritage documentation framework for the public good.
Precious stories are being lost and we can help to make sure that they’re not forgotten. We hope to have many collaborators — our nonprofit’s mission is to share everything we know and we’re openly looking for partners to make all of this possible.
We are all born storytellers, dependent upon our cultures to carry our evolving wisdom to future generations. New tools for the preservation of the wisdom of the world are emerging in a library that anyone can experience and add to, as well as a curriculum where people can use the benefit of our experience to preserve their own heritage. Inside COVID19 is one part of the bigger library that we’ve been creating since 2018.
The emergent properties of 360 filmmaking, cultural storytelling and the evolving technology of headsets is our once-in-a-lifetime opportunity to share what happened in 2020 with our descendants.
What inspires your filmmaking, and your work with software design and VR?
I get the most inspiration from waking up each morning without knowing how I’m going to accomplish the tasks of the day. This has always been the driver of my creativity and this feeling of rising to the challenge started for me in the early days of 3D animation software development when I had the huge responsibility of creating “the AutoCAD of 3D animation” for Autodesk. As the years progressed into the process I became more and more excited about learning the thing I needed to learn that day to get that day’s work done. What I didn’t expect was that after a decade of doing that and I began to feel like I really knew what I was doing, that’s when I began to get bored and wanted to move on to a new challenge.
VR documentary filmmaking is in its infancy and everybody in this field is learning each day how to do it in the best way possible. In some respects it’s similar to how early filmmakers at the turn of the 20th century had to “make it up” as they went, learning and experimenting in the process of developing a new artform. We are in that space of learning which excites me in the same way that 3D software development did!
I’ll share with you a quote that has lived on the wall of my studio for the past 15 years, from the great American poet Wendell Berry. It speaks to this issue of rising to the challenge and the excitement of being in that unknown confused space of beginner’s mind. I read this every day, it is my north star:
“There are, it seems, two muses: the Muse of Inspiration, who gives us inarticulate visions and desires, and the Muse of Realization, who returns again and again to say “It is yet more difficult than you thought.” This is the muse of form. It may be then that form serves us best when it works as an obstruction, to baffle us and deflect our intended course.
It may be that when we no longer know what to do, we have come to our real work and when we no longer know which way to go, we have begun our real journey. The mind that is not baffled is not employed. The impeded stream is the one that sings.”
Are you working on any new projects at the moment, related to COVID-19 and your work with WisdomVR or otherwise?
Inside COVID19 is just the latest immersive film from the WisdomVR Project. Our mission is to preserve the wisdom of our time for future generations using the most immersive medium possible. We see our role as documentarians to be conduits for the most pressing social, environmental and spiritual stories of our time and we learned that even in the face of a global pandemic you need to say yes to important and even potentially dangerous stories that deserve to be told.
VR headsets are finally affordable, mobile and becoming ubiquitous — two huge players are pushing hard on it. Facebook’s Reality Labs/Oculus division now has 10,000 people with an >$18B/year budget and Apple is also spending billions per year on their own upcoming VR ecosystem. While there’s a ton of focus on games and 6DoF synthetic experiences, in the immersive documentary film space there are only a handful of people focusing on making high-quality VR360 content.
My directing partner Adam Loften and I believe in this new art form because it allows unmediated experiences of people and places, preserving them in situ for generations to come. Understanding the highest potential of this storytelling medium is WisdomVR’s mission and something we’re devoting the rest of our lives to.”
One of our projects in development now is a project to provide tools and training to help preserve the land and cultural wisdom of California’s Indigenous peoples by creating virtual reality time capsules that allow future generations the ability to see and hear the unique traditional and ecological knowledge of their people. We’re also working on an ambitious project about the pandemic in which we feature the personal stories of other significant people in science and medicine, along with fully interactive 3D VR opportunities for viewers to explore how our immune system works to fight off viruses.
Is there anything else you’d like to add?
Immersive documentaries have the power to change people’s lives more than any other medium. Our goal at WisdomVR is to give people fully immersive experiences of those who carry wisdom in the world, and we will watch how this unfolds over the next 10 years with great enthusiasm.
Inside COVID19 is just the latest immersive film from the WisdomVR Project. Our mission is to preserve the wisdom of our time for future generations using the most immersive medium possible. We see our role as documentarians to be conduits for the most pressing social, environmental and spiritual stories of our time and we learned that even in the face of a global pandemic you need to say yes to important and even potentially dangerous stories that deserve to be told.
The success of Inside COVID19 galvanizes our commitment and dedication to the craft of VR360 documentary filmmaking and is an affirmation that this work matters. The true stories that we tell each other have the capacity to make us safe and strong… these stories have meaning and help people become resilient in the here and now.
Thanks for reading! For more from Gary, check out the WisdomVR project for immersive, diverse stories as well as the Inside COVID19 documentary.
