The merging of VR/AR films to the cinema industry

The central point of film narrative is to trigger the audience’s empathy and bring them closer to the story. The history of cinema began with the use of moving images to tell stories, then added color and sound, and gradually evolved to 3D, IMAX, CGI, and animation technologies that are now on display on the screen. Today, filmmakers are exploring VR/AR cinema and want to use VR/AR technology to dramatically increase immersive viewing. In traditional movies, whether 2D or 3D, the audience is just a spectator, but in VR/AR movies, the audience is a participant, and VR/AR has always been a highly cross-regional and cross-industry medium. If transcending 2D to 3D allows the viewer to have a sense of three-dimensional, then the feeling brought by VR/AR movies can theoretically be said to be disruptive, it not only allows the audience to get rid of the monitor or theater screen but also increases the interactive effect, the audience no longer sees the predetermined picture, their every move will have an interactive effect. In what follows, I discuss the development of VR/AR technology and its role in the film industry. First, I begin by talking about VR technology used in VR film production. Then, I introduce AR technology and its influences. Next, I move to the AR film that I find interesting and can better demonstrate the use of AR technology in filmmaking. After discussing VR/AR in film production, I conclude by discussing the development trend of VR/AR technology in film industry.

The development of VR technology and VR films

Virtual Reality (VR) technology is the use of computer simulation to generate a three-dimensional virtual world, the user through external interactive devices in the virtual world experiences visual, auditory, tactile, and other real feelings so that users produce a feeling of immersion. The VR movie model is a combination of movies, games, theaters, etc., and has been booming. Many games are now starting to use cinematic language when advancing the plot. Combining games and movies to make the audience highly interactive in an immersive drama will be the future direction of VR movies. Existing VR video production technologies and hardware playback devices can already provide a better experience for users which lets them have more interaction with the movies. With the rapid evolution of virtual reality technology, panoramic video services provided by head-mounted displays or smartphones are widely used, providing users with a new viewing experience. As of now, the combination of VR and film is still in the exploration stage. People wear head-mounted devices to enter films and can get a strong sense of immersion as if they were in the real world. In a VR environment, users can observe their surroundings realistically through their own eyes and perform specific actions themselves. When applied to film, VR technology can simulate sensory functions such as sight, hearing, and touch, making it possible for people to immerse themselves in a virtual realm in a realistic way. For filmmakers, this can be understood as a technique that allows the audience to be fully immersed in the story and deeply immerse themselves in the characters. VR technology allows us to feel that this is not a deliberate recording, but a real scene. If the film is a tool for creating empathy, then VR filmmaking is probably the closest way to do it. The source of VR cinema’s appeal is the highly immersive environment it creates to produce a realistic, immersive experience.  However, interaction in VR film and television has always been a pain point for producers and is currently handled by assigning characters to the audience and having the characters in the show talk to the audience. Overall, VR is a new medium driven by a sense of an engaging experience. VR technology will bring unprecedented breakthroughs in cinematic storytelling and even take it to heights beyond imagination. People are bullish on VR because of the changes brought about by the technological revolution, artificial intelligence, big data, etc. The freedom and subjectivity of VR itself run counter to traditional movies and animation. VR is simple to understand, it presents a completely virtual picture while then using head and motion detection technology to track the user’s movements so that they are reflected in the content and provide an immersive experience. “VR is a tool to emotionally engage viewers with social issues around the world”. (Gillespie, p148) It is more suitable for applications in video games, immersive movies, TV content, etc. than the current near reality 2D displays. Currently, VR games and VR movies are the most widely used. Hollywood is also actively trying to develop virtual reality movies, and some large technology companies have started to build virtual reality amusement parks to achieve a more interactive virtual reality entertainment experience through huge venues, comprehensive motion tracking systems, and video game-like scene design.

Difficulties encountered in VR movie production

“The power of virtual reality (and especially virtual reality cinema) is found in its immersive nature. It is transmissive and completely blends the observer and the art”. (StudioBinder, 2021) However, VR movies currently also have a big bottleneck. One of them is the high cost of VR movies. VR movies are not small, short films, they need to tell a lot of stories and add a lot of special effects. Nowadays, it is difficult to complete a whole movie with high special effects with general cameras and special effects production. The most difficult special effects to produce are rendering effects, which require a long wait because the performance of current machines cannot meet that kind of requirement, so it is very difficult to produce VR with these devices. Second, there is no way to determine the mainline of perspective. This is the problem of filming, VR features the ability to turn the perspective at will, but the audience watching does not know where the best part will appear. The audience may turn to look behind, but the best scene has been played on the other side, it would be a pity. Therefore, the director can’t guide the viewers’ attention to follow the director’s thoughts while shooting. The main problem is that watching a two-hour film while having to constantly rotate your head to follow the director’s storyline, is too tiring.  And there is also a sense of vertigo, this has nothing to do with physical fitness, just like seasickness, some people are not affected by vertigo.

Development and Influences of AR Technology

AR is also well understood as an overlay of digital images based on the real environment, with some of the same motion tracking and feedback technologies. AR devices are usually presented as glasses with some transparency, but also have integrated image projection elements, allowing you to see some digital images in the real environment. A good example is the Iron Man home set of AI systems, to join AR technology. AR in terms of equipment does not seem to have a real consumer machine product launched in the market, Microsoft HoloLens is currently only open to developers, the price is up to $ 3000. AR glasses can also play games, but the effect is like adding some enemies in the real environment. Gaming, film, and television are not the strength of AR devices, but AR in the education and aerospace fields has more advantages. Augmented reality technology not only shows real-world information, but also displays virtual information at the same time, and the two types of information complement and superimpose on each other. The mature use of AR technology can make people’s lives more convenient and sci-fi, making the sci-fi scenes in Harry Potter a reality. AR technology can be combined with film and entertainment, tourism, FMCG, and many other industries. AR is the newest technology, and although it will not replace the deep-rooted influence of TV among the older generation, AR represents the future. Each of our image innovations from black and white to color, from spherical to liquid crystal, from small size to giant curved, each time represents a more enjoyable image experience. Now, VR brings, not only an increase in the size of the home TV, which is unaffordable for most people but also a transformation from a two-dimensional plane to a three-dimensional one.

The production of the AR film, NEST

VR movies are common, but when it comes to AR, it is not directly associated with storytelling. However, in the new era of AR, VR, and MR, new concepts are sprouting and evolving. “The whole idea of using AR in the film is to create a world.” (Gupta, 2021) The AR film which I found interesting is produced by Duncan Walker, founder of Trashgames. Duncan originally majored in filmmaking, animation, and visual effects, but later by mistake engaged in programming, animation, and video games. However, in the new era of VR and AR, these professions have merged, and in a roundabout way, he still engaged in what he initially studied.

The creative worlds of filmmaking and video games are slowly merging into one, creating new potential for storytelling. Traditional filmmaking requires actors in motion capture suits, shaping visual effects (VFX) and computerized 3D animation (CGI) characters, using the green screen and extensive post-production techniques to get the final product. The entire production process takes months or even years to create the film’s rendering and output. Duncan’s approach to filmmaking is a complete reversal of tradition. Now people can shoot CGI characters in real locations and direct them as you would a human actor. In Duncan’s short video NEST, professional dancer Aoi Nakamura is scanned in 3D to replicate her face onto a created character role. He imported CGI characters and alien creatures into his iPhone through ARKit and then filmed them in real-time in a real environment. NEST was created by combining non-specific scenes with specific scenes. Duncan instructed his CGI characters through the non-specific scenes, through a control panel on his phone. He can control any of their movements. On the other hand, specific scenes are pre-made animations, but Duncan can shoot them multiple times from any angle and cinematic scene.

Duncan has said that today, making movies is like making games. 3D resources, lots of animation, not much real acting anymore. He is doing almost the same thing as making a game. Essentially, he is making games, making game characters, and then turning them into movies. Currently, these CGI characters can only achieve simulated character actions, but their emotions have yet to be achieved. However, with the rapid development of machine learning AI today, Duncan believes that emotion realization is not far away. A CGI character can also interact with a real-life actor. Like how a self-driving car can scan its environment and decide how to operate. Duncan explained that the character is as functional as the phone. He hopes that the software for image analysis on the iPhone allows for real-time image processing so that the finished product will be better. For now, the final product will be exported to 2D traditional platforms such as YouTube, cinemas, or TV. It premiered at Raindance Film Festival 2017 Shot on iPhone 7, made with ARKit and Unity. The NEST will still be a 2D movie, but Duncan also proposes to be able to allow users to download an app on their phones that will allow them to watch the movie as it happens in their location. Duncan hopes to continue making AR movies and potentially create a toolkit that will allow filmmakers to create their own AR movies.

The role of VR/AR in the film industry and the development trend

VR/AR cinema may not be just a technological advancement like the change from silent to sound, from black and white to color, or from 2D to 3D. “These new “realities” are at a delicate stage, and they will define a new era of communications and entertainment”. (  Schütze, 2018) It may be more of an artistic leap, like the leap from painting to sculpture, or photography to film. The film pops out of the window of a two-dimensional screen and into a three-dimensional space. VR/AR movies do not have a more mature model in terms of narrative logic and camera language and there are still very few VR/AR filming equipment that can meet the requirements of film level are the reason why VR/AR movies cannot be popularized in the short term.

Work Cited

Ciaran Gillespie (2020) Virtual Humanity—Access, Empathy and Objectivity in VR Film Making, Global Society, 34:2, 145-162, DOI: 10.1080/13600826.2019.1656173

Stephan Schütze, & Anna Irwin-Schütze. (2018). New Realities in Audio : A Practical Guide for VR, AR, MR and 360 Video. CRC Press.

StudioBinder. “What Role Will VR Play in the Future of Movies?” StudioBinder, 8 June 2021, www.studiobinder.com/blog/what-is-vr-film-future-of-vr-movies/.

Gupta, Nitin. “Impact of AR/VR on Filmmaking – Raindance Film Festival.” Raindance, 29 May 2021, raindance.org/impact-of-ar-vr-on-filmmaking/.