At the beginning of the 21st century, with the rapid advancement of real-time computer rendering technology, on-site real-time interactive previsualisation techniques in front of green screens emerged. During actual shooting, live-action elements captured in front of the green screen are composited with virtual elements in real time on set. However, limited by rendering quality and keying performance, the real-time output on site generally serves only as a shooting reference rather than final footage. High-quality final images still rely on post-production compositing.

In recent years, driven by continuous technological progress and accelerated innovation in film industrialisation, a new-generation LED wall virtual production system based on XR technology has emerged — XR / Virtual Production.

What is XR / Virtual Production?

Simply put, it is an innovative shooting method that leverages virtual reality and augmented reality technologies to build virtual environments within real spaces, enabling interactions between real actors and virtual characters to achieve desired shooting results. It is also known as Extended Reality (XR) technology.

↑ XR Virtual Production Workflow

The key to XR shooting lies in positional tracking using specialised cameras combined with real-time rendering on background screens. By merging real shooting environments and virtual backdrops, the “infinitely expandable” background enables comprehensive construction of virtual worlds on screen, delivering more authentic, rich and cinematic footage.

The complete XR / virtual production system consists of core components including compositing & rendering servers, positioning servers, camera tracking systems, cameras, LED screens and synchronisation systems. As the display unit, LED screens can build highly immersive scenes within a limited physical space.

During actual shooting, scenes are usually divided into three layers: background, midground and foreground. Except for the midground (physical objects), the other two layers generate lifelike virtual scenes via 3D rendering engines. This differentiates it from traditional green screen workflows. In conventional green screen shooting, both live-action and virtual elements are filmed in front of green screens. With XR / virtual production, virtual scenes can replace those elements, unlocking unlimited creative possibilities within a constrained shooting volume.

XR / virtual production has gradually become the preferred efficient creative tool for professionals in digital entertainment.

XR / virtual production has brought revolutionary and far-reaching impacts to film and television entertainment, and has become an indispensable part of movie and television production. Compared with traditional green screen shooting, its most prominent advantage is the substantial reduction of manual intervention and post-production workload. Powered by automated and intelligent technologies, XR / virtual production can greatly cut production costs while lifting shooting efficiency and image quality. It allows creators to easily build realistic environments and special effects, greatly enhancing the audience’s viewing experience. In short, XR / virtual production serves as an essential high-efficiency creative tool for creators in film and entertainment, helping them deliver high-quality works with greater ease.

△ CHINGMU built a large-scale XR digital studio for Yaowang Technology, providing creators with a WYSIWYG (What You See Is What You Get) shooting environment.

Within the practical XR / virtual production workflow, camera positioning is a critical step. Inaccurate camera positioning will cause misalignment between the LED wall and physical objects in front of the screen (actors, props, etc.), severely degrading the quality of final footage.

Motion Capture as the Core Technology for Camera Position Tracking in XR / Virtual Production

As one of the key technologies for XR / virtual production, motion capture systems track and record the spatial movement of cameras and synchronise it with virtual backdrops, ensuring precise matching between virtual scenes and real camera positions. Additionally, motion capture technology can track movements of actors and props in real time and map them into the virtual world, enabling seamless blending of reality and virtual elements.

From the perspective of tracking principles, positioning and tracking systems fall into two main categories: Inside-Out tracking and Outside-In tracking.

Inside-Out Tracking

Position sensors are placed inside the tracked object. Common Inside-Out sensors include cameras, infrared (IR) lasers and inertial measurement units (IMUs). Generally, they determine their own position and pose by recognising changes in the surrounding environment. By analogy: humans identify surroundings to judge whether to move forward, backward or turn. This positioning method is similar to humans continuously observing the environment to locate themselves. The approach is straightforward, yet vulnerable to external constraints. For example, people may get disoriented in an entirely unfamiliar setting, much like waking up in a strange room after intoxication, or a small boat losing positional awareness in open sea without buoys or large vessels as references.

Translated to XR / virtual production tracking: when the external environment changes abruptly, Inside-Out tracking cannot achieve self-localisation and loses tracking function. In practical operations, re-calibration of positional data is required, resulting in substantial waste of time and labour and lowering the overall efficiency of the shooting team.

Outside-In Tracking

Position sensors are installed externally to the tracked object, usually fixed and facing the target. Essentially, the pose of the tracked object is calculated by measuring its relative position to the sensors. For instance, satellite positioning is a well-known Outside-In tracking technology, which calculates positions by triangulation via multiple satellites. A simple illustration: no matter whether you are familiar with your surroundings, you can quickly pinpoint your location using map applications on mobile phones or vehicles when lost.

In practical applications, the biggest distinction of Outside-In tracking compared with Inside-Out tracking is that targets within the capture volume can be positioned with high precision.

It is worth noting that optical motion capture requires strict control of ambient natural light. Strong natural light may interfere with capture results and impose certain limits on deployment.

To tackle these challenges, active infrared optical solutions can be adopted. They feature long capture range, strong stability and outstanding anti-jitter performance. Therefore, adopting active luminous solutions for XR positioning extends the camera capture range, enabling motion capture not only indoors but also in outdoor environments with strong natural daylight.

The primary feature of Outside-In positioning is that it is not restricted by camera motion trajectories. It is widely applied in XR, VP and green box virtual production environments. One set of equipment can precisely determine the positions of multiple cameras simultaneously. Within the same capture volume, it can track cameras, live actors, props and full-body performances of actors in motion capture suits at the same time. Real-time interactions between real people and digital humans can be displayed, ensuring shooting results match expectations, saving studio space, shortening post-production cycles and reducing post-production costs.

△ CHINGMU provided XR / virtual production system support and services for the film Born to Fly.

XR / virtual production technology was fully deployed in Born to Fly. Relying on self-developed optical motion capture systems, CHINGMU delivers high precision, low latency and multi-camera tracking. The Outside-In positioning method tracks movement of one or multiple cameras, accurately locating cameras and capturing actor motions for seamless integration with visual effects. Thanks to joint efforts from the production team, realistic flight scenes were reconstructed, offering actors an immersive shooting environment. This innovative XR / virtual production technology helped the creative team of Born to Fly overcome difficulties in presenting spatial depth, perspective and motion, ultimately delivering stunning visual effects to audiences.

Due to the resolution limits of optical systems, XR shooting often encounters issues such as camera occlusion leading to failed capture and marker jitter within the optical camera view. To solve these problems, an optical-inertial hybrid algorithm can be applied. Inertial sensors built into cameras synchronise optical positioning data and inertial positioning data in software, effectively fusing the two datasets and smoothing positioning curves, which greatly optimises XR shooting results.

These algorithms can be implemented via data processing and solving functions built into optical system software. A synchronisation system aligns data from optical motion capture cameras and other XR hardware based on output signals from different devices. Further processing is carried out via mainstream third-party software on the market including Aximmetry, Pixotope, Disguise and hecoos, ultimately generating high-quality finished works.

Extensive Application Scenarios of XR / Virtual Production in Digital Entertainment

The advantages of XR / virtual production — reduced labour and time costs, expanded creative scope, improved production efficiency and real-time visualisation — have enabled wide adoption across film, television dramas, games and other sectors. Moreover, it has become an important development direction of the digital entertainment industry. In specific scenarios including film and television production, commercial shooting, product launches, variety shows, art exhibitions, virtual concerts and livestreams, XR / virtual production delivers high-quality, lifelike visual content, offering audiences richer and more immersive viewing experiences.

△ CHINGMU provided XR / virtual production support for the recording of China’s China Pride.

△ CHINGMU provided XR / virtual production technical support for Shiguang Coordinate and other collaborative creators.

XR / virtual production has opened up brand-new creative domains and brings spectacular audiovisual feasts to the film and entertainment industry! CHINGMU focuses on full-process optical motion capture solutions and XR / virtual production products and services, striving to deliver the most professional and top-tier visual effects. With rich experience and proprietary technology, we build real-time, accurate and immersive visual effects for creators, empowering film and entertainment creation to transcend imagination and realise infinite possibilities

!