Technical Principles of XR+LED Virtual Shooting

01  What is XR

XR (Extended Reality) is a term that describes the fusion or interaction of virtual and real elements. It is generally considered to be a general term for various technologies such as AR (Augmented Reality)/VR (Virtual Reality)/MR (Mixed Reality). Through camera tracking and real-time image rendering technology, the display medium (LED/projection bowl screen) and the virtual scene outside the display medium can track the camera angle of view in real time, and synthesize with the real shooting picture in front of the camera lens, thus creating an infinite sense of space technology.

02 The principle basis of XR

Understanding perspective is an important prerequisite for understanding XR technology. XR technology is a bridge between virtual and reality, which is based on the principle of perspective.

If you want to synthesize a video, the actor is in the room, but there is a whole forest behind him under the lens, is there any other way besides using the traditional green screen + matting method?

If the camera lens is fixed and the position is appropriate, you can even hang a huge forest pattern canvas directly behind the actor. As long as the actor and the background plane show the correct perspective relationship under the lens, some of the most basic needs can be met.

But once the camera moves, the fact that the background forest is actually a flat canvas cannot be concealed anyway.

In fact, the realism of the background of the scene is shaped and enhanced by the movement of the camera. In every frame where the camera is pushed, pulled and panned, such as the large panorama when looking down, and the middle shot when following the shot-there should be a background scene that satisfies the real perspective relationship at this position and angle behind the actor.

It would be nice if the background behind the actor could match the camera lens when the camera rotated or changed angles.

And this is the problem that XR technology is going to solve.

03 Technical Equipment Conditions for XR Realization

After understanding the concept and principle of XR, how to realize it in practical application?

At present, the technical equipment required for the realization of mainstream XR technology in the industry mainly includes: LED display screen, LED control system, virtual rendering system, camera and tracking and positioning system, synchronization system and other auxiliary systems. With the above technical equipment. Through the real-time workflow, performers can be integrated into the virtual world without green screen and post-production, and the final image can be output in real time.

The overall technical framework is shown in the figure below:

The XR scene technology framework includes several major elements such as a shooting site composed of LED displays, a camera, object position tracking and positioning technology, a real-time graphics rendering engine (AR technology), and a high-speed and low-latency network.

Including:

● LED control system

It is mainly responsible for receiving virtual rendering images and outputting them to the LED display screen. An excellent control system needs to have superb video processing capabilities and be able to perform precise color management and brightness adjustment on the LED screen body to maximize the display of the LED screen body. Display effect, and provide technical support for special application requirements of XR (such as multi-camera, high frame rate, etc.) on the hardware side.

● Virtual rendering system

It mainly includes a media server and a real-time rendering engine. The master media server is mainly responsible for controlling the rendering engine, processing camera data, and outputting LED video source images. The real-time rendering engine provides the integration and production of virtual digital assets and builds a complete virtual expansion space.

● Camera system

To capture real images and send them back to the virtual rendering system, a professional-grade film and television camera is required to achieve clear and lossless real image acquisition. Generally, cameras that support acquisition frame rates above 120Hz are used.

● Tracking and positioning system

It is mainly used for positioning to realize real camera displacement and lens rotation data. The key indicators of the positioning system are accuracy, degree of freedom and low delay. More commonly used are mechanical tracking (pan/tilt, robotic arm, etc.), optical positioning, and other infrared tracking , visual camera tracking, etc.

● Synchronization system

It is mainly used for the reference time and picture frame of the overall XR workflow. An independent and accurate synchronization system is a necessary condition for XR projects. The synchronization system can effectively reduce the problems of scanning lines and black fields when the camera shoots the LED screen. It is also possible to achieve multi-camera shooting effects by controlling the synchronization time and matching with the LED control system.

XR technology is an emerging technology that has developed rapidly in recent years. Compared with the traditional green screen + matting virtual shooting method, XR+LED display has significant advantages. In the future, driven by the rapid iteration of technology and the internal drive of demand innovation, the XR technology in the ascendant will surely usher in a vast world of its own.