Design and Implementation of Online Live Streaming System Using A 3D Engine

summary

The gist

The research focuses on designing and implementing a low-latency, multi-peripheral, multi-content live streaming application using a 3D game engine, specifically Unity 3D (U3D)…

In short

The research designed a low-latency live streaming application using Unity 3D to combine video from multiple external cameras into a single stream. By mapping video data onto a virtual 3D scene, the system achieved consistent latency across various capture sources. The solution involves a modular client and server architecture that uses UDP-based RTSP technology for efficient real-time distribution.

Key concepts

One-to-one live video streaming
This stage involved testing different transport protocols like TCP, UDP, HTTP, and WebSocket to understand how best to send live video data from one source to another. The goal was to find the most reliable method for establishing a direct connection with minimal delay.
Multi-channel video data acquisition
Native JavaScript methods failed for accessing multiple cameras simultaneously. The solution used Unity 3D's APIs, specifically World/Screen Space Camera and Webcam Texture APIs, to capture frames from several external devices and integrate them into the 3D engine.

Terminology used across episodes

This episode discusses

The paper

Design and Implementation of Online Live Streaming System Using A 3D Engine · Read on arXiv

With the growing demand for live video streaming, there is an increasing need for low-latency and high-quality transmission, especially with the advent of 5G networks. While 5G offers hardware-level improvements, effective software solutions for minimizing latency remain essential. Current methods, such as multi-channel streaming, fail to address latency issues fundamentally, often only adding new channels without optimizing overall performance. This thesis proposes a novel approach using a 3D engine (e.g., Unity 3D) to stream multi-input video data through a single channel with reduced latency. By leveraging 3D engine capabilities, such as World/Screen Space Cameras, 3D Canvases, and Webcam Textures, the proposed system consolidates video streams from multiple external cameras into a unified, low-latency output. The affiliated project of this thesis demonstrates the implementation of a low-latency multi-channel live video streaming system. It employs the RTSP protocol and examines video encoding techniques, alongside a client-side application based on Unity 3D. The system architecture includes a WebSocket server for persistent connections, an HTTP server for communication, a MySQL database for storage, Redis for caching, and Nginx for load balancing. Each module operates independently, ensuring flexibility and scalability in the system's design. A key innovation of this system is its use of a 3D scene to map multiple video inputs onto a virtual canvas, recorded by an in-engine camera for transmission. This design minimizes redundant data, enabling an efficient and director-guided live streaming network. The thesis concludes by discussing challenges encountered during the project and provides solutions for future improvement.

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "Design and Implementation of Online Live Streaming System Using A 3D Engine".

Jane: Here's the gist of the paper. The research focuses on designing and implementing a low-latency, multi-peripheral, multi-content live streaming application using a 3D game engine, specifically Unity 3D (U3D), to address the growing demand for high-quality video streaming with minimal delay.

Tom: The core innovation proposes obtaining video data from multiple external cameras and streaming them through a single channel with consistent latency, moving beyond existing multi-channel methods that merely add channels without solving latency issues.

Jane: The methodology is structured across six stages: 1.

Tom: One-to-one live video streaming: Evaluating transport protocols (TCP, UDP, HTTP, WebSocket) using the OSI model.

Paper summary: Tom: So, we just covered the paper "Design and Implementation of Online Live Streaming System Using A three dee Engine," and we established that the core idea is using a Unity three dee engine to handle multi-peripheral, multi-content live streaming with consistent low latency <ref:2409.06207#pg1,Design and Implementation of Online Live Streaming System Using A 3D Engine>.

Jane: Exactly. The thesis is that existing methods for increasing streaming content just add more channels without solving the fundamental problem of latency; this paper proposes a method to get video data from multiple external cameras and stream it using only a single channel with lower delay, based on that three dee engine approach <ref:2409.06207#pg1>.

Lu: They claim they designed this by leveraging the interfaces in Unity like the World Space Camera and Webcam Texture API to grab video frames from those different devices. This allows them to build a multi-peripheral, multi-content application where the camera in the world space is aimed at what they want to show on their screen.

Meng: It’s interesting because it moves beyond just adding channels; it suggests a different structural approach for combining that data before it goes out. I wonder if that specific architectural shift actually makes a measurable difference in real-world latency compared to what we see today.

Lalam: The paper tests various transport protocols, looking at TCP, UDP, HTTP, and WebSocket to see which ones handle the frame data distribution most efficiently for this setup.

Tom: And they also look at client optimization across different platforms like Android or web to make sure all these modules—the player module, the message panel, and the streaming module—communicate smoothly together.

Jane: They also refined things by looking at how data packing works, specifically trying to use YUV color space instead of RGB input for encoding efficiency because RGB is harder to manage in a real-time video context.

Lu: The system architecture includes a client application with modules like the Message Panel and Player Module that rely on the Uthree dee Easy Movie Texture plug-in for stable playback, while the server side uses an HTTP server, a WebSocket server for real-time communication, and a MySQL database for persistence.

Meng: That layered backend sounds standard for high availability, but I wonder how much overhead that whole structure adds to the overall latency budget when you're aiming for minimal delay.

Lalam: They use UDP-based RTSP video codec technology on the client side and map the data acquisition onto a virtual "three dee scene" within Unity, recording content onto a canvas using a world-space camera for transmission <ref:2409.06207#pg1>.

Tom: So, they’re essentially creating a "Director-Guided" live network where all these different capture devices are recorded onto that canvas using that three dee environment to be streamed out as one single line <ref:2409.06207#pg1>.

Jane: It’s about achieving the richness and diversity of video content while ensuring that the delay remains unchanged across all those inputs.

Conclusion: Tom: To wrap up, the paper "Design and Implementation of Online Live Streaming System Using A three dee Engine" shows how you can use a Unity three dee engine to successfully integrate multiple external video capture devices into a single low-latency stream <ref:2409.06207#pg1,Design and Implementation of Online Live Streaming System Using A 3D Engine>.

Jane: The implication is that this specific combination of tools—the Uthree dee APIs for capturing diverse inputs and the unified three dee scene for mapping them—provides a concrete solution for multi-peripheral, multi-content live streaming without increasing latency <ref:2409.06207#pg1>.

Lu: It gives us a theoretical framework showing how to design these systems where the interfaces between the world space and screen space are carefully controlled to manage all those different video inputs coherently.

Meng: From my side, it confirms that you don't need some completely new proprietary streaming protocol; you can build a solid, scalable system using existing engine capabilities if you focus on optimizing the data flow itself.

Lalam: The authors acknowledge limitations concerning inter-frame coding efficiency and the complexity of modeling all the database relationships perfectly, but they still deliver a working implementation that proves the concept.

Tom: So, it’s a strong demonstration of using game engine capabilities to solve real-world problems in live video infrastructure. It proves that you can build something functional when you have a clear vision for how the data needs to flow.

More episodes

← Home