Design and Implementation of Online Live Streaming System Using A 3D Engine

arXiv:2409.06207 · cs.NI, cs.MM · Submitted 2024-09-10 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "Design and Implementation of Online Live Streaming System Using A 3D Engine".

Jane: Here's the gist of the paper. The research focuses on designing and implementing a low-latency, multi-peripheral, multi-content live streaming application using a 3D game engine, specifically Unity 3D (U3D), to address the growing demand for high-quality video streaming with minimal delay.

Tom: The core innovation proposes obtaining video data from multiple external cameras and streaming them through a single channel with consistent latency, moving beyond existing multi-channel methods that merely add channels without solving latency issues.

Jane: The methodology is structured across six stages: 1.

Tom: One-to-one live video streaming: Evaluating transport protocols (TCP, UDP, HTTP, WebSocket) using the OSI model.

Paper summary: Tom: So, we just covered the paper "Design and Implementation of Online Live Streaming System Using A three dee Engine," and we established that the core idea is using a Unity three dee engine to handle multi-peripheral, multi-content live streaming with consistent low latency <ref:2409.06207#pg1,Design and Implementation of Online Live Streaming System Using A 3D Engine>.

Jane: Exactly. The thesis is that existing methods for increasing streaming content just add more channels without solving the fundamental problem of latency; this paper proposes a method to get video data from multiple external cameras and stream it using only a single channel with lower delay, based on that three dee engine approach <ref:2409.06207#pg1>.

Lu: They claim they designed this by leveraging the interfaces in Unity like the World Space Camera and Webcam Texture API to grab video frames from those different devices. This allows them to build a multi-peripheral, multi-content application where the camera in the world space is aimed at what they want to show on their screen.

Meng: It’s interesting because it moves beyond just adding channels; it suggests a different structural approach for combining that data before it goes out. I wonder if that specific architectural shift actually makes a measurable difference in real-world latency compared to what we see today.

Lalam: The paper tests various transport protocols, looking at TCP, UDP, HTTP, and WebSocket to see which ones handle the frame data distribution most efficiently for this setup.

Tom: And they also look at client optimization across different platforms like Android or web to make sure all these modules—the player module, the message panel, and the streaming module—communicate smoothly together.

Jane: They also refined things by looking at how data packing works, specifically trying to use YUV color space instead of RGB input for encoding efficiency because RGB is harder to manage in a real-time video context.

Lu: The system architecture includes a client application with modules like the Message Panel and Player Module that rely on the Uthree dee Easy Movie Texture plug-in for stable playback, while the server side uses an HTTP server, a WebSocket server for real-time communication, and a MySQL database for persistence.

Meng: That layered backend sounds standard for high availability, but I wonder how much overhead that whole structure adds to the overall latency budget when you're aiming for minimal delay.

Lalam: They use UDP-based RTSP video codec technology on the client side and map the data acquisition onto a virtual "three dee scene" within Unity, recording content onto a canvas using a world-space camera for transmission <ref:2409.06207#pg1>.

Tom: So, they’re essentially creating a "Director-Guided" live network where all these different capture devices are recorded onto that canvas using that three dee environment to be streamed out as one single line <ref:2409.06207#pg1>.

Jane: It’s about achieving the richness and diversity of video content while ensuring that the delay remains unchanged across all those inputs.

Conclusion: Tom: To wrap up, the paper "Design and Implementation of Online Live Streaming System Using A three dee Engine" shows how you can use a Unity three dee engine to successfully integrate multiple external video capture devices into a single low-latency stream <ref:2409.06207#pg1,Design and Implementation of Online Live Streaming System Using A 3D Engine>.

Jane: The implication is that this specific combination of tools—the Uthree dee APIs for capturing diverse inputs and the unified three dee scene for mapping them—provides a concrete solution for multi-peripheral, multi-content live streaming without increasing latency <ref:2409.06207#pg1>.

Lu: It gives us a theoretical framework showing how to design these systems where the interfaces between the world space and screen space are carefully controlled to manage all those different video inputs coherently.

Meng: From my side, it confirms that you don't need some completely new proprietary streaming protocol; you can build a solid, scalable system using existing engine capabilities if you focus on optimizing the data flow itself.

Lalam: The authors acknowledge limitations concerning inter-frame coding efficiency and the complexity of modeling all the database relationships perfectly, but they still deliver a working implementation that proves the concept.

Tom: So, it’s a strong demonstration of using game engine capabilities to solve real-world problems in live video infrastructure. It proves that you can build something functional when you have a clear vision for how the data needs to flow.

cs.NI, cs.MM

Submitted: 2024-09-10

Updated: 2024-09-10

Journal ref: Excellent Thesis, Beijing University of Technology, 2021

Code: https://github.com/Ezharjan/UnityVideoStreamingPackage

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 77/100

The gist: The research focuses on designing and implementing a low-latency, multi-peripheral, multi-content live streaming application using a 3D game engine, specifically Unity 3D (U3D)…

Key concepts

One-to-one live video streaming
This stage involved testing different transport protocols like TCP, UDP, HTTP, and WebSocket to understand how best to send live video data from one source to another. The goal was to find the most reliable method for establishing a direct connection with minimal delay.
Multi-channel video data acquisition
Native JavaScript methods failed for accessing multiple cameras simultaneously. The solution used Unity 3D's APIs, specifically World/Screen Space Camera and Webcam Texture APIs, to capture frames from several external devices and integrate them into the 3D engine.

Terminology

Summary

The research focuses on designing and implementing a low-latency, multi-peripheral, multi-content live streaming application using a 3D game engine, specifically Unity 3D (U3D), to address the growing demand for high-quality video streaming with minimal delay. p. 2

Where the paper says it

Unity 3D game engine). Based on the interfaces enabling the control of 3D virtual space in 3D Engines, a multi peripheral, multi content and low latency live streaming application is designed and achieved.

The core innovation proposes obtaining video data from multiple external cameras and streaming them through a single channel with consistent latency, moving beyond existing multi-channel methods that merely add channels without solving latency issues. p. 2
Where the paper says it

Existing methods for increasing streaming content is to directly use multi-channel, which does not fundamentally solve the problem of latency while merely adding a new channel into the network. I propose a method to obtain video data from multiple external cameras and stream them using only a single channel with lower latency based on a 3D Engine (eg.

The methodology is structured across six stages:

  1. One-to-one live video streaming: Evaluating transport protocols (TCP, UDP, HTTP, WebSocket) using the OSI model.

  2. One-to-many video distribution: Testing real-time efficiency and applicability of protocols like RTMP, RTP, and RTSP for frame data distribution.

  3. Multi-channel video data acquisition: Identifying that native JavaScript methods are insufficient for multi-camera access across various platforms; U3D's 3D engine proved effective for acquiring multi-camera video frames via its World/Screen Space Camera, 3D Canvas, and Webcam Texture APIs.

  4. Client optimization: Refining peripheral modules to align with server interfaces and achieving cross-platform deployment (Android, web, Linux).

The system architecture is divided into a robust client application and a comprehensive backend infrastructure:

  • Client Application: Composed of modules including the Message Panel, Main Panel, Player Module, and Live Tool Module. The streaming module manages data acquisition from multiple devices via the Main Panel's direction. The Player Module utilizes the U3D Easy Movie Texture plug-in for stable video playback. p. 3

    Where the paper says it

    4 comprehensively describes the specific details of each module in the back-end, Finally, a live video streaming system based on Unity 3D (U3D) game engine is implemented, which supports data input of multiple external video capture devices. The system is composed of client application software based on U3D, long connection dedicated server based on WebSocket protocol, basic communication server based on HTTP protocol, persistent storage dedicated database based on MySQL, cache database based on Redis and load balancing server based on Nginx.

  • Server Architecture: Divided into an HTTP server (for database interaction), a WebSocket server (for long-lived, real-time communication like heartbeats and calls), and a MySQL relational database for persistent storage, complemented by Redis as a caching layer. Nginx handles load balancing and request redirection. p. 3

    Where the paper says it

    4 comprehensively describes the specific details of each module in the back-end, Finally, a live video streaming system based on Unity 3D (U3D) game engine is implemented, which supports data input of multiple external video capture devices. The system is composed of client application software based on U3D, long connection dedicated server based on WebSocket protocol, basic communication server based on HTTP protocol, persistent storage dedicated database based on MySQL, cache database based on Redis and load balancing server based on Nginx.

The streaming pipeline involves using UDP-based RTSP video codec technology on the client side. p. 2

Where the paper says it

The method is illustrated by the affiliated project via the implementation of a low-latency multi-channel live video streaming system. Beginning with the underlying of RTSP protocol, analyzes the underlying protocol and video coding method applicable to the system, and then studied the client side technic, puts forward the requirements and objectives for the final software application, further describes the specific technical implementation principles of each module, and

Data acquisition is mapped onto a virtual 3D scene within U3D, where content from multiple capture devices is recorded onto the canvas using a world-space camera for transmission, effectively forming a Director-Guided live network.

Performance analysis revealed that while algorithmic optimizations for data packing and unpacking were attempted, CPU performance did not significantly improve due to the substantial data volume. This led to a proposed dual approach: reducing RTP packet size and improving encoding efficiency by utilizing YUV color space instead of RGB input, as the latter is more effective for real-time video management.

The backend communication relies on a unified JSON message structure. p. 5

Where the paper says it

2 WebSocket Server for Persistent Communication. 3 WebSocket Dual-End Message Structure.

The HTTP server handles public data queries (GET) and user interactions (POST). p. 3
Where the paper says it

4 comprehensively describes the specific details of each module in the back-end, Finally, a live video streaming system based on Unity 3D (U3D) game engine is implemented, which supports data input of multiple external video capture devices. The system is composed of client application software based on U3D, long connection dedicated server based on WebSocket protocol, basic communication server based on HTTP protocol, persistent storage dedicated database based on MySQL, cache database based on Redis and load balancing server based on Nginx.

The WebSocket server manages the low-latency, persistent connections necessary for one-to-one calls and heartbeat mechanisms. p. 2
Where the paper says it

Based on the interfaces enabling the control of 3D virtual space in 3D Engines, a multi peripheral, multi content and low latency live streaming application is designed and achieved. The research is based on the Control Variable Method, which realizes the richness and diversity of streamed video content under the condition of ensuring the same delay in one single channel.

A Message Panel acts as a bridge between users and the streaming infrastructure, displaying live titles, timings, and user statuses derived from the Public Data Acquisition Module. p. 2
Where the paper says it

Abstract DESIGN AND IMPLEMENTATION OF ONLINE LIVE STREAMING SYSTEM USING A 3D-ENGINE Aizierjiang Aiersilan With the continuous development of communication network and related infrastructure, people’s demand for live video streaming is increasing, and the requirements for lowlatency and high-quality transmission are constantly improving. 5G network, which will be popularized in the nearest future, can bring good news for users’ high requirements for live video streaming on the hardware level.

In summary, the system successfully integrates multiple external video capture devices into a single low-latency stream within a 3D environment using U3D APIs. p. 3

Where the paper says it

4 comprehensively describes the specific details of each module in the back-end, Finally, a live video streaming system based on Unity 3D (U3D) game engine is implemented, which supports data input of multiple external video capture devices. The system is composed of client application software based on U3D, long connection dedicated server based on WebSocket protocol, basic communication server based on HTTP protocol, persistent storage dedicated database based on MySQL, cache database based on Redis and load balancing server based on Nginx.

The modular client design ensures reusability, while the layered server architecture provides necessary real-time communication and data persistence. p. 3
Where the paper says it

4 comprehensively describes the specific details of each module in the back-end, Finally, a live video streaming system based on Unity 3D (U3D) game engine is implemented, which supports data input of multiple external video capture devices. The system is composed of client application software based on U3D, long connection dedicated server based on WebSocket protocol, basic communication server based on HTTP protocol, persistent storage dedicated database based on MySQL, cache database based on Redis and load balancing server based on Nginx.

Although limitations were noted regarding inter-frame coding efficiency and database relationship modeling, the implementation establishes a robust theoretical framework for flexible, multi-source live video broadcasting.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Improved AI systems can integrate multi-channel video capture to create richer live content by seamlessly addition of external video devices to enhance live video content without altering the data volume. This allows the system to push the content from all external video devices to the client without changing the overall data size while maintaining a single-channel stream.

  2. The system can achieve enhanced image quality across diverse client devices by pre-processing color space on the server, as standardizing the image data before streaming on the server side is crucial to ensure that any discrepancies in image quality are not attributable to server-side processing. This ensures no additional processing is required on the client side for brightness and color correction.

  3. AI systems can be designed with decoupled modules for better maintainability, as "the independence of each module, as outlined in the chapter on multi-channel video capture technology, allows the multi-channel video capture module, the pull module, the push module, and other components of the client application to be disassembled and separated from each other without impacting the core functionality." This modularity maximizes reusability.

  4. The client application can support advanced user interactions by implementing control mapping to associate the multiple Raw Images, identified in the initial stage, with various interactive controls in screen space using an orthogonal projection mechanism, enabling users to zoom in on the video content from specific external devices by clicking on the video feed.

  5. The system can provide robust real-time status updates through a heartbeat mechanism that monitors connectivity and facilitates reconnection in case of disconnection when network conditions are unstable or relatively poor, ensuring a stable operational state for online users.

Abstract

With the growing demand for live video streaming, there is an increasing need for low-latency and high-quality transmission, especially with the advent of 5G networks. While 5G offers hardware-level improvements, effective software solutions for minimizing latency remain essential. Current methods, such as multi-channel streaming, fail to address latency issues fundamentally, often only adding new channels without optimizing overall performance. This thesis proposes a novel approach using a 3D engine (e.g., Unity 3D) to stream multi-input video data through a single channel with reduced latency. By leveraging 3D engine capabilities, such as World/Screen Space Cameras, 3D Canvases, and Webcam Textures, the proposed system consolidates video streams from multiple external cameras into a unified, low-latency output. The affiliated project of this thesis demonstrates the implementation of a low-latency multi-channel live video streaming system. It employs the RTSP protocol and examines video encoding techniques, alongside a client-side application based on Unity 3D. The system architecture includes a WebSocket server for persistent connections, an HTTP server for communication, a MySQL database for storage, Redis for caching, and Nginx for load balancing. Each module operates independently, ensuring flexibility and scalability in the system's design. A key innovation of this system is its use of a 3D scene to map multiple video inputs onto a virtual canvas, recorded by an in-engine camera for transmission. This design minimizes redundant data, enabling an efficient and director-guided live streaming network. The thesis concludes by discussing challenges encountered during the project and provides solutions for future improvement.

Related papers