Selecting Person Segmentation Models for Real-Time Video Call Backgrounds on the CPU
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.
Rosa: We're looking at the paper: "Selecting Person Segmentation Models for Real-Time Video Call Backgrounds on the CPU".
Dev: Here's the gist of the paper. blur and background replacement are now normal parts of a video call Many open models and open source pipelines can do this, but it is hard to know which one to pick Speed, edge quality and license all matter, and they are often reported on different hardware, or not reported at all.
Rosa: The authors ran 20 setups on the same laptop CPU with the same 640x480 frames and the same output view.
Dev: These setups included 15 open models and 5 complete open source pipelines.
Rosa: They measured frames per second on a test clip and on a live webcam, compared the masks on a public domain portrait and on three video call photos with a raised hand.
Paper summary: Rosa: So, we're looking at this paper, "Selecting Person Segmentation Models for Real-Time Video Call Backgrounds on the CPU." Basically, they’re trying to figure out which person segmentation models work best for background blur and replacement in video calls when you’re running it on a standard laptop CPU.
Dev: Right. The core idea is that it's not just about finding *a* model, but knowing which one to pick because speed, the quality of the edge—how sharp the mask is—and the license all matter for different uses.
Taro: It sounds like they’re running a test on a laptop CPU with six hundred forty by four hundred eighty frames and comparing fifteen open models against five complete open source pipelines. Rosa That’s right, they ran twenty setups total, which included both the individual models and those full pipelines.
Dev: And what they measured was frames per second on both a test clip and a live webcam feed, plus they compared the masks on some public domain portraits and also on three actual video call photos where someone had their hand raised. Rosa That’s important because it tests how well it handles things that move around, like hands.
Taro: I saw something specific about quality assessment in the abstract—they looked at background leaks, hair, and fingers when comparing the masks visually across those four photos. Dev And they found some specific trade-offs there.
Rosa: They mentioned that models like MODNet and RVM tend to keep loose hair strands, which is good for realism sometimes. But the authors also pointed out that most of the fast segmentation models they tested ended up blurring the hand or losing it completely, even when you filmed it with a close front camera view.
Dev: It seems like there's a direct conflict between getting high speed and getting those fine details, especially with hands and hair. Taro I think that’s the main tension in this work.
Rosa: So, they offered a selection guide mapping common needs to specific models based on what works for weak or old computers. Dev For instance, they recommended MediaPipe Selfie in landscape mode as the fastest setup by a wide margin for less powerful machines.
Taro: And when it comes to those tricky details, like hair and fingers, they suggested MODNet and RVM are the models that keep loose hair strands intact. Rosa That gives us a concrete starting point for developers looking at this research.
Dev: But we have to remember the practical observations from their testing—the runtime library and all the steps surrounding the model can change your speed many times over. Taro And those extra steps, they said, most of that extra time went to the edge step of processing.
Rosa: That’s a big warning for anyone building something in real time on a CPU because those small mistakes in things like gregblur can be easy to miss if you aren't careful. Dev So, even if you pick a fast model, you have to be vigilant about checking the mask itself.
Taro: And they also touched on licensing, which is crucial for shipping any product. They said permissive licenses like Apache-two point zero, MIT, and BSD-three-Clause are the safe bet for closed source products <ref:2611.14558#pg2>. Rosa GPL and AGPL require you to offer a commercial license if you use them.
Dev: It’s a big picture point—the choice of license dictates how that technology can actually be used in a commercial product versus keeping it open source. So, when you look at this paper, remember that the model choice isn't just about accuracy; it's about speed and what you are allowed to do with the resulting software.
Taro: This whole paper is really about making that decision process less of a guessing game for people building these types of video tools. Rosa It points out that real time on a CPU is achievable with smaller models like MediaPipe Selfie, ncnn ERDNet, or even the ML Kit model in the Volcomix pipeline.
Dev: But if you want those soft edges and the best hair quality, you might have to lean towards matting models like RVM or MODNet. Taro And as we talked about earlier, raised hands and fingers are definitely the hardest parts to get right with these fast segmentation models; they often just blur them out entirely.
Rosa: So, the selection guide they put together in section seven is supposed to help developers pick a model without having to run all of them first. Dev It’s about giving people a roadmap based on what their specific needs are and what kind of hardware they're working with.
Taro: The paper really highlights that the speed and quality aren't always in the same place, especially when you bring in complex things like hand segmentation. Rosa It’s a practical guide for anyone working on real-time video processing today.
Conclusion: Rosa: So we’re talking about this paper called "Selecting Person Segmentation Models for Real-Time Video Call Backgrounds on the CPU."
Dev: Yeah, it looks at how to pick the right segmentation model when you need background stuff in a video call running on a standard laptop computer.
Rosa: The authors ran twenty different setups, testing fifteen open models and five full pipelines across the same laptop CPU setup.
Dev: They measured speed in frames per second on both test clips and live webcams, and they also checked how the masks looked on public photos of portraits and actual video calls with hands raised.
Rosa: What they found is that speed isn't everything; you have to balance it with edge quality and licensing.
Dev: They showed that some fast models blur hands or lose them completely, even when the camera is close up, while others like MODNet or RVM keep loose hair strands better.
Rosa: The selection guide they wrote helps people figure out which model to use based on whether their computer is weak or if they need those specific details like hair.
Dev: But they also warned that the way you run the model, the runtime library and all those extra steps, can change your speed by several times over.
Rosa: And that means you gotta check the mask itself really carefully because small mistakes in things like background removal can be easy to miss if you aren't paying attention.
Dev: It also touches on licensing; they said permissive licenses like Apache-two point zero are the safe choice for products, while GPL and AGPL require you to offer a commercial license if you use those models.
Rosa: Basically, this paper gives us a roadmap for picking tools without having to try every single one of them first.
Dev: It shows that real time on a CPU is possible with smaller models like MediaPipe Selfie or ncnn ERDNet, but if you want the best hair and fingers, you might need to use those matting models.
Rosa: So it’s about picking the right tool for the job instead of just chasing the fastest number.
Dev: And that leads us to a big question—how do we know which model is actually going to be good enough for a real video call without spending hours testing everything?
eess.IV
Submitted: 2026-11-04
Updated: 2026-11-04
Code: https://github.com/ultralytics/ultralytics
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 77/100
The gist: Background blur and background replacement are now normal parts of a video call Many open models and open source pipelines can do this, but it is hard to know which one to pick Speed…
Key concepts
- Open Models vs. Pipelines
- Open models are individual AI algorithms you can use directly, while pipelines are complete workflows combining multiple steps (like preprocessing, model inference, and post-processing). The study compared both to see how the structure of the process affects speed and final output quality.
- Frames Per Second (FPS)
- FPS measures how many video frames a system can process in one second. High FPS is crucial for smooth, real-time video calls. The research showed that while some setups reached over 40 FPS on a laptop CPU, the speed can vary significantly based on the model and surrounding code.
- Edge Quality
- Edge quality refers to how sharp and well-defined the boundaries of segmented objects are. This is important for realistic background replacement. The paper found that some fast models blurred hands or lost them entirely, while models like MODNet and RVM preserved loose hair strands.
- Licensing Considerations
- The license attached to a model determines if it can be used in commercial products. Permissive licenses like Apache-2.0 are safe for closed source products, whereas restrictive licenses require offering a commercial license.
Terminology
Summary
Background blur and background replacement are now normal parts of a video call Many open models and open source pipelines can do this, but it is hard to know which one to pick Speed, edge quality and license all matter, and they are often reported on different hardware, or not reported at all. p. 0
Methodology and Testing Setup
The authors ran 20 setups on the same laptop CPU with the same 640x480 frames and the same output view. These setups included 15 open models and 5 complete open source pipelines. p. 0 They measured frames per second on a test clip and on a live webcam, compared the masks on a public domain portrait and on three video call photos with a raised hand. p. 0
Performance Benchmarks
The fastest setups reach more than 40 frames per second on a laptop CPU. The slowest, a high detail model for still images, needs several seconds per frame. p. 0 The runtime library and the steps around the model can change the speed by several times. p. 0
Quality Assessment Findings
The authors compared masks visually on four photos, looking at background leaks, hair, and fingers. For instance, MODNet (setup 5) and RVM (setup 6) keep loose hair strands. p. 6 However, most fast segmentation models blurred the hand or lost it entirely, even in close front camera views. p. 9
Model Selection Guide
The selection guide maps common needs to a recommended model. For weak or old computers, MediaPipe Selfie (landscape) was the fastest setup by a wide margin. p. 7 For hair and fingers, MODNet and RVM keep loose hair strands. p. 7
Practical Observations
The runtime library can change speed many times over. The steps around the model cost time, with most extra time going to the edge step. p. 5 Small mistakes like those in gregblur can be easy to miss, so the mask itself should always be checked. p. 7
Licensing Considerations
The license decides whether a model can be shipped in a product. Permissive licenses like Apache-2. p. 50, MIT, and BSD-3-Clause are the safe choice for closed source products. p. 7 GPL-3.0 and AGPL-3. p. 20 require a commercial license where one is offered. p. 7
Conclusion
Real time on a CPU is easy to reach with small models such as MediaPipe Selfie, ncnn ERDNet, the ML Kit model in the Volcomix pipeline and PP-HumanSegV2. The softest edges and the best hair came from matting models such as RVM and MODNet. p. 9 Raised hands and fingers are the hardest part. p. 9 Most fast segmentation models blurred the hand or lost it entirely, even in close front camera views. p. 9 The runtime library and the steps around the model can also change the speed by several times. p. 0 The selection guide in Section VII helps developers pick a model without running all of them first. p. 9
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Improve model selection by using a specific guide based on needs such as
a weak laptop, the best hair edges, a closed source product, or people behind the speaker.
This allows developers to choose models with known trade-offs regarding speed and quality for their target hardware and use case. -
Optimize pipeline performance by measuring each step individually:
Each step of a pipeline should be measured on its own.
This helps identify bottlenecks, as the paper shows thatMost of the extra time goes to the edge step, which looks at every pixel near the person’s edge.
-
Enhance runtime efficiency by testing different environments:
Before calling a model slow, it is worth trying another runtime or another version.
This suggests that using a specific runtime library can significantly impact speed, as demonstrated by variations in inference time for the same network. -
Improve mask quality for fine details by selecting matting models:
MODNet and RVM keep loose hair strands (Fig. 6).
This indicates that matting models are superior for preserving hair detail compared to general segmentation models like DeepLabV3, which can leave adark haze around the hair.
-
Address issues with hand and finger segmentation by prioritizing specific models:
MODNet and RVM keep separate fingers all three times,
suggesting these are the best choices when hands and fingers matter in video call photos. -
Mitigate flickering masks by incorporating temporal smoothing:
Temporal smoothing mixes in the mask of the previous frame, so a still edge does not flicker.
This step is crucial for improving stability in video streams, as seen withWebinar HumanSeg pipeline
where adding this step helped reduce flicker. -
Improve robustness against background noise by choosing appropriate clean-up steps:
Mask clean up uses thresholds, small blurs or a few pixels of growth to remove specks and holes.
This step can improve results but must be balanced against the time cost, asthe extra time goes to the edge step.
-
Enhance person separation in crowded scenes: Use YOLO26n-seg when people behind the speaker are present because
YOLO is the only model in our test that gives a separate mask for each person, so it can keep only the biggest one.
-
Improve handling of foreground objects: Select BiRefNet or U2-Net when needing clean outlines on portraits or specific foreground elements, as these models were noted to give
the cleanest edge of all
and aclean, smooth outline
for the portrait test image. -
Optimize input size selection: Use smaller inputs when necessary because
A bigger input does not help a fixed size model,
and small models are better for preserving thin parts like hair and fingers at lower resolutions like 256x144.
Sources
- PP-HumanSeg: Connectivity-Aware Portrait Segmentation with a Large-Scale Teleconferencing Video Dataset
- SINet: Extreme Lightweight Portrait Segmentation Networks with Spatial Squeeze Modules and Information Blocking Decoder
- MODNet: Real-Time Trimap-Free Portrait Matting via Objective Decomposition
- Robust High-Resolution Video Matting with Temporal Guidance
- Rethinking Atrous Convolution for Semantic Image Segmentation
- Searching for MobileNetV3
- U$^2$-Net: Going Deeper with Nested U-Structure for Salient Object Detection
- Bilateral Reference for High-Resolution Dichotomous Image Segmentation
Related papers
- HDNet: High-resolution Dual-domain Learning for Spectral Compressive Imaging
- Mask-guided Spectral-wise Transformer for Efficient Hyperspectral Image Reconstruction
- FDIR: Fidelity-Anchored Decompressed Image Restoration for Human and Machine Perception
- A Disease-Centric Vision-Language Foundation Model for Precision Oncology in Kidney Cancer
- CADS: A Comprehensive Anatomical Dataset and Segmentation for Whole-Body Anatomy in Computed Tomography
- Task-Aware Bitrate Thresholds for Codec-Robust Ultrasound Segmentation