Selecting Person Segmentation Models for Real-Time Video Call Backgrounds on the CPU

summary

The gist

Background blur and background replacement are now normal parts of a video call Many open models and open source pipelines can do this, but it is hard to know which one to pick Speed…

In short

The study tested various open models and pipelines for real-time background blur and replacement during video calls on a laptop CPU. While some models achieved high speed, others struggled with quality, particularly around hands and hair. The findings provide a guide to selecting the right model based on hardware constraints and specific quality needs.

Key concepts

Open Models vs. Pipelines
Open models are individual AI algorithms you can use directly, while pipelines are complete workflows combining multiple steps (like preprocessing, model inference, and post-processing). The study compared both to see how the structure of the process affects speed and final output quality.
Frames Per Second (FPS)
FPS measures how many video frames a system can process in one second. High FPS is crucial for smooth, real-time video calls. The research showed that while some setups reached over 40 FPS on a laptop CPU, the speed can vary significantly based on the model and surrounding code.
Edge Quality
Edge quality refers to how sharp and well-defined the boundaries of segmented objects are. This is important for realistic background replacement. The paper found that some fast models blurred hands or lost them entirely, while models like MODNet and RVM preserved loose hair strands.
Licensing Considerations
The license attached to a model determines if it can be used in commercial products. Permissive licenses like Apache-2.0 are safe for closed source products, whereas restrictive licenses require offering a commercial license.

Terminology used across episodes

This episode discusses

The paper

Selecting Person Segmentation Models for Real-Time Video Call Backgrounds on the CPU · Read on arXiv

Transcript

Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.

Rosa: We're looking at the paper: "Selecting Person Segmentation Models for Real-Time Video Call Backgrounds on the CPU".

Dev: Here's the gist of the paper. blur and background replacement are now normal parts of a video call Many open models and open source pipelines can do this, but it is hard to know which one to pick Speed, edge quality and license all matter, and they are often reported on different hardware, or not reported at all.

Rosa: The authors ran 20 setups on the same laptop CPU with the same 640x480 frames and the same output view.

Dev: These setups included 15 open models and 5 complete open source pipelines.

Rosa: They measured frames per second on a test clip and on a live webcam, compared the masks on a public domain portrait and on three video call photos with a raised hand.

Paper summary: Rosa: So, we're looking at this paper, "Selecting Person Segmentation Models for Real-Time Video Call Backgrounds on the CPU." Basically, they’re trying to figure out which person segmentation models work best for background blur and replacement in video calls when you’re running it on a standard laptop CPU.

Dev: Right. The core idea is that it's not just about finding *a* model, but knowing which one to pick because speed, the quality of the edge—how sharp the mask is—and the license all matter for different uses.

Taro: It sounds like they’re running a test on a laptop CPU with six hundred forty by four hundred eighty frames and comparing fifteen open models against five complete open source pipelines. Rosa That’s right, they ran twenty setups total, which included both the individual models and those full pipelines.

Dev: And what they measured was frames per second on both a test clip and a live webcam feed, plus they compared the masks on some public domain portraits and also on three actual video call photos where someone had their hand raised. Rosa That’s important because it tests how well it handles things that move around, like hands.

Taro: I saw something specific about quality assessment in the abstract—they looked at background leaks, hair, and fingers when comparing the masks visually across those four photos. Dev And they found some specific trade-offs there.

Rosa: They mentioned that models like MODNet and RVM tend to keep loose hair strands, which is good for realism sometimes. But the authors also pointed out that most of the fast segmentation models they tested ended up blurring the hand or losing it completely, even when you filmed it with a close front camera view.

Dev: It seems like there's a direct conflict between getting high speed and getting those fine details, especially with hands and hair. Taro I think that’s the main tension in this work.

Rosa: So, they offered a selection guide mapping common needs to specific models based on what works for weak or old computers. Dev For instance, they recommended MediaPipe Selfie in landscape mode as the fastest setup by a wide margin for less powerful machines.

Taro: And when it comes to those tricky details, like hair and fingers, they suggested MODNet and RVM are the models that keep loose hair strands intact. Rosa That gives us a concrete starting point for developers looking at this research.

Dev: But we have to remember the practical observations from their testing—the runtime library and all the steps surrounding the model can change your speed many times over. Taro And those extra steps, they said, most of that extra time went to the edge step of processing.

Rosa: That’s a big warning for anyone building something in real time on a CPU because those small mistakes in things like gregblur can be easy to miss if you aren't careful. Dev So, even if you pick a fast model, you have to be vigilant about checking the mask itself.

Taro: And they also touched on licensing, which is crucial for shipping any product. They said permissive licenses like Apache-two point zero, MIT, and BSD-three-Clause are the safe bet for closed source products <ref:2611.14558#pg2>. Rosa GPL and AGPL require you to offer a commercial license if you use them.

Dev: It’s a big picture point—the choice of license dictates how that technology can actually be used in a commercial product versus keeping it open source. So, when you look at this paper, remember that the model choice isn't just about accuracy; it's about speed and what you are allowed to do with the resulting software.

Taro: This whole paper is really about making that decision process less of a guessing game for people building these types of video tools. Rosa It points out that real time on a CPU is achievable with smaller models like MediaPipe Selfie, ncnn ERDNet, or even the ML Kit model in the Volcomix pipeline.

Dev: But if you want those soft edges and the best hair quality, you might have to lean towards matting models like RVM or MODNet. Taro And as we talked about earlier, raised hands and fingers are definitely the hardest parts to get right with these fast segmentation models; they often just blur them out entirely.

Rosa: So, the selection guide they put together in section seven is supposed to help developers pick a model without having to run all of them first. Dev It’s about giving people a roadmap based on what their specific needs are and what kind of hardware they're working with.

Taro: The paper really highlights that the speed and quality aren't always in the same place, especially when you bring in complex things like hand segmentation. Rosa It’s a practical guide for anyone working on real-time video processing today.

Conclusion: Rosa: So we’re talking about this paper called "Selecting Person Segmentation Models for Real-Time Video Call Backgrounds on the CPU."

Dev: Yeah, it looks at how to pick the right segmentation model when you need background stuff in a video call running on a standard laptop computer.

Rosa: The authors ran twenty different setups, testing fifteen open models and five full pipelines across the same laptop CPU setup.

Dev: They measured speed in frames per second on both test clips and live webcams, and they also checked how the masks looked on public photos of portraits and actual video calls with hands raised.

Rosa: What they found is that speed isn't everything; you have to balance it with edge quality and licensing.

Dev: They showed that some fast models blur hands or lose them completely, even when the camera is close up, while others like MODNet or RVM keep loose hair strands better.

Rosa: The selection guide they wrote helps people figure out which model to use based on whether their computer is weak or if they need those specific details like hair.

Dev: But they also warned that the way you run the model, the runtime library and all those extra steps, can change your speed by several times over.

Rosa: And that means you gotta check the mask itself really carefully because small mistakes in things like background removal can be easy to miss if you aren't paying attention.

Dev: It also touches on licensing; they said permissive licenses like Apache-two point zero are the safe choice for products, while GPL and AGPL require you to offer a commercial license if you use those models.

Rosa: Basically, this paper gives us a roadmap for picking tools without having to try every single one of them first.

Dev: It shows that real time on a CPU is possible with smaller models like MediaPipe Selfie or ncnn ERDNet, but if you want the best hair and fingers, you might need to use those matting models.

Rosa: So it’s about picking the right tool for the job instead of just chasing the fastest number.

Dev: And that leads us to a big question—how do we know which model is actually going to be good enough for a real video call without spending hours testing everything?

More episodes

← Home