UnderOneFacade: Worldwide Facade Semantic Segmentation Benchmark Dataset

arXiv:2607.02018 · cs.CV · Submitted 2026-07-02 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "UnderOneFacade: Worldwide Facade Semantic Segmentation Benchmark Dataset".

Jane: Here's the gist of the paper. Globally consistent semantic digital twins require centimeter-accurate and geographically transferable 3D facade segmentation UnderOneFacade: Worldwide Facade Semantic Segmentation Benchmark Dataset The dataset, evaluation scripts, and pretrained models are available here: https://jiangyuanwangyi.github.io/UnderOneFacade official/ Dataset Introduction The paper introduces UnderOneFacade, the largest cross-country 3D facade benchmark to date, which comprises centimeter-accurate point clouds with hierarchical, harmonized, and architecturally grounded semantic labels totaling 2.7 billion annotated points The dataset spans multiple cities and architectural styles across different countries, enabling systematic evaluation of cross-country and cross-continent generalization Dataset Construction The UnderOneFacade dataset was constructed by prioritizing highest possible geometric accuracy through laser scanners with up to centimeter local accuracy and globally georeferenced centimeter acquisition Three representative regions were selected to capture architectural diversity across continents, including the European subsets (UK and Germany) and the Singapore subset The annotation process involved manual labeling by trained annotators following architectural standards, resulting in roughly 5,500 hours of manual annotation for the full dataset of about 366 facades Semantic Harmonization To ensure consistent semantic interpretation across diverse contexts, the Level of Facade Generalization (LoFG) concept was adopted to unify semantic labels LoFG defines three levels of detail (LoFG1, LoFG2, and LoFG3) allowing precise formulation of segmentation tasks Specifically, the taxonomy includes 15 detailed classes at LoFG3, aggregated into 5 coarser classes at LoFG2, and a single abstract class at LoFG1 Evaluation Challenges UnderOneFacade is designed to evaluate facade segmentation along three benchmark axes: (a) cross-country domain shifts, (b) hierarchical semantic granularity, and (c) realistic long-tailed class distributions The aggregation of multiple countries results in a markedly heavier-tailed distribution compared to single-country datasets, opening new challenges Furthermore, crosscontinental generalization shows a clear asymmetry where Asia→Europe consistently outperforms Europe→Asia at the fine-grained LoFG3 level Experimental Findings Overall performance on hierarchical semantic segmentation remains challenging, with even the strongest model reaching only about 33 mIoU on the combined dataset Graph-based DGCNN consistently outperformed several more recent transformer-based architectures, suggesting current transformer designs may not yet effectively capture the geometric structure of facade point clouds Radiometric features can complement geometric features for small facade elements, increasing mIoU by +7.3 for PointNet++ and +8.1 for PT However, radiometric cues are not universally beneficial and can even degrade performance UnderOneFacade provides an important step toward more robust facade understanding and high definition urban digital twins The benchmark highlights the limited robustness of existing models to architectural variability and long-tailed facade semantics The dataset is released under an open data license (CC BY 4.0, Creative Commons Attribution)

Title and authors: Tom: So we're looking at this paper today called "UnderOneFacade: Worldwide Facade Semantic Segmentation Benchmark Dataset" by Yi Wang and his team, which is pretty big because it tackles the problem of getting consistent digital twins for cities globally.

Jane: It sounds like they've put together a massive dataset, right? What's the main goal here in simple terms for someone who isn't deep into three dee computer vision <ref:2607.02018#pg1>?

Tom: The main point is that to build truly consistent digital twins of buildings around the world, you need three dee data that is super accurate—centimeter accurate—and labels that make sense no matter which country or city you're looking at <ref:2607.02018#pg1>.

Lu: Exactly. They’re solving the problem where datasets are either too local or not precise enough for cross-country work, and this dataset aims to fix that by providing a standardized, large-scale resource for testing models across continents.

Meng: So what's actually in this dataset? Is it just a bunch of random building scans, or is there something specific about how they built it? I worry that if the labeling process isn't rigorous, the results won't be reliable at scale.

Tom: It comprises two point seven billion annotated points, which is huge for a benchmark; they got centimeter-accurate point clouds and those labels are hierarchical and architecturally grounded across multiple cities and countries including Europe and Singapore <ref:2607.02018#pg2,2.7 billion annotated points>.

Jane: Centimeter accuracy is key—so this isn't just a rough idea of a building; it’s precise enough to delineate the smallest structural elements, which I think makes the semantic segmentation much more meaningful for real-world applications.

Lu: And they didn't just do point clouds; they created these hierarchical, harmonized semantic labels using something called the Level of Facade Generalization or LoFG concept, which sets three levels of detail: LoFG1, LoFG2, and LoFG3.

Tom: That structure is important because it lets them systematically evaluate models at different levels of detail for segmentation tasks across these diverse regions.

Jane: So when we talk about this dataset being cross-continental, what's the biggest challenge they found when testing how well current methods transfer from one country to another?

Lu: They found that current methods struggle with cross-country domain shifts, and there’s a clear asymmetry in performance: Asia training data consistently gives stronger supervision for rare classes at the fine detail level.

Meng: Asymmetry is interesting, but from an engineering standpoint, if we're deploying models globally, we need to know how much effort it takes to adapt it when the source domain shifts. Does this paper suggest a specific strategy for handling that unevenness?

Title and authors: Tom: It shows that the robustness of existing models against architectural variability depends on having balanced semantic coverage in the source domain, meaning you can't just train on one region and expect it to work everywhere.

Jane: That makes sense. And I'm also curious about how they handle complexity; with two point seven billion points, how do you keep the semantic labels consistent across all those different architectural styles <ref:2607.02018#pg2>?

Lu: They used architecturally grounded semantics, which means the class definitions are standardized and based on architectural standards rather than just ad hoc labeling for each specific dataset.

Meng: Standardized taxonomies are good in theory, but when you have so many different styles, how much manual annotation time did this whole process take? I need to know the cost of getting this kind of data.

Tom: It involved roughly five thousand five hundred hours of manual annotation for about three hundred sixty-six facades across all those regions. It’s a lot of work, but they argue it’s necessary to achieve that level of geometric and semantic alignment.

Jane: So, what are the actual performance numbers we should expect? Because if the models struggle with cross-country generalization, how much improvement did they actually see on their best results?

Lu: Even the strongest models only reached about thirty-three mIoU on the fine-grained LoFG3 benchmark, which tells us that achieving high performance across this massive scale is still quite challenging <ref:2607.02018#pg2,on the fine-grained LoFG3 benchmark>.

Meng: Thirty-three mIoU sounds low for something this ambitious, especially when we think about deploying this on real traffic where we need better reliability. What did they suggest to boost those numbers?

Tom: They found that combining the centimeter-level geometric precision with radiometric features, or RGBI, can help, increasing the IoU by plus seven point three for PointNet++ and plus eight point one for PT.

Jane: Wait, so adding visual cues from the actual image data helps small elements like windows and doors more than just relying on the three dee geometry alone <ref:2607.02018#pg1>? That’s a practical win for parsing real-world facades.

Lu: It is beneficial for small facade elements, and they found that radiometric cues can actually increase mIoU by plus thirty F1 for windows and plus twenty four F1 for doors, though they also noted those cues aren't universally helpful and can sometimes degrade performance.

Title and authors: Meng: That caveat about degradation is important; if we rely too much on image data for something meant to be purely geometric, we risk introducing noise that breaks deployment consistency. What about the long-tailed distributions they mentioned?

Tom: The paper highlights that the aggregation of multiple countries results in a markedly heavier-tailed distribution compared to single-country datasets, which is a major challenge for current models.

Jane: So, how does the LoFG concept help with those long tails? Is it just about giving models more ways to express what they are seeing?

Lu: Yes, LoFG1 defines one abstract class at the lowest level, LoFG2 has five coarser classes, and LoFG3 has fifteen detailed classes; this hierarchy supports transfer learning and label propagation by allowing models to learn features at different granularities.

Tom: It’s about giving the AI a roadmap for how much detail it needs to focus on when interpreting a facade element, which is crucial when you have rare or small elements appearing sporadically.

Jane: So, looking ahead, what's the big picture here? What does this benchmark actually mean for someone trying to design better urban digital twins or autonomous driving systems that need to understand their surroundings?

Lu: It establishes a rigorous benchmark for developing robust and transferable three dee segmentation models that can handle cross-domain challenges without needing specific target-domain adaptation beforehand <ref:2607.02018#pg1>.

Meng: For production, this means we need to test our current models not just on one city, but across continents to see where they fail structurally before we commit to deployment. It forces us to confront those architectural shifts earlier in the pipeline.

Tom: It’s about moving past local datasets toward a unified understanding of facade semantics that works globally, which is a huge step for any system trying to map the real world digitally.

Jane: So, to wrap up on UnderOneFacade, it’s a really high-quality dataset that forces us to confront the limitations of current models when they encounter real architectural diversity and long-tailed semantic distributions.

Lu: It provides a crucial tool for understanding where we need more robust representation learning techniques, especially concerning how we encode that hierarchy of detail.

Meng: I just hope the community uses this benchmark correctly so we don't end up wasting time training models on data that doesn't actually represent real-world deployment conditions at scale.

Lalam: From my perspective, the paper confirms that while current transformer designs still struggle to capture the geometric structure of these point clouds, combining geometric precision with standardized semantics is definitely the way forward for building reliable digital twins.

The paper's summary: Tom: So, to recap, UnderOneFacade is this massive dataset they put together for facade segmentation that spans multiple cities and countries, aiming for centimeter accuracy across all those different architectural styles globally.

Jane: Right, so it's a huge collection of two point seven billion annotated points trying to solve the problem of getting consistent labels no matter where the building is.

Tom: Exactly, and they built this thing by prioritizing that geometric precision using laser scanners that are accurate within a centimeter locally and georeferenced globally for each acquisition.

Lu: And then they didn't just stop there; they created this way to make sure the semantic labels—the actual categories for the parts of the facade—are harmonized across all these different contexts.

Jane: What does that harmonization mean in plain English? Like, how do you make a label for a window in New York look like it makes sense when applied to a building in Singapore?

Lu: They used this concept called Level of Facade Generalization or LoFG, which sets three levels of detail. You have LoFG one, LoFG two with five coarser classes, and then lo FG three with fifteen really detailed classes for the taxonomy.

Tom: That way, the AI doesn't just get one label; it has a set of choices based on how much detail it needs to focus on for that specific task.

Jane: So if I'm trying to build a system, I can pick the level of detail that fits my need, right? Like if I only care about big structural elements, maybe LoFG two is enough.

Lu: Precisely. It lets you formulate segmentation tasks with different levels of granularity, from very abstract ideas down to fifteen specific classes.

Tom: The challenge they set for this benchmark is three things: how well models handle shifts between different countries and continents, how good the hierarchy of those semantic labels is, and dealing with those really long-tailed class distributions.

Jane: Long-tailed distributions sound tricky; like having a few very common building parts that show up everywhere, but also some super rare or weird elements that only appear in one specific place.

Lu: Exactly. The aggregation of multiple countries makes the distribution much heavier tailed than if you just tested on one country alone, which is what they call a cross-country domain shift challenge.

Tom: And they found that when testing how well models transfer between continents, there's this clear asymmetry; Asian training data actually gives stronger supervision for those rare classes at the fine detail level.

Jane: So if you're training a model, using data from Asia gives it a better head start when it has to figure out what a very specific, rare element is in Europe?

Lu: It means that balanced semantic coverage in your source data really plays a key role when you want models to generalize across different architectural styles.

Tom: Overall performance on this hierarchical segmentation is still tough; even the best model only got about thirty-three mean Intersection over Union on the combined dataset, which shows how hard it is to get perfect understanding across this whole mess.

Jane: So, what's the practical value here? What does this benchmark actually change for someone who's trying to build a digital twin or something that needs to understand buildings globally?

Lu: It provides a rigorous testing ground for developing three dee segmentation models that are robust and transferable without needing specific target-domain adaptation beforehand.

Tom: It forces us to confront the limitations of existing models when they encounter real architectural diversity and those long-tailed semantic distributions, which is a big hurdle in production.

Jane: So, it's about moving past just testing on one city and really seeing how your system holds up when it has to handle a whole world of buildings at once.

Lu: It's about understanding where you need better representation learning techniques, especially concerning how you encode that hierarchy of detail so the AI knows what to focus on.

Tom: And they also showed that adding radiometric features, like those RGBI cues from the actual image data, can actually help improve performance on small facade elements like windows and doors.

Jane: So we're not just relying on three dee shape; we're using visual information too to get better segmentation for those little things.

Lu: It’s a win for small parts, but they also flagged that these radiometric cues aren't always helpful and can sometimes even degrade performance if you use them wrong.

Tom: So, the next thing we look at is how this dataset helps us actually improve those models, and what else is happening in the three dee vision space right now.

The paper's improvements: Tom: So, we've seen how they built this dataset, and now we're looking at what they suggest to actually make these AI models better when they use UnderOneFacade.

Jane: What are the actual tweaks or improvements that the authors propose to get better results from this benchmark?

Lu: They focus on fixing those challenges we talked about: cross-country shifts, hierarchical granularity, and those long-tailed class distributions.

Tom: Specifically, they introduce a concept called Level of Facade Generalization again, LoFG1 through LoFG3. They say this helps support transfer learning and label propagation by letting models learn features at different levels of detail.

Jane: So instead of trying to solve everything at once, the AI can choose how much detail it wants to focus on, right? That sounds like a smart way to handle those complex architectural shifts.

Lu: Exactly. LoFG3 gives you fifteen detailed classes, while LoFG1 is just one abstract class; it's a roadmap for how deep the model needs to go when interpreting the facade.

Tom: Then they look at radiometric features, which are basically the RGBI or color and intensity data from the image itself. They found that combining this with geometry can boost performance on small elements like windows and doors.

Jane: Wait, so it's not just about the three dee shape; we also need to look at the actual colors and lighting in the picture to get a better idea of what those small parts are?

Lu: That's correct. They saw gains, for example, an increase of plus thirty F1 for windows and plus twenty four F1 for doors when they incorporated these radiometric cues.

Tom: But Lalam points out a caveat here; those radiometric cues aren't universally good, and they can even degrade performance in some cases if you use them poorly.

Jane: So it's not a magic fix that just makes everything better; it depends on how the model uses that extra image data.

Lu: Right. And regarding the long-tailed distributions, they suggest that because of this unevenness, models need balanced semantic coverage in their training data to be robust across different regions.

Tom: So the implication is that if you want a system to work anywhere, you can't just train it on one city; you need to expose it to a broader mix of styles and classes upfront.

Jane: It sounds like the focus shifts from just getting a high number on this benchmark to actually building models that are resilient when they encounter real-world variety.

Lu: Precisely. This paper sets up the problem perfectly, but the solution is in how we structure our data and how we define our segmentation tasks through that LoFG concept.

Tom: It's a step toward more robust facade understanding, but it's still a tough test to pass on, which means there’s still a lot of room for improvement in the underlying geometric architectures.

Jane: So, while the benchmark is useful for testing and comparing models, we still need better ways to make those models inherently understand complex 3d structures across continents <ref:2607.02018#pg1>.

Conclusion: Tom: So we've gone through the whole thing on UnderOneFacade, and to wrap up, this dataset is really pushing AI toward more robust facade understanding globally by providing centimeter accuracy across diverse regions like Europe and Singapore.

Jane: It’s a really important step for building those accurate digital twins we talked about earlier because it gives us a massive resource to test models on before they go into the real world.

Lu: This benchmark highlights that current AI models still struggle with architectural variability and those long-tailed semantic distributions across different countries.

Meng: For me, it means we need to stop thinking about just one city and start thinking about how our systems handle massive data shifts when we deploy them globally.

Lalam: I’m sure this will be a crucial tool for anyone building systems that need to understand the physical world at scale, because the paper confirms its importance.

Tom: Yeah, and they did show that combining geometric precision with radiometric features can actually give us some real gains on those small elements like windows and doors.

Jane: It’s not just about the shape anymore; it’s about how the AI uses all the visual information available to make a call.

Lu: The LoFG concept is key here because it gives models a way to handle that complexity by letting them focus on different levels of detail for segmentation tasks.

Meng: But we still need to be careful; they admitted those radiometric cues aren't always helpful and can even cause problems if you don't use them right.

Tom: So, the whole point is that while this is a big step, it shows us where the current models are still lacking in handling real-world architectural messiness.

Jane: It sets a high bar for what we need in terms of generalization, forcing us to design systems that can handle cross-country differences without needing specific training for every new location.

Lu: I think the real implication is that we need better representation learning techniques so the AI can truly learn those underlying structural patterns regardless of which country it sees.

Meng: From an engineering standpoint, it’s good to have this benchmark, but nobody should just assume a model trained on this will work perfectly in production without extra validation against real traffic data.

Lalam: I'm pretty sure the authors are right; the UnderOneFacade dataset is a major contribution because it forces us to confront these hard generalization problems directly.

Tom: So, that’s our take on UnderOneFacade—a huge dataset that shows us exactly where we need to improve our 3d vision models for global application <ref:2607.02018#pg1>.

Jane: Next time we look at a paper, we'll see how other researchers are trying to tackle these same generalization issues in different ways.

Yi Wang, Fan Wang, Wanru Yang, Prabin Gyawali, Ziyang Xu, Anna Klimkowska, Yixiong Jing, Filip Biljecki, Christoph Holst, Benjamin Busam

Technical University of Munich University of Cambridge University of Nottingham National University of Singapore

cs.CV

Submitted: 2026-07-02

Updated: 2026-11-16

Comments: accepted by ECCV 2026

Project page: https://jiangyuanwangyi.github.io/UnderOneFacade_official

License: http://creativecommons.org/licenses/by-sa/4.0/

Importance score: 85/100

The gist: Globally consistent semantic digital twins require centimeter-accurate and geographically transferable 3D facade segmentation UnderOneFacade…

Key concepts

UnderOneFacade
A large benchmark dataset of 2.7 billion annotated points representing facades from various cities and countries worldwide. It provides centimeter-accurate 3D data with detailed semantic labels to test the robustness of facade segmentation AI models.
Level of Facade Generalization (LoFG)
A concept used to unify semantic labels by defining three levels of detail: LoFG1, LoFG2, and LoFG3. This allows for precise segmentation tasks by aggregating 15 detailed classes at the highest level down to a single abstract class.
Cross-country Generalization
The ability of AI models trained on one geographic region (e.g., Asia) to accurately segment facades in a completely different region (e.g., Europe). The dataset shows this ability is asymmetrical, with Asian data providing stronger supervision for rare classes.
Radiometric Features
Visual features derived from RGB images that can be added to geometric features for small facade elements like windows and doors. These cues can significantly improve segmentation performance, increasing metrics like mIoU by substantial amounts.

Terminology

Summary

Globally consistent semantic digital twins require centimeter-accurate and geographically transferable 3D facade segmentation UnderOneFacade: Worldwide Facade Semantic Segmentation Benchmark Dataset The dataset, evaluation scripts, and pretrained models are available here: https://jiangyuanwangyi. p. 1

Where the paper says it

The dataset, evaluation scripts, and pretrained models are available here: https://jiangyuanwangyi. io/UnderOneFacade_official/ Keywords: 3D Point Cloud Segmentation · Urban Scene Understanding · Cross-Domain Generalization 1 Introduction Facade semantic segmentation has been studied since the early days of computer vision [41], owing to the structured yet intricate patterns present in building facades and their importance for urban scene understanding, large-scale mapping, and semantic digital twins.

github.io/UnderOneFacade official/

Dataset Introduction

The paper introduces UnderOneFacade, the largest cross-country 3D facade benchmark to date, which comprises centimeter-accurate point clouds with hierarchical, harmonized, and architecturally grounded semantic labels totaling 2.7 billion annotated points The dataset spans multiple cities and architectural styles across different countries, enabling systematic evaluation of cross-country and cross-continent generalization

Dataset Construction

The UnderOneFacade dataset was constructed by prioritizing highest possible geometric accuracy through laser scanners with up to centimeter local accuracy and globally georeferenced centimeter acquisition Three representative regions were selected to capture architectural diversity across continents, including the European subsets (UK and Germany) and the Singapore subset The annotation process involved manual labeling by trained annotators following architectural standards, resulting in roughly 5,500 hours of manual annotation for the full dataset of about 366 facades

Semantic Harmonization

To ensure consistent semantic interpretation across diverse contexts, the Level of Facade Generalization (LoFG) concept was adopted to unify semantic labels LoFG defines three levels of detail (LoFG1, LoFG2, and LoFG3) allowing precise formulation of segmentation tasks Specifically, the taxonomy includes 15 detailed classes at LoFG3, aggregated into 5 coarser classes at LoFG2, and a single abstract class at LoFG1

Evaluation Challenges

UnderOneFacade is designed to evaluate facade segmentation along three benchmark axes: (a) cross-country domain shifts, (b) hierarchical semantic granularity, and (c) realistic long-tailed class distributions The aggregation of multiple countries results in a markedly heavier-tailed distribution compared to single-country datasets, opening new challenges Furthermore, crosscontinental generalization shows a clear asymmetry where Asia→Europe consistently outperforms Europe→Asia at the fine-grained LoFG3 level

Experimental Findings

Overall performance on hierarchical semantic segmentation remains challenging, with even the strongest model reaching only about 33 mIoU on the combined dataset Graph-based DGCNN consistently outperformed several more recent transformer-based architectures, suggesting current transformer designs may not yet effectively capture the geometric structure of facade point clouds Radiometric features can complement geometric features for small facade elements, increasing mIoU by +7.3 for PointNet++ and +8. p. 2

Where the paper says it

More recent methods process raw points directly. Point-based networks such as PointNet and PointNet++ [35, 36] introduced hierarchical aggregation to capture local geometric context, which is critical for thin and repetitive architectural structures.

1 for PT However, radiometric cues are not universally beneficial and can even degrade performance

Conclusion

UnderOneFacade provides an important step toward more robust facade understanding and high definition urban digital twins The benchmark highlights the limited robustness of existing models to architectural variability and long-tailed facade semantics The dataset is released under an open data license (CC BY 4.0, Creative Commons Attribution) p. 7

Where the paper says it

The dataset is released under an open data license (CC BY 4. 0, Creative Commons Attribution), hosted on a cloud service (i.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Centimeter-accurate geometric fidelity combined with unified semantic annotations allows AI systems to perform fine-grained architectural parsing across different countries and styles, as demonstrated by UnderOneFacade's 2.7 billion annotated points.

  2. The system can systematically evaluate cross-country and cross-continent generalization by testing models on unseen geographic regions without target-domain adaptation, revealing a clear asymmetry in transfer performance where Asian training data provides stronger supervision for rare classes.

  3. AI systems can be trained to handle long-tailed distributions of structural elements by utilizing the Level of Facade Generalization (LoFG) concept, which defines three levels of detail (LoFG1, LoFG2, and LoFG3) to support transfer learning and label propagation.

  4. The system can improve performance on small facade elements like windows and doors by incorporating radiometric features (RGBI), as these cues can lead to gains exceeding +30 F1 for windows and +24 F1 for doors.

  5. AI models can be designed to be more robust against architectural shifts by analyzing the impact of domain shift, which reveals a clear diagonal trend is distinguishable in cross-continental generalization performance, indicating that balanced semantic coverage in the source domain plays a key role.

Abstract

Globally consistent semantic digital twins require centimeter-accurate and geographically transferable 3D facade segmentation. However, progress in facade parsing is limited by the lack of large-scale, standardized benchmarks for evaluating cross-domain generalization. Existing datasets are geographically narrow, semantically inconsistent, or insufficiently precise. We introduce UnderOneFacade, the largest cross-country and cross-continent 3D facade benchmark to date, comprising centimeter-accurate point clouds with hierarchical, harmonized, and architecturally grounded semantic labels totaling 2.7 billion annotated points. Through a systematic evaluation of representative point-, graph- and transformer-based architectures, we show that current methods struggle to recognize fine-grained architectural elements and degrade significantly across geographic domains, with the best models achieving only up to 33 IoU on the fine-grained LoFG3 benchmark. By combining geometric precision with standardized semantics at unprecedented scale, UnderOneFacade establishes a rigorous benchmark for developing robust and transferable 3D segmentation models. The dataset, evaluation scripts, and pretrained models will be released upon publication.

Sources

Related papers