Disarranged Harmonization of Transparency Reporting by Social Media Platforms Under the Digital Services Act

summary

The gist

The European Commission recently introduced new regulation to harmonize transparency reporting of large online platforms under the Digital Services Act (DSA) and this paper presents a systematic…

In short

This study evaluated transparency reporting data quality from eight major social media platforms after new Digital Services Act regulations. Findings show that despite formal harmonization efforts, platforms used different procedures, leading to inconsistent and sometimes contradictory data between their different reporting mechanisms. This suggests that simple standardization isn't enough for reliable oversight.

Key concepts

Transparency Reports (TRs)
These are the reports social media platforms submit detailing their content moderation activities. The study assessed how well these reports met DSA requirements regarding formatting, timeliness, and completeness.
Transparency Database (TDB)
This is the structured data repository that holds the raw information related to platform reporting. Researchers compared this database data against the TRs to check for consistency between the two sources.
Reporting Quality Dimensions
These are five key areas used to judge how good a platform's transparency reporting is, such as how many people receive moderation actions or if they use automated tools for content review.
Operational Interoperability
This refers to whether different platforms and their various reporting systems can effectively work together. The study found that while structures were standardized, the actual operational processes remained inconsistent.

Terminology used across episodes

This episode discusses

The paper

Disarranged Harmonization of Transparency Reporting by Social Media Platforms Under the Digital Services Act · Read on arXiv

Amaury Trujillo, Benedetta Tessa, Stefano Cresci

IIT-CNR · University of Pisa

The European Commission recently introduced new regulation to harmonize transparency reporting of large online platforms under the Digital Services Act (DSA). Here, we present the first systematic evaluation of transparency reporting data quality after this normative change, for the eight largest social media platforms in the European Union. In detail, we run a set of large-scale quantitative analyses on key reporting dimensions, followed by a structured comparative assessment across platforms and reporting mechanisms. Among our findings is that: (i) the analyzed platforms had varying degrees of compliance and data quality, but all exhibited issues on data formatting, timeliness, consistency, and completeness; (ii) some platforms employed differing reporting procedures across mechanisms, which caused them to submit contrasting information; (iii) despite the harmonization, a number of issues still prevent interoperability between reporting mechanisms; and (iv) many of the previously identified issues with transparency reporting are still unresolved. We conclude by discussing implications for transparency auditing and proposing key targeted improvements to strengthen the reliability and interoperability of DSA transparency reporting.

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "Disarranged Harmonization of Transparency Reporting by Social Media Platforms Under the Digital Services Act".

Paper summary: Tom: So, we're looking at this paper now called "Disarranged Harmonization of Transparency Reporting by Social Media Platforms Under the Digital Services Act". Essentially, they’ve taken this new rule from the European Commission that forces big online platforms to report their data in a standardized way and they’ve systematically checked how good that reporting actually is.

Jane: Right, so you're asking, after all this new harmonization effort under the DSA, are these transparency reports actually reliable? And what does that mean for us when we try to understand how these massive platforms are actually running their moderation systems?

Lu: The core idea here is that they run a large-scale quantitative analysis on five key dimensions of platform reporting. These include things like the number of active recipients, how fast those reports are submitted, whether they use automated means for content moderation, actions taken by the platform itself, and actions that happen because of user notices.

Meng: Wait, so you’re looking at how they measure the actual quality? I mean when I’m trying to deploy a model at scale, I care less about the abstract dimensions and more about what breaks when we push millions of users.

Tom: Well, Lu says they assign scores across six different dimensions for both the Transparency Database and the Transparency Reports, looking at things like formatting and consistency. But it also found that some platforms used completely different reporting procedures depending on which mechanism they were using, which caused them to submit totally contrasting information.

Jane: So even though the rules are harmonized now, you still have these differences where one platform might be reporting things very differently than another because of how they report it. That sounds like a big problem for comparison.

Lalam: And they found that when comparing the Transparency Reports to the Transparency Database, there are substantial inconsistencies across all platforms, suggesting those two reporting mechanisms might be working on separate workflows entirely. For instance, LinkedIn and YouTube reported orders of magnitude more own-initiative actions in their reports than what you could see in the actual submitted SoRs.

Tom: That’s a huge red flag for auditors because it means you can't just trust one report over the other without knowing exactly how they connect. It points to a lack of operational interoperability, even if the structure looks standardized on paper.

Paper summary: Jane: So what does that mean for us as observers? If the data isn't consistent between the database and the reports, how can we really get a reliable picture of platform behavior?

Lu: The paper suggests that just having a formal standardization isn't enough to guarantee consistency or auditability, which is what they call semantic consistency and operational interoperability. They found that structural changes didn't automatically fix the practical issues between platforms.

Meng: From my side, if we try to build a unified system based on this, we need more than just a common format; we need the actual underlying operational definitions to match up perfectly across different moderation tools. Otherwise, it’s just prettier paperwork.

Tom: They actually make some suggestions for fixing this infrastructure. They propose making more attributes mandatory, especially for things like the grounds for moderation actions, and they suggest explicitly distinguishing what was targeted versus what caused the violation.

Jane: That makes sense practically; if you don't know *why* something got taken down versus *who* initiated the process, it’s hard to tell if you’re seeing a real pattern or just noise. What about how they think we should validate this?

Lalam: They suggest that harmonized Transparency Reports should go through mandatory automated validation against formal schemas and reporting constraints before they are ever published. That forces them to check their own data against the rules before it goes public.

Tom: And the implications for governance are pretty direct: scholars and practitioners need to be careful when they draw large-scale conclusions from this data without doing extra work to validate it themselves. The infrastructure can look uniform, but in practice, it might still be operationally inconsistent.

Jane: So the big picture here is that we have a new set of rules for transparency under the DSA, but the actual data coming out isn't perfectly aligned or consistent yet. It’s like building a highway with all standardized signs, but the roads themselves are still built in different ways depending on which construction crew worked on that section.

Lu: Exactly. The paper points out that moderation effectiveness really depends on operational factors, like the timing of interventions, which often get lost in those aggregated reports they use for transparency.

Paper summary: Meng: That’s what keeps me up at night when I think about deployment; if the report is missing the timing detail of when an action was taken, we can't debug a failure properly. What would we actually need to see to trust this?

Tom: The paper also flags that independent oversight bodies have pointed out that certain interactions, like those involving law enforcement, might not be represented in these reports at all. That’s another gap in the picture.

Jane: It sounds like the challenge isn't just about having data; it's about making sure the data tells a complete story about what platforms are actually doing on the ground when they make moderation decisions. So, what’s next for this whole process?

Lalam: The authors themselves point out that their analysis focused mostly on operational consistency and compliance with DSA reporting obligations, but they admit that modeling the actual substance of content moderation itself is something they didn't do in this study.

Tom: And they also mention a limitation in their own assessment: the comparative quality assessment involves a degree of expert judgment when weighting those criteria and interpreting what constitutes good or bad reporting.

Lu: They also note that because they only looked at eight originally designated Very Large Online Platforms, it might not fully capture the reporting practices across the entire platform ecosystem.

Jane: So, to sum up this paper on "Disarranged Harmonization of Transparency Reporting by Social Media Platforms Under the Digital Services Act", the main point is that while there's a new rule forcing standardization, they found that platforms still report things differently depending on how they report them, and that formal harmonization alone isn't enough to guarantee real consistency.

Tom: The authors are essentially saying we need more than just common formats; we need operational interoperability and clearer definitions for automated moderation actions to make this data useful for real oversight.

Jane: And for the listener who doesn't read papers, it means when you look at the transparency reports online, don't just see a standardized number; be aware that behind that number, there might be totally different ways the platform is actually tracking those moderation events.

Lalam: The implication is that for effective public oversight and auditing to work under the DSA, we need this level of granular detail and validation to bridge the gap between what the law requires and what platforms are actually doing operationally.

Conclusion: Jane: So we've looked at how the new Digital Services Act is making big social media companies report their moderation data, and this paper, "Disarranged Harmonization of Transparency Reporting by Social Media Platforms Under the Digital Services Act," really digs into whether that harmonization actually works in practice.

Tom: Yeah, it checks eight major platforms and finds that even with the new rules on how to report things, the data isn't always consistent between their different reporting systems.

Lu: They look at five key areas of reporting—like how many people get notified, how fast they submit reports, if they use automated moderation tools, actions taken by the platforms themselves, and what happens because of user notices.

Meng: So they score these things across six dimensions—formatting, timeliness, consistency—and it turns out the overall quality score is a weighted average of all that stuff.

Lalam: The big finding is that there are substantial inconsistencies between the Transparency Database data and those Transparency Reports across all the platforms.

Jane: It means they might be using two separate workflows when they're generating these different kinds of reports, which is a serious issue for anyone trying to compare them reliably.

Tom: They point out that even though there was structural standardization, semantic consistency or operational interoperability didn't happen across all the platforms and mechanisms.

Lu: The authors suggest that just having a formal structure isn't enough to guarantee the data is consistent or easy to audit.

Meng: What they propose is making more things mandatory, especially around the grounds for moderation actions, and making sure they clearly separate what was targeted from what caused the violation.

Lalam: They also suggest that any Transparency Reports need to go through automated validation against formal schemas before they get published.

Jane: So, if we’re looking at this data now, it tells us that while the paperwork looks standardized on the surface, we still have a real challenge in making sure what's being reported operationally makes sense across all those different systems.

Tom: The main point of the paper is that scholars and practitioners need to be very careful when they draw broad conclusions from this transparency data without doing their own validation.

Lu: It suggests that we have to see if the formal standardization actually translates into real-world consistency before we trust it for public oversight.

Jane: It’s a lot to take in, but what does this mean for how we think about accountability when these massive AI systems are making decisions about what content gets removed?

More episodes

← Home