Disarranged Harmonization of Transparency Reporting by Social Media Platforms Under the Digital Services Act

arXiv:2605.17655 · cs.CY, cs.SI · Submitted 2026-05-17 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "Disarranged Harmonization of Transparency Reporting by Social Media Platforms Under the Digital Services Act".

Paper summary: Tom: So, we're looking at this paper now called "Disarranged Harmonization of Transparency Reporting by Social Media Platforms Under the Digital Services Act". Essentially, they’ve taken this new rule from the European Commission that forces big online platforms to report their data in a standardized way and they’ve systematically checked how good that reporting actually is.

Jane: Right, so you're asking, after all this new harmonization effort under the DSA, are these transparency reports actually reliable? And what does that mean for us when we try to understand how these massive platforms are actually running their moderation systems?

Lu: The core idea here is that they run a large-scale quantitative analysis on five key dimensions of platform reporting. These include things like the number of active recipients, how fast those reports are submitted, whether they use automated means for content moderation, actions taken by the platform itself, and actions that happen because of user notices.

Meng: Wait, so you’re looking at how they measure the actual quality? I mean when I’m trying to deploy a model at scale, I care less about the abstract dimensions and more about what breaks when we push millions of users.

Tom: Well, Lu says they assign scores across six different dimensions for both the Transparency Database and the Transparency Reports, looking at things like formatting and consistency. But it also found that some platforms used completely different reporting procedures depending on which mechanism they were using, which caused them to submit totally contrasting information.

Jane: So even though the rules are harmonized now, you still have these differences where one platform might be reporting things very differently than another because of how they report it. That sounds like a big problem for comparison.

Lalam: And they found that when comparing the Transparency Reports to the Transparency Database, there are substantial inconsistencies across all platforms, suggesting those two reporting mechanisms might be working on separate workflows entirely. For instance, LinkedIn and YouTube reported orders of magnitude more own-initiative actions in their reports than what you could see in the actual submitted SoRs.

Tom: That’s a huge red flag for auditors because it means you can't just trust one report over the other without knowing exactly how they connect. It points to a lack of operational interoperability, even if the structure looks standardized on paper.

Paper summary: Jane: So what does that mean for us as observers? If the data isn't consistent between the database and the reports, how can we really get a reliable picture of platform behavior?

Lu: The paper suggests that just having a formal standardization isn't enough to guarantee consistency or auditability, which is what they call semantic consistency and operational interoperability. They found that structural changes didn't automatically fix the practical issues between platforms.

Meng: From my side, if we try to build a unified system based on this, we need more than just a common format; we need the actual underlying operational definitions to match up perfectly across different moderation tools. Otherwise, it’s just prettier paperwork.

Tom: They actually make some suggestions for fixing this infrastructure. They propose making more attributes mandatory, especially for things like the grounds for moderation actions, and they suggest explicitly distinguishing what was targeted versus what caused the violation.

Jane: That makes sense practically; if you don't know *why* something got taken down versus *who* initiated the process, it’s hard to tell if you’re seeing a real pattern or just noise. What about how they think we should validate this?

Lalam: They suggest that harmonized Transparency Reports should go through mandatory automated validation against formal schemas and reporting constraints before they are ever published. That forces them to check their own data against the rules before it goes public.

Tom: And the implications for governance are pretty direct: scholars and practitioners need to be careful when they draw large-scale conclusions from this data without doing extra work to validate it themselves. The infrastructure can look uniform, but in practice, it might still be operationally inconsistent.

Jane: So the big picture here is that we have a new set of rules for transparency under the DSA, but the actual data coming out isn't perfectly aligned or consistent yet. It’s like building a highway with all standardized signs, but the roads themselves are still built in different ways depending on which construction crew worked on that section.

Lu: Exactly. The paper points out that moderation effectiveness really depends on operational factors, like the timing of interventions, which often get lost in those aggregated reports they use for transparency.

Paper summary: Meng: That’s what keeps me up at night when I think about deployment; if the report is missing the timing detail of when an action was taken, we can't debug a failure properly. What would we actually need to see to trust this?

Tom: The paper also flags that independent oversight bodies have pointed out that certain interactions, like those involving law enforcement, might not be represented in these reports at all. That’s another gap in the picture.

Jane: It sounds like the challenge isn't just about having data; it's about making sure the data tells a complete story about what platforms are actually doing on the ground when they make moderation decisions. So, what’s next for this whole process?

Lalam: The authors themselves point out that their analysis focused mostly on operational consistency and compliance with DSA reporting obligations, but they admit that modeling the actual substance of content moderation itself is something they didn't do in this study.

Tom: And they also mention a limitation in their own assessment: the comparative quality assessment involves a degree of expert judgment when weighting those criteria and interpreting what constitutes good or bad reporting.

Lu: They also note that because they only looked at eight originally designated Very Large Online Platforms, it might not fully capture the reporting practices across the entire platform ecosystem.

Jane: So, to sum up this paper on "Disarranged Harmonization of Transparency Reporting by Social Media Platforms Under the Digital Services Act", the main point is that while there's a new rule forcing standardization, they found that platforms still report things differently depending on how they report them, and that formal harmonization alone isn't enough to guarantee real consistency.

Tom: The authors are essentially saying we need more than just common formats; we need operational interoperability and clearer definitions for automated moderation actions to make this data useful for real oversight.

Jane: And for the listener who doesn't read papers, it means when you look at the transparency reports online, don't just see a standardized number; be aware that behind that number, there might be totally different ways the platform is actually tracking those moderation events.

Lalam: The implication is that for effective public oversight and auditing to work under the DSA, we need this level of granular detail and validation to bridge the gap between what the law requires and what platforms are actually doing operationally.

Conclusion: Jane: So we've looked at how the new Digital Services Act is making big social media companies report their moderation data, and this paper, "Disarranged Harmonization of Transparency Reporting by Social Media Platforms Under the Digital Services Act," really digs into whether that harmonization actually works in practice.

Tom: Yeah, it checks eight major platforms and finds that even with the new rules on how to report things, the data isn't always consistent between their different reporting systems.

Lu: They look at five key areas of reporting—like how many people get notified, how fast they submit reports, if they use automated moderation tools, actions taken by the platforms themselves, and what happens because of user notices.

Meng: So they score these things across six dimensions—formatting, timeliness, consistency—and it turns out the overall quality score is a weighted average of all that stuff.

Lalam: The big finding is that there are substantial inconsistencies between the Transparency Database data and those Transparency Reports across all the platforms.

Jane: It means they might be using two separate workflows when they're generating these different kinds of reports, which is a serious issue for anyone trying to compare them reliably.

Tom: They point out that even though there was structural standardization, semantic consistency or operational interoperability didn't happen across all the platforms and mechanisms.

Lu: The authors suggest that just having a formal structure isn't enough to guarantee the data is consistent or easy to audit.

Meng: What they propose is making more things mandatory, especially around the grounds for moderation actions, and making sure they clearly separate what was targeted from what caused the violation.

Lalam: They also suggest that any Transparency Reports need to go through automated validation against formal schemas before they get published.

Jane: So, if we’re looking at this data now, it tells us that while the paperwork looks standardized on the surface, we still have a real challenge in making sure what's being reported operationally makes sense across all those different systems.

Tom: The main point of the paper is that scholars and practitioners need to be very careful when they draw broad conclusions from this transparency data without doing their own validation.

Lu: It suggests that we have to see if the formal standardization actually translates into real-world consistency before we trust it for public oversight.

Jane: It’s a lot to take in, but what does this mean for how we think about accountability when these massive AI systems are making decisions about what content gets removed?

Amaury Trujillo, Benedetta Tessa, Stefano Cresci

IIT-CNR · University of Pisa

cs.CY, cs.SI

Submitted: 2026-05-17

Updated: 2026-11-17

Comments: Accepted to ICWSM 2027

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 79/100

The gist: The European Commission recently introduced new regulation to harmonize transparency reporting of large online platforms under the Digital Services Act (DSA) and this paper presents a systematic…

Key concepts

Transparency Reports (TRs)
These are the reports social media platforms submit detailing their content moderation activities. The study assessed how well these reports met DSA requirements regarding formatting, timeliness, and completeness.
Transparency Database (TDB)
This is the structured data repository that holds the raw information related to platform reporting. Researchers compared this database data against the TRs to check for consistency between the two sources.
Reporting Quality Dimensions
These are five key areas used to judge how good a platform's transparency reporting is, such as how many people receive moderation actions or if they use automated tools for content review.
Operational Interoperability
This refers to whether different platforms and their various reporting systems can effectively work together. The study found that while structures were standardized, the actual operational processes remained inconsistent.

Terminology

Summary

The European Commission recently introduced new regulation to harmonize transparency reporting of large online platforms under the Digital Services Act (DSA) and this paper presents a systematic evaluation of transparency reporting data quality after this normative change for eight major social media platforms in the EU. p. 0

Where the paper says it

it Abstract The European Commission recently introduced new regulation to harmonize transparency reporting of large online platforms under the Digital Services Act (DSA). Here, we present the first systematic evaluation of transparency reporting data quality after this normative change, for the eight largest social media platforms in the European Union.

Among its findings is that some platforms employed differing reporting procedures across mechanisms, which caused them to submit contrasting information. p. 0
Where the paper says it

In detail, we run a set of large-scale quantitative analyses on key reporting dimensions, followed by a structured comparative assessment across platforms and reporting mechanisms. Among our findings is that: (i) the analyzed platforms had varying degrees of compliance and data quality, but all exhibited issues on data formatting, timeliness, consistency, and completeness; (ii) some platforms employed differing reporting procedures across mechanisms, which caused them to submit contrasting information; (iii) despite the harmonization, a number of issues still prevent interoperability between reporting mechanisms; and (iv) many of the previously identified issues with transparency reporting are still unresolved.

Evaluation of Reporting Quality Dimensions

The quantitative analyses examine five key dimensions of platform reporting: (i) number of active recipients, (ii) timeliness of submitted SoRs, (iii) use of automated means for content moderation, (iv) moderation actions taken on the platforms’ own initiative, and (v) actions resulting from notice mechanisms. These aspects correspond to central DSA transparency obligations and capture core operational components of platform moderation and governance. p. 2

Where the paper says it

, users), (ii) timeliness of submitted SoRs, (iii) use of automated means for content moderation, (iv) moderation actions taken on the platforms’ own initiative, and (v) actions resulting from notice mechanisms. These aspects correspond to central DSA transparency obligations and capture core operational components of platform moderation and governance (Trujillo, Fagni, and Cresci 2025; Kaushal et al.

The structured comparative assessment assigned scores along six evaluation dimensions—three for the TDB and three for TRs—covering aspects such as formatting, timeliness, consistency, and completeness. p. 2
Where the paper says it

Next, we build on the results of the quantitative analyses to carry out a structured comparative assessment of the reporting quality of each platform across both transparency mechanisms. Specifically, we assigned scores along six evaluation dimensions—three for the TDB and three for TRs— covering aspects such as formatting, timeliness, consistency, and completeness.

The overall reporting quality score is the weighted mean of all other factors. p. 7
Where the paper says it

Scores were assigned based on the empirical patterns, inconsistencies, omissions, and reporting limitations identified throughout the quantitative analyses. We further computed an overall reporting quality score for each platform and transparency mechanism as the weighted mean of the corresponding criterion scores.

Inconsistencies in Data Consistency

RQ2 questions whether harmonized Transparency Reports data are consistent with the corresponding Transparency Database data. The analysis revealed substantial inconsistencies between the TDB and TRs across platforms, indicating that the two reporting mechanisms might be implemented via partially independent workflows. p. 9

Where the paper says it

Overall, reporting data in the TDB were generally less complete and internally consistent than the corresponding information in the TRs, despite both mechanisms being intended to describe the same moderation activity. These discrepancies suggest that the two reporting mechanisms might be implemented via partially independent workflows, with limited internal interoperability and heterogeneous interpretations of the DSA obligations.

For example, LinkedIn, Snapchat, and YouTube reported orders of magnitude more own-initiative actions in their TRs than were observable in the corresponding SoRs. p. 6
Where the paper says it

While Facebook, Instagram, Pinterest, and YouTube ads exhibit strong consistency across the two sources, other platforms show substantial divergences. In particular, LinkedIn, Snapchat, and YouTube reported orders of magnitude more own-initiative actions in their TRs than were observable in the corresponding SoRs.

Limitations of Harmonization

RQ1 addresses the extent to which harmonized Transparency Reports and Transparency Database records comply with the European Commission’s instructions. The results indicate that while structural standardization occurred, semantic consistency or operational interoperability across platforms and reporting mechanisms was not fully achieved. p. 9

Where the paper says it

On the one hand, these findings signal that the EC might need to revise parts of the current reporting mechanisms and provide further reporting guidance. On the other hand, they suggest that the current harmonization framework improved structural standardization, but did not fully achieve semantic consistency or operational interoperability across platforms and reporting mechanisms.

This suggests that formal harmonization alone is insufficient to guarantee interoperability, consistency, or auditability of transparency data. p. 9
Where the paper says it

Several inconsistencies identified throughout our analyses appear to stem from heterogeneous operational interpretations of the same regulatory obligations across different reporting mechanisms within the same platform. At the regulatory level, our results highlight that formal harmonization alone is insufficient to guarantee interoperability, consistency, or auditability of transparency data.

Suggestions for Infrastructure Improvement

The study suggests several targeted modifications to the DSA transparency infrastructure could substantially improve the interoperability, consistency, and auditability of transparency reporting. Proposed improvements include making more attributes mandatory (especially those concerning moderation grounds), explicitly distinguishing moderation targets from the entities causing the violation, and clarifying the semantics of automated moderation attributes. p. 9

Where the paper says it

In this regard, our analyses further support prior proposals aimed at improving the TDB, including making more attributes mandatory (especially those concerning moderation grounds), explicitly distinguishing moderation targets from the entities causing the violation (e. , content versus accounts), separately identifying synthetic or AI-generated targets, and clarifying the semantics of automated moderation attributes (Kaushal et al.

Furthermore, harmonized TRs should undergo mandatory automated validation against formal schemas and reporting constraints before publication. p. 9
Where the paper says it

Several platforms did not fully comply with the EC templates and reporting instructions, substantially complicating data processing and interoperability. Harmonized TRs should therefore undergo mandatory automated validation against formal schemas and reporting constraints before publication.

Implications for Governance

The findings imply that scholars and practitioners should interpret DSA transparency data with caution when drawing comparative or large-scale conclusions without additional validation. The results suggest that transparency infrastructures can become formally standardized while remaining operationally inconsistent in practice. p. 9

Where the paper says it

This is particularly relevant for VLOPs, whose reporting practices directly affect the transparency and accountability of some of the largest online moderation systems operating in the EU. More broadly, our findings suggest that transparency infrastructures can become formally standardized while remaining operationally inconsistent in practice.

Ensuring the reliability and interoperability of transparency data is a necessary condition for effective public oversight, independent auditing, and regulatory enforcement. p. 1
Where the paper says it

Several formal proceedings initiated by the EC against major platforms already involve alleged failures concerning regulatory and transparency compliance. In this context, assessing the consistency, interoperability, and reliability of DSA transparency data is not only necessary to evaluate platform reporting practices, but also to understand the practical capacity of the EC and the DSA framework to support meaningful enforcement at scale.

Limitations and Future Work

The analysis focused primarily on operational consistency, interoperability, and compliance with DSA reporting obligations rather than modeling the substantive dynamics of content moderation itself. A second limitation concerns the comparative assessment of reporting quality, which inevitably involves a degree of expert judgment in the weighting and interpretation of reporting criteria. p. 7

Where the paper says it

Given the central role of notice-based mechanisms in the DSA framework, these stark discrepancies raise serious concerns. Qualitative Evaluation To synthesize the empirical results into a comparable and actionable evaluation, we conducted a structured comparative assessment of reporting quality across platforms and transparency mechanisms based on the previous quantitative analyses as well as the ones reported in the Appendix.

The study focuses on eight originally designated VLOPs and may not fully capture reporting practices across the broader platform ecosystem. p. 2
Where the paper says it

Our work contributes to this broader effort by assessing whether the recent harmonization initiative effectively advanced transparency, interoperability, and independent oversight under the DSA. Data We focus on the newly-harmonized transparency reports for the second semester of 2025 (2025H2) published by the eight social media platforms originally designated as very large online platforms (VLOPs)—namely, Facebook, Instagram, LinkedIn, Pinterest, Snapchat, TikTok, X, and YouTube.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Enhanced Compliance Monitoring for DSA Reporting Systems: AI systems can be designed to perform automated validation against formal schemas and reporting constraints before publication for harmonized Transparency Reports, which would eliminate many of the formatting issues identified herein.

  2. Improved Data Consistency Checks Between Reporting Mechanisms: AI can implement cross-checking procedures between the Transparency Database and Harmonized Transparency Reports to assess the extent to which harmonized Transparency Reports data is consistent with the corresponding Transparency Database data, addressing findings that show substantial inconsistencies between the TDB and TRs.

  3. Semantic Guidance for Moderation Concepts: AI systems can be used to provide prescriptive semantic guidance by developing tools that clarify the semantics of automated moderation attributes and explicitly define concepts like solely, partially, and non-automated moderation, thereby improving semantic consistency and crossplatform comparability.

  4. Automated Risk Scoring for Platform Reporting Quality: AI can use the qualitative evaluation criteria to assign a weighted mean score to each platform's reporting quality, allowing for the identification of platforms with specific weaknesses based on factors like "timeliness (TDB); consistency (TDB & TRs); completeness (TDB & TRs); and formatting (TRs)."

  5. Targeted Recommendations for Data Collection: AI can analyze discrepancies in AMAR reporting, such as those seen in YouTube, to flag potential overcounting or inconsistencies with DSA methodological indications, informing platforms on how to compute figures consistently with regulatory requirements.

Abstract

The European Commission recently introduced new regulation to harmonize transparency reporting of large online platforms under the Digital Services Act (DSA). Here, we present the first systematic evaluation of transparency reporting data quality after this normative change, for the eight largest social media platforms in the European Union. In detail, we run a set of large-scale quantitative analyses on key reporting dimensions, followed by a structured comparative assessment across platforms and reporting mechanisms. Among our findings is that: (i) the analyzed platforms had varying degrees of compliance and data quality, but all exhibited issues on data formatting, timeliness, consistency, and completeness; (ii) some platforms employed differing reporting procedures across mechanisms, which caused them to submit contrasting information; (iii) despite the harmonization, a number of issues still prevent interoperability between reporting mechanisms; and (iv) many of the previously identified issues with transparency reporting are still unresolved. We conclude by discussing implications for transparency auditing and proposing key targeted improvements to strengthen the reliability and interoperability of DSA transparency reporting.

Sources

Related papers