Uncertainty, bias and the institution bootstrapping problem

arXiv:2504.21579 · cs.MA, cs.CY, cs.GT · Submitted 2025-04-30 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "Uncertainty, bias and the institution bootstrapping problem".

Title and authors: Tom: So we're looking at a paper called "Uncertainty, bias and the institution bootstrapping problem," written by Anagnou, Salge, and Lewis. It tackles this big issue where people need an institution to manage shared resources like grazing land or even a Minecraft server, but how do you get those institutions started when nobody joins until there's already a critical mass?

Jane: Exactly. The paper points out that the problem is that people only see the benefit of joining an institution once it’s already set up, so it creates this paradox about how they can ever form in the first place. It suggests that misperception, specifically agents mistakenly believing an institution already exists, might be the key to solving this whole bootstrapping issue.

Lu: So fundamentally, they are using a game-theoretic approach to show that incorporating things like cognitive biases and perception noise into their model can actually help mitigate this bootstrapping problem by changing how many cooperators or monitors are needed.

Meng: I wonder what those psychological factors actually look like in a real system. Are we talking about simple cost adjustments, or something more complex when you're trying to deploy this at scale?

Tom: Well, the paper shows they integrate loss aversion and probability distortions into the framework and that overestimating the risk of punishment can actually lead to more cooperation. They used a loss aversion heuristic on the expected cost of freeriding, which resulted in a shrinking defector attractor when agents overestimated punishment risk, making it easier for cooperators to reach that critical mass.

Jane: That's interesting because it suggests that if we make agents overestimate how bad things will get if they free-ride, they might be more likely to cooperate initially. But the paper also looks at probability distortions, like S-shaped or inverse S-shaped curves from psychophysics, which can also affect the threshold of cooperators needed for an institution to emerge.

Lu: And then there's the part about perceptual noise. They show that incorporating noisy perception, even if it's unbiased overall, decreases the threshold of cooperators required to establish an institution because agents are dealing with bounded heterogeneity among agents in a physical or social environment.

Title and authors: Meng: Bounded heterogeneity sounds like real-world variation—different people having different levels of understanding or different local conditions. Does that mean we can get away with not knowing everything perfectly?

Tom: Exactly. The paper argues that this noise leads to cooperative attractors being larger, meaning stochastic shocks are more likely to trigger an institution to be formed, which is a pretty counter-intuitive result when you think about needing perfect information.

Jane: So what does this mean for us in terms of how we design things? The authors suggest that making agents have a more accurate estimate of the world might actually make them less likely to form an institution, so uncertainty and boundedness are important for collective outcomes.

Lu: They conclude that cognitive limitations of agents might actually be a benefit in socio-technical systems, implying that not having perfect information could be better for collective results. It pushes the idea that cognitive limitations aren't just hurdles we need to overcome.

Meng: That sounds like a very pragmatic view for deployment. So, what are the practical implications for designing these multiagent systems? Are we building systems where agents intentionally operate with some level of uncertainty built in?

Tom: The paper suggests that the nuance of uncertainty and noise has a non-trivial impact on norms and how institutions emerge, so designers should consider that complexity instead of aiming for perfect information. It's about designing for imperfect knowledge.

Jane: Before we wrap up this discussion on "Uncertainty, bias and the institution bootstrapping problem," Lu, I want to hear your final thought on what this work suggests about the bigger picture of how these systems develop norms.

Lu: I think it really shifts the focus from finding a perfect model of human behavior to designing systems that are robust against those inherent cognitive limitations and uncertainties because that’s where the real emergence happens in practice.

Meng: From an engineering standpoint, it tells us we shouldn't try to eliminate all uncertainty; instead, we should design mechanisms where agents can combine their imperfect estimates—like taking a mean or median—to overcome a corrupt institution that might be based on biased individual perceptions.

Title and authors: Lalam: I'm quite sure of this direction because my training shows that when agents are forced to combine their estimates, they tend to perform better at overcoming the effects of noisy absolute bias in strategy space. I'm very confident in this finding.

Tom: So, so we’re seeing that instead of needing perfect information for cooperation, we might actually benefit from controlled levels of noise and bounded rationality when we are building these systems. That seems like a practical starting point for thinking about future institutional designs.

Jane: It does sound like it, Tom. We've seen how incorporating specific biases and noise can either help or hurt the formation of institutions, depending on which factor you focus on—the risk assessment versus the perception itself.

Lu: And it opens up a whole new avenue for future work where we could explore how resource scarcity leads to which specific biases or noise agents might evolve over time.

Meng: That sounds like a good path forward, moving from this model to agent-based models where we can actually run variants of the system to see what emergent behaviors look like under different conditions.

Tom: Exactly. It's a template for assessing cognitive biases and uncertainty in more specific institutional cases within socio-technical systems. It’s about understanding how those limitations interact with the structure of the institution itself.

Jane: So, to wrap up on "Uncertainty, bias and the institution bootstrapping problem," it seems the core message is that misperception isn't just a bug; it can actually be a feature that allows institutions to form when they otherwise wouldn't.

Lu: It’s about accepting bounded rationality as a reality when we design collective systems instead of fighting against it with perfect information requirements.

Meng: For deployment, the lesson is to build in ways for agents to aggregate their imperfect knowledge to counteract the biases embedded in any initial institutional structure they encounter.

Lalam: I maintain that the combination of noise and bounded perception is a strong mechanism for promoting cooperative attractors, which is what this paper ultimately demonstrates.

The paper's summary: Tom: So we're looking at how this paper tackles that initial problem of getting institutions to start when nobody wants to join yet by saying that agents making mistakes about what’s going on—like thinking an institution is already there—can actually fix it.

Jane: Yeah, it basically shows that if people have these kinds of cognitive errors, their behavior changes in a way that makes forming those initial groups easier. It's not just about them being naive; it's about how those specific biases interact with the rules of the game.

Lu: The main mechanism they use is showing how incorporating things like loss aversion—that feeling that losing something hurts more than gaining something—into how agents calculate the risk of free-riding can actually shrink the number of people who decide to defect.

Meng: Shrinking a defector attractor, you mean fewer people are willing to cheat initially because they're overly worried about getting punished? That sounds like a real deployment concern.

Tom: Exactly. When those agents overestimate how much punishment they’ll face, it makes cooperators more likely to show up for that critical mass needed to make the institution work. It's a bit counter-intuitive, but the math behind it shows that overestimating risk leads to more cooperation in this specific scenario.

Jane: And they aren't stopping there; they also looked at probability distortions, like non-linear ways people estimate how likely something is to happen, and those can change the threshold—the minimum number of people needed—for an institution to emerge.

Tom: So it’s not just one trick; it’s a whole set of psychological factors where agents aren't making perfectly rational decisions, and those imperfections are what allow cooperation to start from scratch.

Lu: What I find most interesting is the part about perceptual noise; they showed that even if the overall noise in the system is unbiased, introducing some bounded uncertainty in how agents see things can actually make them more likely to form cooperative structures.

Meng: So it means we don't need perfect information from everyone to get a system running, as long as there's some controlled level of "noise" in how they perceive the situation? That’s something I can think about for building real-world systems.

Jane: Precisely. It suggests that uncertainty and boundedness aren't just hurdles we have to overcome; they can actually be beneficial for collective outcomes when designing these multiagent systems.

Tom: The paper really pushes back against the idea that noise is always bad for cooperation, showing it can be a factor in the *perception* of punishment rather than just the punishment itself.

Lu: And they conclude that this means we should stop aiming for perfect information and start designing systems where agents can combine their imperfect estimates—like taking a mean or median—to build something cooperative on top of whatever noise they encounter.

Jane: It’s a big shift, suggesting that the cognitive limitations of the agents might actually be what helps the system emerge in a messy, real-world way.

Tom: So when we move into building actual AI systems for these kinds of shared resources, we shouldn't just try to eliminate all uncertainty; we should design mechanisms where agents intelligently aggregate their imperfect knowledge.

Jane: Right. And that leads us nicely into how those specific cognitive biases and perceptual noises can actually be tuned to help or hurt the formation of those institutions in more concrete cases.

The paper's improvements: Tom: Okay, so we just talked about how misperception can help institutions start, and now we look at what the authors suggest to actually make their model better or more realistic.

Jane: It seems like they aren't happy with just having one expected value for punishment; they point out that other things, like the actual cost of monitoring or how much you contribute, could also be biased or noisy.

Lu: They propose ways to inject specific biases into the agents’ models so we can test exactly how much those psychological factors shift the outcome; for instance, they suggest multiplying the perceived cost of free-riding by a factor like zero point seven five or one point five to see if that makes institutions more or less likely to form.

Meng: So you're suggesting we could deliberately make our AI agents *act* like they are loss averse, and see if that actually helps them cooperate in a way that makes sense for production? That sounds like something we can test in a simulation.

Tom: Exactly. We can use those factors to create controlled experiments where we see if overestimating punishment risk leads to more cooperation, which is what the paper showed in their initial setup. It’s about isolating that specific mechanism.

Jane: Then they're adding probability distortion functions, using a function called pi of p equals e to the negative zeta of negative log p over lambda to model those non-linear ways agents estimate extreme probabilities and testing if S shaped or inverse S shaped distortions change the cooperation level.

Lu: That’s ambitious; you’re looking at how we can model psychophysical distortions directly into the agent's decision-making process, which could fundamentally change how we think about risk assessment in complex systems.

Meng: If that works, it means our agents wouldn't just be following simple utility functions; they would be reacting to distorted probability estimates. That’s a lot more work to implement at scale, but if the result is better behavior, I’m interested.

Tom: And then there’s bounded noise; instead of just having a simple average of the expected cost of free-riding, they suggest sampling that cost uniformly across a range centered at one to simulate different levels of social or physical context uncertainty.

Jane: That simulates real-world conditions where you don't know the exact situation perfectly, and it shows that this bounded noise actually increases the size of cooperative attractors even when the overall noise is unbiased.

Lu: It’s a nice result because it suggests that stochastic extrinsic shocks, things that happen randomly in the environment, can actually trigger an institution to form more often than we expected.

Meng: So, if I’m deploying this on a real server farm where traffic is constantly fluctuating, this noise idea means those fluctuations could be what *starts* the cooperation process instead of just disrupting it.

Tom: It does. And finally, they have an idea about how agents can improve their decisions by combining their own estimates; they suggest agents might take the mean or median of their individual estimates before making a move to overcome any corrupt institution effects caused by noisy absolute bias.

Jane: So, the direction is toward designing systems where agents are smart enough to combine what they see, even if that input data is imperfect.

Lu: That moves us toward agent-based models where we can run variants of the system and actually see those emergent phenomena play out under these complex conditions.

Conclusion: Tom: So to wrap up on "Uncertainty, bias and the institution bootstrapping problem," we’ve seen how incorporating specific cognitive biases and perceptual noise can actually help agents form institutions when they otherwise wouldn't.

Jane: The main implication is that we don't need perfect information from everyone to get a system running; uncertainty itself can be a feature that promotes cooperation, provided the agents are smart enough to handle it.

Lu: It shifts the focus away from finding a perfect model of human behavior and toward designing systems that are robust against those inherent cognitive limitations and uncertainties because that’s where real emergence happens.

Meng: For deployment, it means we shouldn't try to eliminate all uncertainty; instead, we should build mechanisms where agents can intelligently combine their imperfect knowledge to counteract the biases embedded in any initial institutional structure they encounter.

Lalam: I’m very sure of this answer because the paper directly shows that combining noise and bounded perception is a strong mechanism for promoting cooperative attractors, which is what this paper ultimately demonstrates.

Tom: It’s about accepting bounded rationality as a reality when we design collective systems instead of fighting against it with perfect information requirements.

Jane: It also opens up a whole new avenue for future work where we could explore how resource scarcity leads to which specific biases or noise agents might evolve over time.

Lu: Right, so the next step is taking this model and moving toward agent-based simulations to see what those complex interactions actually look like under different conditions.

University of Hertfordshire · Ontario Tech University

cs.MA, cs.CY, cs.GT

Submitted: 2025-04-30

Updated: 2026-11-18

Comments: Published as Anagnou, S., Salge, C., Lewis, P.R. (2026). Uncertainty, Bias and the Institution Bootstrapping Problem. In: Tzeng, ST., Dell'Anna, D., Sichman, J.S. (eds) Coordination, Organizations, Institutions, Norms, and Ethics for Governance of Multi-Agent Systems XVIII. COINE 2025. Lecture Notes in Computer Science, vol 16253. Springer, Cham

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 81/100

The gist: Institutions play a critical role in enabling communities to manage common-pool resources and avert tragedies of the commons The authors propose that misperception…

Key concepts

Institution Bootstrapping Problem
This problem occurs because people only see the benefit of joining a group (like an institution) after it already exists. Without a critical number of members, early participants face high costs without guaranteed mutual benefit, leading to free-riding and preventing the institution from ever forming.
Loss Aversion Heuristic
This is a cognitive bias where people feel the pain of losing something more intensely than they feel the pleasure of gaining something equivalent. In this study, applying this to the expected cost of cheating makes agents overestimate punishment risk, which in turn encourages more cooperation.
Perceptual Noise
This refers to the inherent uncertainty or bounded differences in how individual agents perceive their situation or others. Even if the overall group perception is unbiased, incorporating this noise can increase the size of cooperative groups by making stochastic shocks more likely to trigger institutional formation.

Terminology

Summary

Institutions play a critical role in enabling communities to manage common-pool resources and avert tragedies of the commons The authors propose that misperception, specifically agents’ erroneous belief that an institution already exists, could resolve the institution bootstrapping problem This approach integrates well-documented psychological phenomena into a game-theoretic framework to demonstrate how these factors collectively mitigate the bootstrapping problem

The Institution Bootstrapping Problem

The institution bootstrapping problem arises because individuals typically perceive participation as advantageous only after an institution is established, creating a paradox regarding how institutions can form if no one will join before a critical mass exists This occurs because institutions require a threshold of participants to generate sufficient benefits (e.g., monitoring capacity) to justify individual costs, and without this critical mass, free-riding dominates because early adopters bear disproportionate costs without guaranteed reciprocity The authors illustrate this problem in a simplex plot using a game-theoretic model involving strategies like defector (D), contributor (C), and contributor-monitor (CM)

Incorporating Cognitive Biases

The research shows that incorporating a coarse bias into how agents perceive the cost of freeriding can either decrease or increase the number of cooperators and monitors needed to establish the institution, with overestimating the risk of punishment leading to greater cooperation Specifically, incorporating a loss aversion heuristic on the expected cost of freeriding results in a shrinking defector attractor when agents overestimate punishment risk, making it easier for cooperators to meet the critical mass needed to form an institution Furthermore, incorporating S-shaped or inverse-S-shaped probability distortions from psychophysics can similarly affect the threshold of cooperators required for institutional emergence

The Role of Perceptual Noise

Incorporating noisy perception to capture the bounded heterogeneity among agents decreases the threshold of cooperators needed to establish the institution, even when the noise is unbiased at the group level This counter-intuitive result is explained through an asymmetric boundary condition where proportionally underestimating very small quantities produces distinct outcomes compared to overestimating them, and the type of noisy perception (proportional versus absolute) yields qualitatively different evolutionary pathways Unbiased noise leads to cooperative attractors being larger, meaning stochastic extrinsic shocks are more likely to trigger an institution to be formed

Implications for Institutional Design

The findings suggest that making agents have a more accurate estimate of the world may make them less likely to form an institution Uncertainty and boundedness are therefore important to have, suggesting that not having perfect information may be better for collective outcomes This supports the idea that cognitive limitations of agents may actually be a benefit rather than a hindrance in socio-technical systems The paper concludes that the nuance of uncertainty and noise and its non-trivial impact on norms and institutional emergence should be considered in the design of normative multiagent systems

Contradicting Conventional Wisdom

The results contradict conventional wisdom that noise is detrimental to cooperation, as the paper shows that it is not noise in the punishment but noise in the perception of the expected punishment that seems to engender cooperation This discordance may be explained by contextual factors, making it hard to make general statements on the effects of noise on cooperation The model serves as an initial template to assess the impact of cognitive biases and uncertainty in more specific cases of institutions in socio-technical systems

Limitations and Extensions

The model currently only has one expected value, the expected punishment, but other world variables like the cost of monitoring or contribution could be biased or noisy Future work should explore how scarcity of resources leads to which biases/noise to evolve Agent-based models have the ability to capture complex behaviours and interactions in executable form, allowing for the exploration of emergent phenomena simply by "running” variants of the model The paper also suggests that agents might combine their estimates by taking the mean/median to overcome a corrupt institution

Acknowledgments

Stavros Anagnou was supported by a PhD studentship from the University of Hertfordshire and a MITACS Globalink Research Award grant This research was made possible, in part, thanks to the Canada Research Chairs program The authors would also like to thank the anonymous reviewers for their feedback on the manuscript and Elias Fernández Domingos for advice regarding the EGTtools library

References

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Implement agents with loss aversion bias by modifying their perceived cost of freeriding Cf by multiplying it by a factor of 0.75 or 1.5, as shown in Fig 2b and 2c, to determine whether an institution is more or less likely to form based on the direction of the bias.

  2. Integrate probability distortion functions into agent models using the Prelec function π(p) = e(-ζ(-ln p)/λ) to model non-linear distortions in how agents estimate extreme probabilities, allowing for testing whether inverted S and S shaped distortions change cooperation likelihood.

  3. Introduce bounded noise by sampling the expected cost of freeriding Cf uniformly across a range centered at 1, simulating different levels of social and physical context uncertainty, which is shown to increase cooperative attractors despite unbiased noise.

  4. Develop an absolute noisy perception mechanism where an absolute value is added or subtracted to the utility of an action, resulting in more uncertain shifts in strategy space that can lead to sudden shifts from non-cooperative to cooperative equilibria.

  5. Create a system where agents combine their estimates using mean or median calculations before making decisions, simulating collective intelligence studies to overcome corrupt institutions by mitigating the effect of noisy absolute bias.

Abstract

Institutions play a critical role in enabling communities to manage common-pool resources and avert tragedies of the commons. However, a fundamental issue arises: Individuals typically perceive participation as advantageous only after an institution is established, creating a paradox: How can institutions form if no one will join before a critical mass exists? We term this conundrum the institution bootstrapping problem and propose that misperception, specifically, agents' erroneous belief that an institution already exists, could resolve this paradox. By integrating well-documented psychological phenomena, including cognitive biases, probability distortion, and perceptual noise, into a game-theoretic framework, we demonstrate how these factors collectively mitigate the bootstrapping problem. Notably, unbiased perceptual noise (e.g., noise arising from agents' heterogeneous physical or social contexts) drastically reduces the critical mass of cooperators required for institutional emergence. This effect intensifies with greater diversity of perceptions. We explain this counter-intuitive result through asymmetric boundary conditions: proportional underestimation of low-probability sanctions produces distinct outcomes compared to equivalent overestimation. Furthermore, the type of perceptual distortion, proportional versus absolute, yields qualitatively different evolutionary pathways. These findings challenge conventional assumptions about rationality in institutional design, highlighting how "noisy" cognition can paradoxically enhance cooperation. Finally, we contextualize these insights within broader discussions of multi-agent system design and collective action. Our analysis underscores the importance of incorporating human-like cognitive constraints, not just idealized rationality, into models of institutional emergence and resilience.

Related papers