Mapping the Ethical Dimensions of Trust & Safety Work

Jul 8

In this blog, Michael Ong, trust and safety expert, reflects on how trust and safety work is fundamentally an exercise in governance and ethical judgment, not just technical policy enforcement. It maps the field's core tensions and offers ethical frameworks and practical habits for navigating them.

Trust and safety work has never been a purely technical discipline, however much the industry has occasionally treated it as one. Every moderation decision, policy limit, or detection rule encodes a hidden value judgment beneath its operational surface. For example, when we calibrate classifiers for hate speech, we’re making implicit determinations about what our communities can collectively bear. Adjudicating standard fraud disputes exercises a form of discretion recognizable to anyone with familiarity inside a prosecutor's office. Drafting suspension criteria in any honest accounting frequently resembles legislative oversight more than product compliance.

Unsurprisingly, the field has developed its technical infrastructure more deliberately than it has pursued shared vocabulary for ethics. Most practitioners accumulate informal moral intuitions through the work itself, refined through empirical touchpoints, social cues, and the accumulated weight of decisions that touch real people at planetary scale; however they rarely have familiar language to ascribe towards those intuitions, or common frameworks for adjudicating between conflicts. This piece attempts to remedy some of these disconnects, with no claim as either a prescription nor a hierarchy of ethical frameworks ranked by correctness. Rather, it is an invitation to become more deliberate about terrain we’re already navigating, whether explicitly named or not.

As a definitional note before proceeding, throughout this piece, I’m using the term “trust and safety” as understood in its broadest institutional sense, encompassing content moderation, fraud and financial crimes, risk operations, and all other adjacent integrity functions that collectively constitute online governance. The ethical considerations explored here apply across this spectrum, even where specific examples draw more directly from one domain than another.

The Inherent Ethical Bounding of Trust & Safety

It is worth stating plainly what practitioners often sense but rarely articulate with precision: trust and safety work is governance. Tarleton Gillespie argues in Custodians of the Internet that platforms exercise consequential authority through decisions that remain largely hidden from public scrutiny, with the word hidden acting as the key operative here. These are not decisions that default as neutral when left unexamined, as they’re made regardless, simply with less accountability and intentionality. The question isn’t whether T&S practitioners exercise discretionary authority but rather whether they do so with sufficient self-awareness to honor the gravity of that prerogative. Structurally, this is an unavoidable outcome for most platforms.

From my days in the world of criminal justice reform, I can’t help but identify the parallels here. Specifically Kenneth Culp Davis's foundational work on discretionary justice comes to mind, which established that institutional decision making is nearly always more discretionary than its formal rules acknowledge, and that the space between codified policy and applied judgment is where power actually resides. Or consider Angela Davis' examination of prosecutorial discretion, in which the prosecutor ultimately determines which cases to charge, which plea arrangements to sanction, and when to pursue the full weight of the law, thus wielding a level of authority that no statutory framework fully illuminates. T&S practitioners at every level occupy a structurally analogous position; policy may help establish parameters, but it is human judgment filling the considerable space between them. Consequently, at the scale of modern platforms that gap constitutes something closer to social policy than isolated cases of individual adjudication. Big tech’s growing investment in consistency mechanisms, appeals infrastructure, and transparency reporting reflects an implicit acknowledgment of this reality. What remains is to validate these assumptions by naming the governance functions of trust & safety work outright, and accepting the ethical responsibility that such a function necessarily carries.

Conscious and subconscious influence is a more subtle dimension here that also warrants attention. We don’t make decisions in a vacuum of pure deliberation so much as we operate within institutional contexts that covertly shape the criteria for a reasonable decision. The framing choices embedded in a PRD, the default configurations of a review queue, and the informal communication norms that accumulate within a Slack channel over time all exert ethical force that rarely surfaces in case-by-case deliberation. A genuinely ethical practice attends not only to individual decisions, but also acknowledges the superseding structures that render certain choices more attractive, weighted, or probable than others.

Landscape: Core Tensions & Structural Challenges

Discussions of T&S work often involve the impossible tradeoffs that teams are routinely asked to make. The trade-off centered framework developed by Jiang and colleagues offers a rigorous map of moderation's constitutive tensions that reverberate across platforms and enforcement contexts with remarkable consistency. Notably, the conclusions drawn don’t aim to reach a solvable endpoint so much as a navigable wireframe.

Unsurprisingly, the most foundational of these tradeoffs remains that of user safety versus expression. Every content restriction involves a tradeoff between protecting users from demonstrable harm and preserving the expressive freedom that grants platforms their social value. This is not a dial that can be calibrated once and left to self regulate. It requires continuous recalibration as platform communities evolve, the nature of online harm mutates, and interpersonal contexts shift in ways that no static policy can fully manage.

The paradox of consistency versus context is equally intractable. Uniform application produces legibility, resists manipulation, and signals equality as a virtue, but also generates unjust outcomes when bluntly imposed without sensitivity to circumstance. Subjectivity and individual biases can distort enforcement patterns in ways that compound across decision scale, yet rigid consistency pursued without contextual judgment strips away crucial nuance.

Balancing transparency against appropriate platform defense occupies similarly uncomfortable territory. Effective enforcement mechanisms require some degree of opacity; it’s somewhat self-apparent that publishing precise intervention signals extends a welcome invitation for their circumvention. Yet over time, obscurity also corrodes the legitimacy upon which communal trust ultimately depends. While the Santa Clara Principles represent a principled attempt to navigate this tension through structured transparency and accountability, like Jiang’s framework they do not dissolve the underlying tensions so much as manage them.

Tradeoffs between speed & accuracy also receive less sustained attention in the literature than warranted. Operational pressure (queue backlog, technical debt, aggressive ship cadences) systematically displaces the deliberative sandbox that intentional ethical judgment requires. This is not, in the first instance, a failure of individual practitioners, but instead a structural condition that organizations must bear responsibility for if they wish for ethical diligence to mean something more than aspirational talking points.

Finally, the ethical practitioner must consider individual harm versus systemic harm, as decisions made to protect a particular user may generate second order effects that erode community cohesion when applied at scale. My friends at Reddit understand this firsthand, where policies designed to interdict coordinated abuse may over-correct in ways that suppress legitimate civic discourse. Optimizing relentlessly for the harm that is visible and proximate may unintentionally exacerbate the harm that is diffuse, delayed, and equally offensive.

What makes this so?

Beyond these tensions, some structural features of trust and safety work compound the landscape of ethical challenge. For example, take pervasive information asymmetry; as Binns and colleagues demonstrate in Towards an Epistemic Compass for Online Content Moderation, practitioners routinely make high impact decisions under conditions of epistemic constraint, namely partial context, unreliable signals, and the conspicuous absence of the black and white. Regrettably, this is an inexorable condition, a workplace hazard so to speak, only missing the conspicuous neon signage.

Sarah Roberts's infamous documentary account in Behind the Screen highlights the problem of broken feedback loops. Consider that we almost never learn whether our decisions were ultimately correct, however much mental or operational horsepower went into it. Appeals are rarely adjudicated within the same circuit as the suspension, and downstream consequences remain shrouded at best. The corrective signal that moral learning requires simply does not exist in most scaled moderation. Thus this system tends to foreclose the reflective feedback loop through which our judgment instincts would ordinarily develop, allowing us to self correct for bias, fatigue, and other variables.

Two additional pressure points warrant explicit acknowledgment. One, the fragmented global regulatory landscape creates conditions in which cross-cultural practitioners may operate under meaningfully different normative expectations without full awareness that this divergence exists. Two, the structural asymmetry of power between platforms and users looms large; platforms exercise increasingly outsized authority over user experience, access, and exposure with limited mechanisms for meaningful internal accountability.

A Brief Taxonomy of Ethical Lenses for Consideration

What follows is neither a ranking nor a prescription, but a brief classroom-to-cubicle mapping that I find helpful in this conversation. The most exemplary practitioners I’ve met tend to move between ethical frameworks with fluency, deploying each as an interrogative instrument rather than a decision mapping algorithm with set outputs.

Consequentialism directs attention to outcomes, which in our context means sustained attention on scaled effects, including both the proximate harm addressed by a risk action and the implicative downstream effects on expression, community dynamics, and other incentive structures that shape future user behavior. Mill's utilitarian inheritance is the most recognizable anchor here, but the consequentialist obligation in T&S runs deeper than outcome tallying: it demands honest reckoning with second & third order effects that are easy to discount when they are diffuse, latent, and distributed across populations lacking an institutional voice.

Deontological thinking considers whether there are foundational constraints that hold irrespective of outcome, actions that remain impermissible even when the projected outcomes are favorable. Kant's categorical imperative finds a practical analog in the enforcement context: would we endorse this rule if applied universally across every case sharing the same characteristics? Rule-based reasoning also provides a structural check against the slippery slope reasoning which consequentialist framing is susceptible to. Ends only justify means insofar as quantitative objectives are preeminent, and I needn’t press further on the danger of this heuristic. We would do well to mark this consideration with proportionate wariness.

Procedural fairness carries perhaps the most immediate operational salience for the field. Tom Tyler has some landmark research in Why People Obey the Law, establishing that institutional legitimacy is constructed less by the outcomes people receive than by the perceived justice of the processes that produce those outcomes. Pan and colleagues' work on perceived legitimacy in content moderation extends this finding directly into our domain, showing that users who viewed enforcement as arbitrary or inconsistent withdrew trust from platforms regardless of the underlying determination’s veracity. Thus, procedural design deserves a seat at the builder’s table as a fundamental ethical obligation.

Care ethics reorients the frame from A/B dynamics and aggregate outcomes toward relationships and the particular vulnerabilities of those affected. Similar to Rawl’s veil of ignorance, it asks, who bears the greatest exposure in this situation, and are we attending to them with appropriate moral seriousness? Lindsay Blackwell's Content Moderation Futures argues that this view is a requirement for high integrity moderation practice, and that current institutions systematically underinvest in the care dimension, both on the user and practitioner side. Care ethics serves as a corrective salve to frameworks that optimize for systemic consistency at the expense of attentiveness to particular human situations that enforcement decisions actually impact.

Some miscellaneous thoughts - how about proportionality? As a general rule are we disciplined in our matching of enforcement severity to harm severity? How about due process (think DSA) on the consumer side of the equation when it comes to notice of violation, appeal pathways, & or fair access? Finally, is it even possible to achieve equal application and therefore enforcement of policy across ideological, demographic, & geographic contexts? What would that look like?

Application

This piece is not intended as a directive or a compliance recommendation. It is, as the title suggests, a gentle check on posture, an attempt to characterize what it might look like to maintain ethical intentionality throughout a complex ecosystem, rather than reserving moral deliberation for compelled cases. Subsequently, here are three quick practices I find helpful as a flashcard for dissecting daily problems.

The first practice is calling out the specific values in play before scrambling to ship, deploy, or send. Most trust and safety determinations involve tradeoffs between genuinely competing goods, such as safety and expression, consistency and contextual sensitivity, operational velocity and deliberative care. Explicitly rendering those tradeoffs even briefly tends to produce better decisions and more defensible reasoning in the long term. It also fertilizes the conditional soil for healthy institutional disagreement; teams articulating the values at stake are equipped to debate about the appropriate tradeoffs, whereas teams that cannot often land in less nuanced opposition, mistaking a values conflict for a factual one.

The second is building dissent and structured review into process architecture, not merely into team culture. An organization that values disagreement in principle but has constructed no actualized mechanism for surfacing will reliably converge on the path of institutional least resistance. The Meta Oversight Board seeks to address this issue, instructive as an archetype for codifying adversarial scrutiny of platform decision making. Internally, analogous structures may include designated dissent roles (think ethical red teamers) in high-stakes deliberations, four (or maybe 6) eyes approval on important crossroads, or explicit criteria requiring engagement with the strongest available counterargument before decisions are finalized.

Finally, we have to embolden epistemic humility as an institutional commitment, not just an individual virtue. Given the information asymmetries and attenuated feedback loops described earlier, institutional overconfidence in enforcement determinations can’t be overstated. Humility in this context doesn’t necessitate paralysis; it prescribes the building of uncertainty into product design, maintaining progressive appeals infrastructure, and treating reversibility as a feature and not a bug of healthy governance.

Conclusion

The trust and safety field is maturing across the board. Its institutional infrastructure, the depth and rigor of its academic literature, and the recent spotlight lent by AI risks have pushed the industry to the degrees with which it engages broad questions of morality. Part of that maturation is the development of a shared ethical vocabulary, not as an instrument of universal agreement on every hard case, but as a precondition for north star-setting.

Again, the frameworks offered here are intended for discipline, not judgement. Despite the elevated calling of online safety work, we’re not immune to putting our heads down and operating informed by business goals but blind to conscientiousness; in this day and age characterized by executive pressures, layoffs, regulatory capture, & an inflated Overton window, I can fully admit that ethical navigation rarely occupies an enduring thought in my head. Yet this is exactly why I was inspired to write the piece, surfacing the values already embedded in the decisions we make daily, and providing a more rigorous common language for the work already underway.

The ultimate goal is not, and can never be a unanimous consensus. We are aiming for a sort of ambient visibility in which the ethical dimensions of trust and safety work are sufficiently surfaced, shared, and openly contested such that their working capital cannot be negated, outsourced, or resolved by industry winds or institutional default. That is a more modest aspiration than a unified theory of platform ethics. It is also, for the practitioners doing this work at scale and under pressure, the aspiration that matters most.

Bibliography

Gillespie, T. (2018). Custodians of the Internet. Yale University Press.

Roberts, S.T. (2019). Behind the Screen. Yale University Press.

Gorwa, R. (2024). The Politics of Platform Regulation. Oxford University Press.

Jiang, J. et al. (2022). A Trade-off-centered Framework of Content Moderation. arXiv.

Santa Clara Principles (2018/2023). On Transparency and Accountability in Content Moderation.

Davis, K.C. (1969). Discretionary Justice: A Preliminary Inquiry. University of Illinois Press.

Davis, A.J. (2007). Arbitrary Justice: The Power of the American Prosecutor. Oxford University Press.

Stuntz, W.J. (2011). The Collapse of American Criminal Justice. Harvard University Press.

Tyler, T.R. (2006). Why People Obey the Law. Princeton University Press.

Pan, S. et al. (2022). Comparing the Perceived Legitimacy of Content Moderation Processes. arXiv.

Blackwell, L. (2025). Content Moderation Futures. arXiv.