What T&S loses with a compliance mindset
I'm Ben Whitelaw and this is Trust & Safety Insider. I wanted to write about something that's been on my mind: the downstream effect of compliance-as-a-mindset on T&S professionals.
While we're on the topic of interacting with peers, why not register to join the next EiM Community Call on 30th November? The topic is yet to be decided so now is the time to pitch your ideas or the problem you're working on.
As always, thanks for reading — Ben
Powered by data from trusted sources and Thorn’s issue expertise, Safer helps trust and safety teams proactively detect and prioritize CSAM and child sexual exploitation conversations with:
- Proprietary hashing and matching for known CSAM
- CSAM Classifier for finding new or previously unreported content in images or videos
- Context Labels for classified CSAM that can be used to filter, sort, and prioritize flagged content.
- Text Classifier provides risk scores for conversations that contain sextortion, grooming or requests for self-generated content from a minor.
Online harms continue to evolve. Get purpose-built solutions backed by experts in child safety technology.
The other cost of a compliance mindset
When I think about the future of the internet, I imagine it being shaped by T&S professionals: people who have spent years watching, up close and at scale, what internet users do to — and for — each other online; who know which design choices invite harm and which protect people; and who, most of all, understand what it means to make trade-offs between speech and safety.
Last week, at an event I co-organised about the UK social media ban, someone asked a question that threw that into doubt.
The question was about whether we’ll be having the same conversation about restricting the internet to teens 20 years from now. It stopped everyone — including me — in their tracks.
After a long pause, one panellist said that, as a society, we found it “very easy to talk about the problems of the internet and much harder to talk about what’s good about it”.
Based on the conversations at the event, I’d go even further than that: many people working in platform T&S, tech policy or product safety struggle to envisage what a better version of the internet is.
That's not to say that they can't. It's more that many do not have the headspace to think about anything other than the next incoming regulatory obligation.
The chance, a number of people told me last Thursday, to spend a bit of time imagining something different from, or better than, the current version of the web was novel and very welcome.
My hypothesis, which I will try to explain in today’s T&S Insider, is that regulation has not only changed the actual work that T&S involves but has also become the default mindset for many platform professionals. The people who previously said “what could we build to make people’s online experience better and safer and more creative?” now say “what do we need to avoid?”.
From where I sit, that’s something to be concerned about.
The rise of compliance culture
Many EiM readers will have read Daphne Keller’s piece about the “compliance-isation” of Trust & Safety In it, the Stanford law professor describes how regulation has pushed T&S teams into focusing on "the process for making decisions about users’ speech rather than the underlying speech rules or individual decisions affecting users." Mechanisms like risk assessments, transparency reports and Requests for Information (RFIs) are generating such overhead that some very well-resourced platforms have tried, and failed, to push on how much work it entails.
I should say that I am in principle in favour of platform regulation, and I understand how hard it is to write rules that work in practice. What I didn’t anticipate was the side effect: regulation atrophying the thinking of people who built safety interventions for user bases in the hundreds of millions, and who experimented with new ways of giving communities a say in the rules.
From the conversations I had on Thursday, I think the cause is time and headspace. T&S people who know the internet better than most are too busy meeting obligations to imagine what comes next.
Out in the wild
The day after the event, I did three things that showed me where T&S professionals are absent in the conversation about building what comes next.
The first was that I watched a panel discussion from MozFest, the annual convening run by Mozilla. In it, Rudy Fraser, the founder and CEO of Blacksky Algorithms, explained how its community members gave input on how Rudy himself and Blacksky use AI. Through a vote, they advocated for open source models, local hardware and transparency about when Blacksky used technologies provided by the major frontier labs. Go and have a watch, it's impressive.
The second was that I read a Tech Policy Press essay co-authored by Lisa Macpherson of Public Knowledge and Anna Tumadóttir of Creative Commons in which they argued that a new exchange model is forming on the open web, driven by AI training and the fight over fair use, and that without intervention it could close off the knowledge commons, including for beneficial uses. The authors proposed principles to build solutions for a better internet, including prioritising pro-competition ideas and being mindful of the “unintended consequences and unexpected uses of solutions we may develop” — regulation included.
The third was that I received an invitation to join a futures-visioning roundtables being organised by Amnesty International. The goal is to “strengthen the collective power of the tech accountability movement, build the infrastructure needed for a more just digital future, and articulate a compelling alternative vision where technology empowers people, strengthens civic space, and advances human rights.” Who doesn't want to do that, eh?
What struck me was how each of these three ideas, concept and frames was radically different from the mindset of the T&S professionals at our event.
One is consumed by a regulatory checklist, almost at the cost of all else. The other — full of builders, academics and civil society actors — refuses to believe that that's the best we can do.
The cost of doing business
Conferences are a decent guide to where a profession’s attention is targeted. Another event I went to last week was Marketplace Risk, the self-proclaimed "world's largest digital risk conference" and a must-attend for many T&S professionals.
Compare that with the places where the future of the internet is being argued over. I don’t know for sure, but I’d guess the number of T&S workers who attended MozFest this year can be counted on one or two hands. The same can probably be said of the Chaos Communication Congress or AtmosphereConf, two other events with a focus on rewilding the web.
What does that tell me? That the people with the deepest knowledge of user harm, and of building mitigations and interventions for when harm happens, are mostly absent from conversations that matter: about creating spaces where the incentives are different, which work in new ways and which treat user data as something to protect rather than harvest.
You might argue that having T&S workers involved in regulatory preparation is the right deployment of their experience and knowledge. That’s a fair point and I wouldn't necessarily disagree. But it leaves a worrying gap in who builds the new internet.
Danielle Keats Citron and Ari Ezra Waldman made a related point in their recently published paper, The Evolution of Trust and Safety. They describe today’s political climate as a unique opportunity for scholars, policymakers and practitioners “to build anew” and to imagine what trust and safety might look like in future.
The concern I have from talking to people at last week’s events is that T&S workers don’t have time to “build anew” if their bandwidth and expertise are exhausted by playing regulatory whack-a-mole. Meanwhile, the next iteration of the internet is being created without them.
Member discussion