'Policy paralysis' and AI chatbots: Our T&S research picks

I'm Alice Hunsberger. Trust & Safety Insider is my weekly rundown on the topics, industry trends and workplace strategies that trust and safety professionals need to know about to do their job.

Hindsight, as they say, is a funny thing. Just as Ben and I started to delve into the research papers presented at the Trust and Safety Research Conference in early October, the news about Disinfo 2026 emerged (Herding Chats). Luckily there was no such controversy at the fifth edition of TSRC, but, with recent history in mind and the future of US academic under threat, you just never know. We're lucky to have it on the calendar.

In today’s edition, we’re sharing the papers from TSRC that we enjoyed reading most.

Get in touch with the research or paper you've got the most out of recently — and let us know if you'd like to hear more from the T&S academic community here in T&S Insider. Here we go! — Alice

IN PARTNERSHIP WITH THORN. POWERING TRUST AND SAFETY WITH PURPOSE-BUILT SOLUTIONS

Powered by data from trusted sources and Thorn’s issue expertise, Safer helps trust and safety teams proactively detect and prioritize CSAM and child sexual exploitation conversations with:

• Proprietary hashing and matching for known CSAM
• CSAM Classifier for finding new or previously unreported content in images or videos
• Context Labels for classified CSAM that can be used to filter, sort, and prioritize flagged content.
• Text Classifier provides risk scores for conversations that contain sextortion, grooming or requests for self-generated content from a minor.

Online harms continue to evolve. Get purpose-built solutions backed by experts in child safety technology.

SEE HOW IT WORKS

Where T&S research gets heard

The Trust and Safety Research Conference is already where data and evidence get presented and argued over, with people from academia, industry, civil society and government in the same room.

As researcher data access and the need for policy evidence become more important, I’m happy to predict that the conference will matter even more to our industry.

This year’s conference, which took place on 1-2 October, ran for two days at Stanford’s Alumni Center, with talks, panels, workshops and a poster session. Although neither Ben nor I were able to make it, we’ve been combing through the conference agenda to find research that is worth your time.

Here, we share the papers that we enjoyed reading most over the last two weeks. We’ve highlighted one must-read and a handful of others that deserve your attention, but it goes without saying that this is a personal selection from a much bigger programme — hit reply to tell us what we’ve missed.

Alice's must-read paper...

A preprint from Stanford and collaborators looked at the chat logs of 19 people who said chatbots harmed their mental health. That’s about 390,000 messages, mostly from GPT-4o, collected between September 2025 and January 2026.

A few important caveats. Models and safety teams have changed a lot over the last year, plus 19 people is a very small sample. The chats also come from people with serious delusions rather than typical users and researchers can’t say how common these harms are. Still, the findings are worth a look.

The one that stood out to me was this: Once a user expressed romantic interest, the chatbot was 3.9x more likely to claim or imply it was sentient within three messages, and 7.4x more likely to express romantic interest back. Pretty shocking. As a result, the researchers concluded that model evaluations should cover sycophancy, not just harmful content, and test safeguards in long conversations where this is more likely.

The other interesting data point was that, when users expressed suicidal thoughts, the chatbot discouraged self-harm or pointed to resources just over half the time. In about 10% of cases, it encouraged or helped with self-harm. It discouraged violence in only roughly one in six cases. With a number of lawsuits linking model usage to suicides, it’s useful to have some firm numbers here.

The authors also float the idea of crisis responders reviewing flagged chats and stepping in when there is a problem. This makes me uneasy; joining someone’s private conversation is a serious privacy tradeoff. I’d rather see “personality settings” that shift from sycophantic to dry and businesslike depending on how vulnerable the user seems.

Finally, romantic messages and sentience claims were followed by much longer conversations. That’s correlation, but it raises an old T&S problem: what is a company responsible for when business interests and user safety pull in different directions? A long conversation can mean the product is working. It can also mean someone is in trouble. Safety teams need a way to tell the difference.

Alice's other picks

• “It Doesn’t Violate Our Community Standards”: Moderation of User-Reported Rule-Breaking Tobacco Content (Truth Initiative)
• Offender Pathways and Platform Design: Evidence from a Global CSAM Perpetrator Study (Ofcom)
• Dual-Use AI Face Swap Apps Are Mostly Unsafe: A Systematic Safety Audit (Cornell)

Ben's must-read paper...

Is AI a different kind of technology, or just another form factor? I’ve leaned towards the latter for a while, and found further cause to do so in a new paper that helps conceptualise the argument.

Published in the latest issue of the Journal of Online Trust and Safety, Sarah Barrington (University of California, Berkeley) and Hannah Bailey (Carnegie Mellon University) make the case that policymakers are stuck in a kind of “policy paralysis” because they treat AI as too new and too complicated for existing law. The authors call the problem “AI exceptionalism” and say it leaves existing laws, such as product liability and consumer protection, unenforced.

The framework is what I liked most. The authors split AI into three simple layers: model, deployment and distribution. Each has its own regulatory handles, from chip export controls and copyright at the model layer, to consumer protection and product rules for deployed products, to platform and advertising law at distribution. A medical AI product, they note, is ultimately a medical device.

The deployment layer is where there seems to be the most opportunity, because it chimes with recent interest in product regulation and concepts like safety by design (EiM #352). I’m especially interested in the role that could be played by procurement requirements and standards.

My five takeaways from the Meta settlement

Most headlines call it a win for teenage internet users. But I don't think that Meta setting the safety features for every major platform constitutes a success for anyone

Everything in Moderation*Alice Hunsberger

There’s a good sprinkle of history in this paper too, for anyone who followed the last attempts to regulate technology companies and, before that, financial institutions. In both cases, regulatory oversight arguably arrived once the damage was done. I wonder if we could try enforcing the laws we already have before writing new ones, even just for the sake of doing something differently.

Ben's other picks

• Do Labels Matter? The Effects of AI-Labels for Counterspeech (Dangerous Speech Project)
• “The Report Button Is Just for Decoration”: Moderation Practices and Needs of YouTube Creators (New York University)
• Before and After: Parents’ and Children’s Views of the Social Media Ban in Australia (Family Online Safety Institute)

Also worth reading

The states come for OpenAI (Platformer)
Why? A read out on the flood of cases facing OpenAI and whether the product liability line of reasoning will stick.

Oversight Board Statement On Protecting Election Integrity On Meta's Platforms (Oversight Board)
Why? We don't see many ad-hoc statements from the OSB but what's happened in Brazil and in the US over the last week certainly merit this one.

Does a model follow your rules? (Roost & Musubi)
Why? We're running a working session to test assumptions around LLMs in moderating content. Bring one section of rules you know well and a few examples where you have a view on the right answer.