How to use AI for policy creation & iteration
I'm Alice Hunsberger. Trust & Safety Insider is my weekly rundown on the topics, industry trends and workplace strategies that trust and safety professionals need to know about to do their job.
This is the third installment of my series on platform policy, and for this one, we’re talking AI. As a reminder, the first covered how to write user-facing community guidelines, the second covered how to operationalize them across human reviewers, LLMs, and ML models. Today we’re going to look at how to use AI to produce and pressure-test those policies faster and more thoroughly than used to be possible.
Get in touch with questions, comments, feedback, rants, raves, existential problems…. I'm hoping to have the next issue be answering questions from readers (that's YOU!), so if there’s anything you want to share, please don’t hesitate.
Here we go! — Alice
Stress-test policies at scale with AI
What’s great is, you don't need specialized software for any of this. You can start today with a model you already have and a policy you're already working on. Before we get into the exercises you can do, some general (maybe obvious) AI tips:
- Use the latest models: such as a large thinking model — I tend to use Claude Opus 4.7.
- Use technical industry jargon: weirdly this makes AI answers better quality. Use words like bias, precision, recall, tradeoffs, taxonomy, etc. for the best responses.
- Build a custom skill/ GPT/ gem for these stress tests: with step-by-step instructions and context for your specific platform, especially if you’re going to run these regularly. (Here's a policy analysis skill I created for Claude, feel free to copy/ use/ modify).
- Don’t let AI do all the thinking for you! It’s best when used as a mirror to reflect back your policies to you, warts and all. Don’t use it as an end-to-end solution or replacement for critical thinking.
Now here are some actionable ideas on how you can use AI to improve and build on your policies. Let me know if you try any of these, or if you use AI in any other ways.