5 min read

An new AI model for moderation, Snap's new family feature and 'shadow promotion'

The week in content moderation - edition #171
Screengrab of Snap's VP of Global Public Policy, Jennifer Stout, at a congressional hearing in October 2021
Snap's VP of Global Public Policy, Jennifer Stout, at a congressional hearing in October 2021 (screengrab via CSPAN)

Hello and welcome to Everything in Moderation, your one-stop shop for content moderation and online safety news and analysis. It's written by me, Ben Whitelaw.

This week marks the fourth anniversary of the first edition of EiM which, frankly, is crazy. If you've been here from the start, you'll have noticed that the newsletter has evolved into an early warning system of sorts, designed to help people working in online speech and content moderation stay ahead of threats and risks by keeping them up-to-date about what is happening in the space. I hope you feel that it delivers on that.

I recently got a new full-time job but am no less committed to producing EiM each week in my spare time. I'm grateful to each and every subscriber that reads, shares, provides feedback, and supports the newsletter by becoming an EiM member. 171 editions down; hopefully, plenty more to go.

It's that time of the year when I take a short break so the next newsletter will be in your inbox on Friday 2nd September.

Here's this week's round-up — BW


New and emerging internet policy and online speech regulation

Kenya's recent threat to ban Facebook following its failure to remove hate-filled ads in the country is a reminder of the "the false choice between safety and content platform banning", according to an op-ed from Julie Owono, executive director of Stanford's Content Policy & Society Lab (CPSL). In an interesting and important piece, the Oversight Board member says it is:

"urgent to demonstrate that it is possible to propose a content moderation that doesn’t renounce fundamental rule of law and democratic principles".

She also urges platforms to work with civil society organizations and governments to find "context-specific solutions", of which there are too few, especially outside the US. My read of the week.

A new report has found that Russian TikTok users continue to post content despite the supposed suspension of uploads to the platform back in March (EiM #150).  Research by the digital rights nonprofit Tracking Exposed found that content appeared on the For You Page of the app, despite not showing on the poster's profile. This so-called 'shadow promotion' is "something we’ve never seen before," according to Tracking Exposed's head of research Salvatore Romano and merits more research.

After her inclusion in last week's newsletter (EiM #170), Marika Tedroff has followed up with an interesting post on the trade-offs implicit in body image content guidelines. The former TikTok policy manager explains why the video streaming company banned ads about weight loss products but, when it came to organic content of the same ilk, questioned whether it was "a platform problem, a product problem, a societal problem, or a user problem?".


Features, functionality and startups shaping online speech

Parents and guardians will now be able to quickly report concerns directly to Snap’s Trust and Safety team after the camera company (still feels weird saying that) launched the ability to view their child’s friends list and a list of accounts they’ve interacted with over the last seven days.

The Family Center (sic) was mentioned at a US congressional hearing by Jennifer Stout, Snap's VP of Global Public Policy, back in October and will be rolled out in the next few weeks as part of an effort to "reflect the dynamics of real-world relationships", according to a spokesperson. Kids needn't worry; you have to approve mum and dad viewing your friends list.

If you've been frustrated by Dalle2's confusing content policy or confounded by any of OpenAi's other clients, there's good news: the research company has released a new moderation endpoint that better "assesses whether the content is sexual, hateful, violent, or promotes self-harm". Time will tell whether it improves detection of harmful content, although new research from Raphaël Millière at Columbia University suggests a process he calls "macaronic prompting" could, in principle, bypass such endpoints. to In the meantime, folks can pore over a the evaluation data and a technical paper released by the company on Wednesday.

New Q&A: Where does trust and safety fit alongside cybersecurity? Is it — as one security website put it — "just about putting a positive spin on the same old task"?

It's those questions that led me to speak to Allison Nixon, Chief Research Officer at cybersecurity company Unit 221b. She's my third interviewee in the Getting to Know series, in collaboration with the Integrity Institute. Have a read and let me know what you think.

I've got two more Getting to Know Q&As lined up in the coming weeks, which are both great (if I say so myself).

If you're interested in supporting the series and showcasing trust and safety professionals like Allison, become a member today.


Social networks and the application of content guidelines  

For the second week running, the biggest platform story of this week involves TikTok and its relationship with a third-party moderation company: Forbes has found that child sexual abuse material (CSAM) was being stored in training documents widely available to staff and shared with Teleperformance, a global outsourcing company headquartered in Paris.

In doing so, the company may have breached US federal law and acted in a way that was "ham-handed and cavalier at best, and harmful and re-traumatizing at worst", according to experts. Other people's reactions, including this piece from Techdirt,  say way more than I could on this one.

Clubhouse continues to have problems identifying and preventing harassment, according to multiple accounts in a long read from Dame Magazine. One user said a room was created "specifically to attack me, body shame me, and being openly xenophobic toward my Nigerian heritage" while others say the abuse continued elsewhere after the attacks.

Back in April, I spoke to Jess Mason, Clubhouse's Head of Global Policy and Public Affairs, about how the company was trying to move on from its early negative press so it's notable that this is remains a problem.


Those impacting the future of online safety and moderation

This week saw the release of the schedule for the Trust and Safety Research Conference and boy have the organisers done a good job with it.

The Stanford Internet Observatory and the Trust and Safety Foundation deserve credit for putting together some great sessions over two days in late September, including this one on harm reduction and an interesting practical session on pulling data from Twitter. The event immediately follows TrustCon 2022, which is aimed at trust and safety professionals.

It looks like I won't be able to attend due to new job commitments but tickets are very reasonably priced (especially for academia/civil society/government attendees) so, if you can make it, I'd recommend doing so.

Tweets of note

Handpicked posts that caught my eye this week

  • "platforms were mostly consistent, but we did find many examples of identical content being treated differently." Assistant professor Samantha Bradshaw shares newly published work on whether platforms were consistent when labelling posts.
  • "like EXACTLY the same. like literally nothing has gotten even a little bit better" - Lena reacts to last week's story on Moroccan moderators facing poor working conditions.
  • "ahh just got an article accepted by new media and society!" - I really felt the joy in Rae Jereza's tweet and am sending congrats.

Job of the week

Share and discover jobs in trust and safety, content moderation and online safety. Become an EiM member to share your job ad for free with 1300+ EiM subscribers. This week's job of the week is from an EiM member.

ActiveFence, a leading Trust & Safety start-up based in Tel Aviv, is looking for a Human Exploitation Researcher to provide valuable insights regarding online threats, ways of distributing content, motivations for harm, and macro trends.

A successful applicant should have at least 2+ years in the academia, journalism or private sector and will be tasked with creating daily intelligence reports for Fortune 50 clients.

If that role isn't for you, ActiveFence is also looking for an Illegal Goods Researcher to join its growing Intelligence team. Again, you'll be required to stay on top of trends and developments in this field and will utilise your excellent research skills to process large amounts of both qualitative and quantitative data to reach important analyses.