Is this the end of 'content moderation?'

There are four problems for Mark Zuckerberg to worry about

Mark Zuckerberg has been waiting years for the opportunity to scale back Meta’s ‘content moderation’ system - that intricate set of rules that dictates what’s allowed on this platforms. The election of Trump (and Musk) gave him the chance. Unsurprisingly, he took it.

Critics say it’s opportunistic – a chance for Zuck to jump on the MAGA train, to curry favour, to save money. (Content moderation teams cost Meta millions each year). Some think he’s just ‘giving in to the right’.

I don’t think that’s quite right. Meta – like Twitter, YouTube, TikTok and the rest – never wanted to get into the content moderation game in the first place. They’ve always disliked it, and felt bounced into becoming ‘arbiters of truth’.

But if Mr Z thinks that controversies over content will melt away, he’s mistaken. MAGA now expects a flourishing of wild, vigorous debate about issues they’ve long felt censored by big-tech: immigration, Islam, transgenderism, Covid, Ukraine. That might happen. But there will also be a surge in co-ordinated anti-American propaganda and manipulation campaigns, often directed from overseas: Iran, Russia, China. We’ll see how MAGA likes that kind of ‘free speech’. And while America might value the new mood, in other parts of the world where Meta rules all, it could be disastrous.

Why do we have content moderation anyway?

To understand Zuckerberg’s decision, you need to know how they expanded content moderation in the first place. It’s a 15-year-long slippery slope, which started with well-intended efforts to fight spam; and ended with content moderators trying to decide if saying Covid originated in a lab was ‘misinformation’.

The animating dream of social media was to free the world’s information. “People being able to express themselves at scale is a new force in the world. It is a fifth estate.” As Mark said, many years ago. Decades before that, proto-social media platforms – bulletin board systems, the Well, blogging sites – were popping up all over the place, promising to liberate information from the stranglehold of establishment media magnates and editors. That dream found legal backing in 1996, when section 230 of the Communications Decency Act was passed in the US, granting internet websites legal immunity over the content they carried: “No provider or user of an interactive computer service shall be treated as the publisher or speaker of any information provided by another information content provider.” (Importantly, section 230 also said that interactive computer services could censor content without becoming liable.)

Incidentally the whole story of section 230, which is the basis of all modern social media, is fascinating, and involves the Wolf of Wall Street, Jordan Belfort.

No-one at the time predicted what would follow. Social media platforms pursued a wildly successful ‘growth-at-all-costs’ model. Suddenly we were all online, and we started causing problems. Illegal pornography proliferated. Bullying. Holocaust denial. Hate speech. Pro-suicide material. Terrorist propaganda. It started to strain American – and especially other countries’ – speech laws. Not to mention making these platform unpleasant places for users and advertisers.

In Silicon Valley, the naïve 20-something utopians running these sites were surprised that so many people were doing bad things on their platforms. Slowly, and unhappily, they introduced ‘community guidelines’. Rules about what was and wasn’t allowed on their sites. Pretty soon tens of thousands of people were sitting at desks in the Philippines, Kenya and elsewhere, working 12 hour days to review and remove millions of rule-breaking posts. As new problems were identified the rules expanded. Sometimes it was pressure from users. Sometimes governments.

And more than the platforms would ever admit, it was advertisers who didn’t like their products appearing next to ISIS videos or self-harm images.

Mark Zuckerberg and all the tech guys didn’t like it. I think they did want to get rid of the bad stuff, but it meant getting tangled up in every complex, nuanced, and thorny social question of the day. These were meant to be free-wheeling people-powered neutral platforms! By attempting to keep it PG, they were becoming arbiters of truth. Meta tried anything it could to outsource the responsibility, in a desperate attempt to appear neutral: third party fact-checkers, an oversight board. But every decision they made annoyed someone.

Even a perfect system of content moderation (impossible, by the way) would be divisive. But the scale of user-generated content meant mistakes were inevitable. A one-in-a-million error happens hundreds of times a day at places like Twitter or Facebook. So there were always lots of mistakes too, which didn’t help.

To the bosses, content moderation was the least-bad solution to the irritating problem of people shit-posting. Yishan Wong, a former CEO of Reddit, summed this up best in a long Twitter thread a couple of years back.

Here’s a flavour:

It’s summed up well by Yishan’s line: “They would like you (the users) to stop squabbling over stupid shit and causing drama so that they can spend their time writing more features and not have to adjudicate your stupid little fights”.

Everything changed after the 2016 election of Trump. It wasn’t just the result that upset them, although some Facebook execs were in tears. It was the realisation that their platforms had been systemically abused by foreign actors trying to influence the result. (Whether it worked or not is a different question).

In January 2017, a few weeks after Trump’s inauguration, US intelligence agencies declassified a report. It showed that 129 million Facebook users in America had interacted with Russian propaganda pumped online by the Internet Research Agency. Over on Twitter almost four thousand accounts were judged to have been part of a co-ordinated network all geared towards electing Trump. Yoel Roth – head of Twitter’s ‘site integrity’ until 2022 told me recently that he found that report ‘scary’ and ‘deeply upsetting’. It was a ‘wake up call for Twitter and Silicon Valley’, he said.

The journalist Maria Ressa had noticed it happening in her home country of the Philippines two years earlier. Presidential candidate Duterte was running a huge, co-ordinated campaign of false stories, pushed by false people in a false network. But no-one in at Facebook seemed to care that much – at least not until it happened in the US two years later.

Post-2016 content moderation ramped up. The platforms were even tougher censors – and were increasingly looking out for what they called ‘inauthentic’ activity: bots or phoney accounts, run from overseas, pushing certain stories in order to move public opinion.

Yoel Roth was trying to prevent intentional manipulation of the public sphere by a foreign adversary. But that’s not how Republicans saw it. They felt like it was all a cover to censor right-wing views. People like Yoel were dubbed ‘censor-in-chief’ – and even bombarded with death threats. (He had to move house after the Daily Mail published a photo of his property).

The platforms did get it wrong sometimes too - it wasn’t all just Republican paranoia. The Hunter Biden laptop story was suppressed by both Meta and Twitter, and it shouldn’t have been.

Having taken on this responsibility for content moderation, the platforms soon found themselves in an even tighter spot. When the pandemic hit they had to work out what was fair debate, and what was dangerous health misinformation that could endanger lives. “It felt like trying to boil the ocean”, one former Twitter exec told me. It proved almost impossible.

As the pandemic lifted, it seemed everyone was pissed off with these information overlords. The left felt they were permitting constant hate and election interference; the right felt their voices were being subtly censored. And both agreed that children especially were exposed to harmful content.

For a while it was starting to look like section 230 might even be reformed. Committees were formed, working groups established. Donald Trump himself even said he would abolish section 230 if he won. It looked like the experiment in free information could be over.

Enter Elon Musk, and everything changed. Reform of Section 230 now looks like a pipe-dream. And with Zuckerberg’s announcement, there will be less content moderation, not more.

Although a lot of people seemed shocked by the announcement, it has been on the cards for a while. Content moderation budgets were being cut even before MZ’s intervention. Experts have been worried about looser censorship across Silicon Valley ever since Musk took over at Twitter.

Back in 2023 I interviewed a former Twitter employee called Anika Collier Navaroli for my podcast series The Gatekeepers. She was running a content moderation team there when the 2021 riots on Capitol Hill broke out. She was terrified by any prospective reduction in content moderation. She told me that she “doesn't really see a way that like Americans come out of the 2024 election with a peaceful transition of power. I genuinely believe that I may have witnessed the last peaceful transition of power within American democracy in my lifetime”.

Similarly, Maria Ressa, who I also interviewed around this time, was even more worried. “We cannot be insidiously manipulated because if we are, then…there's no shared reality. Democracy will fail. I mean, I don't know what else to say. 2024 is a tipping point…So, yeah, I sound really dystopian. So sorry.”

The people who used to work for these companies in content moderation see its faults. They see the mistakes, the overreach. They realise that by becoming the information gatekeepers they became tangled up in difficult – maybe impossible – decisions about what’s true and what’s not. But all of them think the alternative – less content moderation – is even worse.

Especially if the underlying business model – user engagement ruling all– doesn’t change. Which it won’t.

Although Zuckerberg et al hope Trump’s election and shift away from content moderation will free them from the impossible responsibility of being ‘arbiters of truth’, it won’t be straight-forward. He’ll face at least four big problems.

Inauthentic co-ordinated responses

When he ran content moderation at Twitter, Yoel Roth wasn’t just worried about specific pieces of content. He was worried about co-ordinated inauthentic activity: often driven by accounts that were fake people, pushing divisive stories to stoke division and anger. “The most subtle and sophisticated parts of a disinformation campaign are not ones where a bad actor tries to convince people to believe something untrue” he told me. “It's where things people already believe are pushed to more and more extreme ends of the spectrum.”

Zuck wants to see more ‘debate’. Me too. But what if that ‘debate’ is actually being run by Iranian controlled bots, Chinese controlled fake accounts and Russian trolls, with the clear and explicit aim of dividing Americans, or manipulating public opinion? (Which it will. I bet they’re already planning it right now). Meta will have to tread a line between defending free debate; but still removing foreign campaigns of interference. If they don’t, it won’t be long before MAGA turns on them, for allowing ‘un-American’ content to proliferate. (Which it will). That won’t be easy.

Will this apply in ‘high risk’ countries?

Zuckerberg has said that his new ‘community notes’ approach will apply to the US first. It’s not clear where else, or when. This is an important question. In places like Myanmar and Ethiopia – where ethnic tension runs deep – the issue of content moderation is more than just contentious politics. It’s life and death. In 2023 I spoke to Abrham Maereg – whose Ethiopian Tigrayan father was murdered in his front garden after false information about him being a thief and traitor was posted repeatedly on a popular anti-Tigrayan Facebook group. It was one tragic story in a wider pattern. In October 2023, the UN-appointed International Commission of Human Rights Experts on Ethiopia concluded that “[t]he prevalence of hate speech in Ethiopia, in particular online, stoked community tensions and created a climate in which individuals and groups became targets of incitement and calls to violence.” Similarly a recent UN report found that Facebook had allowed hate speech to spread on its platform in Myanmar – which they said ‘facilitated’ a genocide against Rohnigya Muslims there.

Shifting away from active content moderation in places like America is one thing. Doing it in places like Myanmar or Ethiopia is something different. What’s the plan, Mark? If this goes global, there will be huge consequences. When Frances Haugen, the ‘Facebook Whistleblower’ testified to Congress about all this a couple of years ago, she was clearly terrified about what the future held - and that was before Mark’s new policy. “What we saw in Myanmar and are now seeing in Ethiopia are only the opening chapters of a story so terrifying, no one wants to read the end of it.”

What about self-hate?

After hearing Zuckerberg’s announcement, I immediately emailed Ian Russell. His daughter Molly Russell died by suicide aged 14 in 2017 – after consuming vast amounts of self-hate and self-harm content online, especially Instagram. For years he’s been pushing Meta to strengthen their content moderation on the sorts of content Molly viewed before she died: depressing, dark, self-loathing material. The company has improved its efforts (in the last quarter, they removed around 12 million pieces of self-injury and suicide content). But for many, including Ian, it’s still not enough. Will relaxing content rules mean more of this stuff too?

Here’s the good news: Ian has been informed that the rules on self-harm and suicide content will not be weakened. However, he is worried that the guidelines on generic depressive content might be, which can also be really harmful for young people. ‘There is likely to be unintended consequences’, he told me.

Advertisers will get scared

The press – here in the UK but even more so in the US – consider social media platform as direct rivals for advertising money. Mark Zuckerberg’s decision will, invariably, result in more extreme, angry, nasty, divisive content being posted. He even said so himself. Each example will be gleefully reported by the press, served up as evidence that Facebook or Instagram is becoming unsafe and cruel. Mark might say it’s just ‘legacy media’ whining. But it will still worry advertisers, who in turn will complain to Mark’s marketing people. Advertisers are the world’s most quietly powerful lobbyists. Mark might find that he has to choose between what advertisers (and therefore shareholders) want, and what MAGA wants. Musk was able to stick two fingers up at the ad-men. I’m not sure if Mark will.

And finally, spare a thought for all those content moderators. No-one seems to talk about them much. But for several years these poor souls have been cleaning up our digital shit. They often work very long, repetitive hours, doing nothing but watching self-harm content, terrorist propaganda, beheadings, and so on. Many have complained about the long-term mental health strain that work has had. Sad as it is, this is still a pretty decent job in many parts of the world. I guess a lot them are about to be made redundant. And that sucks too.