Content moderation, in the context ofwebsites that facilitateuser-generated content, is the systematic process of identifying, reducing,[1] or removing user contributions that are irrelevant, obscene, illegal, harmful, or insulting. This process may involve either direct removal of problematic content or the application ofwarning labels to flagged material. As an alternative approach, platforms may enable users to independentlyblock andfilter content based on their preferences.[2] This practice operates within the broader domain oftrust and safety frameworks.
Various types of Internet sites permituser-generated content such as posts, comments, videos includingInternet forums,blogs, and news sites powered by scripts such asphpBB, awiki,PHP-Nuke, etc. Depending on the site's content and intended audience, the site's administrators will decide what kinds of user comments are appropriate, then delegate the responsibility of sifting through comments to lessermoderators. Most often, they will attempt to eliminatetrolling,spamming, orflaming, although this varies widely from site to site.
Major platforms use a combination of algorithmic tools, user reporting and human review.[2]Social media sites may also employ content moderators to manually flag or remove content flagged forhate speech,incivility or other objectionable content. Other content issues includerevenge porn,graphic content,child abuse material andpropaganda.[2] Some websites must also make their content hospitable to advertisements.[2]
In the United States, content moderation is governed bySection 230 of theCommunications Decency Act, and has seen several cases concerning the issue make it to theUnited States Supreme Court, such as the currentMoody v. NetChoice, LLC.
Content moderation can result in a range of outcomes, includingblocking and visibility moderation such asshadow banning.[3]
Content moderation together withparental controls can helpparents filterage appropriateness of content for their children.[4]
Also known as unilateral moderation, this kind of moderation system is often seen onInternet forums. A group of people are chosen by the site's administrators (usually on a long-term basis) to act as delegates, enforcing the community rules on their behalf. Thesemoderators are given special privileges to delete or edit others' contributions and/or exclude people based on theire-mail address orIP address, and generally attempt to remove negative contributions throughout the community.[5]
Commercial Content Moderation is a term coined bySarah T. Roberts to describe the practice of "monitoring and vettinguser-generated content (UGC) forsocial media platforms of all types, in order to ensure that the content complies with legal and regulatory exigencies, site/community guidelines, user agreements, and that it falls within norms of taste and acceptability for that site and its cultural context".[6]
The content moderation industry is estimated to be worth US$9 billion. While no official numbers are provided, there are an estimates 10,000 content moderators forTikTok; 15,000 forFacebook and 1,500 forTwitter as of 2022.[7]
Theglobal value chain of content moderation typically includes social media platforms, largeMNE firms and the content moderation suppliers. The social media platforms (e.g. Facebook, Google) are largely based in the United States, Europe and China. The MNEs (e.g.Accenture,Foiwe) are usually headquartered in the global north or India while suppliers of content moderation are largely located inglobal southern countries like India and the Philippines.[8]: 79–81
While at one time this work may have been done by volunteers within theonline community, for commercial websites this is largely achieved throughoutsourcing the task to specialized companies, often in low-wage areas such as India and the Philippines. Outsourcing of content moderation jobs grew as a result of thesocial media boom. With the overwhelming growth of users and UGC, companies needed many more employees to moderate the content. In the late 1980s and early 1990s, tech companies began to outsource jobs to foreign countries that had an educated workforce but were willing to work for cheap.[9]
Employees work by viewing, assessing and deleting disturbing content.[10]Wired reported in 2014, they may suffer psychological damage.[11][12][13][5][14] In 2017, the Guardian reportedsecondary trauma may arise, with symptoms similar toPTSD.[15] Some large companies such as Facebook offer psychological support[15] and increasingly rely on the use ofartificial intelligence to sort out the most graphic and inappropriate content, but critics claim that it is insufficient.[16] In 2019, NPR called it a job hazard.[17]Non-disclosure agreements are the norm when content moderators are hired. This makes moderators more hesitant to speak up about working conditions or organize.[7]
Psychological hazards including stress andpost-traumatic stress disorder, combined with theprecarity ofalgorithmic management and low wages make content moderation extremely challenging.[18][19]: 123 The number of tasks completed, for examplelabeling content as copyright violation, deleting a post containing hate-speech or reviewing graphic content are quantified for performance andquality assurance.[7]
In February 2019, an investigative report byThe Verge described poor working conditions atCognizant's office inPhoenix, Arizona.[20] Cognizant employees tasked with content moderation for Facebook developedmental health issues, includingpost-traumatic stress disorder, as a result of exposure tographic violence,hate speech, andconspiracy theories in the videos they were instructed to evaluate.[20][21] Moderators at the Phoenix office reporteddrug abuse,alcohol abuse, andsexual intercourse in the workplace, and feared retaliation fromterminated workers who threatened to harm them.[20][22] In response, a Cognizant representative stated the company would examine the issues in the report.[20]
The Verge published a follow-up investigation of Cognizant'sTampa, Florida, office in June 2019.[23][24] Employees in the Tampa location described working conditions that were worse than the conditions in the Phoenix office.[23][25][26] Similarly, Meta's outsourced moderation company in Kenya and Ghana reported mental illness, self-harm, attempted suicide, poor working conditions, low pay, and retaliation for advocating for better working conditions.[27]
Moderators were required to sign non-disclosure agreements with Cognizant to obtain the job, although three former workers broke the agreements to provide information toThe Verge.[23][28] In the Tampa office, workers reported inadequate mental health resources.[23][29] As a result of exposure to videos depicting graphic violence,animal abuse, andchild sexual abuse, some employees developedpsychological trauma and post-traumatic stress disorder.[23][30] In response to negative coverage related to its content moderation contracts, a Facebook director indicated that Facebook is in the process of developing a "global resiliency team" that would assist its contractors.[23]
Facebook had increased the number of content moderators from 4,500 to 7,500 in 2017 due to legal requirements and othercontroversies. In Germany, Facebook was responsible for removing hate speech within 24 hours of when it was posted.[31] In late 2018, Facebook created anoversight board or an internal "Supreme Court" to decide what content remains and what content is removed.[17]
According toFrances Haugen, the number of Facebook employees responsible for content moderation was much smaller as of 2021.[32]
Social media site Twitterhas a suspension policy. Between August 2015 and December 2017, it suspended over 1.2 million accounts for terrorist content to reduce the number of followers and amount of content associated with the Islamic State.[33] Following the acquisition of Twitter by Elon Musk in October 2022, content rules have been weakened across the platform in an attempt to prioritize free speech.[34] However, the effects of this campaign have been called into question.[35][36]
User moderation allows any user to moderate any other user's contributions. Billions of people are currently making decisions on what to share, forward or give visibility to on a daily basis.[37] On a large site with a sufficiently large active population, this usually works well, since relatively small numbers of troublemakers are screened out by the votes of the rest of the community.
User moderation can also be characterized by reactive moderation. This type of moderation depends on users of a platform or site to report content that is inappropriate and breachescommunity standards. In this process, when users are faced with an image or video they deem unfit, they can click the report button. The complaint is filed and queued for moderators to look at.[38]
150 content moderators, who contracted forMeta,ByteDance andOpenAI gathered inNairobi, Kenya to launch the first African Content Moderators Union on 1 May 2023. This union was launched 4 years after Daniel Motaung was fired and retaliated against for organizing a union atSama, which contracts for Facebook.[39]
The Digital Services Act (DSA) is an EU regulation that entered into force in 2022, establishing a comprehensive framework for digital services accountability, content moderation, and platform transparency across the European Union. Users can contest moderation decisions by online platforms restricting their accounts or sanctioning their content in several ways. This right also applies to notices of illegal content that were rejected by the platform. According to the DSA, users may appeal through the internal complaint-handling system of platforms. Platforms are required to promptly review their decisions.[40]
One out-of-court dispute settlement body isAppeals Centre Europe, they challenge decisions by social media platforms and there is no charge for users. The platforms they currently review are Facebook, Instagram, Tiktok, Pinterest, Threads and YouTube.[41]
{{citation}}: CS1 maint: work parameter with ISBN (link){{citation}}: CS1 maint: work parameter with ISBN (link){{citation}}: CS1 maint: work parameter with ISBN (link)