The price of a "clean" internet | Hans Block and Moritz Riesewieck

TL;DR

Moritz Riesewieck and Hans Block explore the disturbing world of content moderators, the hidden guardians of our "clean" and "safe" social media platforms.

Transcript

[This talk contains mature content] Moritz Riesewieck: On March 23, 2013, users worldwide discovered in their news feed a video of a young girl being raped by an older man. Before this video was removed from Facebook, it was already shared 16,000 times, and it was even liked 4,000 times. This video went viral and infected the net. Hans Block: And t... Read More

Key Insights

• 😮 The spread of harmful content on social media, like the viral video of a rape, raises questions about how such content can be allowed on platforms like Facebook and Twitter.
• 🔎 Image recognition software struggles to accurately distinguish between explicit content and harmless imagery, making human content moderators necessary to make delete or keep decisions.
• 😷 Content moderators, who work in secret, are subjected to trauma and psychological damage from exposure to disturbing and graphic material, with some even developing post-traumatic stress disorders.
• 🌍 Social media plays a pivotal role in spreading information quickly, but the speedy deletion of potentially disturbing content can hinder efforts to document war crimes or raise awareness of social injustices.
• 🛡️ Content moderators face the difficult task of deciding whether to delete or keep controversial and ambivalent content, such as that posted by civil rights activists or citizen journalists.
• 💻 Personal filter settings proposed by Mark Zuckerberg may lead to isolated online bubbles, preventing meaningful discussions and the awareness of social issues.
• 🌐 The balance between freedom of expression and the need for protection in the digital space is a crucial consideration, and it poses a challenge to the design of an open or closed society online.
• 🌧️ The content moderation industry, which operates largely in outsourcing firms in places like the Philippines, represents an outsourcing of responsibilities once held by state authorities. Open debate about the moderation process should be held in new institutions that reflect the diversity of users.

Questions & Answers

Q: How did a video of a young girl being raped go viral on Facebook?

The video went viral on Facebook because it was shared 16,000 times and received 4,000 likes before it was removed. This raises questions about how such content could make its way onto the platform in the first place.

Q: Why don't we often see revolting content on social media platforms like Facebook, Twitter, or Google?

The difficulty lies in the limitations of image-recognition software, which has trouble distinguishing between explicit and non-explicit content. It struggles to differentiate pornographic material from harmless images like holiday pictures or educational campaigns.

Q: Who is responsible for deciding what content should be deleted and what should remain?

The responsibility falls on content moderators, who work in secret and sign nondisclosure agreements that prohibit them from sharing their experiences. They are threatened with fines and monitored by private security firms to ensure they do not speak to journalists.

Q: What are the potential psychological effects on content moderators?

Content moderators are exposed to horrifying and disturbing images and videos, which can have long-lasting effects on their mental health. These include eating disorders, loss of libido, anxiety disorders, alcoholism, and even post-traumatic stress disorders similar to those experienced by soldiers.

Q: How does the rapid spread of content on social media impact society?

Content posted on social media platforms can quickly reach millions of people, influencing their thoughts and actions. Once such content spreads, it can incite hatred and anger, leading people to either engage in online hate speech or take to the streets in violent protests.

Q: What is the dilemma faced by content moderators when deciding what content to delete?

Content moderators often have to make swift decisions regarding controversial or ambivalent content, such as that posted by civil rights activists or citizen journalists. The challenge lies in balancing freedom of expression with the need for protection and safety, as determining the potential impact of certain images or videos is difficult.

Q: Should potentially disturbing content be deleted or preserved for investigative purposes?

There is a debate about the value of preserving potentially disturbing content, especially when it relates to war crimes or human rights abuses. While some argue that such content should be deleted to protect viewers, others believe it has the potential to raise awareness and prompt social change.

Q: How can we find a balance between freedom of expression and the need for protection on social media platforms?

It is essential to have open discussions about freedom of expression in new institutions that reflect the diversity of users, rather than leaving these debates solely to platforms like Facebook or Twitter. Creating new cosmopolitan parliaments and institutions can help ensure a more balanced approach to addressing the challenges of the digital age.

Summary & Key Takeaways

• A video of a young girl being raped went viral on Facebook, raising questions about how such content gets on the platform and why it isn't seen more often.

• Image-recognition software has difficulty distinguishing between pornographic content and other images, so humans are needed to decide which content should be deleted.

• Content moderators, who work in secret and are monitored by private security firms, suffer from mental health issues and post-traumatic stress disorders from the disturbing content they view.