Google is releasing an open source harassment filter for journalists

Illustration by Alex Castro / The Verge

The code for an open source anti-harassment tool called Harassment Manager is being released. The tool is intended for journalists and other public figures to sort through potentially abusive comments on social media. It will be a source code for developers to build on and be launched as a functional application for Thomson Reuters Foundation journalists in June.

The Harassment Manager can combine moderation options like hiding replies and blocking accounts with a reporting system. Perspective checks messages for toxicity based on elements like threats, insults and profanity. It sorts messages into queue on a dashboard, where users can address them in batches rather than individually. They can blur the text of the messages so they don't need to read them, and they can use the automatically generated queue to search for words.

Google

The Harassment Manager allows users to download a report containing abusive messages, which can be used to track down illegal content like direct threats to law enforcement. Users can not download a stand alone application for now. Developers can freely build apps that incorporate its functions and services, which will be launched by partners like the Thomson Reuters Foundation.

The tool was announced on International Women's Day, and it was framed as relevant to female journalists who face gender-based abuse. In a Medium post, the team says it hopes developers can tailor it for other-risk social media users.

Perspective has been used for automated moderation before. It released a browser extension that lets social media users avoid seeing messages with a high chance of being toxic, and it has been used by many commenting platforms to supplement human moderation. The language analysis model has been far from perfect. It sometimes misclassifies satire or fails to detect abusive messages. The toxic workplace culture has been criticized by the company.

The Harassment Manager isn't a platform-side moderation feature. Even if they can't use it for now, it's a sort tool that could be useful for people far away from journalism.