After Uvalde, social media monitoring apps struggle to justify surveillance

Photo by Amelia Holowaty Krales / The Verge

With the country still reeling from the mass shooting in Uvalde, Texas, officials are scrambling for more ways to stop mass shootings and facing hard truths about how ineffective many of our existing tools really are. The Uvalde school district has been under scrutiny for experimenting with a service that claims to identify and alert schools to threats based on social media conversations.

It is an increasingly common service as schools grapple with the chaos of social media, often raising serious privacy and speech concerns along the way. The systems like Social Sentinel promise to give genuine insight into the huge volume of information posted on social media every day, so that educators can be informed of threats before harm takes place. For these firms, it can be a lucrative business, but often they are mining shallow insights from available data, providing few benefits to outweigh the privacy harms.

The lack of evidence for the technology's effectiveness means that there aren't enough grounds for the potential violations of privacy that come with its use. The researcher at Human Rights Watch said that using technology on children could cause harm.

Is it possible for schools to use toxic materials to build classrooms even if they haven't met safety standards? Han said that to use untested technologies on children exposes them to an unacceptable risk of harm.

There were multiple requests for comment sent to the company, but they did not receive a response.

The Uvalde school district bought monitoring capability from Social Sentinel in 2020 but it is not certain if the subscription was still active at the time of the shooting. Even if it had been, the technology wouldn't have flagged the posts of the shooter. There are reports of the shooter making threats to young women and girls via chat apps, sending images of guns to acquaintances, and discussing carrying out the school shooting in an online chat. Private messages would not have had access to any of the content shared in public posts.

There are privacy concerns with the software. The Brennan Center for Justice outlined a range of civil and human rights concerns stemming from expanded social media monitoring in K-12 schools, among them the questionable effectiveness of the technology in combination with a tendency to disproportionately impact students from minority communities. In the same year, Education Week reported on the dramatic expansion of digital surveillance in schools, as well as the large number of false positives generated by the technology. The alert was triggered by a student who was happy with their credit score and by a student who was happy with the Mark Wahlberg movie.

Texas has been the most enthusiastic about the use of digital snooping for school children. Texas has more school districts contracting with digital spy companies than any other state. The results of the Texas districts that took out these contracts were mixed, with a number of them declining to renew their contracts because of the lack of actionable alerts.

There is a suggestion that the accessibility of data sources is more important than the importance of the platform in which it is monitored. The EFF shared a client presentation from the company that lists a number of social media sources for monitoring.

The company skewed heavily towards Twitter monitoring, according to data obtained under the Freedom of Information Act. Even though social media sites are used more by younger people, 98 percent of the alerts provided to BuzzFeed were related to social media. It is easier to monitor than it is to ignore the vast majority of posts because of the conventions of the social media site.

The success of social media monitoring programs is dependent on how few options school officials have. School staff are aware that they need at least some insight into their students' online activities to carry out their duty of care. Tackling cyberbullying is a big challenge for teachers, and the UN recently noted that it is a major concern of parents whose children use the internet. Mental health advocates have long highlighted the negative effects of social media on self-esteem, while other online harms like revenge porn are also sadly a part of young people's lives online.

A range of companies promising to safeguard the wellbeing of children and teens through a range of monitoring services that will keep tabs on their digital lives has come into this landscape, with Gaggle and Securly often cited as main alternatives.

Social media monitoring companies have been backed by investors to the tune of tens of millions of dollars, betting on the longevity of digital surveillance as a feature of the educational landscape. The perception of social monitoring as a high-tech, relatively low-cost safety solution has proved to be attractive for school districts across the country. It seems likely that the expansion of technology will continue until schools can be made safe by other means.