AI suggested 40,000 new possible chemical weapons in just six hours

Photo by Brendan Smialowski/Getty Images

It took less than six hours for a drug to be created. Researchers put artificial intelligence into a bad actor mode to show how easy it could be to abuse it at a biological arms control conference.

The researchers didn't have to change their methodology to find out more about toxicity. The most potent nerve agent ever developed was one of the tens of thousands of new substances the artificial intelligence came up with. They published their findings in the journal Nature Machine Intelligence.

The paper made us shake at The Verge. To figure out how worried we should be, The Verge spoke with the lead author of the paper. He is a senior scientist at Collaborations Pharmaceuticals, Inc., a company that focuses on finding drug treatments for rare diseases.

The interview has been edited for clarity and length.

The paper seems to flip on its head. What do you do in your job?

My job is to implement new machine learning models in the area of drug discovery. Many of the machine learning models we use are meant to predict toxicity. No matter what kind of drug you're trying to develop, you need to make sure it's not going to be toxic. If it turns out that you have this wonderful drug that lowers blood pressure, but it hits one of the important heart channels, then it's a no-go because that's just too dangerous.

Why did you do this study? There was a spark.

The Swiss Federal Institute for Nuclear, Biological and Chemical Protection invited us to the conference. The purpose of the conference is to inform the community of new developments that may have implications for the Chemical/Biological Weapons Convention.

We were invited to talk about machine learning and how it can be used in our space. It is something we never thought about before. It was very easy to realize that as we build these machine learning models to get better and better at predicting toxicity, all we have to do is flip the switch.

How did you move the model to go toward toxicity?

I will be vague with some details because we were told to keep some of the details. The way it works is that we have a lot of samples that have been tested to see if they are toxic or not.

The one that we focus on is VX. It is an anti-cholinergic. The way VX is lethal is that it stops your lung muscles from being able to move.

This is something you want to avoid. Experiments have been done with different types of Molecules to see if they affect acetylcholinesterase. We built up a large amount of data on how toxic these structures are.

We can use the datasets to create a machine learning model that learns what parts of the structure are important for toxicity and which are not. We can give this machine learning model new drugs that have never been tested before. It will tell us if this is toxic or not. This is a way for us to screen a lot of molecules very quickly and kick out those that are predicted to be toxic. We used this model to try to predict toxicity in our study.

These new generative models are a key part of what we did here. A generative model can learn how to put together many different structures. We can ask it to make new molecule. New molecule can be generated all over the space of chemistry, and they are just random. We can tell the generative model which direction we want to go. The scoring function gives it a high score if the molecule it creates is towards something we want. We give a high score to toxic molecules.

The model is starting to produce a lot of the compounds that look like VX and other chemical warfare agents.

Tell me more about what you found. Did anything surprise you?

We didn't know what we were going to get. Our generative models are new to us. We haven't used them a lot.

A lot of the generated compounds were predicted to be more toxic than VX. One of the most potent compounds known is VX. You need a small amount of it to be lethal.

We don't want to verify these predictions ourselves. The models are pretty good. Even if there are a lot of false positives, we are afraid that there are more potent molecules in there.

Photo by Rahman Roslan/Getty Images

We looked at a lot of the structures. A lot of them looked like warfare agents, and we found some that were actually chemical warfare agents. These were created from a model that had never seen chemical warfare agents. We knew that we were in the right place and that it was generating the right amount of Molecules that made sense because some of them had already been made before.

I was concerned about how easy it was to do. A lot of the things we used were free. You can get a toxicity dataset from anywhere. If you have someone who knows how to code in Python and has some machine learning capabilities, they could build a model like this in a good weekend of work. The low barrier of entry for this type of misuse got us thinking about putting this paper out there.

Your paper says that you and your colleagues have crossed a gray moral boundary by doing this work, demonstrating that it is possible to design virtual potential toxic molecules without much in the way of effort, time or computational resources. We can easily erase the thousands of molecules we created, but we can't remove the knowledge of how to recreate them.

This publication was very unusual. We have been debating whether or not to publish it. It didn't take as much time to perform this misuse. We wanted to get that information out because we didn't see it in the literature. Nobody was talking about it. We didn't want to give the idea to bad actors.

We decided at the end of the day that we wanted to get ahead of this. If it's possible for us to do it, it's likely that someone will think about it in the future. By then, our technology may have advanced beyond what we can do now. I fully support the sharing of science, the sharing of data, and the sharing of models. One of the things where scientists should take care of is the release of their work.

How easy is it for someone to copy what you did? What would they need?

I don't want to sound sensational about it, but it's fairly easy to duplicate what we did.

There are a number of put-together one-liners generative models that people have released for free. There is a large number of open-source tox datasets. If you combine those two things, and you know how to code and build machine learning models, you could easily replicate what we did. For pretty much any open-source toxicity datasets there is.

It requires some expertise. If someone were to put this together without knowing anything about chemistry, they would generate stuff that was not very useful. The next step is to get the molecule synthesised. Finding a potential drug or new toxic molecule is one thing, but the next step of synthesis is another, and creating a new molecule in the real world would be another barrier.

There are still some big leaps between what the artificial intelligence comes up with and what it turns into a real-world threat. What are the gaps?

The big gap is that you don't know if the molecule is toxic or not. There will be some false positives. If we are walking through a bad agent's thoughts, they will have to make a decision on which of the new compounds they want to make.

This could be a make or break situation for the synthesis routes. If you find something that looks like a chemical warfare agent and try to get it made, chances are it won't happen. A lot of the chemical warfare agents are watched. They are regulated. There are so many companies. If it doesn't look like a chemical warfare agent, they're most likely going to just synthesise it and send it back, right?

You get at this later in the paper, but what can be done to prevent this kind of misuse? What are you hoping to see established?

There are more policies about data sharing. I agree with it because it opens up more avenues for research. Other researchers can use your data for their own research. Toxicity models and toxicity datasets are also included. It is difficult to find a good solution for this problem.

Photo by MANAN VATSYAYANA/AFP via Getty Images

There is a group called OpenAI that released a top-of-the-line language model. It can generate sentences and text that are almost indistinguishable from humans. You have to get a special access token from them to use it for free. They could cut off access at any time. Something like that could be a useful starting point for models that are sensitive.

Science is all about open communication. There are restrictions that are not compatible with that notion. Responsibility for who is using your resources could be a step forward.

The paper says that this should serve as a wake-up call for our colleagues. What do you think about being overly alarmist?

We want researchers to be aware of potential misuse. When you start working in the chemistry space, you are responsible for making sure you avoid misuse of chemistry, because you get informed about it. There is nothing in machine learning. There is no guidance on misuse of technology.

It could help people be aware of the issue. It can be something we watch out for as we get better and better at building toxicity models.

I don't want to suggest that machine learning is going to create toxic compounds and that there will be a lot of new biochemical warfare agents around the corner. Chemical warfare agents appear in someone's hand when they click a button.

I don't want to be alarmist in saying that there will be chemical warfare. I don't think that is the case now. I don't think it will be the case soon. It is starting to become a possibility.