Training artificial intelligence is an energy intensive process. New estimates suggest that the carbon footprint of training a single AI is as much as 284 tonnes of carbon dioxide equivalent – five times the lifetime emissions of an average car.
Emma Strubell at the University of Massachusetts Amherst in the US and colleagues have assessed the energy consumption required to train four large neural networks, a type of AI used for processing language.
Language-processing AIs underpin the algorithms that power Google Translate as well as OpenAI’s GPT-2 text generator, which can convincingly pen fake news articles when given a few lines of text.
These AIs are trained via deep learning, which involves processing vasts amounts of data. “In order to learn something as complex as language, the models have to be large,” says Strubell.
A common approach involves giving an AI billions of written articles so that it learns to understands the meaning of words and how sentences are constructed.
To measure the environmental impact of this approach, the researchers trained four different AIs – Transformer, ELMo, BERT, and GPT-2 – for one day each, and sampled the energy consumption throughout.
They calculated the total power required to train each AI by multiplying this figure by the total training time reported by each model’s original developers. A carbon footprint was then estimated based on the average carbon emissions used in power production in the US.
Renewables for AI
A process called the neural architecture search (NAS) – which involves automating the design of a neural network through trial and error – was particularly energy intensive and time-consuming. Training Transformer without NAS takes 84 hours, but more than 270,000 hours with it, requiring 3000 times the amount of energy. Such training is split over dozens of chips and takes months to complete.
Its inefficiency stems from the need to fine-tune the model for very specific tasks, such as translating from one language to another, says Strubell.
Big tech firms such as Amazon and Google offer cloud-based platforms that researchers can pay to use remotely for training AIs. To get a more accurate picture of the associated carbon footprint the analysis would have to account for the actual energy mix these companies use.
Amazon’s energy sources are comparable to the breakdown across the US, says Strubell. However, this may be changing as the company is investing in wind and solar farms, and according to its website was powered by more than 50 per cent renewable energy last year. Amazon declined to comment on the research.
Similarly, Google has long-term agreements with renewable energy suppliers, which reduces the carbon emissions associated with AI training processed by its data centres.
“From an energy perspective, and from a carbon reduction perspective, we should be thinking about designing the services and making sure the algorithms are efficient as possible,” says Chris Priest at the University of Bristol.
The research will be presented at the Annual Meeting of the Association for Computer Linguistics in Florence, Italy in July.