Calvin Qi works for a search startup called Glean and would love to make his products more intelligent by using the most recent artificial intelligence algorithms.
Glean allows you to search through applications such as Salesforce, Slack and Gmail. Qi claims that Gleans customers will be able to find the right file or conversation faster using new AI techniques to parse language.
However, it is expensive to train such an advanced AI algorithm. Glean therefore uses smaller, less efficient AI models that are not as capable of extracting meaning from text.
Qi states that it is difficult for smaller companies with lower budgets to achieve the same results as Amazon and Google. He says that the most powerful AI models are impossible to achieve.
In the last decade, AI has produced some amazing breakthroughs. Programs can now beat humans in complex games, drive cars through cities under certain conditions, respond quickly to spoken commands and create coherent text from a brief prompt. Recent advances in computer technology have made it possible to manipulate and parse language.
These advances can be attributed to giving algorithms more text to use as examples and giving them more chips to process it. This is expensive.
OpenAIs language model GPT-3 is a large mathematically simulated neural network that was fed vast amounts of text from the internet. GPT-3 is able to predict with remarkable coherence which words should be followed by other words using statistical patterns. GPT-3 performs better than AI models in answering questions, summarizing texts, and correcting grammatical mistakes. It is 1000 times more powerful than GPT-2, its predecessor. Training GPT-3 was expensive, at some estimates, about $5 million.
Qi believes that GPT-3 would be easily available and affordable, which would greatly enhance our search engine. This would be really, really powerful.
Smaller companies with smaller budgets may not be able to achieve the same results as Amazon or Google. Calvin Qi, Glean
Established companies are also facing problems with the rising cost of advanced AI training.
One division of Optum is home to Dan McCreary, who leads a team that analyzes transcripts to determine if there are any high-risk patients. Even training a model of language that is only one-thousandth as large as GPT-3 can quickly drain a team's budget. The cost of training models for specific tasks can run into the thousands. This is because they are paid to cloud computing companies that rent their computers and programs.
McCreary claims that cloud computing providers don't have any reason to lower their cost. He says that we cannot trust cloud providers to lower our costs of building our AI models. He is considering buying specialized chips to accelerate AI training.
AI's rapid progress has been due to the fact that many startups and academic labs were able to download the most recent ideas and techniques. Algorithms that made breakthroughs in image processing were created by academic labs. They used off-the-shelf hardware, openly-shared data sets, and were built using these tools.
It has been clear over time that AI progress is closely tied to an exponential rise in the underlying computing power.
Large companies have always enjoyed advantages in budget, scale and reach. In industries such as drug discovery, large amounts of computer power can be a major advantage.
It reduces innovation, I believe. Chris Manning, Stanford professor of machine learning.
Some are now pushing for even greater scale. Microsoft claimed this week that it has built a language model with Nvidia that is more than twice the size of GPT-3. Chinese researchers claim they have built a language model four times bigger than GPT-3.
According to David Kanter (executive director of MLCommons), which tracks the performance and cost of AI-related chips, the cost of training AI is on the rise. He says that the idea of larger models unlocking valuable new capabilities is evident in many sectors of tech industry. This may be why Tesla has created its own chips to train AI models for autonomous driving.
There are concerns that the rising cost to tap the latest tech might slow down innovation. This is because it will be reserved for the largest companies and those who lease their tools.
It reduces innovation, according to Chris Manning, a Stanford professor whose research focuses on AI and language. There are only a few places that people can explore the inner workings of such models, which has to severely limit the creativity that occurs.
