Google is beta testing its AI future

It's clear that the future of the company is tied to the models of language. At this year's I/O conference, the company announced a raft of updates that rely on this technology.

There are major problems with the field of artificial intelligence language research. It has mishandled internal criticism, firing employees who raised issues with bias in language models and damaging its reputation with the community. Researchers continue to find issues with the models, from the fact that they have a tendency to make things up to the fact that they don't deliver reliable information.

The company seems to be taking a slower step forward. At I/O this year, there has been a new focus on projects designed to test and remedy problems like artificial intelligence bias, as well as a new way to measure skin tones that the company hopes will help with diversity in machine-vision models. Think of it as a test for the future.

Step into the AI Test Kitchen

Over a video call ahead of I/O, Josh Woodward, senior director of product management at Google, is asking the company's latest language model to imagine a volcano.

You're at a volcano of marshmallows. You feel the ground shake when you hear a giant rumble. The marshmallows are flying.

Woodward is happy with the answer and wants to know what it smells like.

Woodward is showing me an app that will give limited access to the latest and greatest artificial intelligence language model, LaMDA 2. The model itself is an update to the original LaMDA announced at last year's I/O and has the same basic functions: you talk to it, and it will talk back. Test Kitchen wraps the system in a new interface that encourages users to give feedback.

The idea is to create an experimental space for the new models.

Image: Google

The app has three modes, with each intended to test a different aspect of the app.

The Play Store will not be the only place where you can download the app, as it will be rolling out in the US in the coming months. Woodward says that the company hasn't decided how it will offer access, but that it will be on an invitation-only basis, with the company reaching out to academics, researchers, and policymakers to see if they are interested in trying it out.

Woodward explains that the app will say inaccurate things if people don't know what they're signing up for. It will say things that are not representative of a finished product.

The announcement and framing tell us a few things. It's not something a company like Google thinks it can do without outside help, because the artificial intelligence language models are so complex and testing them to find all the possible error cases isn't something that can be done without outside help. Thirdly, that the machine learning models are prone to fail, and that the company wants to manage expectations.

Image: Google

The results can be disastrous when organizations push new systems into the public sphere. Remember the Microsoft chatbot that was taught to be racist? Ask Delphi, the ethics advisor that could be prompted to condone genocide. The aim of the new app is to control the flow of feedback and to invite criticism of its systems.

The approach will limit what third-parties can learn about the system, according to Deborah Raji. Some companies like Facebook have been more open with their research, releasing models in a way that allows more scrutiny.

Exactly how the approach will work in the real world isn't yet clear, but the company expects some things to go wrong.

We have done a red-teaming process to test the weaknesses of the system, but we still think people will try and break it. There is a lot to figure out. We need to open it up so that we can figure it out.

Hunting for the future of search

It's hard to imagine how technology like this will change search in the future. The company's services are very obviously connected with the function of the test kitchen. The mode in Test Kitchen is very similar to the one in the Google Assistant.

A research paper published last year fueled such speculation. Four of the company's engineers suggested in the paper that future search engines would act like a middleman, using artificial intelligence to analyze the content of the results and then lift out the most useful information. There are new problems stemming from the models themselves, from bias in results to the systems making up answers.

There are tools like featured snippets and knowledge panels that are used to answer queries. Artificial intelligence has the potential to accelerate this process. Last year, the company showed off an experimental model that answered questions from the perspective of the former planet itself, and this year, the slow trickle of artificial intelligence-powered features continues.

Speculation about a sea change to search is not true. When I asked the vice president of research at Google how artificial intelligence will change the search engine, his answer was something of an anticlimax.

I think it will be gradual, but I think it matches reality. He is careful to say that the search box should be the beginning of the search journey for people.

Quality, safety, and groundedness are some of the key criteria that will be evaluated by the company.

These are essentially unsolved problems and until they are more tractable, it will be cautious about applying this technology. There is a big gap between what we can build as a research prototype and what can actually be deployed as a product.

It should be taken with some skepticism. Users who immediately found problems were the first to benefit from the latest artificial intelligence-powered assistive writing feature. It's clear that the technology is badly wanted by the company and it's dedicated to working out its problems one test app at a time.