How Microsoft built its smart Surface camera

If you buy something from a link, it may earn you a commission. Our ethics statement can be found here.

Steven Bathiche, who oversees all hardware innovation for Microsoft devices, explained in an interview with The Verge that they knew they were going to make their cameras smart from day one of Surface Hub 2. The $799.99 Surface Hub 2 Smart Camera is the first conference room camera to offer automatic reframing without the warping and distortions you would typically see on other cameras.

It can detect faces and bodies in order to make sure that everyone is visible in a room. The Surface Hub 2 Smart Camera has a 136-degree field of view, which allows it to see the entire conference room.

Microsoft had always planned to upgrade its Surface Hub 2 camera before the Pandemic put hybrid meetings into focus, that's why it's modular and can be detached from the top of the 55 or 85-inch displays. We didn't know exactly how, but we knew that was going to change and needed to change with people's needs, the evolution of the conference room, and even how our culture will essentially adapt towards meetings, says Bathiche.

The Surface Hub 2 presented challenges for capturing everyone in a meeting room with a traditional camera.

The Surface Hub 2 Smart Camera is powered by Microsoft's own optics, artificial intelligence model, and edge computer, created by Bathiche and his team.

That means that all of the work is done on the camera itself, and never sent to the cloud or the Surface Hub 2 to process it. The camera runs the model, processes the data, and makes a decision to crop the image. The Smart Camera will use tilt compensation to adjust the image for the camera position and create more natural eye contact instead of using automatic framing. The fish eye effect can be removed from wide-angle lens so people don't look stretched inside meeting rooms.

Bathiche explains that they designed an 11 element, completely glass lens with super sharp focus and close to the refraction limits. The camera can look behind itself because the lens is a 184-degree field of view.

Image: Microsoft

The Surface Smart Camera is powered by the same models that power all of this hardware. Microsoft started this project before the Pandemic, but it had to train its models in order to fill meeting rooms with people.

Bathiche says that they went to New Zealand because there were no COVID-19 cases there. Our data set is huge.

Microsoft trained its artificial intelligence model on faces and bodies to make sure it is fully inclusive and will detect people that aren't always facing the camera. It used synthetic people and faces to improve its diversity.

The Smart Camera isn't trained to detect animals. If an office cat or dog steps into view, that should mean it won't try to change the meeting's agenda. The project includes a committee and a set of tools to ensure the fairness and inclusiveness of the project.

Image: Microsoft

The data set we have is amazing in terms of the disparity between the different groups: race, gender, skin tone, hair styles, etc.

Bathiche says Microsoft has changed the Smart Camera's autoframing capabilities to make sure it doesn't jumpy or slow to miss content.

You might be wondering if you could use the camera on a regular PC, but it's not that simple. The Surface Hub 2 Smart Camera isn't designed to be a regular webcam. Bathiche explains that the elevation, the angles, and the artificial intelligence was designed for multiple people.

This isn't the first time Microsoft has focused on improving its cameras. The Surface Pro X has an eye contact feature that makes it look like you are always making eye contact, no matter what you are looking at. Bathiche says that the faces used in the camera are the same as those used in eye contact.

Microsoft clearly designed this Smart Camera for the Surface Hub 2, but with persistent rumors around Surface- branded webcams, it's possible we'll see a powerfulWebcam from Microsoft one day. Bathiche says that they will continue evolving their devices as they see in the Surface Pro X.