Microsoft fixes harebrained Y2K22 Exchange bug that disrupted email worldwide



The Exchange server bug that shut down on-premises mail delivery around the world just as clocks were chiming in the new year was fixed by Microsoft.

The Y2K22 bug was caused by a date check failure in Exchange server that made it impossible for the server to accommodate the year 2022, prompting some to call it the mass disruption. The mail programs have dates and times in signed integers. The first two numbers of an update version are used by Microsoft. Everything worked well if the year was earlier.

What about Microsoft?

On-premises server crashed when Microsoft released version 2201010001 on New Year's Eve. Messages got stuck in the queue. The admins around the world were left trying to fix things instead of ringing in the new year with friends and family. They only had to go on with the two cryptic log messages.

Advertisement

Log name is application
The source is the fips.
Logged on 1/12/22 at 1:03:42 AM.
The event is called: 5300.
There is a level of error.
The computer is server1.contoso.com.
The FIP-FS "Microsoft" Scan Engine failed to load. The error code is 0x80004005. Can't convert "2201010001" to long.

Log name is application
The source is the fips.
Logged on January 1st, 2017: 11:47:16 AM.
The event is called 1106
There is a level of error.
The computer is server1.contoso.com.
The FIP-FS Scan Process failed to start. The error is 0x80004005. There is an unspecified error.

One admin wrote in this thread that Microsoft was in the absolute hell. On New Year's Eve? You guys save my life when I check out on Reddit.

Microsoft released a fix the next day. An automated script or a manual solution is what it comes in, as reported by some admins. The fixes must be performed on every Exchange server inside the affected organization. The script can run on multiple server. The automated script might take some time to run, and the software maker urged admins to be patient.

Exchange checked the version of the FIP-FS, a scanning engine that is part of Exchange anti-malware protections, and performed a date and time check. Mail delivery was halted once FIP-FS versions began with the number 22. The fix stops the Microsoft Filtering Management and Microsoft Exchange Transport services.

Things were getting back to normal by Monday. The buggy date storage may have been introduced when Exchange server 2016 was under development.