Tuesday 3 March 2015

20 epic Microsoft Windows Automatic Update meltdowns

20 Windows Automatic Updates from hell

Fifteen years ago, Microsoft introduced automatic updating to the unwashed Windows masses. Fifteen years later, it’s hard to find a Windows user who hasn’t bumped into at least one problem with a Windows update or knows someone who has. That’s a billion and a half people.

From inscrutable driver problems to bricked machines and everywhere in between, Automatic Update is a poster child in “what’s wrong with Windows” circles -- rightfully so.

Hope springs eternal that Windows 10 will finally bring relief, but much depends on the determination and deep pockets of Those in Charge. One thing’s for sure: In the land of Win10 milk and honey, customers don’t want to be treated like cannon fodder.

Here’s my take on the 20 worst Microsoft Automatic Update patches of all time. Based on either the amount of pain inflicted or the number of people afflicted -- or both -- they deserve their notoriety.

Those who cannot remember the past are condemned to repeat it.

November 2001: The UPnP patch debacle
Microsoft introduced Windows automatic updating as one of the great new benefits in Windows Me, around September 2000. A year later, we were treated to a Keystone Kops episode in the guise of MS01-059 -- ostensibly, a patch to the Windows Universal Plug 'n Play subsystem that prevented a buffer overrun. In fact, I think it was the first (though hardly the last) security bulletin conceived and scripted by Comedy Central.

Microsoft patched, repatched, and re-repatched the patch. The FBI's National Infrastructure Protection Center followed along like a kid cleaning up after his dog: NIPC issued a warning about the security hole, an update, another update, and ultimately an advisory that Microsoft had finally solved the problem.

April 2004: Windows 2000 bricked
In April 2004, Microsoft sent a slew of patches down the automatic update chute, one of which (MS04-014) locked up a sizable percentage of all Windows 2000 machines. That patch was supposed to fix a hole in the Jet Database Engine.

Knowledge Base article 841382 tells the tale:

[Y]ou may experience any one of the following symptoms:

• Your computer appears to stop responding at startup.
• You cannot log on to Windows.
• Your CPU usage for the System process approaches 100 percent.

The company sure plugged that one.

April 2006: The pretax predicament
On Black Tuesday in April 2006, Microsoft released MS06-015, a patch for Windows Explorer. By the weekend, most Windows users with Automatic Update turned on got it -- right across the face. The weekend before tax day, many Windows customers found they couldn't navigate to the Documents or Pictures folder, couldn't open or save files, had to type http:// into Internet Explorer to keep it from freezing, and much more.

We ultimately discovered that the patch messed up any machine with an older HP scanner program or an older Nvidia video driver.

Microsoft's ultimate workaround (KB 918165) included a manual fix procedure that any computer-science grad would be proud to explain, if they can figure it out.

April 2006: Windows Genuine Spyware -- er, Advantage
Microsoft uses the Automatic Update channel (and permissions) to install Black Tuesday security patches, as well as non-security-related patches. My favorite example came in late April 2006, when somebody at Microsoft decided Automatic Update would be a great way to install the new Windows Genuine Advantage, uh, feature.

That version of WGA (in addition to throwing off bogus "not genuine" messages) installed a component called WGA Notification that phoned home -- sent information to Microsoft about the current computer -- with absolutely no notification to or approval from the customer. Lawsuits ensued. I called it Windows Genuine Spyware.

August 2006: The IE patch that created a new buffer overflow hole in IE
Let's hear it for MS06-042, the cumulative security update for Internet Explorer that not only caused IE to crash, but also introduced a security hole of its very own.

In late August, Microsoft owned up to problems in KB 923762: the part where IE6 crashes while looking at a valid website. Solution? Install the latest, greatest version of MS06-042.

Then in September, Microsoft had to reissue the patch again to "address a vulnerability documented in the Vulnerability Details section as Long URL Buffer Overflow -- CVE-2006-3873."

KB 918899 lists 15 separately identified problems with this patch, from crashes to freezes to inexplicable behavior.

December 2007: Internet Explorer crashes on sites with lots of graphics -- like msn.com
Yet another cumulative security update for IE, MS07-069 patched IE so well that many WinXP SP2 customers reported IE6 freezes on sites with many graphics. If you had automatic updates turned on and were running plain-vanilla WinXP SP2, after the patch was installed, you couldn't let IE go to the default IE6 home page, msn.com.

If you installed the patch for Internet Explorer 7, your (third-party) firewall might not have recognized IE. As a result, it may have kept IE from going out to the Internet. IE produced the marvelously informative error message "Webpage cannot be displayed."

It took weeks, but Microsoft finally acknowledged the problem and posted a downloadable fix program in KB 946627.

April 2008: Quicken suddenly stops working
Nobody seems to know why, but Microsoft suddenly released the .Net 2.0 Service Pack 1 on a Thursday, one week before tax time, via the automatic update chute. The patch itself had been available as an optional, manual download for months, but somebody flipped the auto update switch.

Within minutes, Quicken users were complaining. QuickBooks got hit, as did TurboTax and software from Commerce Clearing House.

How bad was it? If you were bit, uninstalling, then reinstalling QuickBooks didn't solve the problem. You had to uninstall, then reinstall .Net 2.0 -- if you could get it to uninstall.

All through 2009, 2010, 2011: Bad .Net patches
Over and over again, we saw botched .Net patches -- some refused to install, others left .Net dead, others clobbered programs that relied on .Net. It started in January 2009 with a patch that claimed to push .Net Framework 3.5 to Service Pack 1, but didn't.

Another patch, in March 2009, also identified as .Net Framework 3.5 SP1, installed .Net Framework 2.0 SP2 and .Net Framework 3.0 SP2 as well. It was an unholy mess that had us going in circles for months.

We saw many more .Net patching problems in 2010 and 2011, all compliments of Automatic Update.

March 2009: The XP AutoRun blocker that didn’t
It took Microsoft forever to post a patch that disabled AutoRun in Windows XP. AutoRun, indicted as the culprit behind mass Conficker infections, deserved to die, but Microsoft's first and second attempts to talk people through the disabling procedure didn't work.

The final solution is so incredibly convoluted that pages of KB 967715 are devoted to explaining the interactions of all the patches, both delivered via automatic update and manually downloaded. It's complicated. Bottom line: If you installed only one automatic update, you might've thought that you fixed AutoRun, but you didn't. It took several patches over several months to finally get it right.

December 2010: Patch brings down Task Scheduler
MS10-092 was an innocuous patch, designed to plug a hole in Windows Task Scheduler.

But shortly after people started installing it, they saw messages saying, "The task image is corrupt or has been tampered with." In some cases, the task was killed. In other cases, the machine froze. Simply uninstalling the patch didn't solve the problem -- great prelude to the holiday season.

KB 2305420 has pages and pages of manual workarounds.

January 2011: A reliability update that wasn’t
On January's Black Tuesday, Microsoft pushed a nonsecurity patch into the Automatic Update black hole. Known as KB 2454826, Microsoft claimed it was a "performance and functionality update." Details about the patch at the time were sketchy, but the 0x7F blue screen crashes weren't.

Microsoft's advice: Manually uninstall the patch. That's your reward for turning automatic updates on, bucko.

It wasn't until the next month that we discovered the real reason why Microsoft pushed this nonsecurity patch out the Black Tuesday chute: It's a prerequisite for installing the Internet Explorer 9 Release Candidate, which Microsoft was flaunting at the time.

April 2012: TurboTax won’t print
Just before tax day -- tell me if this is starting to sound familiar -- Microsoft released MS12-025, yet another botched .Net patch.

(For the sake of brevity, I didn't bother to list separately MS10-070, MS11-039, MS11-044, MS11-066, or MS11-069, all of which were incredibly botched .Net patches.)

This particular patch kept TurboTax from printing tax forms ... on tax day. #epicfail

May 8, 2012: Duqu patch installation failure
A massive patch known as MS12-034 (with many associated KB numbers) left some Windows customers who used Automatic Update wondering what had gone wrong. Some found that the installer failed with an Error Code 0x8007F0F4. When they checked the KB 2686509 support article, they were instructed to delete a keyboard log file. Many people couldn't find the file.

The instructions in KB 2686509 go on for pages, explaining how to modify and move keyboard layout files -- in response to a known, anticipated error thrown off by the installer. Microsoft finally got around to creating a Fix it that made the patching easier. But lots of unsuspecting Windows consumers wasted hours trying to make heads from tails out of this automatically updated disaster.

February 2013: Blue screens on Internet Explorer 9
Once again, Microsoft threw a bunch of machines into a tizzy by releasing a nonsecurity patch on the fourth Tuesday of the month -- and sending it down the Automatic Update chute.

This time, KB 2670838, a "Platform Update for Windows 7 x64-Edition" messed with IE9 so badly that it would put a black bar on the right side of the screen. Click on the bar, and your PC died with a blue screen.

Fortunately, the fix is to uninstall the bad patch.

April 2013: More blue screens
This time, MS13-036/KB 2823324 -- a Black Tuesday security patch designed to replace a kernel-mode driver -- triggered all sorts of bogus warnings and frequently froze machines. Primary suspects include a common IE add-in from Brazil and Kaspersky Antivirus.

Microsoft pulled the patch, then issued a replacement patch: "Microsoft has released security update 2840149. This security update resolves the issue that was introduced by security update 2823324."

August 2013: The biggest, baddest bungled batch ever
Within 48 hours of the month's automatic update, Microsoft publicly admitted six Windows patches were bad and pulled four of them, all associated with MS13-066 and Active Directory Federation Services.

As far as I can tell, that's a record. It's not only a record for bad patches. It's a record for how quickly Microsoft acknowledged, documented, and in some cases, pulled the offending patches. We’ve seen bad Patch Tuesdays since, but this one stands out, in both good and bad ways.

November 2013: Outlook 2013 gets special treatment
One of the patches in the November 2013 set caused no end of problems with Outlook 2013 -- Outlook hangs when trying to sync IMAP accounts; Out of Office replies on Exchange Server triggered "currently unavailable" messages; Free/Busy data for the Outlook Calendar didn't download; S/MIME certificates wouldn't validate; and more.

Unfortunately a second patch released in the November crop made it impossible to fix all of those Outlook 2013 problems by simply uninstalling the bad patch. In the end, users in the know discovered they could resuscitate Outlook 2013 by uninstalling both patches, then deleting and rebuilding the Outlook profile.

Microsoft, which did so well in August -- “well” in the sense it cleaned up quickly -- really blew it in November.

May 2014: Windows 8.1 Update won’t install, Microsoft backs off its deadline
In a scene straight out of Dante’s "Inferno," Microsoft cracked the whip and told all Windows 8.1 users that they had to install the KB 2919355 update (so-called Win8.1 Update 1) by May 13, or they wouldn’t get any new patches. Predictably and with much wailing, a vocal subset of Windows 8.1 customers discovered Update 1 wouldn’t install, for love nor money -- or anything resembling either or both.

Quite dramatically (tell me if you can visualize the seventh ring), Microsoft finally relented on May 12 and said it would allow the tardy minority to receive updates -- but only this one last time.

(I think it’s poetic justice that Win 8.1 Update 2 stalled, then fizzled completely, ultimately leading to a re-release that didn’t do much.)

August 2014: Blue screens all around, Microsoft recommends you manually yank the patches
Four patches in August were credited with driving blue screens on Windows 7, 8, 8.1, and RT machines. Microsoft pulled the patches on Sunday, then issued a very unusual notice, buried deep in a Knowledge Base article: Even if you weren’t having any problem with the patches, you were supposed to manually uninstall them.

Apparently the patches continue to cause problems, even after they were installed, in certain unusual circumstances.

It’s a bit much to tell your Aunt Mabel to manually uninstall a handful of patches, based on a warning in a KB article, but there you have it.

December 2014: Roughly a quarter of all the patches this month generated problems
With only a few of the bad patches fixed before the end of the year, December 2014 represents the worst combination of bad patches and lackadaisical responses I can recall. In response to the unprecedented number of screw-ups, Microsoft pulled a few patches, released two Fix its, created a Silver Bullet patch that specifically killed one of the bad patches, and wrote up numerous manual work-arounds.

Even at this late date, more than two months later, the problems brought down on Excel macro programmers haven’t been fixed.

Would you say Automatic Update is getting better?

Come out of the Automatic Update cave and into the light
That's by no means an exhaustive list. Some problems are inevitable when you're dealing with a Windows hardware and software gene pool that looks like the La Brea Tar Pit, but I think you can draw three important conclusions:

First, patching Windows is hard.

Second, Microsoft needs to do a better job of tracking and reporting on problems as they appear.

Third, for Pete's sake, set Automatic Update to Notify but Don't Download on any machine controlled by a reasonably savvy Windows jockey.

If somebody tells you differently, point them to this list. If they're still convinced Automatic Update is the way to go, ask them to refrain from dragging their knuckles on the floor.



Best Microsoft MCTS Certification, Microsoft MCITP Training at certkingdom.com

No comments:

Post a Comment