Fixing Windows Updates

Seriously Microsoft... wtf!

I have no idea what travesty I committed in a past life to deserve the pain I have recently experienced with Windows Updates1. However, I do know that my punishment is to know way more about the Windows Update module than any sane IT Manager has the right to know.

By way of atonement for my past sins, I offer the following in the hope that it helps some other poor bugger out there who might be in a similar situation.

What happened?

Our IT Estate has roughly 40 workstations. We use Shavlik to push out Windows patches. I noticed one month that Shavlik was pushing out patches but in many cases could not confirm that they had been installed properly. After a bit of digging around, we decided to have a go at patching those machines manually. That is when things went south.

The machines in question refused to push out updates using Windows Update (seriously… Microsoft… WTF!)

We Googled every single error code and tried every single internet “fix” that we could find. Nothing was working. It was time to admit defeat and do the unthinkable… the last resort… we had no choice… we had to get on the phone to Microsoft support.

What Microsoft found

On the first pass, even Microsoft gave up and recommended an OS re-install. Of course, we all laughed heartily at that suggestion (seriously… Microsoft… WTF!)

After the required amount of pushback, Microsoft relented and escalated internally.

For several more days, Microsoft worked remotely on one of our affected PCs and discovered that a number of patches had been “Staged”, but never installed. They were stuck in this “Staged” state and this was preventing all other patches from installing.

How did we get into this mess

This is our working theory. One word. McAfee.

It turns out that one of our EPO policies was actively preventing some Windows Updates from installing. If you have landed on this page because you have the same problem, you have to disable the following option:

How we got out of this mess

This is where it gets “interesting”. The solution on paper sounds simple:

  • Reset Windows Update:
    • Stop all Windows Update related services
    • Rename the SoftwareDistribution folder
    • Rename the catroot folder
  • Extract a list of all installed/staged packages to a text file using dsim
  • Restart Windows Update services
  • Parse the file containing the installed packages, extract any module name in the “Staged” state, and use dsim to remove those packages.

The problem is that the list of Staged packages was different on every single machine - not the sort of task you want to be doing by hand on that many machines.

The good news is that is is scriptable, but has to be done in phases.

Phase One

Run the following Powershell script on every affected machine (it can be run in a remote Powershell session without disturbing the target machine user):

$res = "\\<yourservershare>\WindowsUpdates\StagedResult." + $env:COMPUTERNAME + ".txt"

"Stopping BACKGROUND INTELLIGENT TRANSFER SERVICE..."
net stop BITS
"Stopping CRYPTOGRAPHIC SERVICES..."
net stop Cryptsvc
"Stopping WINDOWS MODULES INSTALLER..."
net stop trustedinstaller
"Stopping WINDOWS UPDATE..."
net stop wuauserv
 
"Renaming Windows Update Folders..."
ren $env:systemroot\SoftwareDistribution SoftwareDistribution.old
ren $env:WINDIR\system32\catroot2 catroot2.old
 
"Getting packages..."
dism /online /get-packages | out-file -encoding ascii $res
 
"Starting BACKGROUND INTELLIGENT TRANSFER SERVICE..."
net start BITS
"Starting CRYPTOGRAPHIC SERVICES..."
net start Cryptsvc
"Starting WINDOWS MODULES INSTALLER..."
net start trustedinstaller
"Starting WINDOWS UPDATE..."
net start wuauserv
"Complete."

Note that this script outputs the installed packages for that machine to a file called:

StagedResult.<COMPUTERNAME>.txt

In our case, we output this file to a shared folder on a server so that we could collate all the files in preparation for phase two.

Here is an example of the dsim output file:

Deployment Image Servicing and Management tool
Version: 6.1.7600.16385
 
Image Version: 6.1.7601.18489
 
Packages listing:
 
Package Identity : Package_for_KB272945231bf3856ad364e35amd646.1.1.0
State : Staged
Release Type : Security Update
Install Time : 17/02/2017 23:04

Phase Two

I have a bit of background in Linux system admin, so I chose to use Cygwin bash scripts for this phase. The principle is pretty simple, so if you are not comfortable with bash, grep or sed, feel free to script in a language of your choice.

The following bash script: stagedupdateremoval.sh takes a generated package list file from phase one as a parameter, then uses grep to extract the list of “Staged” package names, and pipes that output to sed to create a Powershell script that can be run on the source machine to remove all of the Staged packages:

1
2
3
4
#!/bin/bash
cat $1 | grep -Pzo '(?<=^Package Identity \: )(.*)(?=\n^State : Staged$)' | \
    sed 's/^/dism \/online \/remove-package \/packagename:/' > \
    $1.remove.ps1

As we had collated all of the files from the affected machines in a shared folder, I used the following script to process them all in the one go:

1
2
3
4
5
6
7
8
#!/bin/bash
for f in ./StagedResult*.txt; do
 
    if [ -e "$f" ]; then
        ./stagedupdateremoval.sh "$f"
    fi
 
done

This is an example of the output script for a single machine:

dism /online /remove-package /packagename:Package_for_KB272945231bf3856ad364e35amd646.1.1.0
dism /online /remove-package /packagename:Package_for_KB300374331bf3856ad364e35amd646.1.1.1
dism /online /remove-package /packagename:Package_for_KB302038731bf3856ad364e35amd646.1.1.2

The output filename of the new script is: StagedResult.<COMPUTERNAME>.txt.removal.ps1

Phase Three

We now have to run the targeted output scripts from Phase Two on each of the affected machines. Again, this can be done using a remote Powershell session without disturbing the machine user.

When this phase has been completed, Windows Updates will work again.


  1. The post was composed on a nice, clean, virus free (and Anti Virus free) Mac. A world where OS updates magically work every time. As if by magic. 

View comments.

more ...


Cameron to Ban End-to-End Encryption?

It seems that every time we have a high profile terrorist attack, politicians, with alarming alacrity, seize on the opportunity to demand extra snooping powers. This used to be New Labour’s domain, but now it’s the turn of David Cameron. Our esteemed PM wants to allow our security services to view the content of encrypted messaging services:

Mr Cameron told ITV News: “I think we cannot allow modern forms of communication to be exempt from the ability, in extremis, with a warrant signed by the Home Secretary, to be exempt from being listened to.”

(That is very nearly a coherent sentence)

On the same day, The Independent spun similar comments made by Cameron into the following article:

WhatsApp and iMessage could be banned under new surveillance plans

David Cameron could block WhatsApp and Snapchat if he wins the next election, as part of his plans for new surveillance powers announced in the wake of the shootings in Paris.

The Prime Minister said today that he would stop the use of methods of communication that cannot be read by the security services even if they have a warrant. But that could include popular chat and social apps that encrypt their data, such as WhatsApp.

Apple’s iMessage and FaceTime also encrypt their data, and could fall under the ban along with other encrypted chat apps like Telegram.

The Guardian was kind enough to supply the original quote:

In extremis, it has been possible to read someone’s letter, to listen to someone’s call, to mobile communications,” Cameron said. “The question remains: are we going to allow a means of communications where it simply is not possible to do that? My answer to that question is: no, we must not.”

David, you are not the Prime Minister of North Korea. Policies like this will only lose you votes.

There are dozens of encrypted chat clients out there. Do you really want the security service spending all of their time play Whack-a-Mole with the online development community? The government and security services will be chasing their own tails for eternity trying to solve this problem. It is stupid idea, unenforcible, futile, and counter-productive.

Any half baked measure to introduce back door access or lower encryption standards will be an open invitation to hackers. It might give the government what they want, but we will all be worse off for it.

Also, as Guido points out:

However, Dave can take heart that he’s not in this fight alone, the other countries where there are known domestic controls on the use of encryption are Russia, China, Mongolia, Vietnam, Pakistan, Iran, Kazakhstan, Belarus, Ukraine, Moldova, Israel, Tunisia and Morocco.

That’s one hell of a club Dave’s trying to sign us up to…

View comments.

more ...