Batch jobs in PowerShell

| 1 Comment
Say you want to execute a series of commands on a bunch of Windows machines and your AV considers 'psexec' to be malware. What are you to do? Remote PowerShell can be used. What's more, it can be done in parallel. Say you've got to run a malware cleanup script on 1200 computer-lab machines RIGHT NOW, you can't just put it as a scheduled task in a GPO and wait for the GPOs to apply.

$JobTracker=new-object system.collections.hashtable
$Finished=new-object system.collections.hashtable
$MachineArray=$Args[0]

foreach ($Mcn in $MachineArray) {
    $JobTracker["$Mcn"]=invoke-command -AsJob -ComputerName "$Mcn" -ScriptBlock {
        $ProcList=get-wmiobject win32_Process
        foreach ($Proc in $ProcList) {
           if ($Proc.Name -eq "evil.exe") {$Proc.Terminate}
        }
        remove-item c:\windows\system32\evil.exe
    }
}
$JobCount=$JobTracker.Count
$Machines=$MachineArray.Keys
$Completed=0
while ($Completed -lt $JobCount) {
    foreach ($Mcn in $Machines) {
        if (($JobTracker["$Mcn"].State -eq "Completed") -and (-not $Finished.Contains("$Mcn"))) {
            remove-job -id $JobTracker["$Mcn"].ID
            $Completed++
            $Finished["$Mcn"]="Completed"
        } elseif (($JobTracker["$Mcn"].State -eq "Failed") -and (-not $Finished.Contains("$Mcn"))) {
            $FailReason=$JobTracker["$Mcn"].JobStateInfo.Reason
            write-warning "$Mcn failed: $FailReason"
            remove-job -i $JobTracker["$Mcn"].ID
            $Completed++
            $Finished["$Mcn"]="Failed"
        }
    }
    sleep 1
}

Heck, psexec can't be run in parallel like this, so this is even better!

What this does:

$JobTracker=new-object system.collections.hashtable
$Finished=new-object system.collections.hashtable
$MachineArray=$Args[0]

foreach ($Mcn in $MachineArray) {
    $JobTracker["$Mcn"]=invoke-command -AsJob -ComputerName "$Mcn" -ScriptBlock {
        $ProcList=get-wmiobject win32_Process
        foreach ($Proc in $ProcList) {
            if ($Proc.Name -eq "evil.exe") {$Proc.Terminate}
        }
        remove-item c:\windows\system32\evil.exe
    }
}

This sets up a trio of variables. Two for tracking, and the third to handle the input. $MachineArray comes in as an argument to the script, presumably out of some other script that assembles an array of lab-machines that need to be hit.

The foreach loop iterates over every item in the passed-in array.

Now for the interesting bit. The invoke-command part uses the "-AsJob" parameter to cause the execution of this job to happen as a PowerShell job. This job will execute in the background, which allows the script to continue. The script specified in the ScriptBlock will execute on the remote side while the script moves on on the local machine. Since this is invoked in a loop, in short order we should have as many background jobs as we have entries in $MachineArray.

The 'invoke-command' statement will return an object of type powershell-job, which we then assign to the hashtable-value.

When all is done, $JobTracker should contain a hashlist of job-objects tracking the execution progress on the remote machines.

$JobCount=$JobTracker.Count
$Machines=$MachineArray.Keys
$Completed=0
while ($Completed -lt $JobCount) {
    foreach ($Mcn in $Machines) {
        if (($JobTracker["$Mcn"].State -eq "Completed") -and (-not $Finished.Contains("$Mcn"))) {
            remove-job -id $JobTracker["$Mcn"].ID
            $Completed++
            $Finished["$Mcn"]="Completed"
        } elseif (($JobTracker["$Mcn"].State -eq "Failed") -and (-not $Finished.Contains("$Mcn"))) {
            $FailReason=$JobTracker["$Mcn"].JobStateInfo.Reason
            write-warning "$Mcn failed: $FailReason"
            remove-job -i $JobTracker["$Mcn"].ID
            $Completed++
            $Finished["$Mcn"]="Failed"
        }
    }
    sleep 1
}
This bit cleans up after the above loop. First it grabs the number of jobs in the tracker, then dumps a list of the hashkeys, then zeroes a completed-tracker variable.

The while loop will run until the number of completed jobs equals the number of jobs that were created. For extra credit you can build in a timeout loop, but I didn't do that here; an exercise for the reader.

The inner foreach loop iterates over all the hash-keys. If the job-state for that job is completed, and the job hasn't already been checked into the $Finished hashtable, then it will remove the job from the queue, increment the completed count, and add it to the $Finished hashtable. If the status is failed, it does much the same thing but dumps a warning that a certain machine failed.

Then it sleeps for one second before doing it all again.

If you want to get output from the running remote code, the `receive-job` cmd-let will extract such and send it to a variable. Each line of output will be a new Arraylist node of type system.string.

In order for this to work, the remote machines need to have WinRM up and running and preferably in the same domain as the station the above code is run on (though not required, it just makes things easier). WinRM isn't on by default, but it most certainly can be turned in via Group Policy. For workstations like computer lab-stations, ensuring such remote-management ports are open is a very good idea anyway.

This example shows how PowerShell jobs can be used in conjunction with remote-powershell to do awesome things. The 'invoke-command' cmd-let has some very interesting possibilities in addition to what is presented here. Perhaps I'll get into that later on.

1 Comment

Wow, this is awesome. I can definitely see some scenarios where this will be useful.