Runspaces made simple

PowerShell is great at simplifying and automating your daily tasks. And it’s only natural that after you have automated or scripted most of your simple tasks, you start creating bigger scripts. If you work in a large environment, running for instance a script that uses WMI to query computers for data can take a long time to complete. The solution? Multi-threading of course. One way of doing this is using the built-in ‘Jobs’ cmdlets, but what this post is about is Runspaces.

Runspaces lets you create separate PowerShell sessions running in separate runspaces (hence the name), within the same process. This means that you won’t see any difference when looking at Task Manager (except CPU and memory will probably go up, depending on the number of runspaces you use).

Working with runspaces have for some been a daunting task, as it means you need to work with .NET directly. I have created a set of wrappers for working with runspaces that simplifies this and lets you work with functions that behave as common PowerShell cmdlets.

The following functions are included:

  • New-RunspacePool
  • New-RunspaceJob
  • Receive-RunspaceJob
  • Show-RunspaceJob
  • Clear-RunspaceJobs

New-RunspacePool

This function will create a new runspace pool. A runspace pool can be thought of as the object that is controlling the behaviour of you runspaces, as well as tying them all together. Setting up the runspace pool lets you control how many concurrent threads you want to run, as well as control snapins, modules, functions and variables that you want each runspace to have access to when spawned. The New-RunspacePool function returns a RunspacePool object that you will need to capture in a variable, as this is used together with the next function: New-RunspaceJob.

New-RunspaceJob

This is the function that takes the supplied code (in the form of a script block), creates a new runspace and injects the code together with any parameters (in the form of a hashtable) that you want the code to use. Each new runspace job is connected to the runspace pool, and the pool takes care of how many runspaces are current running at the same time, and will automatically start a new one when a new opening in the queue comes up. All the runspace jobs are saved in the global ‘runspaces’ variable. You can name your jobs as well, so if you are running several different jobs at the same time, it’s easy to differentiate between them when looking in the ‘runspaces’ array.

Receive-RunspaceJob

Receive-RunspaceJob will check if the job is finished and return the result if it is. This function have a ‘Wait’ parameter that you can use (together with a ‘Timeout’ parameter) to let it run until all jobs are finished (or the timeout value is exceeded). It also supports getting only runspaces belonging to a certain job, or even by the unique ID of each job.

Show-RunspaceJob

Show-RunspaceJob will list out all jobs currently in the global runspaces array. It only have one parameter, ‘JobName’, that lets you list only runspaces belonging to a particular job.

Clear-Runspaces

Lastly we have the Clear-Runspaces function. The only thing this one does, is remove everything from the global runspaces variable (it is in fact deleting the variable). Hopefully you would never need this, but working with WMI I know from experience that sometimes a WMI session can hang, and it’s nice to be able to completely remove any jobs left.

Creating these functions, I had a goal of creating something that I could use both in scripts, but also directly from the console, and I hope I have succeeded. To get you going I have also created a basic template that you can use when running code against an array of computers. It is by no means a complete script, and should only be used as a starting-point you can work on, but it should be enough to give you a picture of how to use the different functions.

# Basic Script Template for Runspace Jobs against an array of computers
# This is just an example to show the basic layout of a script using the Runspace functions
# If you are creating a script to be run in production, please add logging and error handling
# Also consider writing it to support parameters for input instead of hardcoding it like in this example
#region verify that runspace functions are present
$missingFunctions = $false
$functions = (
'New-RunspacePool',
'New-RunspaceJob',
'Show-RunspaceJob',
'Receive-RunspaceJob'
)
foreach ($function in $functions) {
try {
$thisFunction = Get-Item LiteralPath "function:$function" ErrorAction Stop
}
catch {
$missingFunctions = $true
Write-Warning "$function not found"
}
}
if ($missingFunctions) {
break
}
#endregion
# computers to run against
# re-write to read from script parameter, file, Active Directory etc as needed
$computers = (
'computer01',
'computer02',
'computer03',
'computer04'
)
# define timeout for Receive-RunspaceJob (in seconds)
$timeout = 30
# code to run in each runspace
$code = {
Param([string]$ComputerName)
# code goes here
}
# create new runspace pool
$thisRunspacePool = New-RunspacePool
# define results array
$results = @()
# iterate through each computer and create new runspace jobs
# also run Receive-RunspaceJob to collect any already finished jobs
foreach ($computer in $computers) {
New-RunspaceJob RunspacePool $thisRunspacePool ScriptBlock $code Parameters @{ComputerName = $computer}
$results += Receive-RunspaceJob
}
# if any jobs left, wait until all jobs are finished, or timeout is reached
if ([bool](Show-RunspaceJob)) {
$results += Receive-RunspaceJob Wait TimeOut $timeout
}
Write-Output $results

As always, if you spot any bugs, or have any suggestions for future improvements, or just need some input on how to use these functions, don’t hesitate to contact me!

function New-RunspacePool{
<#
.SYNOPSIS
Create a new runspace pool
.DESCRIPTION
This function creates a new runspace pool. This is needed to be able to run code multi-threaded.
.EXAMPLE
$pool = New-RunspacePool
Description
———–
Create a new runspace pool with default settings, and store it in the pool variable.
.EXAMPLE
$pool = New-RunspacePool -Snapins 'vmware.vimautomation.core'
Description
———–
Create a new runspace pool with the VMWare PowerCli snapin added, and store it in the pool variable.
.NOTES
Name: New-RunspacePool
Author: Øyvind Kallstad
Date: 10.02.2014
Version: 1.0
#>
[CmdletBinding()]
param(
# The minimun number of concurrent threads to be handled by the runspace pool. The default is 1.
[Parameter(HelpMessage='Minimum number of concurrent threads')]
[ValidateRange(1,65535)]
[int32]$minRunspaces = 1,
# The maximum number of concurrent threads to be handled by the runspace pool. The default is 15.
[Parameter(HelpMessage='Maximum number of concurrent threads')]
[ValidateRange(1,65535)]
[int32]$maxRunspaces = 15,
# Using this switch will set the apartment state to MTA.
[Parameter()]
[switch]$MTA,
# Array of snapins to be added to the initial session state of the runspace object.
[Parameter(HelpMessage='Array of SnapIns you want available for the runspace pool')]
[string[]]$Snapins,
# Array of modules to be added to the initial session state of the runspace object.
[Parameter(HelpMessage='Array of Modules you want available for the runspace pool')]
[string[]]$Modules,
# Array of functions to be added to the initial session state of the runspace object.
[Parameter(HelpMessage='Array of Functions that you want available for the runspace pool')]
[string[]]$Functions,
# Array of variables to be added to the initial session state of the runspace object.
[Parameter(HelpMessage='Array of Variables you want available for the runspace pool')]
[string[]]$Variables
)
# if global runspace array is not present, create it
if(-not $global:runspaces){
$global:runspaces = New-Object System.Collections.ArrayList
}
# if global runspace counter is not present, create it
if(-not $global:runspaceCounter){
$global:runspaceCounter = 0
}
# create the initial session state
$iss = [System.Management.Automation.Runspaces.InitialSessionState]::CreateDefault()
# add any snapins to the session state object
if($Snapins){
foreach ($snapName in $Snapins){
try{
$iss.ImportPSSnapIn($snapName,[ref]'') | Out-Null
Write-Verbose "Imported $snapName to Initial Session State"
}
catch{
Write-Warning $_.Exception.Message
}
}
}
# add any modules to the session state object
if($Modules){
foreach($module in $Modules){
try{
$iss.ImportPSModule($module) | Out-Null
Write-Verbose "Imported $module to Initial Session State"
}
catch{
Write-Warning $_.Exception.Message
}
}
}
# add any functions to the session state object
if($Functions){
foreach($func in $Functions){
try{
$thisFunction = Get-Item LiteralPath "function:$func"
[String]$functionName = $thisFunction.Name
[ScriptBlock]$functionCode = $thisFunction.ScriptBlock
$iss.Commands.Add((New-Object TypeName System.Management.Automation.Runspaces.SessionStateFunctionEntry ArgumentList $functionName,$functionCode))
Write-Verbose "Imported $func to Initial Session State"
Remove-Variable thisFunction, functionName, functionCode
}
catch{
Write-Warning $_.Exception.Message
}
}
}
# add any variables to the session state object
if($Variables){
foreach($var in $Variables){
try{
$thisVariable = Get-Variable $var
$iss.Variables.Add((New-Object TypeName System.Management.Automation.Runspaces.SessionStateVariableEntry ArgumentList $thisVariable.Name, $thisVariable.Value, ''))
Write-Verbose "Imported $var to Initial Session State"
}
catch{
Write-Warning $_.Exception.Message
}
}
}
# create the runspace pool
$runspacePool = [System.Management.Automation.Runspaces.RunspaceFactory]::CreateRunspacePool($minRunspaces, $maxRunspaces, $iss, $Host)
Write-Verbose 'Created runspace pool'
# set apartmentstate to MTA if MTA switch is used
if($MTA){
$runspacePool.ApartmentState = 'MTA'
Write-Verbose 'ApartmentState: MTA'
}
else {
Write-Verbose 'ApartmentState: STA'
}
# open the runspace pool
$runspacePool.Open()
Write-Verbose 'Runspace Pool Open'
# return the runspace pool object
Write-Output $runspacePool
}
function New-RunspaceJob{
<#
.SYNOPSIS
Create a new runspace job.
.DESCRIPTION
This function creates a new runspace job, executed in it's own runspace (thread).
.EXAMPLE
New-RunspaceJob -JobName 'Inventory' -ScriptBlock $code -Parameters $parameters
Description
———–
Execute code in $code with parameters from $parameters in a new runspace (thread).
.NOTES
Name: New-RunspaceJob
Author: Øyvind Kallstad
Date: 10.02.2014
Version: 1.0
#>
[CmdletBinding()]
param(
# Optionally give the job a name.
[Parameter()]
[string]$JobName,
# The code you want to execute.
[Parameter(Mandatory = $true)]
[ScriptBlock]$ScriptBlock,
# A working runspace pool object to handle the runspace job.
[Parameter(Mandatory = $true)]
[System.Management.Automation.Runspaces.RunspacePool]$RunspacePool,
# Hashtable of parameters to add to the runspace scriptblock.
[Parameter()]
[HashTable]$Parameters
)
# increment the runspace counter
$global:runspaceCounter++
# create a new runspace and set it to use the runspace pool object
$runspace = [System.Management.Automation.PowerShell]::Create()
$runspace.RunspacePool = $RunspacePool
# add the scriptblock to the runspace
$runspace.AddScript($ScriptBlock) | Out-Null
# if any parameters are given, add them into the runspace
if($parameters){
foreach ($parameter in ($Parameters.GetEnumerator())){
$runspace.AddParameter("$($parameter.Key)", $parameter.Value) | Out-Null
}
}
# invoke the runspace and store in the global runspaces variable
[void]$runspaces.Add(@{
JobName = $JobName
InvokeHandle = $runspace.BeginInvoke()
Runspace = $runspace
ID = $global:runspaceCounter
})
Write-Verbose 'Code invoked in runspace'
}
function Receive-RunspaceJob{
<#
.SYNOPSIS
Receive data back from a runspace job.
.DESCRIPTION
This function checks for completed runspace jobs, and retrieves the return data.
.EXAMPLE
Receive-RunspaceJob -Wait
Description
———–
Will wait until all runspace jobs are complete and retrieve data back from all of them.
.EXAMPLE
Receive-RunspaceJob -JobName 'Inventory'
Description
———–
Will get data from all completed jobs with the JobName 'Inventory'.
.NOTES
Name: Receive-RunspaceJob
Author: Øyvind Kallstad
Date: 10.02.2014
Version: 1.0
#>
[CmdletBinding()]
param(
# Only get results from named job.
[Parameter()]
[string]$JobName,
# Only get the results from job with this ID.
[Parameter()]
[int] $ID,
# Wait for all jobs to finish.
[Parameter(HelpMessage='Using this switch will wait until all jobs are finished')]
[switch]$Wait,
# Timeout in seconds until breaking free of the wait loop.
[Parameter()]
[int] $TimeOut = 60,
# Not implemented yet!
[Parameter(HelpMessage='Not implemented yet!')]
[switch]$ShowProgress
)
$startTime = Get-Date
do{
$more = $false
# handle filtering of runspaces
$filteredRunspaces = $global:runspaces.Clone()
if($JobName){
$filteredRunspaces = $filteredRunspaces | Where-Object {$_.JobName -eq $JobName}
}
if ($ID) {
$filteredRunspaces = $filteredRunspaces | Where-Object {$_.ID -eq $ID}
}
# iterate through the runspaces
foreach ($runspace in $filteredRunspaces){
# If job is finished, write the result to the pipeline and dispose of the runspace.
if ($runspace.InvokeHandle.isCompleted){
Write-Output $runspace.Runspace.EndInvoke($runspace.InvokeHandle)
$runspace.Runspace.Dispose()
$runspace.Runspace = $null
$runspace.InvokeHandle = $null
$runspaces.Remove($runspace)
Write-Verbose 'Job received'
}
# If invoke handle is still in place, the job is not finished.
elseif ($runspace.InvokeHandle -ne $null){
$more = $true
}
}
# break free of wait loop if timeout is exceeded
if ((New-TimeSpan Start $startTime).TotalSeconds -ge $TimeOut) {
Write-Verbose 'Timeout exceeded – breaking out of loop'
$more = $false
}
}
while ($more -and $PSBoundParameters['Wait'])
}
function Show-RunspaceJob{
<#
.SYNOPSIS
Show info about current runspace jobs.
.DESCRIPTION
This function will show you information about current (non-received) runspace jobs.
.EXAMPLE
Show-RunspaceJob
Description
———–
Will list all current (non-received) runspace jobs.
.EXAMPLE
Show-RunspaceJob -JobName 'Inventory'
Description
———–
Will list all jobs with the name 'Inventory' that are not received yet.
.NOTES
Name: Show-RunspaceJob
Author: Øyvind Kallstad
Date: 10.02.2014
Version: 1.0
#>
[CmdletBinding()]
param(
# Use the JobName parameter to optionally filter on the name of the job.
[Parameter()]
[string]$JobName
)
# if JobName parameter is used filter the runspaces
if($JobName){
$filteredRunspaces = $global:runspaces | Where-Object {$_.JobName -eq $JobName}
}
# else use all runspaces
else{
$filteredRunspaces = $global:runspaces
}
# iterate through all runspaces
foreach ($runspace in $filteredRunspaces){
# and create and output object for each job
Write-Output (,([PSCustomObject] [Ordered] @{
JobName = $runspace.JobName
ID = $runspace.ID
InstanceId = $runspace.Runspace.InstanceId
Status = $runspace.Runspace.InvocationStateInfo.State
Reason = $runspace.Runspace.InvocationStateInfo.Reason
Completed = $runspace.InvokeHandle.IsCompleted
HadErrors = $runspace.Runspace.HadErrors
}))
}
}
function Clear-RunspaceJobs {
Remove-Variable Name 'Runspaces' Scope 'Global'
}

8 comments

  1. Awesome – thanks!

    Why does New-RunspaceJob require a jobname? (I get an error with your example until I give it an arbitratary name):

    # iterate through each computer and create new runspace jobs
    # also run Receive-RunspaceJob to collect any already finished jobs
    foreach ($computer in $computers) {
    New-RunspaceJob -RunspacePool $thisRunspacePool -ScriptBlock $code -Parameters @{ComputerName = $computer} -JobName LeBeuf
    $results += Receive-RunspaceJob
    }

    Like

  2. Ah.. Because I had accidentally defined the JobName parameter as mandatory. That was not my intention. I have updated the Gist now. Thanks for letting me know 🙂

    Like

  3. Great.

    You’ve just a really great job at making threading easy to include in scripts, so that all the advanced stuff is seperated from the maincode.
    If you like feedback/suggestions – here’s some..
    As I see your intention, what you want is for the caller to just worry about the following (which is a really great idea):
    1: List of computernames
    2. How many threads
    3. Timeout

    How would it sound to you, if the following, from you maincode, where part of the module?:
    ——————————————————————————————————————–
    # create new runspace pool
    $thisRunspacePool = New-RunspacePool

    # define results array
    $results = @()

    # iterate through each computer and create new runspace jobs
    # also run Receive-RunspaceJob to collect any already finished jobs
    foreach ($computer in $computers) {
    New-RunspaceJob -RunspacePool $thisRunspacePool -ScriptBlock $code -Parameters @{ComputerName = $computer}
    $results += Receive-RunspaceJob
    }

    # if any jobs left, wait until all jobs are finished, or timeout is reached
    if ([bool](Show-RunspaceJob)) {
    $results += Receive-RunspaceJob -Wait -TimeOut $timeout
    }

    Write-Output $results
    ——————————————————————————————————————–

    Anders

    Like

  4. I can see why that would be tempting to do. There is however a couple of reason I wouldn’t do that.

    First, if you take a look at the New-RunspacePool function, it takes a lot of parameters. If you want to have any third-party/custom functions, modules, snapins or variables available to you runspaces, this is all set up along the runspace pool. I have no way of knowing what kind of information the users wants to be part of the pool, and by putting it in the main module, I’m forcing the creation of a runspace pool on the user.

    The same with the New-RunspaceJob really, this is where you add on any parameters you need for your code block. Usually this includes at least ComputerName. That’s why I added it to my example. But more often than not, some other parameters would be needed as well (for instance a credential parameter).

    That’s why I have opted for the fully user-customizable approach. What you can do of course, is to add a line in your powershell profile that automatically creates a ‘default’ runspace pool, always ready for when you want to quickly run some code in multiple threads.

    And if you’d like, feel free to play with my code and customize it to your own needs 🙂

    I had originally not intended to publish anything but the functions themselves, but thought I’d add a basic template as an example of how you could use the functions, and also as a starting point for further development. But that’s all it is really, a very basic template, meant to be further customized to individual needs.

    You could very easily convert my basic template into a function btw, and use this function the way you describe: As an entry-point to all the other functions, for any scenarios where you just have a script block and an array of computer names.

    But thanks a lot for the positive and constructive feedback! If you need any help customizing this to your needs, let me know 🙂

    Like

  5. Hi Øyvind – great name btw – There’s an eagle in the danish version of Donald Duck called Øjvind Ørn (he’s one of the bad guys though) 🙂

    That was a very informative explanation – thanks for taking your time! I see your intention and understand the choices you’ve taken.
    I’ve made a module out of your code, that will be put to use some how I’m sure.

    Thanks for the generosity of sharing your great work – I’ll let you know any other questions surfaces or theres a need to enjoy your friendlyness!

    Anders (the danish Donald Duck is called Anders And 🙂

    Like

  6. Would you mind providing an example of how I’d pass a function and then call one?
    I’ve been trying with no luck.

    I’ve been trying to do something like
    New-RunspacePool -Functions ${function:Get-SQLDBValues}

    and then calling the function from the RunSpace like
    Get-SQLDBValues -SQLConnection $Connection -Query $Query

    Like

Leave a Reply to Anders Cancel reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s