PowerShell is great at simplifying and automating your daily tasks. And it’s only natural that after you have automated or scripted most of your simple tasks, you start creating bigger scripts. If you work in a large environment, running for instance a script that uses WMI to query computers for data can take a long time to complete. The solution? Multi-threading of course. One way of doing this is using the built-in ‘Jobs’ cmdlets, but what this post is about is Runspaces.
Runspaces lets you create separate PowerShell sessions running in separate runspaces (hence the name), within the same process. This means that you won’t see any difference when looking at Task Manager (except CPU and memory will probably go up, depending on the number of runspaces you use).
Working with runspaces have for some been a daunting task, as it means you need to work with .NET directly. I have created a set of wrappers for working with runspaces that simplifies this and lets you work with functions that behave as common PowerShell cmdlets.
The following functions are included:
- New-RunspacePool
- New-RunspaceJob
- Receive-RunspaceJob
- Show-RunspaceJob
- Clear-RunspaceJobs
New-RunspacePool
This function will create a new runspace pool. A runspace pool can be thought of as the object that is controlling the behaviour of you runspaces, as well as tying them all together. Setting up the runspace pool lets you control how many concurrent threads you want to run, as well as control snapins, modules, functions and variables that you want each runspace to have access to when spawned. The New-RunspacePool function returns a RunspacePool object that you will need to capture in a variable, as this is used together with the next function: New-RunspaceJob.
New-RunspaceJob
This is the function that takes the supplied code (in the form of a script block), creates a new runspace and injects the code together with any parameters (in the form of a hashtable) that you want the code to use. Each new runspace job is connected to the runspace pool, and the pool takes care of how many runspaces are current running at the same time, and will automatically start a new one when a new opening in the queue comes up. All the runspace jobs are saved in the global ‘runspaces’ variable. You can name your jobs as well, so if you are running several different jobs at the same time, it’s easy to differentiate between them when looking in the ‘runspaces’ array.
Receive-RunspaceJob
Receive-RunspaceJob will check if the job is finished and return the result if it is. This function have a ‘Wait’ parameter that you can use (together with a ‘Timeout’ parameter) to let it run until all jobs are finished (or the timeout value is exceeded). It also supports getting only runspaces belonging to a certain job, or even by the unique ID of each job.
Show-RunspaceJob
Show-RunspaceJob will list out all jobs currently in the global runspaces array. It only have one parameter, ‘JobName’, that lets you list only runspaces belonging to a particular job.
Clear-Runspaces
Lastly we have the Clear-Runspaces function. The only thing this one does, is remove everything from the global runspaces variable (it is in fact deleting the variable). Hopefully you would never need this, but working with WMI I know from experience that sometimes a WMI session can hang, and it’s nice to be able to completely remove any jobs left.
Creating these functions, I had a goal of creating something that I could use both in scripts, but also directly from the console, and I hope I have succeeded. To get you going I have also created a basic template that you can use when running code against an array of computers. It is by no means a complete script, and should only be used as a starting-point you can work on, but it should be enough to give you a picture of how to use the different functions.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # Basic Script Template for Runspace Jobs against an array of computers | |
| # This is just an example to show the basic layout of a script using the Runspace functions | |
| # If you are creating a script to be run in production, please add logging and error handling | |
| # Also consider writing it to support parameters for input instead of hardcoding it like in this example | |
| #region verify that runspace functions are present | |
| $missingFunctions = $false | |
| $functions = ( | |
| 'New-RunspacePool', | |
| 'New-RunspaceJob', | |
| 'Show-RunspaceJob', | |
| 'Receive-RunspaceJob' | |
| ) | |
| foreach ($function in $functions) { | |
| try { | |
| $thisFunction = Get-Item -LiteralPath "function:$function" -ErrorAction Stop | |
| } | |
| catch { | |
| $missingFunctions = $true | |
| Write-Warning "$function not found" | |
| } | |
| } | |
| if ($missingFunctions) { | |
| break | |
| } | |
| #endregion | |
| # computers to run against | |
| # re-write to read from script parameter, file, Active Directory etc as needed | |
| $computers = ( | |
| 'computer01', | |
| 'computer02', | |
| 'computer03', | |
| 'computer04' | |
| ) | |
| # define timeout for Receive-RunspaceJob (in seconds) | |
| $timeout = 30 | |
| # code to run in each runspace | |
| $code = { | |
| Param([string]$ComputerName) | |
| # code goes here | |
| } | |
| # create new runspace pool | |
| $thisRunspacePool = New-RunspacePool | |
| # define results array | |
| $results = @() | |
| # iterate through each computer and create new runspace jobs | |
| # also run Receive-RunspaceJob to collect any already finished jobs | |
| foreach ($computer in $computers) { | |
| New-RunspaceJob -RunspacePool $thisRunspacePool -ScriptBlock $code -Parameters @{ComputerName = $computer} | |
| $results += Receive-RunspaceJob | |
| } | |
| # if any jobs left, wait until all jobs are finished, or timeout is reached | |
| if ([bool](Show-RunspaceJob)) { | |
| $results += Receive-RunspaceJob -Wait -TimeOut $timeout | |
| } | |
| Write-Output $results |
As always, if you spot any bugs, or have any suggestions for future improvements, or just need some input on how to use these functions, don’t hesitate to contact me!
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| function New-RunspacePool{ | |
| <# | |
| .SYNOPSIS | |
| Create a new runspace pool | |
| .DESCRIPTION | |
| This function creates a new runspace pool. This is needed to be able to run code multi-threaded. | |
| .EXAMPLE | |
| $pool = New-RunspacePool | |
| Description | |
| ———– | |
| Create a new runspace pool with default settings, and store it in the pool variable. | |
| .EXAMPLE | |
| $pool = New-RunspacePool -Snapins 'vmware.vimautomation.core' | |
| Description | |
| ———– | |
| Create a new runspace pool with the VMWare PowerCli snapin added, and store it in the pool variable. | |
| .NOTES | |
| Name: New-RunspacePool | |
| Author: Øyvind Kallstad | |
| Date: 10.02.2014 | |
| Version: 1.0 | |
| #> | |
| [CmdletBinding()] | |
| param( | |
| # The minimun number of concurrent threads to be handled by the runspace pool. The default is 1. | |
| [Parameter(HelpMessage='Minimum number of concurrent threads')] | |
| [ValidateRange(1,65535)] | |
| [int32]$minRunspaces = 1, | |
| # The maximum number of concurrent threads to be handled by the runspace pool. The default is 15. | |
| [Parameter(HelpMessage='Maximum number of concurrent threads')] | |
| [ValidateRange(1,65535)] | |
| [int32]$maxRunspaces = 15, | |
| # Using this switch will set the apartment state to MTA. | |
| [Parameter()] | |
| [switch]$MTA, | |
| # Array of snapins to be added to the initial session state of the runspace object. | |
| [Parameter(HelpMessage='Array of SnapIns you want available for the runspace pool')] | |
| [string[]]$Snapins, | |
| # Array of modules to be added to the initial session state of the runspace object. | |
| [Parameter(HelpMessage='Array of Modules you want available for the runspace pool')] | |
| [string[]]$Modules, | |
| # Array of functions to be added to the initial session state of the runspace object. | |
| [Parameter(HelpMessage='Array of Functions that you want available for the runspace pool')] | |
| [string[]]$Functions, | |
| # Array of variables to be added to the initial session state of the runspace object. | |
| [Parameter(HelpMessage='Array of Variables you want available for the runspace pool')] | |
| [string[]]$Variables | |
| ) | |
| # if global runspace array is not present, create it | |
| if(-not $global:runspaces){ | |
| $global:runspaces = New-Object System.Collections.ArrayList | |
| } | |
| # if global runspace counter is not present, create it | |
| if(-not $global:runspaceCounter){ | |
| $global:runspaceCounter = 0 | |
| } | |
| # create the initial session state | |
| $iss = [System.Management.Automation.Runspaces.InitialSessionState]::CreateDefault() | |
| # add any snapins to the session state object | |
| if($Snapins){ | |
| foreach ($snapName in $Snapins){ | |
| try{ | |
| $iss.ImportPSSnapIn($snapName,[ref]'') | Out-Null | |
| Write-Verbose "Imported $snapName to Initial Session State" | |
| } | |
| catch{ | |
| Write-Warning $_.Exception.Message | |
| } | |
| } | |
| } | |
| # add any modules to the session state object | |
| if($Modules){ | |
| foreach($module in $Modules){ | |
| try{ | |
| $iss.ImportPSModule($module) | Out-Null | |
| Write-Verbose "Imported $module to Initial Session State" | |
| } | |
| catch{ | |
| Write-Warning $_.Exception.Message | |
| } | |
| } | |
| } | |
| # add any functions to the session state object | |
| if($Functions){ | |
| foreach($func in $Functions){ | |
| try{ | |
| $thisFunction = Get-Item -LiteralPath "function:$func" | |
| [String]$functionName = $thisFunction.Name | |
| [ScriptBlock]$functionCode = $thisFunction.ScriptBlock | |
| $iss.Commands.Add((New-Object -TypeName System.Management.Automation.Runspaces.SessionStateFunctionEntry -ArgumentList $functionName,$functionCode)) | |
| Write-Verbose "Imported $func to Initial Session State" | |
| Remove-Variable thisFunction, functionName, functionCode | |
| } | |
| catch{ | |
| Write-Warning $_.Exception.Message | |
| } | |
| } | |
| } | |
| # add any variables to the session state object | |
| if($Variables){ | |
| foreach($var in $Variables){ | |
| try{ | |
| $thisVariable = Get-Variable $var | |
| $iss.Variables.Add((New-Object -TypeName System.Management.Automation.Runspaces.SessionStateVariableEntry -ArgumentList $thisVariable.Name, $thisVariable.Value, '')) | |
| Write-Verbose "Imported $var to Initial Session State" | |
| } | |
| catch{ | |
| Write-Warning $_.Exception.Message | |
| } | |
| } | |
| } | |
| # create the runspace pool | |
| $runspacePool = [System.Management.Automation.Runspaces.RunspaceFactory]::CreateRunspacePool($minRunspaces, $maxRunspaces, $iss, $Host) | |
| Write-Verbose 'Created runspace pool' | |
| # set apartmentstate to MTA if MTA switch is used | |
| if($MTA){ | |
| $runspacePool.ApartmentState = 'MTA' | |
| Write-Verbose 'ApartmentState: MTA' | |
| } | |
| else { | |
| Write-Verbose 'ApartmentState: STA' | |
| } | |
| # open the runspace pool | |
| $runspacePool.Open() | |
| Write-Verbose 'Runspace Pool Open' | |
| # return the runspace pool object | |
| Write-Output $runspacePool | |
| } | |
| function New-RunspaceJob{ | |
| <# | |
| .SYNOPSIS | |
| Create a new runspace job. | |
| .DESCRIPTION | |
| This function creates a new runspace job, executed in it's own runspace (thread). | |
| .EXAMPLE | |
| New-RunspaceJob -JobName 'Inventory' -ScriptBlock $code -Parameters $parameters | |
| Description | |
| ———– | |
| Execute code in $code with parameters from $parameters in a new runspace (thread). | |
| .NOTES | |
| Name: New-RunspaceJob | |
| Author: Øyvind Kallstad | |
| Date: 10.02.2014 | |
| Version: 1.0 | |
| #> | |
| [CmdletBinding()] | |
| param( | |
| # Optionally give the job a name. | |
| [Parameter()] | |
| [string]$JobName, | |
| # The code you want to execute. | |
| [Parameter(Mandatory = $true)] | |
| [ScriptBlock]$ScriptBlock, | |
| # A working runspace pool object to handle the runspace job. | |
| [Parameter(Mandatory = $true)] | |
| [System.Management.Automation.Runspaces.RunspacePool]$RunspacePool, | |
| # Hashtable of parameters to add to the runspace scriptblock. | |
| [Parameter()] | |
| [HashTable]$Parameters | |
| ) | |
| # increment the runspace counter | |
| $global:runspaceCounter++ | |
| # create a new runspace and set it to use the runspace pool object | |
| $runspace = [System.Management.Automation.PowerShell]::Create() | |
| $runspace.RunspacePool = $RunspacePool | |
| # add the scriptblock to the runspace | |
| $runspace.AddScript($ScriptBlock) | Out-Null | |
| # if any parameters are given, add them into the runspace | |
| if($parameters){ | |
| foreach ($parameter in ($Parameters.GetEnumerator())){ | |
| $runspace.AddParameter("$($parameter.Key)", $parameter.Value) | Out-Null | |
| } | |
| } | |
| # invoke the runspace and store in the global runspaces variable | |
| [void]$runspaces.Add(@{ | |
| JobName = $JobName | |
| InvokeHandle = $runspace.BeginInvoke() | |
| Runspace = $runspace | |
| ID = $global:runspaceCounter | |
| }) | |
| Write-Verbose 'Code invoked in runspace' | |
| } | |
| function Receive-RunspaceJob{ | |
| <# | |
| .SYNOPSIS | |
| Receive data back from a runspace job. | |
| .DESCRIPTION | |
| This function checks for completed runspace jobs, and retrieves the return data. | |
| .EXAMPLE | |
| Receive-RunspaceJob -Wait | |
| Description | |
| ———– | |
| Will wait until all runspace jobs are complete and retrieve data back from all of them. | |
| .EXAMPLE | |
| Receive-RunspaceJob -JobName 'Inventory' | |
| Description | |
| ———– | |
| Will get data from all completed jobs with the JobName 'Inventory'. | |
| .NOTES | |
| Name: Receive-RunspaceJob | |
| Author: Øyvind Kallstad | |
| Date: 10.02.2014 | |
| Version: 1.0 | |
| #> | |
| [CmdletBinding()] | |
| param( | |
| # Only get results from named job. | |
| [Parameter()] | |
| [string]$JobName, | |
| # Only get the results from job with this ID. | |
| [Parameter()] | |
| [int] $ID, | |
| # Wait for all jobs to finish. | |
| [Parameter(HelpMessage='Using this switch will wait until all jobs are finished')] | |
| [switch]$Wait, | |
| # Timeout in seconds until breaking free of the wait loop. | |
| [Parameter()] | |
| [int] $TimeOut = 60, | |
| # Not implemented yet! | |
| [Parameter(HelpMessage='Not implemented yet!')] | |
| [switch]$ShowProgress | |
| ) | |
| $startTime = Get-Date | |
| do{ | |
| $more = $false | |
| # handle filtering of runspaces | |
| $filteredRunspaces = $global:runspaces.Clone() | |
| if($JobName){ | |
| $filteredRunspaces = $filteredRunspaces | Where-Object {$_.JobName -eq $JobName} | |
| } | |
| if ($ID) { | |
| $filteredRunspaces = $filteredRunspaces | Where-Object {$_.ID -eq $ID} | |
| } | |
| # iterate through the runspaces | |
| foreach ($runspace in $filteredRunspaces){ | |
| # If job is finished, write the result to the pipeline and dispose of the runspace. | |
| if ($runspace.InvokeHandle.isCompleted){ | |
| Write-Output $runspace.Runspace.EndInvoke($runspace.InvokeHandle) | |
| $runspace.Runspace.Dispose() | |
| $runspace.Runspace = $null | |
| $runspace.InvokeHandle = $null | |
| $runspaces.Remove($runspace) | |
| Write-Verbose 'Job received' | |
| } | |
| # If invoke handle is still in place, the job is not finished. | |
| elseif ($runspace.InvokeHandle -ne $null){ | |
| $more = $true | |
| } | |
| } | |
| # break free of wait loop if timeout is exceeded | |
| if ((New-TimeSpan -Start $startTime).TotalSeconds -ge $TimeOut) { | |
| Write-Verbose 'Timeout exceeded – breaking out of loop' | |
| $more = $false | |
| } | |
| } | |
| while ($more -and $PSBoundParameters['Wait']) | |
| } | |
| function Show-RunspaceJob{ | |
| <# | |
| .SYNOPSIS | |
| Show info about current runspace jobs. | |
| .DESCRIPTION | |
| This function will show you information about current (non-received) runspace jobs. | |
| .EXAMPLE | |
| Show-RunspaceJob | |
| Description | |
| ———– | |
| Will list all current (non-received) runspace jobs. | |
| .EXAMPLE | |
| Show-RunspaceJob -JobName 'Inventory' | |
| Description | |
| ———– | |
| Will list all jobs with the name 'Inventory' that are not received yet. | |
| .NOTES | |
| Name: Show-RunspaceJob | |
| Author: Øyvind Kallstad | |
| Date: 10.02.2014 | |
| Version: 1.0 | |
| #> | |
| [CmdletBinding()] | |
| param( | |
| # Use the JobName parameter to optionally filter on the name of the job. | |
| [Parameter()] | |
| [string]$JobName | |
| ) | |
| # if JobName parameter is used filter the runspaces | |
| if($JobName){ | |
| $filteredRunspaces = $global:runspaces | Where-Object {$_.JobName -eq $JobName} | |
| } | |
| # else use all runspaces | |
| else{ | |
| $filteredRunspaces = $global:runspaces | |
| } | |
| # iterate through all runspaces | |
| foreach ($runspace in $filteredRunspaces){ | |
| # and create and output object for each job | |
| Write-Output (,([PSCustomObject] [Ordered] @{ | |
| JobName = $runspace.JobName | |
| ID = $runspace.ID | |
| InstanceId = $runspace.Runspace.InstanceId | |
| Status = $runspace.Runspace.InvocationStateInfo.State | |
| Reason = $runspace.Runspace.InvocationStateInfo.Reason | |
| Completed = $runspace.InvokeHandle.IsCompleted | |
| HadErrors = $runspace.Runspace.HadErrors | |
| })) | |
| } | |
| } | |
| function Clear-RunspaceJobs { | |
| Remove-Variable -Name 'Runspaces' -Scope 'Global' | |
| } |
Communary 
Awesome – thanks!
Why does New-RunspaceJob require a jobname? (I get an error with your example until I give it an arbitratary name):
# iterate through each computer and create new runspace jobs
# also run Receive-RunspaceJob to collect any already finished jobs
foreach ($computer in $computers) {
New-RunspaceJob -RunspacePool $thisRunspacePool -ScriptBlock $code -Parameters @{ComputerName = $computer} -JobName LeBeuf
$results += Receive-RunspaceJob
}
LikeLike
Ah.. Because I had accidentally defined the JobName parameter as mandatory. That was not my intention. I have updated the Gist now. Thanks for letting me know 🙂
LikeLike
Great.
You’ve just a really great job at making threading easy to include in scripts, so that all the advanced stuff is seperated from the maincode.
If you like feedback/suggestions – here’s some..
As I see your intention, what you want is for the caller to just worry about the following (which is a really great idea):
1: List of computernames
2. How many threads
3. Timeout
How would it sound to you, if the following, from you maincode, where part of the module?:
——————————————————————————————————————–
# create new runspace pool
$thisRunspacePool = New-RunspacePool
# define results array
$results = @()
# iterate through each computer and create new runspace jobs
# also run Receive-RunspaceJob to collect any already finished jobs
foreach ($computer in $computers) {
New-RunspaceJob -RunspacePool $thisRunspacePool -ScriptBlock $code -Parameters @{ComputerName = $computer}
$results += Receive-RunspaceJob
}
# if any jobs left, wait until all jobs are finished, or timeout is reached
if ([bool](Show-RunspaceJob)) {
$results += Receive-RunspaceJob -Wait -TimeOut $timeout
}
Write-Output $results
——————————————————————————————————————–
Anders
LikeLike
I can see why that would be tempting to do. There is however a couple of reason I wouldn’t do that.
First, if you take a look at the New-RunspacePool function, it takes a lot of parameters. If you want to have any third-party/custom functions, modules, snapins or variables available to you runspaces, this is all set up along the runspace pool. I have no way of knowing what kind of information the users wants to be part of the pool, and by putting it in the main module, I’m forcing the creation of a runspace pool on the user.
The same with the New-RunspaceJob really, this is where you add on any parameters you need for your code block. Usually this includes at least ComputerName. That’s why I added it to my example. But more often than not, some other parameters would be needed as well (for instance a credential parameter).
That’s why I have opted for the fully user-customizable approach. What you can do of course, is to add a line in your powershell profile that automatically creates a ‘default’ runspace pool, always ready for when you want to quickly run some code in multiple threads.
And if you’d like, feel free to play with my code and customize it to your own needs 🙂
I had originally not intended to publish anything but the functions themselves, but thought I’d add a basic template as an example of how you could use the functions, and also as a starting point for further development. But that’s all it is really, a very basic template, meant to be further customized to individual needs.
You could very easily convert my basic template into a function btw, and use this function the way you describe: As an entry-point to all the other functions, for any scenarios where you just have a script block and an array of computer names.
But thanks a lot for the positive and constructive feedback! If you need any help customizing this to your needs, let me know 🙂
LikeLike
Hi Øyvind – great name btw – There’s an eagle in the danish version of Donald Duck called Øjvind Ørn (he’s one of the bad guys though) 🙂
That was a very informative explanation – thanks for taking your time! I see your intention and understand the choices you’ve taken.
I’ve made a module out of your code, that will be put to use some how I’m sure.
Thanks for the generosity of sharing your great work – I’ll let you know any other questions surfaces or theres a need to enjoy your friendlyness!
Anders (the danish Donald Duck is called Anders And 🙂
LikeLike
Cool. In Norwegian he is called Ørnulf Ørn 🙂
LikeLike
🙂
LikeLike
Would you mind providing an example of how I’d pass a function and then call one?
I’ve been trying with no luck.
I’ve been trying to do something like
New-RunspacePool -Functions ${function:Get-SQLDBValues}
and then calling the function from the RunSpace like
Get-SQLDBValues -SQLConnection $Connection -Query $Query
LikeLike