Powershell steppable pipelines

May 16, 2022 powershell

Steppable pipelines can be used to manually control the execution of cmdlets

Example

First we need to declare a scriptblock with a single statement. Then execute the GetSteppablePipeline() method on the scriptblock object to get a SteppablePipeline object.

To start the execution call Begin($true). The $true parameter specifies that the pipeline is expecting to receive input, which will usually be the case.

Then to have the pipeline process an object call Process() with the object to process as the parameter.

Finally call End() to stop the pipeline. The pipeline object is now disposed and to start a new one we have to call GetSteppablePipeline() again.

$scriptBlock = { Measure-Object -Sum }
$pipeline = $scriptBlock.GetSteppablePipeline()
$pipeline.Begin($true)
$pipeline.Process(1)
$pipeline.Process(2)
$pipeline.End() # Returns { Count = 2; Sum = 3}

The statement in the scriptblock can itself be a more complex pipeline

$pipeline = {
    ForEach-Object { $_ * 2 } `
    | Where-Object { $_ -lt 10 } `
    | Measure-Object -Sum
}.GetSteppablePipeline()

$pipeline.Begin($true)
1,2,10 | % { $pipeline.Process($_) }
$pipeline.End()

# Returns { Sum = 6 }. 1 and 2 gets doubled and summed, while 10 is doubled and exceeds the condition in the Where-Object step

Be mindful that the call to Begin() on a SteppablePipeline cannot return a value, so if a cmdlet emits a value here it will be returned with the first call to Process().

Also be mindful that values can be emitted from both Process() and then End() blocks, depending on cmdlets in the scriptblock.

Function Test {
    Begin { "Begin" }
    Process { "Process" }
    End { "End" }
}

$pipeline = { Test }.GetSteppablePipeline()
$pipeline.Begin($false)
$pipeline.Process() # Returns "Begin","Process"
$pipeline.Process() # Returns "Process"
$pipeline.End() # Returns "End"

One of the most useful applications for this is if you want to process a large amount of data without buffering it, while also routing different objects to different pipelines. A simple example here is a cmdlet that can take the output of Get-ChildItem and selectively send it to different pipelines for further processing.

Function Get-FileAndFolderCount {
    Param(
        [Parameter(ValueFromPipeline)][System.IO.FileSystemInfo]$FileInfo
    )

    Begin {
        $filePipe = { Measure-Object -Sum Length }.GetSteppablePipeline()
        $filePipe.Begin($true)
        $folderPipe = { Measure-Object }.GetSteppablePipeline()
        $folderPipe.Begin($true)
    }

    Process {
        if ($FileInfo.GetType().IsAssignableTo([System.IO.FileInfo])) {
            $filePipe.Process($FileInfo)
        } elseif ($FileInfo.GetType().IsAssignableTo([System.IO.DirectoryInfo])) {
            $folderPipe.Process($FileInfo)
        }
    }

    End {
        $fileResult = $filePipe.End()
        [PSCustomObject]@{
            FileCount = $fileResult | Select-Object -ExpandProperty Count
            FileSizeSum = $fileResult | Select-Object -ExpandProperty Sum
            FolderCount = $folderPipe.End() | Select-Object -ExpandProperty Count
        }
    }
}

This can then be called like this.

Get-ChildItem $env:APPDATA -Recurse | Get-FileAndFolderCount