在PowerShell中运行N个并行作业

Question

我有以下powershell脚本

$list = invoke-sqlcmd 'exec getOneMillionRows' -Server...
$list | % {
    GetData $_ > $_.txt
    ZipTheFile $_.txt $_.txt.zip
    ...
}

如何以{ GetDatta $_ > $_.txt ....}有限的最大作业数并行运行脚本块(),例如,一次最多可以生成8个文件？

Answer 1

与发布用户"Start-Automating"相同的想法,但更正了在他的示例中点击else子句时忘记启动阻止的作业的错误:

$servers = @('a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n')

foreach ($server in $servers) {
    $running = @(Get-Job | Where-Object { $_.State -eq 'Running' })
    if ($running.Count -ge 4) {
        $running | Wait-Job -Any | Out-Null
    }

    Write-Host "Starting job for $server"
    Start-Job {
        # do something with $using:server. Just sleeping for this example.
        Start-Sleep 5
        return "result from $using:server"
    } | Out-Null
}

# Wait for all jobs to complete and results ready to be received
Wait-Job * | Out-Null

# Process the results
foreach($job in Get-Job)
{
    $result = Receive-Job $job
    Write-Host $result
}

Remove-Job -State Completed

Answer 2

Start-Job cmdlet允许您在后台运行代码.要做你要问的事情,下面的代码应该有用.

foreach ($server in $servers) {
    $running = @(Get-Job | Where-Object { $_.State -eq 'Running' })
    if ($running.Count -le 8) {
        Start-Job {
             Add-PSSnapin SQL
             $list = invoke-sqlcmd 'exec getOneMillionRows' -Server...
             ...
        }
    } else {
         $running | Wait-Job
    }
    Get-Job | Receive-Job
}

希望这可以帮助.

Answer 3

使用SplitPipeline模块的Split-Pipelinecmdlet 应该非常简单.代码看起来很简单:

Import-Module SplitPipeline
$list = invoke-sqlcmd 'exec getOneMillionRows' -Server...
$list | Split-Pipeline -Count 8 {process{
    GetData $_ > $_.txt
    ZipTheFile $_.txt $_.txt.zip
    ...
}}

Answer 4

旧线程，但我认为这可能会有所帮助：

$List = C:\List.txt
$Jobs = 8

Foreach ($PC in Get-Content $List)
{
Do
    {
    $Job = (Get-Job -State Running | measure).count
    } Until ($Job -le $Jobs)

Start-Job -Name $PC -ScriptBlock { "Your command here $Using:PC" }
Get-Job -State Completed | Remove-Job
}

Wait-Job -State Running
Get-Job -State Completed | Remove-Job
Get-Job

当“运行”的作业量超过允许运行的“$jobs”量时，“Do”循环会暂停“foreach”。而不是等待剩余的作业完成并显示失败的作业......

Answer 5

后台工作就是答案.您还可以使用[System.Collection.Queue]限制运行队列中的作业.PowerShell团队就此主题发表了一篇博文:http://blogs.msdn.com/b/powershell/archive/2011/04/04/scaling-and-queuing-powershell-background-jobs.aspx

使用排队方法可能是限制后台作业的最佳答案.