如何异步调用列表中的多个URL

Question

我需要调用几十万个URL.这些是对应用程序服务器的调用,应用程序服务器将处理它们并将状态代码写入表.我不需要等待响应(成功/失败),只需要服务器获得请求.我还希望能够指定一次可以运行多少并发作业,因为我还没有确定tomcat可以处理多少并发请求.

这是我到目前为止所得到的,基本上取自某人的其他人尝试做类似的事情,而不是使用网址调用.文本文件包含各自的URL.网址如下所示:

http://webserver:8080/app/mwo/services/create?server=ServerName&e1user=admin&newMWONum=123456&sourceMWONum=0&tagNum=33-A-1B

和代码:

$maxConcurrentJobs = 10
$content = Get-Content -Path "C:\Temp\urls.txt"

foreach ($url in $content) {
    $running = @(Get-Job | Where-Object { $_.State -eq 'Running' })
    if ($running.Count -le $maxConcurrentJobs) {
        Start-Job {
             Invoke-WebRequest -UseBasicParsing -Uri $using:url
        }
    } else {
         $running | Wait-Job -Any
    }
    Get-Job | Receive-Job
}

我遇到的问题是每个"工作"给出2个错误,我不知道为什么.当我转储url数组$ content时,它看起来很好,当我逐个运行我的Invoke-WebRequest时,它们可以正常工作.

126    Job126          BackgroundJob   Running       True            localhost            ...                
Invalid URI: The hostname could not be parsed.
    + CategoryInfo          : NotSpecified: (:) [Invoke-RestMethod], UriFormatException
    + FullyQualifiedErrorId : System.UriFormatException,Microsoft.PowerShell.Commands.InvokeRestMethodComman 
   d
    + PSComputerName        : localhost

Invalid URI: The hostname could not be parsed.
    + CategoryInfo          : NotSpecified: (:) [Invoke-RestMethod], UriFormatException
    + FullyQualifiedErrorId : System.UriFormatException,Microsoft.PowerShell.Commands.InvokeRestMethodComman 
   d
    + PSComputerName        : localhost

任何帮助或替代实施将不胜感激.我愿意不使用PowerShell,但我只限于Windows 7桌面或Windows 2008 R2服务器,而且我可能会使用网址中的localhost在服务器上运行最终脚本,以减少网络延迟.

Answer 1

使用Jobs会产生大量开销,因为每个新Job都会生成一个新进程.

请使用Runspaces!

$maxConcurrentJobs = 10
$content = Get-Content -Path "C:\Temp\urls.txt"

# Create a runspace pool where $maxConcurrentJobs is the 
# maximum number of runspaces allowed to run concurrently    
$Runspace = [runspacefactory]::CreateRunspacePool(1,$maxConcurrentJobs)

# Open the runspace pool (very important)
$Runspace.Open()

foreach ($url in $content) {
    # Create a new PowerShell instance and tell it to execute in our runspace pool
    $ps = [powershell]::Create()
    $ps.RunspacePool = $Runspace

    # Attach some code to it
    [void]$ps.AddCommand("Invoke-WebRequest").AddParameter("UseBasicParsing",$true).AddParameter("Uri",$url)

    # Begin execution asynchronously (returns immediately)
    [void]$ps.BeginInvoke()

    # Give feedback on how far we are
    Write-Host ("Initiated request for {0}" -f $url)
}

如链接的ServerFault帖子中所述,您还可以使用更通用的解决方案,例如Invoke-Parallel,基本上可以执行上述操作