Foreach 对象比 foreach -parallel 更快？

Question

Foreach 对象比 foreach -parallel 更快？

Den*_*020 2 parallel-processing powershell foreach multithreading

我刚刚开始使用 powershell，我想知道为什么我的并行脚本比正常的 foreach 对象脚本更慢？

我的普通 foreachobject 脚本：

function Get-ADUsers {  #get all users in nested groups }

 function Get-NestedGroupUsers {
    param ( 
        [Parameter(Mandatory = $true)][String]$FileName,
        [Parameter(Mandatory = $true)][String]$searchFileURL
    )
    $storageHolder = @()
    # $storageHolder | Export-Csv -Path "C:\Users\demandx\Desktop\AD User Lists\$FileName.csv" -NoTypeInformation -Force 
    $groupList = Get-Content $searchFileURL 
    $groupList |  ForEach-Object { 
        $allusers = Get-ADUsers -GroupName $_
        $storageHolder += $allusers  
       
    }
    $storageHolder | select ParentGroup, Name, EmployeeNumber, Enabled, LastLogonDate, PasswordLastSet  |Export-Csv -Path "C:\Users\demandx\Desktop\$FileName.csv" -NoTypeInformation -Force
}

Run Code Online (Sandbox Code Playgroud)

我的 foreach -parallel 脚本（我将函数存储在 psm1 内，然后在此处导入。）

Function Get-Members {
    param ( 
        [Parameter(Mandatory = $true)][String]$FileName,
        [Parameter(Mandatory = $true)][String]$searchFileURL
    )
    $groupList = Get-Content $searchFileURL 
    $storageHolder = $groupList |  ForEach-Object -Parallel {
        Import-Module -Name "C:\Users\demandx\Desktop\Get-ADUserMembers.psm1" 
        Get-ADUserMembers -GroupName $_ | Select-Object ParentGroup, Name, EmployeeNumber, Enabled, LastLogonDate, PasswordLastSet
    }   -ThrottleLimit 5
    
    $storageHolder |  Export-Csv -Path "C:\Users\demandx\Desktop\AD User Lists\$FileName.csv" -NoTypeInformation -Force

}

Run Code Online (Sandbox Code Playgroud)

脚本或我的 get-adusers （获取嵌套组中的所有用户）

 function Get-ADUsers { 
    param ( 
        [Parameter(ValuefromPipeline = $true, mandatory = $true)][String] $GroupName
    ) 
    [int]$circular = $null

    # result holder
    $resultHolder = @()
        $table = $null 
        $nestedmembers = $null 
        $adgroupname = $null     
  function Get-ADUsers { 
    param ( 
        [Parameter(ValuefromPipeline = $true, mandatory = $true)][String] $GroupName
    ) 
    [int]$circular = $null

    # result holder
    $resultHolder = @()
        $table = $null 
        $nestedmembers = $null 
        $adgroupname = $null     

        # get members of the group and member of
        $ADGroupname = get-adgroup $groupname -properties memberof, members

        # list all members as list (no headers) and save to var
        $memberof = $adgroupname | select -expand memberof 
       
        if ($adgroupname) {  
            if ($circular) { 
                $nestedMembers = Get-ADGroupMember -Identity $GroupName -recursive 
                $circular = $null 
            } 
            else { 
                $nestedMembers = Get-ADGroupMember -Identity $GroupName | sort objectclass -Descending
                # if get adgroupmember returns nothing, it uses the members for ordinary getADGroup
                if (!($nestedmembers)) {
                    $unknown = $ADGroupname | select -expand members
                    if ($unknown) {
                        $nestedmembers = @()
                        foreach ($member in $unknown) {
                            $nestedmembers += get-adobject $member
                        }
                    }
                }
            } 
            # loops through each member
            ForEach($nestedmember in $nestedmembers){ 
                # creates the properties into a custom object. 
                $Props = @{
                    Type            = $nestedmember.objectclass;
                    Name            = $nestedmember.name;
                    DisplayName     = "";
                    ParentGroup     = $ADgroupname.name;
                    Enabled         = "";
                    Nesting         = $nesting;
                    DN              = $nestedmember.distinguishedname;
                    Comment         = ""
                    EmployeeNumber  = "";
                    LastLogonDate   = "";
                    PasswordLastSet = "";
                } 
                # if member object is a user
                if ($nestedmember.objectclass -eq "user") { 
                    # saves all the properties in the table. 
                    $nestedADMember = get-aduser $nestedmember.Name -properties enabled, displayname, EmployeeNumber, LastLogonDate, PasswordLastSet
                    $table = new-object psobject -property $props 
                    $table.enabled = $nestedadmember.enabled
                    $table.name = $nestedadmember.samaccountname
                    $table.displayname = $nestedadmember.displayname
                    $table.EmployeeNumber = $nestedadmember.EmployeeNumber
                    $table.LastLogonDate = $nestedadmember.LastLogonDate
                    $table.PasswordLastSet = $nestedadmember.PasswordLastSet

                    #save all in 1 storage
                    $resultHOlder += $table | select type, name, displayname, parentgroup, nesting, enabled, dn, comment , EmployeeNumber, LastLogonDate, PasswordLastSet
                } 


                # if member object is group
                elseif ($nestedmember.objectclass -eq "group") {  
                    $table = new-object psobject -Property $props 
                    # if circular, meaning the groups member of list contains one of its members. 
                    # e.g. if group 2 is a member of group 1 and group 1 is a member of grou 2
                    if ($memberof -contains $nestedmember.distinguishedname) { 
                        $table.comment = "Circular membership" 
                        $circular = 1 
                    } 
                    # for circular output
                    #$table | select type, name, displayname, parentgroup, nesting, enabled, dn, comment 
                    #calling function itself
                    $resultHOlder += Get-ADUsers -GroupName $nestedmember.distinguishedName                             
                } 
                else { 
                    if ($nestedmember) {
                        $table = new-object psobject -property $props
                        $resultHolder += $table | select type, name, displayname, parentgroup, nesting, enabled, dn, comment, EmployeeNumber, LastLogonDate, PasswordLastSet
                    }
                } 
            } 
        } 



    return   $resultHOlder

}
function Get-NestedGroupUsers {
    param ( 
        [Parameter(Mandatory = $true)][String]$FileName,
        [Parameter(Mandatory = $true)][String]$searchFileURL
    )
    $storageHolder = @()
    # $storageHolder | Export-Csv -Path "C:\Users\demandx\Desktop\AD User Lists\$FileName.csv" -NoTypeInformation -Force 
    $groupList = Get-Content $searchFileURL #| ForEach-Object { $_ }
    $groupList |  ForEach-Object { 
        $allusers = Get-ADUsers -GroupName $_
        $storageHolder += $allusers  
       
    }
    $storageHolder | select ParentGroup, Name, EmployeeNumber, Enabled, LastLogonDate, PasswordLastSet  |Export-Csv -Path "C:\Users\demandx\Desktop\$FileName.csv" -NoTypeInformation -Force
}



        # get members of the group and member of
        $ADGroupname = get-adgroup $groupname -properties memberof, members

        # list all members as list (no headers) and save to var
        $memberof = $adgroupname | select -expand memberof 
       
        if ($adgroupname) {  
            if ($circular) { 
                $nestedMembers = Get-ADGroupMember -Identity $GroupName -recursive 
                $circular = $null 
            } 
            else { 
                $nestedMembers = Get-ADGroupMember -Identity $GroupName | sort objectclass -Descending
                # if get adgroupmember returns nothing, it uses the members for ordinary getADGroup
                if (!($nestedmembers)) {
                    $unknown = $ADGroupname | select -expand members
                    if ($unknown) {
                        $nestedmembers = @()
                        foreach ($member in $unknown) {
                            $nestedmembers += get-adobject $member
                        }
                    }
                }
            } 
            # loops through each member
            ForEach($nestedmember in $nestedmembers){ 
                # creates the properties into a custom object. 
                $Props = @{
                    Type            = $nestedmember.objectclass;
                    Name            = $nestedmember.name;
                    DisplayName     = "";
                    ParentGroup     = $ADgroupname.name;
                    Enabled         = "";
                    Nesting         = $nesting;
                    DN              = $nestedmember.distinguishedname;
                    Comment         = ""
                    EmployeeNumber  = "";
                    LastLogonDate   = "";
                    PasswordLastSet = "";
                } 
                # if member object is a user
                if ($nestedmember.objectclass -eq "user") { 
                    # saves all the properties in the table. 
                    $nestedADMember = get-aduser $nestedmember.Name -properties enabled, displayname, EmployeeNumber, LastLogonDate, PasswordLastSet
                    $table = new-object psobject -property $props 
                    $table.enabled = $nestedadmember.enabled
                    $table.name = $nestedadmember.samaccountname
                    $table.displayname = $nestedadmember.displayname
                    $table.EmployeeNumber = $nestedadmember.EmployeeNumber
                    $table.LastLogonDate = $nestedadmember.LastLogonDate
                    $table.PasswordLastSet = $nestedadmember.PasswordLastSet

                    #save all in 1 storage
                    $resultHOlder += $table | select type, name, displayname, parentgroup, nesting, enabled, dn, comment , EmployeeNumber, LastLogonDate, PasswordLastSet
                } 


                # if member object is group
                elseif ($nestedmember.objectclass -eq "group") {  
                    $table = new-object psobject -Property $props 
                    # if circular, meaning the groups member of list contains one of its members. 
                    # e.g. if group 2 is a member of group 1 and group 1 is a member of grou 2
                    if ($memberof -contains $nestedmember.distinguishedname) { 
                        $table.comment = "Circular membership" 
                        $circular = 1 
                    } 
                    # for circular output
                    #$table | select type, name, displayname, parentgroup, nesting, enabled, dn, comment 
                    #calling function itself
                    $resultHOlder += Get-ADUsers -GroupName $nestedmember.distinguishedName                             
                } 
                else { 
                    if ($nestedmember) {
                        $table = new-object psobject -property $props
                        $resultHolder += $table | select type, name, displayname, parentgroup, nesting, enabled, dn, comment, EmployeeNumber, LastLogonDate, PasswordLastSet
                    }
                } 
            } 
        } 



    return   $resultHOlder

}

Run Code Online (Sandbox Code Playgroud)

并行结果

-------------------------------------------
Days              : 0
Hours             : 0
Minutes           : 1
Seconds           : 2
Milliseconds      : 283
Ticks             : 622833415
TotalDays         : 0.000720872008101852
TotalHours        : 0.0173009281944444
TotalMinutes      : 1.03805569166667
TotalSeconds      : 62.2833415
TotalMilliseconds : 62283.3415

Run Code Online (Sandbox Code Playgroud)

非平行结果

-------------------------------------------
Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 35
Milliseconds      : 322
Ticks             : 353221537
TotalDays         : 0.00040882122337963
TotalHours        : 0.00981170936111111
TotalMinutes      : 0.588702561666667
TotalSeconds      : 35.3221537
TotalMilliseconds : 35322.1537

Run Code Online (Sandbox Code Playgroud)

Answer 1

HAL*_*256 6

总而言之：

\n

有3个原因：

\n

为了充分利用ForEach-Object -Parallel性能，脚本块的处理时间需要明显大于设置线程和环境的时间。
这Import-Module将带来开销。
这两个因素单独来看都很小，但是乘以 1000 或更大的数字，它们就会变得很大。

\n

ForEach-Object -Parallel运行方式与正常运行方式非常不同ForEach-Object。

\n

首先，法线ForEach-Object在当前的 PowerShell 线程内运行，可以访问所有变量、加载的内存和管道。这对于我们运行的 98% 的作业来说都很好，1 秒的执行时间也还可以。在 2% 的情况下，我们有一个超级 CPU 密集型进程，它会耗尽单个 CPU 核心并永远运行，或者当其他执行可以发生时我们需要等待响应（例如 API 请求），那么我们就需要执行以下操作-Parallel：需要看看。

\n

并行执行背后的想法是利用具有 64 核/128 线程的全新 AMD Ryzen\xe2\x84\xa2 Threadripper\xe2\x84\xa2 3990X，并将进程拆分为可以跨多个运行的单独“作业” CPU 核心和多个线程同时存在。这可以将您的速度提高几个数量级，例如可能快 128 倍。

\n

为了实现这一点，ForEach-Object -Parallel为您执行的每个脚本块创建一个新的“作业”，并开始将作业分布在 CPU 核心上执行。当您有长时间运行的 CPU 密集型进程时，这非常有用，但是当您有非常短且小的作业时，您就会遇到并行执行的症结，其中设置比实际执行花费更多的时间。ForEach-Object -Parallel必须为您运行的每个“作业”完全设置环境，例如，它必须为每个要运行的作业启动多个新线程和多个新的 PowerShell 实例。

\n

为了说明所需的设置时间量，如果我们向当前线程写入一次“Hello World”，则需要 1 毫秒：

\n

PS C:\\> Measure-Command { Write-Host "Hello World" }\nHello World\n\nSeconds           : 0\nMilliseconds      : 1\nTotalMilliseconds : 1.9798\n

Run Code Online (Sandbox Code Playgroud)\n

并行运行 1 个“Hello World”需要 26 毫秒：

\n

PS C:\\> Measure-Command { 1 | ForEach-Object -Parallel { Write-Host "Hello World" } }\nHello World\n\nSeconds           : 0\nMilliseconds      : 26\nTotalMilliseconds : 26.052\n

Run Code Online (Sandbox Code Playgroud)\n

这意味着它花费了大约 25 毫秒来启动一个新线程并设置环境，以及 1 毫秒的实际工作时间。

\n

在当前运行的线程上写入 100 次大约需要 83ms：

\n

PS C:\\> Measure-Command { 1..100 | ForEach-Object { Write-Host "Hello World" } }\nHello World\n...\nHello World\nHello World\n\nSeconds           : 0\nMilliseconds      : 83\nTotalMilliseconds : 83.1846\n

Run Code Online (Sandbox Code Playgroud)\n

-Parallel使用a运行-ThrottleLimit 5需要 294ms：

\n

PS C:\\> Measure-Command { 1..100 | ForEach-Object -Parallel { Write-Host "Hello World" }  -ThrottleLimit 5 }\nHello World\n...\nHello World\nHello World\n\nSeconds           : 0\nMilliseconds      : 294\nTotalMilliseconds : 294.3205\n

Run Code Online (Sandbox Code Playgroud)\n

这表明并行运行对于微小的单个操作可能是不利的。但另一方面，如果你有一些需要 1 秒才能运行的东西，你可以开始看看它如何更好地工作：

\n

例如，运行 5 个进程，每个进程需要 1 秒。首先在单线程上：

\n

PS C:\\> Measure-Command { 1..5 | ForEach-Object { Start-Sleep -Seconds 1 } }\n\nSeconds           : 5\nMilliseconds      : 46\nTotalSeconds      : 5.046348\nTotalMilliseconds : 5046.348\n

Run Code Online (Sandbox Code Playgroud)\n

正如预期的那样，只需要 5 秒多一点的时间。现在，并行：

\n

PS C:\\> Measure-Command { 1..5 | ForEach-Object -Parallel { Start-Sleep -Seconds 1 } -ThrottleLimit 5 }\n\nSeconds           : 1\nMilliseconds      : 73\nTotalSeconds      : 1.0732423\nTotalMilliseconds : 1073.2423\n

Run Code Online (Sandbox Code Playgroud)\n

它在一秒钟多一点的时间内完成。如果处理时间明显多于设置时间，那么-Parallel很有用。

\n

此外，就您而言，不仅需要额外的设置时间开销，而且加载模块（设置新环境所需）也会显着增加版本的时间ForEach-Object -Parallel。

\n

例如，让我们在脚本中导入一个模块5AzureAD 次ForEach-Object：

\n

PS C:\\> Measure-Command { 1..5 | ForEach-Object { Import-Module AzureAD } }\n\nSeconds           : 0\nMilliseconds      : 18\nTotalSeconds      : 0.0185406\nTotalMilliseconds : 18.5406\n

Run Code Online (Sandbox Code Playgroud)\n

现在ForEach-Object -Parallel：

\n

PS C:\\> Measure-Command { 1..5 | ForEach-Object -Parallel { Import-Module AzureAD } -ThrottleLimit 5 }\n\nSeconds           : 0\nMilliseconds      : 125\nTotalSeconds      : 0.1256923\nTotalMilliseconds : 125.6923\n

Run Code Online (Sandbox Code Playgroud)\n

我们可以看到有一个显着的差异，因为它必须加载模块 5 次，而不是在线程内只加载一次，然后注意到它仍然加载，而不是重新加载它。

\n

归档时间：	5 年前
查看次数：	2696 次
最近记录：	5 年前