PowerShell脚本读取文件的性能太慢

Question

我目前正在开发一个PowerShell脚本,该脚本将作为构建步骤的一部分在TeamCity中使用.该脚本必须:

• 递归检查文件夹中具有特定扩展名(.item)的所有文件,
• 读取每个文件的第三行(包含GUID)并检查这些行中是否有任何重复项,
• 记录包含重复GUID的文件的路径并记录GUID本身,
• 如果找到一个或多个重复项,则使TeamCity构建失败

我是PowerShell脚本的新手,但到目前为止,我已经做了一些我期望它做的事情:

Write-Host "Start checking for Unicorn serialization errors."

$files = get-childitem "%system.teamcity.build.workingDir%\Sitecore\serialization" -recurse -include *.item | where {! $_.PSIsContainer} | % { $_.FullName }
$arrayOfItemIds = @()
$NrOfFiles = $files.Length
[bool] $FoundDuplicates = 0

Write-Host "There are $NrOfFiles Unicorn item files to check."

foreach ($file in $files)
{
    $thirdLineOfFile = (Get-Content $file)[2 .. 2]

    if ($arrayOfItemIds -contains $thirdLineOfFile)
    {
        $FoundDuplicates = 1
        $itemId = $thirdLineOfFile.Split(":")[1].Trim()

        Write-Host "Duplicate item ID found!"
        Write-Host "Item file path: $file"
        Write-Host "Detected duplicate ID: $itemId"
        Write-Host "-------------"
        Write-Host ""
    }
    else
    {
        $arrayOfItemIds += $thirdLineOfFile
    }
}

if ($foundDuplicates)
{
    "##teamcity[buildStatus status='FAILURE' text='One or more duplicate ID's were detected in Sitecore serialised items. Check the build log to see which files and ID's are involved.']"
    exit 1
}

Write-Host "End script checking for Unicorn serialization errors."

问题是:它很慢!此脚本必须检查的文件夹当前包含超过14.000个.item文件,并且该数量很可能在将来继续增加.我知道打开和读取这么多文件是一个广泛的操作,但我没想到它需要大约半个小时才能完成.这太长了,因为这意味着每个(快照)构建的构建时间将延长半个小时,这是不可接受的.我原本希望剧本能在几分钟内完成.

我不可能相信没有更快的方法来做到这一点..所以在这方面的任何帮助都非常感谢!

解

我不得不说到目前为止我收到的所有3个答案都帮助了我.我首先开始直接使用.NET框架类,然后使用字典来解决不断增长的数组问题.运行我自己的脚本所花费的时间大约是30分钟,然后使用.NET框架类只需要2分钟.使用词典解决方案后,它也只有6或7秒!我使用的最终脚本:

Write-Host "Start checking for Unicorn serialization errors."

[String[]] $allFilePaths = [System.IO.Directory]::GetFiles("%system.teamcity.build.workingDir%\Sitecore\serialization", "*.item", "AllDirectories")
$IdsProcessed = New-Object 'system.collections.generic.dictionary[string,string]'
[bool] $FoundDuplicates = 0
$NrOfFiles = $allFilePaths.Length

Write-Host "There are $NrOfFiles Unicorn item files to check."
Write-Host ""

foreach ($filePath in $allFilePaths)
{
    [System.IO.StreamReader] $sr = [System.IO.File]::OpenText($filePath)
    $unused1 = $sr.ReadLine() #read the first unused line
    $unused2 = $sr.ReadLine() #read the second unused line
    [string]$thirdLineOfFile = $sr.ReadLine()
    $sr.Close()

    if ($IdsProcessed.ContainsKey($thirdLineOfFile))
    {
        $FoundDuplicates = 1
        $itemId = $thirdLineOfFile.Split(":")[1].Trim()
        $otherFileWithSameId = $IdsProcessed[$thirdLineOfFile]

        Write-Host "---------------"
        Write-Host "Duplicate item ID found!"
        Write-Host "Detected duplicate ID: $itemId"
        Write-Host "Item file path 1: $filePath"
        Write-Host "Item file path 2: $otherFileWithSameId"
        Write-Host "---------------"
        Write-Host ""
    }
    else
    {
        $IdsProcessed.Add($thirdLineOfFile, $filePath)
    }
}

if ($foundDuplicates)
{
    "##teamcity[buildStatus status='FAILURE' text='One or more duplicate ID|'s were detected in Sitecore serialised items. Check the build log to see which files and ID|'s are involved.']"
    exit 1
}

Write-Host "End script checking for Unicorn serialization errors. No duplicate ID's were found."

谢谢大家!

Answer 1

尝试更换Get-Content用[System.IO.File]::ReadLines.如果这仍然太慢,请考虑使用System.IO.StreamReader- 这会导致您编写更多代码但允许您只读取前3行.