查询的性能调优

Question

寻求帮助以提高此查询性能。

SQL Server 2008 R2 Enterprise，最大 RAM 16 GB，CPU 40，最大并行度 4。

SELECT DsJobStat.JobName AS JobName
    , AJF.ApplGroup AS GroupName
    , DsJobStat.JobStatus AS JobStatus
    , AVG(CAST(DsJobStat.ElapsedSec AS FLOAT)) AS ElapsedSecAVG
    , AVG(CAST(DsJobStat.CpuMSec AS FLOAT)) AS CpuMSecAVG 
FROM DsJobStat, AJF 
WHERE DsJobStat.NumericOrderNo=AJF.OrderNo 
AND DsJobStat.Odate=AJF.Odate 
AND DsJobStat.JobName NOT IN( SELECT [DsAvg].JobName FROM [DsAvg] )         
GROUP BY DsJobStat.JobName
, AJF.ApplGroup
, DsJobStat.JobStatus
HAVING AVG(CAST(DsJobStat.ElapsedSec AS FLOAT)) <> 0;

执行消息，

(0 row(s) affected)
Table 'AJF'. Scan count 11, logical reads 45, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'DsAvg'. Scan count 2, logical reads 1926, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'DsJobStat'. Scan count 1, logical reads 3831235, physical reads 85, read-ahead reads 3724396, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.

(1 row(s) affected)

SQL Server Execution Times:
      CPU time = 67268 ms,  elapsed time = 90206 ms.

表的结构：

-- 212271023 rows
CREATE TABLE [dbo].[DsJobStat](
    [OrderID] [nvarchar](8) NOT NULL,
    [JobNo] [int] NOT NULL,
    [Odate] [datetime] NOT NULL,
    [TaskType] [nvarchar](255) NULL,
    [JobName] [nvarchar](255) NOT NULL,
    [StartTime] [datetime] NULL,
    [EndTime] [datetime] NULL,
    [NodeID] [nvarchar](255) NULL,
    [GroupName] [nvarchar](255) NULL,
    [CompStat] [int] NULL,
    [RerunCounter] [int] NOT NULL,
    [JobStatus] [nvarchar](255) NULL,
    [CpuMSec] [int] NULL,
    [ElapsedSec] [int] NULL,
    [StatusReason] [nvarchar](255) NULL,
    [NumericOrderNo] [int] NULL,
CONSTRAINT [PK_DsJobStat] PRIMARY KEY CLUSTERED 
(   [OrderID] ASC,
    [JobNo] ASC,
    [Odate] ASC,
    [JobName] ASC,
    [RerunCounter] ASC
));

-- 48992126 rows
CREATE TABLE [dbo].[AJF](  
    [JobName] [nvarchar](255) NOT NULL,
    [JobNo] [int] NOT NULL,
    [OrderNo] [int] NOT NULL,
    [Odate] [datetime] NOT NULL,
    [SchedTab] [nvarchar](255) NULL,
    [Application] [nvarchar](255) NULL,
    [ApplGroup] [nvarchar](255) NULL,
    [GroupName] [nvarchar](255) NULL,
    [NodeID] [nvarchar](255) NULL,
    [Memlib] [nvarchar](255) NULL,
    [Memname] [nvarchar](255) NULL,
    [CreationTime] [datetime] NULL,
CONSTRAINT [AJF$PrimaryKey] PRIMARY KEY CLUSTERED 
(   [JobName] ASC,
    [JobNo] ASC,
    [OrderNo] ASC,
    [Odate] ASC
));

-- 413176 rows
CREATE TABLE [dbo].[DsAvg](
    [JobName] [nvarchar](255) NULL,
    [GroupName] [nvarchar](255) NULL,
    [JobStatus] [nvarchar](255) NULL,
    [ElapsedSecAVG] [float] NULL,
    [CpuMSecAVG] [float] NULL
);

CREATE NONCLUSTERED INDEX [DJS_Dashboard_2] ON [dbo].[DsJobStat] 
(   [JobName] ASC,
    [Odate] ASC,
    [StartTime] ASC,
    [EndTime] ASC
)
INCLUDE ( [OrderID],
[JobNo],
[NodeID],
[GroupName],
[JobStatus],
[CpuMSec],
[ElapsedSec],
[NumericOrderNo]) ;

CREATE NONCLUSTERED INDEX [Idx_Dashboard_AJF] ON [dbo].[AJF] 
(   [OrderNo] ASC,
[Odate] ASC
)
INCLUDE ( [SchedTab],
[Application],
[ApplGroup]) ;

CREATE NONCLUSTERED INDEX [DsAvg$JobName] ON [dbo].[DsAvg] 
(   [JobName] ASC
)

执行计划：

https://www.brentozar.com/pastetheplan/?id=rkUVhMlXM

---

得到答复后更新

非常感谢@Joe Obbish

你对这个查询的问题是正确的，它是关于 DsJobStat 和 DsAvg 之间的。关于如何加入和不使用 NOT IN 并不多。

确实有你猜到的桌子。

CREATE TABLE [dbo].[DSJobNames](
    [JobName] [nvarchar](255) NOT NULL,
 CONSTRAINT [DSJobNames$PrimaryKey] PRIMARY KEY CLUSTERED 
(   [JobName] ASC
) ); 

我试过你的建议，

SELECT DsJobStat.JobName AS JobName
, AJF.ApplGroup AS GroupName
, DsJobStat.JobStatus AS JobStatus
, AVG(CAST(DsJobStat.ElapsedSec AS FLOAT)) AS ElapsedSecAVG
, Avg(CAST(DsJobStat.CpuMSec AS FLOAT)) AS CpuMSecAVG 
FROM DsJobStat
INNER JOIN DSJobNames jn
    ON jn.[JobName]= DsJobStat.[JobName]
INNER JOIN AJF 
    ON DsJobStat.Odate=AJF.Odate 
    AND DsJobStat.NumericOrderNo=AJF.OrderNo 
WHERE NOT EXISTS ( SELECT 1 FROM [DsAvg] WHERE jn.JobName =  [DsAvg].JobName )      
GROUP BY DsJobStat.JobName, AJF.ApplGroup, DsJobStat.JobStatus
HAVING AVG(CAST(DsJobStat.ElapsedSec AS FLOAT)) <> 0;   

执行消息：

(0 row(s) affected)
Table 'DSJobNames'. Scan count 5, logical reads 1244, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'DsAvg'. Scan count 5, logical reads 2129, physical reads 0, read-ahead reads 24, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'DsJobStat'. Scan count 8, logical reads 84, physical reads 0, read-ahead reads 83, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'Worktable'. Scan count 0, logical reads 0, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'AJF'. Scan count 5, logical reads 757999, physical reads 944, read-ahead reads 757311, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'Worktable'. Scan count 0, logical reads 0, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.

(1 row(s) affected)

 SQL Server Execution Times:
   CPU time = 21776 ms,  elapsed time = 33984 ms.

执行计划：https : //www.brentozar.com/pastetheplan/?id=rJVkLSZ7f

Answer 1

让我们从考虑连接顺序开始。查询中有三个表引用。哪种连接顺序可以为您提供最佳性能？查询优化器认为连接 from DsJobStattoDsAvg将消除几乎所有行（基数估计从 212195000 下降到 1 行）。实际计划向我们表明估计非常接近现实（11 行在连接中幸存下来）。但是，连接是作为右反半合并连接实现的，因此DsJobStat扫描表中的所有 2.12 亿行只是为了生成 11 行。这肯定会导致较长的查询执行时间，但我想不出更好的连接的物理或逻辑运算符会更好。我确定DJS_Dashboard_2index 用于其他查询，但所有额外的键和包含的列将只需要为此查询更多的 IO 并减慢您的速度。因此，您可能会遇到表上索引扫描的表访问问题DsJobStat。

我将假设 join toAJF不是很有选择性。它目前与您在查询中看到的性能问题无关，因此我将在本答案的其余部分忽略它。如果表中的数据发生变化，这可能会发生变化。

从计划中显而易见的另一个问题是行计数假脱机操作符。这是一个非常轻量级的操作符，但它的执行次数超过 2 亿次。运算符在那里，因为查询是用NOT IN. 如果其中只有一个 NULL 行，DsAvg则必须消除所有行。线轴是该检查的实现。这可能不是您想要的逻辑，因此最好编写该部分以使用NOT EXISTS. 重写的实际好处将取决于您的系统和数据。

我根据查询计划模拟了一些数据来测试一些查询重写。我的表定义与您的有很大不同，因为为每一列模拟数据会花费太多精力。即使使用缩写的数据结构，我也能够重现您遇到的性能问题。

CREATE TABLE [dbo].[DsAvg](
    [JobName] [nvarchar](255) NULL
);

CREATE CLUSTERED INDEX CI_DsAvg ON [DsAvg] (JobName);

INSERT INTO [DsAvg] WITH (TABLOCK)
SELECT TOP (200000) ROW_NUMBER() OVER (ORDER BY (SELECT NULL))
FROM master..spt_values t1
CROSS JOIN master..spt_values t2
OPTION (MAXDOP 1);

CREATE TABLE [dbo].[DsJobStat](
    [JobName] [nvarchar](255) NOT NULL,
    [JobStatus] [nvarchar](255) NULL,
);

CREATE CLUSTERED INDEX CI_JobStat ON DsJobStat (JobName)

INSERT INTO [DsJobStat] WITH (TABLOCK)
SELECT [JobName], 'ACTIVE'
FROM [DsAvg] ds
CROSS JOIN (
SELECT TOP (1000) 1
FROM master..spt_values t1
) c (t);

INSERT INTO [DsJobStat] WITH (TABLOCK)
SELECT TOP (1000) '200001', 'ACTIVE'
FROM master..spt_values t1;

根据查询计划，我们可以看到表中大约有 200000 个唯一JobName值DsAvg。根据连接到该表后的实际行数，我们可以看到几乎所有的JobName值DsJobStat也在DsAvg表中。因此，该DsJobStat表的JobName列有 200001 个唯一值，每个值有 1000 行。

我相信这个查询代表了性能问题：

SELECT DsJobStat.JobName AS JobName, DsJobStat.JobStatus AS JobStatus
FROM DsJobStat
WHERE DsJobStat.JobName NOT IN( SELECT [DsAvg].JobName FROM [DsAvg] );

查询计划中的所有其他内容（GROUP BY、、HAVING古代样式连接等）都在结果集减少到 11 行之后发生。从查询性能的角度来看，目前这无关紧要，但可能存在其他问题，这些问题可能会通过表中更改的数据来揭示。

我正在 SQL Server 2017 中进行测试，但我得到的基本计划形状与您相同：

在我的机器上，该查询需要 62219 毫秒的 CPU 时间和 65576 毫秒的运行时间来执行。如果我重写查询以使用NOT EXISTS：

SELECT DsJobStat.JobName AS JobName, DsJobStat.JobStatus AS JobStatus
FROM DsJobStat
WHERE NOT EXISTS (SELECT 1 FROM [DsAvg] WHERE DsJobStat.JobName = [DsAvg].JobName);

线轴不再执行 2.12 亿次，它可能具有供应商的预期行为。现在查询在 34516 毫秒的 CPU 时间和 41132 毫秒的运行时间中执行。大部分时间用于扫描索引中的 2.12 亿行。

该索引扫描对于该查询来说非常不幸。平均每个 的唯一值有 1000 行JobName，但在读取第一行后我们知道是否需要前面的 1000 行。我们几乎从不需要这些行，但无论如何我们仍然需要扫描它们。如果我们知道表中的行不是很密集，并且几乎所有行都将被连接消除，我们可以想象索引上可能更有效的 IO 模式。如果 SQL Server 读取 的每个唯一值的第一行JobName，检查该值是否在 中DsAvg，然后直接跳到下一个值（JobName如果是）呢？不需要扫描 2.12 亿行，而是可以执行需要大约 20 万次执行的搜索计划。

这主要可以通过使用递归以及此处描述的 Paul White 首创的技术来实现。我们可以使用递归来完成我上面描述的 IO 模式：

WITH RecursiveCTE
AS
(
    -- Anchor
    SELECT TOP (1)
        [JobName]
    FROM dbo.DsJobStat AS T
    ORDER BY
        T.[JobName]

    UNION ALL

    -- Recursive
    SELECT R.[JobName]
    FROM
    (
        -- Number the rows
        SELECT 
            T.[JobName],
            rn = ROW_NUMBER() OVER (
                ORDER BY T.[JobName])
        FROM dbo.DsJobStat AS T
        JOIN RecursiveCTE AS R
            ON R.[JobName] < T.[JobName]
    ) AS R
    WHERE
        -- Only the row that sorts lowest
        R.rn = 1
)
SELECT js.*
FROM RecursiveCTE
INNER JOIN dbo.DsJobStat js ON RecursiveCTE.[JobName]= js.[JobName]
WHERE NOT EXISTS (SELECT 1 FROM [DsAvg] WHERE RecursiveCTE.JobName = [DsAvg].JobName)
OPTION (MAXRECURSION 0);

该查询需要查看的内容很多，因此我建议仔细检查实际计划。首先，我们对索引执行 200002 索引查找DsJobStat以获取所有唯一JobName值。然后我们加入DsAvg并消除除一行之外的所有行。对于剩余的行，返回DsJobStat并获取所有必需的列。

IO 模式完全改变。在我们得到这个之前：

表'DsJobStat'。扫描计数 1，逻辑读取 1091651，物理读取 13836，预读读取 181966

通过递归查询，我们得到：

表'DsJobStat'。扫描计数 200003，逻辑读 1398000，物理读 1，预读 7345

在我的机器上，新查询仅在 6891 毫秒的 CPU 时间和 7107 毫秒的运行时间中执行。请注意，需要以这种方式使用递归表明数据模型中缺少某些内容（或者可能只是在发布的问题中未说明）。如果有一个包含所有可能的相对较小的表，JobNames那么使用该表比在大表上递归要好得多。它归结为如果您有一个包含所有JobNames您需要的结果集，那么您可以使用索引搜索来获取其余的缺失列。但是，您不能使用不需要的结果集来做到这JobNames一点。