MySQL:避免由GROUP BY子句引起的临时/文件排序

Question

我有一个相当简单的查询,试图显示订阅的电子邮件地址的数量以及取消订阅的数字,按客户分组.

查询:

SELECT
    client_id,
    COUNT(CASE WHEN subscribed = 1 THEN subscribed END) AS subs,
    COUNT(CASE WHEN subscribed = 0 THEN subscribed END) AS unsubs
FROM
    contacts_emailAddresses
LEFT JOIN contacts ON contacts.id = contacts_emailAddresses.contact_id
GROUP BY
    client_id

下面是相关表格的模式.contacts_emailAddresses是联系人(具有client_id)和emailAddresses(在此查询中实际未使用)之间的联结表.

CREATE TABLE `contacts` (
  `id` int(11) unsigned NOT NULL AUTO_INCREMENT,
  `firstname` varchar(255) NOT NULL DEFAULT '',
  `middlename` varchar(255) NOT NULL DEFAULT '',
  `lastname` varchar(255) NOT NULL DEFAULT '',
  `gender` varchar(5) DEFAULT NULL,
  `client_id` mediumint(10) unsigned DEFAULT NULL,
  `datasource` varchar(10) DEFAULT NULL,
  `external_id` int(10) unsigned DEFAULT NULL,
  `created` timestamp NULL DEFAULT NULL,
  `trash` tinyint(1) NOT NULL DEFAULT '0',
  `updated` timestamp NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  PRIMARY KEY (`id`),
  KEY `client_id` (`client_id`),
  KEY `external_id combo` (`client_id`,`datasource`,`external_id`),
  KEY `trash` (`trash`),
  KEY `lastname` (`lastname`),
  KEY `firstname` (`firstname`),
  CONSTRAINT `contacts_ibfk_1` FOREIGN KEY (`client_id`) REFERENCES `clients` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=14742974 DEFAULT CHARSET=utf8 ROW_FORMAT=COMPACT

CREATE TABLE `contacts_emailAddresses` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `contact_id` int(10) unsigned NOT NULL,
  `emailAddress_id` int(11) unsigned DEFAULT NULL,
  `primary` tinyint(1) unsigned NOT NULL DEFAULT '0',
  `subscribed` tinyint(1) unsigned NOT NULL DEFAULT '1',
  `modified` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  PRIMARY KEY (`id`),
  KEY `contact_id` (`contact_id`),
  KEY `subscribed` (`subscribed`),
  KEY `combo` (`contact_id`,`emailAddress_id`) USING BTREE,
  KEY `emailAddress_id` (`emailAddress_id`) USING BTREE,
  CONSTRAINT `contacts_emailAddresses_ibfk_1` FOREIGN KEY (`contact_id`) REFERENCES `contacts` (`id`),
  CONSTRAINT `contacts_emailAddresses_ibfk_2` FOREIGN KEY (`emailAddress_id`) REFERENCES `emailAddresses` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=24700918 DEFAULT CHARSET=utf8

这是EXPLAIN:

+----+-------------+-------------------------+--------+---------------+---------+---------+-------------------------------------------+----------+---------------------------------+
| id | select_type | table                   | type   | possible_keys | key     | key_len | ref                                       | rows     | Extra                           |
+----+-------------+-------------------------+--------+---------------+---------+---------+-------------------------------------------+----------+---------------------------------+
| 1  | SIMPLE      | contacts_emailAddresses | ALL    | NULL          | NULL    | NULL    | NULL                                      | 10176639 | Using temporary; Using filesort |
| 1  | SIMPLE      | contacts                | eq_ref | PRIMARY       | PRIMARY | 4       | icarus.contacts_emailAddresses.contact_id | 1        |                                 |
+----+-------------+-------------------------+--------+---------------+---------+---------+-------------------------------------------+----------+---------------------------------+
2 rows in set (0.08 sec)

这里的问题显然是GROUP BY子句,因为我可以删除JOIN(以及依赖它的项目)并且性能仍然很糟糕(40+秒).contacts_emailAddresses中有10m记录,联系人中有12m记录,分组中有10-15个客户记录.

来自doc:

可以在以下条件下创建临时表:

如果存在ORDER BY子句和不同的GROUP BY子句,或者ORDER BY或GROUP BY包含连接队列中第一个表以外的表中的列,则会创建临时表.

DISTINCT与ORDER BY结合使用可能需要临时表.

如果使用SQL_SMALL_RESULT选项,MySQL将使用内存中的临时表,除非查询还包含需要磁盘存储的元素(稍后描述).

我显然没有将GROUP BY与ORDER BY结合起来,我尝试了多种方法来确保GROUP BY位于应该正确放置在连接队列中的列上(包括重写查询以将联系人放入FROM中)而是加入到contacts_emailAddresses),一切都无济于事.

任何性能调整的建议将非常感谢!

Answer 1

我认为你唯一能够摆脱"使用临时;使用文件排序"操作(给定当前模式,当前查询和指定结果集)的实际镜头将是在SELECT列表中使用相关子查询.

SELECT c.client_id
     , (SELECT IFNULL(SUM(es.subscribed=1),0)
          FROM contacts_emailAddresses es
          JOIN contacts cs
            ON cs.id = es.contact_id
         WHERE cs.client_id = c.client_id
       ) AS subs
     , (SELECT IFNULL(SUM(eu.subscribed=0),0)
          FROM contacts_emailAddresses eu
          JOIN contacts cu
            ON cu.id = eu.contact_id
         WHERE cu.client_id = c.client_id
       ) AS unsubs
  FROM contacts c
 GROUP BY c.client_id

这可能比原始查询运行得更快,或者可能不会.这些相关的子查询将为外部查询返回的每个子查询运行.如果那个外部查询返回了一大堆行,那就是一大堆子查询执行.

这是一个输出EXPLAIN:

id  select_type        table type  possible_keys                       key        key_len  ref   Extra
--  ------------------ ----- ----- ----------------------------------- ---------- ------- ------ ------------------------
 1  PRIMARY            c     index (NULL)                              client_id  5       (NULL) Using index
 3  DEPENDENT SUBQUERY cu    ref   PRIMARY,client_id,external_id combo client_id  5       func   Using where; Using index
 3  DEPENDENT SUBQUERY eu    ref   contact_id,combo                    contact_id 4       cu.id  Using where
 2  DEPENDENT SUBQUERY cs    ref   PRIMARY,client_id,external_id combo client_id  5       func   Using where; Using index
 2  DEPENDENT SUBQUERY es    ref   contact_id,combo                    contact_id 4       cs.id  Using where

为了获得此查询的最佳性能,我们非常希望在eu和es表的说明的Extra列中看到"Using index" .但要获得的是,我们需要一个合适的指数,一个与领先的列contact_id和包括subscribed列.例如:

CREATE INDEX cemail_IX2 ON contacts_emailAddresses (contact_id, subscribed);

随着新索引的可用,EXPLAIN输出显示MySQL将使用新索引:

id  select_type        table type  possible_keys                       key        key_len ref    Extra                     
--  ------------------ ----- ----- ----------------------------------- ---------- ------- ------ ------------------------
 1  PRIMARY            c     index (NULL)                              client_id  5       (NULL) Using index
 3  DEPENDENT SUBQUERY cu    ref   PRIMARY,client_id,external_id combo client_id  5       func   Using where; Using index
 3  DEPENDENT SUBQUERY eu    ref   contact_id,combo,cemail_IX2         cemail_IX2 4       cu.id  Using where; Using index
 2  DEPENDENT SUBQUERY cs    ref   PRIMARY,client_id,external_id combo client_id  5       func   Using where; Using index
 2  DEPENDENT SUBQUERY es    ref   contact_id,combo,cemail_IX2         cemail_IX2 4       cs.id  Using where; Using index

笔记

这是一种引入少量冗余可以提高性能的问题.(就像我们在传统的数据仓库中一样.)

为了获得最佳性能,我们真正喜欢的是client_id在contacts_emailAddresses表中提供列,而无需JOINI到contacts表.

在当前模式中,与contacts表的外键关系得到了我们client_id(相反,原始查询中的JOIN操作是为我们提供的.)如果我们可以完全避免该JOIN操作,我们可以完全从单个查询中满足查询index,使用索引进行聚合,并避免"使用临时;使用filesort"和JOIN操作的开销......

在client_id列可用的情况下,我们将创建一个覆盖索引,如...

... ON contacts_emailAddresses (client_id, subscribed)

然后,我们有一个非常快速的查询......

SELECT e.client_id
     , SUM(e.subscribed=1) AS subs
     , SUM(e.subscribed=0) AS unsubs
  FROM contacts_emailAddresses e
GROUP BY e.client_id

这将使我们在查询计划中获得"使用索引",并且此结果集的查询计划没有比这更好.

但是,这需要更改你的scheam,它并没有真正回答你的问题.

如果没有client_id专栏,那么我们可能做的最好的事情就是像Gordon在他的回答中发布的那样的查询(尽管你仍然需要添加GROUP BY c.client_id以获得指定的结果.)Gordon推荐的指数将有益...

... ON contacts_emailAddresses(contact_id, subscribed)

定义了该索引后,contact_id上的独立索引是多余的.新索引将是支持现有外键约束的合适替代.(刚才contact_id可以删除索引.)

另一种方法是在执行JOIN之前首先在"大"表上进行聚合,因为它是外连接的驱动表.实际上,由于该外键列被定义为NOT NULL,并且有一个外键,它根本不是一个"外部"连接.

SELECT c.client_id
     , SUM(s.subs) AS subs
     , SUM(s.unsubs) AS unsubs 
  FROM ( SELECT e.contact_id
              , SUM(e.subscribed=1) AS subs
              , SUM(e.eubscribed=0) AS unsubs
           FROM contacts_emailAddresses e
          GROUP BY e.contact_id
       ) s
 JOIN contacts c
   ON c.id = s.contact_id
GROUP BY c.client_id

同样,我们需要一个索引contact_id作为前导列并包含subscribed列,以获得最佳性能.(s应该显示"使用索引" 的计划.)不幸的是,这仍然会将相当大的结果集(派生表s)实现为临时MyISAM表,并且MyISAM表不会被索引.