oli*_*ive 3 mysql sql performance join
Really hoping that some kind of performance whiz can explain to me why a single join is causing the query to become 10x slower. (Also, please don't laugh about the size of this query! I wanted to get out the whole catalog in my database to output with one query. I'm not sure if it would be faster to break it into smaller queries but that doesn't seem right.)
SELECT `c`.`categoryID`,
`cl`.`name` AS `category_name`,
`v`.*,
TRUE AS `categoried`,
GROUP_CONCAT(DISTINCT t_v.iso_3166_1_alpha_2) AS `video_territories`,
GROUP_CONCAT(DISTINCT t_c.iso_3166_1_alpha_2) AS `category_territories`,
`vl`.*,
GROUP_CONCAT(DISTINCT kl.name) AS `keywords`
FROM `tblCategories` AS `c`
INNER JOIN `tblCategoryLocalisedData` AS `cl` ON c.categoryID = cl.categoryID
LEFT JOIN `tblCategoryDurations` AS `cd` ON c.categoryID = cd.categoryID
LEFT JOIN `tblCategoryRules` AS `cr` ON c.categoryID = cr.categoryID
LEFT JOIN `tblCategoryVideos` AS `cv` ON c.categoryID = cv.categoryID
LEFT JOIN `tblVideos` AS `v` ON cv.videoID = v.videoID
LEFT JOIN `tblVideoTerritories` AS `vt` ON vt.videoID = v.videoID
LEFT JOIN `tblCategoryTerritories` AS `ct` ON ct.categoryID = c.categoryID
INNER JOIN `tblTerritories` AS `t_v` ON t_v.territoryID = vt.territoryID
INNER JOIN `tblTerritories` AS `t_c` ON t_c.territoryID = ct.territoryID
INNER JOIN `tblVideoLocalisedData` AS `vl` ON vl.videoID = v.videoID
LEFT JOIN `tblVideoKeywords` AS `vk` ON v.videoID = vk.videoID
LEFT JOIN `tblKeywords` AS `k` ON vk.keywordID = k.keywordID
LEFT JOIN `tblKeywordLocalisedData` AS `kl` ON kl.keywordID = k.keywordID
INNER JOIN `tblLanguages` AS `l`
WHERE (cv.disabled IS NULL)
AND (cd.start_date < NOW() OR cd.start_date IS NULL)
AND (cd.end_date > NOW() OR cd.end_date IS NULL)
AND (cr.name IS NULL)
AND (l.languageID = cl.languageID OR cl.languageID IS NULL)
AND (l.languageID = kl.languageID OR kl.languageID IS NULL)
AND (l.languageID = vl.languageID OR vl.languageID IS NULL)
AND (l.iso_639_1 = 'en')
GROUP BY `v`.`videoID`, `c`.`categoryID`
ORDER BY `c`.`categoryID` ASC
When I run the above query it takes 1 whole second to finish. I tried running an EXPLAIN on it and it gave me this:
+----+-------------+-------+--------+--------------------------------------------------------------------------------------+-----------------------------------------+---------+------------------------+------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+--------------------------------------------------------------------------------------+-----------------------------------------+---------+------------------------+------+----------------------------------------------+
| 1 | SIMPLE | cv | ALL | fk_tblCategoryVideos_tblCategories1,fk_tblCategoryVideos_tblVideos1 | NULL | NULL | NULL | 2 | Using where; Using temporary; Using filesort |
| 1 | SIMPLE | c | eq_ref | PRIMARY | PRIMARY | 4 | db.cv.categoryID | 1 | Using index |
| 1 | SIMPLE | cd | ref | fk_tblCategoryDurations_tblCategories | fk_tblCategoryDurations_tblCategories | 4 | db.cv.categoryID | 1 | Using where |
| 1 | SIMPLE | cr | ref | fk_tblCategoryRules_tblCategories1 | fk_tblCategoryRules_tblCategories1 | 4 | db.cv.categoryID | 1 | Using where; Not exists |
| 1 | SIMPLE | vt | ref | fk_tblVideoTerritories_tblVideos1,fk_tblVideoTerritories_tblTerritories1 | fk_tblVideoTerritories_tblVideos1 | 4 | db.cv.videoID | 1 | Using where |
| 1 | SIMPLE | t_v | eq_ref | PRIMARY | PRIMARY | 4 | db.vt.territoryID | 1 | |
| 1 | SIMPLE | v | eq_ref | PRIMARY | PRIMARY | 4 | db.vt.videoID | 1 | Using where |
| 1 | SIMPLE | vk | ref | fk_tblVideoKeywords_tblVideos1 | fk_tblVideoKeywords_tblVideos1 | 4 | db.cv.videoID | 6 | |
| 1 | SIMPLE | k | eq_ref | PRIMARY | PRIMARY | 4 | db.vk.keywordID | 1 | Using index |
| 1 | SIMPLE | kl | ref | fk_tblKeywordLocalisedData_tblKeywords1 | fk_tblKeywordLocalisedData_tblKeywords1 | 4 | db.k.keywordID | 1 | |
| 1 | SIMPLE | cl | ALL | fk_tblCategoryLocalisedData_tblCategories1,fk_tblCategoryLocalisedData_tblLanguages1 | NULL | NULL | NULL | 5 | Using where; Using join buffer |
| 1 | SIMPLE | l | eq_ref | PRIMARY | PRIMARY | 4 | db.cl.languageID | 1 | Using where |
| 1 | SIMPLE | ct | ALL | fk_tblCategoryTerritories_tblCategories1,fk_tblCategoryTerritories_tblTerritories1 | NULL | NULL | NULL | 2 | Using where; Using join buffer |
| 1 | SIMPLE | vl | ALL | fk_tblVideoLocalisedData_tblLanguages1,fk_tblVideoLocalisedData_tblVideos1 | NULL | NULL | NULL | 9 | Using where; Using join buffer |
| 1 | SIMPLE | t_c | eq_ref | PRIMARY | PRIMARY | 4 | db.ct.territoryID | 1 | |
+----+-------------+-------+--------+--------------------------------------------------------------------------------------+-----------------------------------------+---------+------------------------+------+----------------------------------------------+
However I have no idea what that means. How do I troubleshoot this? I do, thankfully, know which parts of the query cause the massive slow-down. If I remove the joins from either tblVideoTerritories (vt) to tblTerritories (t_v) or tblCategoryTerritories (ct) to tblTerritories (t_c) then everything speeds up considerably. I thought to begin with it might have been because of the GROUP_CONCAT or DISTINCT but I tried removing these and it made barely any change. It seems that the performance problem is caused by joining to the same table 'tblTerritories' twice. If I only have one of those joins the query only takes 0.1 seconds or 0.2 seconds to run -- this is still a long time but it's a better start!
我想知道的是如何解决这个性能问题？为什么两次加入同一个表会导致查询占用10倍的时间？!
谢谢你的帮助!
编辑: tblVideoTerritories上的SHOW CREATE TABLE给了我:
CREATE TABLE `tblVideoTerritories` (
`videoTerritoryID` int(10) unsigned NOT NULL AUTO_INCREMENT,
`videoID` int(10) unsigned NOT NULL,
`territoryID` int(10) unsigned NOT NULL,
PRIMARY KEY (`videoTerritoryID`),
KEY `fk_tblVideoTerritories_tblVideos1` (`videoID`),
KEY `fk_tblVideoTerritories_tblTerritories1` (`territoryID`),
CONSTRAINT `fk_tblVideoTerritories_tblTerritories1` FOREIGN KEY (`territoryID`) REFERENCES `tblTerritories` (`territoryID`) ON DELETE NO ACTION ON UPDATE NO ACTION,
CONSTRAINT `fk_tblVideoTerritories_tblVideos1` FOREIGN KEY (`videoID`) REFERENCES `tblVideos` (`videoID`) ON DELETE NO ACTION ON UPDATE NO ACTION
) ENGINE=InnoDB AUTO_INCREMENT=8 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
关于tblCategoryTerritories的SHOW CREATE TABLE给了我:
CREATE TABLE `tblCategoryTerritories` (
`categoryTerritoryID` int(10) unsigned NOT NULL AUTO_INCREMENT,
`categoryID` int(10) unsigned NOT NULL,
`territoryID` int(10) unsigned NOT NULL,
PRIMARY KEY (`categoryTerritoryID`),
KEY `fk_tblCategoryTerritories_tblCategories1` (`categoryID`),
KEY `fk_tblCategoryTerritories_tblTerritories1` (`territoryID`),
CONSTRAINT `fk_tblCategoryTerritories_tblCategories1` FOREIGN KEY (`categoryID`) REFERENCES `tblCategories` (`categoryID`) ON DELETE NO ACTION ON UPDATE NO ACTION,
CONSTRAINT `fk_tblCategoryTerritories_tblTerritories1` FOREIGN KEY (`territoryID`) REFERENCES `tblTerritories` (`territoryID`) ON DELETE NO ACTION ON UPDATE NO ACTION
) ENGINE=InnoDB AUTO_INCREMENT=6 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
关于tblTerritories的SHOW CREATE TABLE给了我:
CREATE TABLE `tblTerritories` (
`territoryID` int(10) unsigned NOT NULL AUTO_INCREMENT,
`iso_3166_1_alpha_2` char(2) COLLATE utf8_unicode_ci DEFAULT NULL,
`iso_3166_1_alpha_3` char(3) COLLATE utf8_unicode_ci DEFAULT NULL,
`defaultLanguageID` int(10) unsigned DEFAULT NULL,
PRIMARY KEY (`territoryID`),
KEY `fk_tblTerritories_tblLanguages1` (`defaultLanguageID`),
KEY `iso_3166_1_alpha_2` (`iso_3166_1_alpha_2`),
CONSTRAINT `fk_tblTerritories_tblLanguages1` FOREIGN KEY (`defaultLanguageID`) REFERENCES `tblLanguages` (`languageID`) ON DELETE NO ACTION ON UPDATE NO ACTION
) ENGINE=InnoDB AUTO_INCREMENT=8 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
edit2: 两次加入同一地区的原因是我需要使用查询顶部的GROUP_CONCAT生成两个单独的地区列表.我需要一个用于视频,一个用于它所属的类别.
edit3: 有趣的是,如果我将查询减少到它的裸骨,那么即使加入同一个表两次,它也非常快(0.00秒):
SELECT `c`.`categoryID`,
`v`.`videoID`,
GROUP_CONCAT(DISTINCT t_v.iso_3166_1_alpha_2) AS `video_territories`,
GROUP_CONCAT(DISTINCT t_c.iso_3166_1_alpha_2) AS `category_territories`
FROM `tblCategories` AS `c`
LEFT JOIN `tblCategoryVideos` AS `cv` ON c.categoryID = cv.categoryID
LEFT JOIN `tblVideos` AS `v` ON cv.videoID = v.videoID
LEFT JOIN `tblVideoTerritories` AS `vt` ON vt.videoID = v.videoID
LEFT JOIN `tblCategoryTerritories` AS `ct` ON ct.categoryID = c.categoryID
INNER JOIN `tblTerritories` AS `t_v` ON t_v.territoryID = vt.territoryID
INNER JOIN `tblTerritories` AS `t_c` ON t_c.territoryID = ct.territoryID
GROUP BY `v`.`videoID`, `c`.`categoryID`
edit4: If I switch from using the WHERE as a makeshift ON then I still have a query which takes 0.98 seconds:
SELECT `c`.`categoryID`,
`cl`.`name` AS `category_name`,
`v`.*,
TRUE AS `categoried`,
GROUP_CONCAT(DISTINCT t_v.iso_3166_1_alpha_2) AS `video_territories`,
GROUP_CONCAT(DISTINCT t_c.iso_3166_1_alpha_2) AS `category_territories`,
`vl`.*,
GROUP_CONCAT(DISTINCT kl.name) AS `keywords`
FROM `tblCategories` AS `c`
INNER JOIN `tblCategoryLocalisedData` AS `cl` ON c.categoryID = cl.categoryID
LEFT JOIN `tblCategoryDurations` AS `cd` ON c.categoryID = cd.categoryID
LEFT JOIN `tblCategoryRules` AS `cr` ON c.categoryID = cr.categoryID
LEFT JOIN `tblCategoryVideos` AS `cv` ON c.categoryID = cv.categoryID
LEFT JOIN `tblVideos` AS `v` ON cv.videoID = v.videoID
LEFT JOIN `tblVideoTerritories` AS `vt` ON vt.videoID = v.videoID
LEFT JOIN `tblCategoryTerritories` AS `ct` ON ct.categoryID = c.categoryID
INNER JOIN `tblTerritories` AS `t_v` ON t_v.territoryID = vt.territoryID
INNER JOIN `tblTerritories` AS `t_c` ON t_c.territoryID = ct.territoryID
INNER JOIN `tblVideoLocalisedData` AS `vl` ON vl.videoID = v.videoID
LEFT JOIN `tblVideoKeywords` AS `vk` ON v.videoID = vk.videoID
LEFT JOIN `tblKeywords` AS `k` ON vk.keywordID = k.keywordID
LEFT JOIN `tblKeywordLocalisedData` AS `kl` ON kl.keywordID = k.keywordID
INNER JOIN `tblLanguages` AS `l` ON (l.languageID = cl.languageID OR cl.languageID IS NULL) AND (l.languageID = kl.languageID OR kl.languageID IS NULL) AND (l.languageID = vl.languageID OR vl.languageID IS NULL)
WHERE (cv.disabled IS NULL)
AND (cd.start_date < NOW() OR cd.start_date IS NULL)
AND (cd.end_date > NOW() OR cd.end_date IS NULL)
AND (cr.name IS NULL) AND (l.iso_639_1 = 'en')
GROUP BY `v`.`videoID`, `c`.`categoryID`
ORDER BY `c`.`categoryID` ASC
edit5: If I remove the keyword related joins the query happens in 0.09 seconds... Removing the tblKeyword and tblKeywordLocalisedData but leaving tblVideoKeywords gives me 0.80 seconds. Removing tblVideoKeywords gives me 0.09 seconds.
But it seems to have indexes so yet again I don't get it:
CREATE TABLE `tblVideoKeywords` (
`videoKeywordID` int(10) unsigned NOT NULL AUTO_INCREMENT,
`videoID` int(10) unsigned NOT NULL,
`keywordID` int(10) unsigned NOT NULL,
PRIMARY KEY (`videoKeywordID`),
KEY `fk_tblVideoKeywords_tblVideos1` (`videoID`),
KEY `fk_tblVideoKeywords_tblKeywords1` (`keywordID`),
CONSTRAINT `fk_tblVideoKeywords_tblKeywords1` FOREIGN KEY (`keywordID`) REFERENCES `tblKeywords` (`keywordID`) ON DELETE NO ACTION ON UPDATE NO ACTION,
CONSTRAINT `fk_tblVideoKeywords_tblVideos1` FOREIGN KEY (`videoID`) REFERENCES `tblVideos` (`videoID`) ON DELETE NO ACTION ON UPDATE NO ACTION
) ENGINE=InnoDB AUTO_INCREMENT=18 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
edit6: Using the query provide by DRapp makes everything a lot faster. The EXPLAIN of his query now gives me:
+----+-------------+---------+--------+--------------------------------------------------------------------------------------+-----------------------------------------+---------+------------------------+------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+---------+--------+--------------------------------------------------------------------------------------+-----------------------------------------+---------+------------------------+------+----------------------------------------------+
| 1 | SIMPLE | c | index | PRIMARY | PRIMARY | 4 | NULL | 3 | Using index; Using temporary; Using filesort |
| 1 | SIMPLE | cl | ALL | fk_tblCategoryLocalisedData_tblCategories1,fk_tblCategoryLocalisedData_tblLanguages1 | NULL | NULL | NULL | 5 | Using where; Using join buffer |
| 1 | SIMPLE | lang_cl | ALL | PRIMARY | NULL | NULL | NULL | 2 | Using where; Using join buffer |
| 1 | SIMPLE | cd | ref | fk_tblCategoryDurations_tblCategories | fk_tblCategoryDurations_tblCategories | 4 | db.c.categoryID | 1 | |
| 1 | SIMPLE | cr | ref | fk_tblCategoryRules_tblCategories1 | fk_tblCategoryRules_tblCategories1 | 4 | db.c.categoryID | 1 | Using where; Not exists |
| 1 | SIMPLE | cv | ALL | fk_tblCategoryVideos_tblCategories1,fk_tblCategoryVideos_tblVideos1 | NULL | NULL | NULL | 2 | Using where; Using join buffer |
| 1 | SIMPLE | ct | ALL | fk_tblCategoryTerritories_tblCategories1,fk_tblCategoryTerritories_tblTerritories1 | NULL | NULL | NULL | 2 | Using where; Using join buffer |
| 1 | SIMPLE | t_c | eq_ref | PRIMARY | PRIMARY | 4 | db.ct.territoryID | 1 | |
| 1 | SIMPLE | v | eq_ref | PRIMARY | PRIMARY | 4 | db.cv.videoID | 1 | Using where |
| 1 | SIMPLE | vt | ref | fk_tblVideoTerritories_tblVideos1,fk_tblVideoTerritories_tblTerritories1 | fk_tblVideoTerritories_tblVideos1 | 4 | db.v.videoID | 1 | Using where |
| 1 | SIMPLE | t_v | eq_ref | PRIMARY | PRIMARY | 4 | db.vt.territoryID | 1 | |
| 1 | SIMPLE | vl | ALL | fk_tblVideoLocalisedData_tblLanguages1,fk_tblVideoLocalisedData_tblVideos1 | NULL | NULL | NULL | 9 | Using where; Using join buffer |
| 1 | SIMPLE | lang_vl | eq_ref | PRIMARY | PRIMARY | 4 | db.vl.languageID | 1 | Using where |
| 1 | SIMPLE | vk | ALL | fk_tblVideoKeywords_tblVideos1,fk_tblVideoKeywords_tblKeywords1 | NULL | NULL | NULL | 15 | Using where; Using join buffer |
| 1 | SIMPLE | k | eq_ref | PRIMARY | PRIMARY | 4 | db.vk.keywordID | 1 | Using where; Using index |
| 1 | SIMPLE | kl | ref | fk_tblKeywordLocalisedData_tblKeywords1,fk_tblKeywordLocalisedData_tblLanguages1 | fk_tblKeywordLocalisedData_tblKeywords1 | 4 | db.k.keywordID | 1 | Using where |
| 1 | SIMPLE | lang_kl | eq_ref | PRIMARY | PRIMARY | 4 | db.kl.languageID | 1 | Using where |
+----+-------------+---------+--------+--------------------------------------------------------------------------------------+-----------------------------------------+---------+------------------------+------+----------------------------------------------+
17 rows in set (0.01 sec)
对于其他几个问题,我也回答过类似的问题,只需添加一个"STRAIGHT_JOIN",轻微的重组就可以提供帮助.查询优化器实际上会尝试针对所有表进行思考,尝试查找记录较少的表并将其连接到较大的表,从而导致总混乱.这种情况发生在我通过查询超过14个子表时查看1400多万条记录的数据...非常类似于你在这里发生的事情.它在一个专用的独立服务器上运行了30多个小时的查询并将其挂起,不到2小时......请尝试以下操作:
除了对我习惯的连接进行一些视觉清理/排序之外,我还采用了一些NOW()vs NULL并将它们移动到连接中.如果查询左连接并将日期作为连接限定符的一部分,则将排除那些超出范围的记录,从而保留NULL结果集或有效条目,无需将该限定符加倍.
SELECT STRAIGHT_JOIN
c.categoryID,
cl.name AS category_name,
v.*,
TRUE AS categoried,
GROUP_CONCAT(DISTINCT t_v.iso_3166_1_alpha_2) AS video_territories,
GROUP_CONCAT(DISTINCT t_c.iso_3166_1_alpha_2) AS category_territories,
vl.*,
GROUP_CONCAT(DISTINCT kl.name) AS keywords
FROM
tblCategories AS c
INNER JOIN tblCategoryLocalisedData AS cl
ON c.categoryID = cl.categoryID
INNER JOIN tblLanguages AS lang_cl
ON l.languageID = lang_cl.languageID
AND lang_cl.iso_639_1 = 'en'
LEFT JOIN tblCategoryDurations AS cd
ON c.categoryID = cd.categoryID
AND cd.start_date < NOW()
AND cd.end_date > NOW()
LEFT JOIN tblCategoryRules AS cr
ON c.categoryID = cr.categoryID
LEFT JOIN tblCategoryVideos AS cv
ON c.categoryID = cv.categoryID
LEFT JOIN tblCategoryTerritories AS ct
ON c.categoryID = ct.categoryID
INNER JOIN tblTerritories AS t_c
ON ct.territoryID = t_c.territoryID
LEFT JOIN tblVideos AS v
ON cv.videoID = v.videoID
LEFT JOIN tblVideoTerritories AS vt
ON v.videoID = vt.videoID
INNER JOIN tblTerritories AS t_v
ON vt.territoryID = t_v.territoryID
INNER JOIN tblVideoLocalisedData AS vl
ON v.videoID = vl.videoID
INNER JOIN tblLanguages AS lang_vl
ON vl.languageID = lang_vl.languageID
AND lang_vl.iso_639_1 = 'en'
LEFT JOIN tblVideoKeywords AS vk
ON v.videoID = vk.videoID
LEFT JOIN tblKeywords AS k
ON vk.keywordID = k.keywordID
LEFT JOIN tblKeywordLocalisedData AS kl
ON k.keywordID = kl.keywordID
INNER JOIN tblLanguages AS lang_kl
ON kl.languageID = lang_kl.languageID
AND lang_kl.iso_639_1 = 'en'
WHERE
( cv.disabled IS NULL)
AND ( cr.name IS NULL)
GROUP BY
v.videoID,
c.categoryID
ORDER BY
c.categoryID ASC
正如我上面解释的那样,STRAIGHT_JOIN基本上告诉优化器"不要为我考虑"....按照我告诉你的顺序进行查询.在这种情况下,使用"tblCategories"作为主表并链接其他所有内容.优化器,即使有解释可能会尝试慢,并在下次运行查询时尝试不同的方法.因此,它可以尝试首先使用Languages表,然后通过其他表进行反向划分并阻塞.此外,通过将"AND"部分(例如日期)指向那些左连接,这些连接简化了WHERE,如您所见......正如您在NULL或它存在的位置,只是应用于该特定连接.保持干净的地方.
此外,通过保持关系的直接和缩进分别与他们加入的内容相比,它更容易理解与哪里有关联...
我还想看看最后的"EXPLAIN",看看它是什么.
| 归档时间： |
|
| 查看次数： |
3822 次 |
| 最近记录： |