缓慢的 InnoDB 插入/更新/删除

And*_*ndy 5 mysql innodb

背景

几周前,我所有的 INSERT/UPDATE/DELETE 变得整体缓慢(不是 SELECT)。在此之前,一切正常(mysql 语句在 <0.01 秒内执行)。现在一个简单的 INSERT 大约需要 0.1 秒。对于数据库中的所有表都是一样的,即使对于没有数据且只有一个索引的表中最简单的查询,它也与任何高级或复杂查询无关。

例子:

mysql> INSERT INTO test (name) VALUES ('Andy');
Query OK, 1 row affected (0.09 sec)
Run Code Online (Sandbox Code Playgroud)

该表如下所示:

mysql> SHOW CREATE TABLE test;
+-----------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table     | Create Table                                                                                                                                                                                |
+-----------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| test | CREATE TABLE `test` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `name` varchar(50) NOT NULL DEFAULT '',
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=791 DEFAULT CHARSET=latin1 |
+-----------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)
Run Code Online (Sandbox Code Playgroud)

服务器重启后

我测试的第一件事是重新启动服务器。之后一切都很好,恢复正常!但是几天后,缓慢的查询又回来了。

由于 mysql 在重新启动之前甚至之后都快得多,几天以来,我认为一定存在某种问题,还是我应该接受 mysql 查询需要大约 0.1 秒才能执行?

我的.cnf

[mysql]
# CLIENT #
port                           = 3306
socket                         = /var/run/mysqld/mysqld.sock
[mysqld]
# GENERAL #
user                           = mysql
default_storage_engine         = InnoDB
socket                         = /var/run/mysqld/mysqld.sock
pid_file                       = /var/run/mysqld/mysql.pid
# MyISAM #
key_buffer_size                = 32M
myisam_recover                 = FORCE,BACKUP
# SAFETY #
max_allowed_packet             = 1G
max_connect_errors             = 1000000
innodb                         = FORCE
# DATA STORAGE #
datadir                        = /var/lib/mysql/
# BASIC SETTINGS #
basedir                = /usr
tmpdir                 = /tmp
# BINARY LOGGING #
log_bin                        = /var/log/mysql/mysql-bin
expire_logs_days               = 4
sync_binlog                    = 1
# CACHES AND LIMITS #
tmp_table_size                 = 32M
max_heap_table_size            = 32M
query_cache_type               = 0
query_cache_size               = 0
max_connections                = 500
thread_cache_size              = 50
open_files_limit               = 65535
table_definition_cache         = 4096
table_open_cache               = 1024
# INNODB #
innodb_flush_method            = O_DIRECT
innodb_log_files_in_group      = 2
innodb_log_file_size           = 256M
innodb_flush_log_at_trx_commit = 1
innodb_file_per_table          = 1
innodb_buffer_pool_size        = 12G
# LOGGING #
log_error                      = /var/log/mysql/error.log
log_queries_not_using_indexes  = 0
long_query_time                = 5
slow_query_log                 = 1
slow_query_log_file            = /var/log/mysql/mysql-slow.log
# REPLICATION RELATED #
server-id              = 1
log-slave-updates
auto_increment_increment       = 10
auto_increment_offset          = 1
relay_log              = /var/log/mysql/mysql-relay
sync_binlog            = 1
innodb_support_xa          = 1
sync_master_info           = 1
sync_relay_log             = 1
sync_relay_log_info        = 1
#replicate-ignore-db           = mysql
binlog-format              = ROW
[mysqldump]
quick
quote-names
max_allowed_packet  = 16M
Run Code Online (Sandbox Code Playgroud)

最佳

top - 14:34:24 up 28 days, 13:16,  2 users,  load average: 0.21, 0.46, 0.48
Tasks: 104 total,   1 running, 103 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.1%us,  0.0%sy,  0.0%ni, 99.9%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  16419976k total, 12208664k used,  4211312k free,   209064k buffers
Swap:  8376316k total,        0k used,  8376316k free,  9348992k cached
Run Code Online (Sandbox Code Playgroud)

笔记

服务器内存: 16GB

服务器硬盘: 3 x 146 GB SAS (RAID 5)

版本: Percona MySQL 5.5.30-30.2-log

复制:是的,服务器是复制设置中的主服务器

数据库大小: ~2GB


1 月 7 日更新 1(测试 1)

innodb_flush_log_at_trx_commit更改为 0。现在查询运行得更快一点,但是将innodb_flush_log_at_trx_commit设置为 0 是我听说的不是一个好的解决方案。这是一个查询配置文件。

mysql> INSERT INTO test (name) VALUES ('Andy');
Query OK, 1 row affected (0.02 sec)

mysql> SHOW PROFILE;
+----------------------+----------+
| Status               | Duration |
+----------------------+----------+
| starting             | 0.000036 |
| checking permissions | 0.000006 |
| Opening tables       | 0.000048 |
| System lock          | 0.000006 |
| init                 | 0.000012 |
| update               | 0.000069 |
| end                  | 0.000002 |
| query end            | 0.021729 |
| closing tables       | 0.000015 |
| freeing items        | 0.000019 |
| logging slow query   | 0.000002 |
| cleaning up          | 0.000002 |
+----------------------+----------+
12 rows in set (0.00 sec)
Run Code Online (Sandbox Code Playgroud)

1 月 14 日更新 2

问题似乎是间歇性的,即问题消失了一段时间然后又出现。不仅在重新启动服务器之后。几天前问题消失了,但昨天又回来了。这是与上述相同查询的查询配置文件:

mysql> INSERT INTO test (name) VALUES ('Andy');
Query OK, 1 row affected (0.08 sec)

mysql> SHOW PROFILE;
+----------------------+----------+
| Status               | Duration |
+----------------------+----------+
| starting             | 0.000025 |
| checking permissions | 0.000004 |
| Opening tables       | 0.000014 |
| System lock          | 0.000004 |
| init                 | 0.000007 |
| update               | 0.000036 |
| end                  | 0.000003 |
| query end            | 0.082652 |
| closing tables       | 0.000012 |
| freeing items        | 0.000016 |
| logging slow query   | 0.000002 |
| cleaning up          | 0.000001 |
+----------------------+----------+
12 rows in set (0.00 sec)
Run Code Online (Sandbox Code Playgroud)

这篇文章的评论很清楚地表明 MySQL/InnoDB 不应该在 RAID 5 上运行。在我得出 RAID 5 是问题的结论之前,还有什么我可以测试的吗?间歇性缓慢的 INSERT/UPDATE/DELETE 是否表明问题是 RAID 5?

And*_*ndy 2

发现问题了。与 MySQL/InnoDB 无关。问题出在 RAID 控制器电池上,该电池正在经历“重新学习”周期,然后禁用回写策略并启用直写,因此写入变得非常慢。如果您有戴尔服务器并出现类似症状,请查看系统日志和电池。

Jan 13 17:34:21 Server Administrator: Storage Service EventID: 2188  The controller write policy has been changed to Write Through.:  Battery 0 Controller 0
Run Code Online (Sandbox Code Playgroud)

欲了解更多信息:http://agiletesting.blogspot.se/2011/09/slow-database-check-raid-battery.html

也许这个问题应该重命名并移至服务器故障。