在并发截断命令期间服务器崩溃后 MySQL INNODB 损坏

Question

我的服务器今天崩溃了，我认为是由于我们的一个 INNODB 表上的并发 truncate table 命令。服务器可以重新启动，但是在它启动后，每次我尝试发出 SQL 命令时，都会出现以下错误：

ERROR 2006 (HY000): MySQL server has gone away

这是日志中发生的事情：

121206 01:11:12  mysqld restarted
121206  1:11:13  InnoDB: Started; log sequence number 275 559321759
InnoDB: !!! innodb_force_recovery is set to 1 !!!
121206  1:11:13 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.95-log'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
InnoDB: Error: trying to load index PRIMARY for table 
InnoDB: but the index tree has been freed!
121206  1:11:37 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=134217728
read_buffer_size=1048576
max_used_connections=1
max_connections=400
threads_connected=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 950272 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=0x9900950
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0x46353fa0, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
(nil)
New value of fp=0x9900950 failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/using-stack-trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x993e500 =
thd->thread_id=1
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

我在网上搜索过，我得到提示，这是一个 MySQL 错误，但我不知道如何解决它。我使用的是 MySQL 5.0.95 版。

似乎我必须创建一个新数据库并将旧数据转储到新数据库中，但是如果我什至无法向当前数据库发出任何 SQL 命令，我该怎么做呢？

--- 更新 ---
版本：'5.0.95-log' 套接字：'/var/lib/mysql/mysql.sock' 端口：3306 源分发 InnoDB：错误：尝试加载表 InnoDB 的索引 PRIMARY：但是索引树已被释放！121206 4:13:41 - mysqld 收到信号 11；这可能是因为您遇到了错误。也可能是这个二进制文件或它所链接的库之一已损坏、构建不当或配置错误。此错误也可能由硬件故障引起。我们将尽最大努力收集一些信息，希望能帮助诊断问题，但由于我们已经崩溃了，肯定出了点问题，这可能会失败。

key_buffer_size=134217728
read_buffer_size=1048576
max_used_connections=1
max_connections=400
threads_connected=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 950272 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=0x17fb8950
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0x464a3fa0, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
(nil)
New value of fp=0x17fb8950 failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/using-stack-trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x17ff6500 =
thd->thread_id=3
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

Number of processes running now: 0
121206 04:13:41  mysqld restarted
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
121206  4:13:42  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
121206  4:13:43  InnoDB: Started; log sequence number 275 559323148
121206  4:13:43 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.95-log'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution

Answer 1

方面#1

首先引起我注意的是这条线

InnoDB：错误：尝试为表 / 加载索引 PRIMARY

这表明您有一个使用 InnoDB 存储引擎的表

InnoDB 的有趣之处在于 PRIMARY KEY 的存储方式。它存储在一个称为gen_clust_index的结构中，或更常见的称为聚集索引。

我的直接猜测是某个 PRIMARY KEY 条目太大了

请考虑一些关于使用长 PRIMARY KEY 的好处、坏处和丑陋的文章：

• Innodb 表的长 PRIMARY KEY
• 优化 InnoDB 表的存储布局
• Mysql int vs varchar 作为主键（InnoDB 存储引擎？

然后看看是否<DB Hidden>.<Table Hidden>需要重新设计。

方面#2

就您关于平行截断表的猜想而言，这听起来有点危险。为什么？InnoDB 执行 TRUNCATE TABLE 为DDLnot DML。我以前写过这个：

• Jul 09, 2012：什么会导致 TRUNCATE TABLE 花费很长时间？
• Jan 17, 2012：InnoDB“每表”文件大小的问题
• Sep 28, 2011:如何恢复文件被移动的 InnoDB 表

方面#3

一些调整建议

请将以下内容添加到 my.ini

[mysqld]
max_allowed_packet=1G
innodb_fast_shutdown=0

启动mysql

在另一个会话中，运行tail -f <errorlogfile>并观看 InnoDB Crash Recovery。

如果 mysql 已完全启动备份并且 InnoDB 崩溃恢复已完成，请尝试立即关闭 mysql。您可能需要调整 InnoDB 事务日志的大小。

很抱歉这些疯狂的建议，但我在这里盲目飞行。

请在问题中发布以下内容：

• 你的整个 my.cnf
• 板上有多少 RAM

更新 2012-12-05 12:09 EDT

请执行以下操作：

STEP 01) 将这些更改添加到 my.cnf

[mysqld]
max_allowed_packet=1G
innodb_fast_shutdown=0
innodb_thread_concurrency=0

步骤 02) service mysql restart

确保 mysql 出现

STEP 03) 您需要调整 ib_logfile0 和 ib_logfile1 的大小（24M 可能太小了）

service mysql stop
cd /var/lib/mysql
mv ib_logfile0 ib_logfile0.bak
mv ib_logfile1 ib_logfile1.bak

STEP 04) 将这些更改添加到 my.cnf

[mysqld]
innodb_log_file_size=512M
innodb_log_buffer_size=8M

步骤 05) service mysql start

mysqld 将分别重新创建 ib_logfile0 和 ib_logfile1 512M

现在，试着看看会发生什么......

更新 2012-12-05 12:18 EDT

同时，请阅读我关于 mysql 数据包的 ServerFault 帖子及其对innodb_log_file_size和innodb_log_buffer_size 的大小影响，正如我从其他人的 ServerFault 帖子中了解到的。

更新 2012-12-05 14:28 EDT

我从这个问题中编辑了对客户表的所有引用。

根本原因是一个损坏的页面，ibdata1其中混合了数据和索引页面。我帮助 Andrew 将数据迁移出来，使用innodb_file_per_table重新创建 ibdata1 ，然后 Andrew 重新加载了数据。