高负载下的 iSCSI 超时

Question

我有两台通过千兆以太网连接的服务器。一个是 iSCSI 目标，第二个是启动器。当我mkfs.ext4在启动器上运行时，一段时间后磁盘 IO 在目标中严重减慢。在目标主机中，我可以在 syslog 中看到以下内容：

Sep 14 09:40:03 sh11 tgtd: abort_task_set(1139) found 119668c 0
Sep 14 09:40:03 sh11 tgtd: abort_cmd(1115) found 119668c 6
Sep 14 09:40:03 sh11 tgtd: abort_task_set(1139) found 119668d 0
Sep 14 09:40:03 sh11 tgtd: abort_cmd(1115) found 119668d 6
Sep 14 09:40:03 sh11 tgtd: abort_task_set(1139) found 119668e 0
Sep 14 09:40:03 sh11 tgtd: abort_cmd(1115) found 119668e 6
Sep 14 09:40:03 sh11 tgtd: abort_task_set(1139) found 1196696 0
Sep 14 09:40:03 sh11 tgtd: abort_cmd(1115) found 1196696 6
Sep 14 09:40:03 sh11 tgtd: abort_task_set(1139) found 119669e 0
Sep 14 09:40:03 sh11 tgtd: abort_cmd(1115) found 119669e 6
Sep 14 09:40:04 sh11 tgtd: abort_task_set(1139) found 119669f 0
Sep 14 09:40:04 sh11 tgtd: abort_cmd(1115) found 119669f 6

平均负载增长到 12 甚至更多：

# uptime
 12:37:00 up 23 days, 13:25,  1 user,  load average: 12.00, 7.00, 4.00

• CentOS 6.3
• tgtd 1.0.24
• 英特尔奔腾 4 2.4GHz
• 1Gb 内存
• 2Tb WD Cavlar 绿色 SATA 2.0

#lspci
00:00.0 Host bridge: Intel Corporation 82845G/GL[Brookdale-G]/GE/PE DRAM Controller/Host-Hub Interface (rev 02)
00:01.0 PCI bridge: Intel Corporation 82845G/GL[Brookdale-G]/GE/PE Host-to-AGP Bridge (rev 02)
00:1d.0 USB controller: Intel Corporation 82801DB/DBL/DBM (ICH4/ICH4-L/ICH4-M) USB UHCI Controller #1 (rev 02)
00:1d.1 USB controller: Intel Corporation 82801DB/DBL/DBM (ICH4/ICH4-L/ICH4-M) USB UHCI Controller #2 (rev 02)
00:1d.2 USB controller: Intel Corporation 82801DB/DBL/DBM (ICH4/ICH4-L/ICH4-M) USB UHCI Controller #3 (rev 02)
00:1d.7 USB controller: Intel Corporation 82801DB/DBM (ICH4/ICH4-M) USB2 EHCI Controller (rev 02)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 82)
00:1f.0 ISA bridge: Intel Corporation 82801DB/DBL (ICH4/ICH4-L) LPC Interface Bridge (rev 02)
00:1f.1 IDE interface: Intel Corporation 82801DB (ICH4) IDE Controller (rev 02)
00:1f.3 SMBus: Intel Corporation 82801DB/DBL/DBM (ICH4/ICH4-L/ICH4-M) SMBus Controller (rev 02)
00:1f.5 Multimedia audio controller: Intel Corporation 82801DB/DBL/DBM (ICH4/ICH4-L/ICH4-M) AC'97 Audio Controller (rev 02)
01:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI RV200 QW [Radeon 7500]
02:01.0 Ethernet controller: D-Link System Inc DGE-530T Gigabit Ethernet Adapter (rev 11) (rev 11)
02:02.0 RAID bus controller: VIA Technologies, Inc. VT6421 IDE/SATA Controller (rev 50)
02:03.0 RAID bus controller: VIA Technologies, Inc. VT6421 IDE/SATA Controller (rev 50)
02:04.0 RAID bus controller: Silicon Image, Inc. SiI 3114 [SATALink/SATARaid] Serial ATA Controller (rev 02)
02:08.0 Ethernet controller: Intel Corporation 82801DB PRO/100 VE (CNR) Ethernet Controller (rev 82)

有没有办法调整目标主机以避免这些超时？

更新 失败磁盘显示以下值：

# smartctl -A /dev/sdb
smartctl 5.42 2011-10-20 r3458 [i686-linux-2.6.32-279.2.1.el6.i686] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED >     RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   167   167   021    Pre-fail  Always       -       6633
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       93
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   088   088   000    Old_age   Always       -       9444
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       91
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       64
193 Load_Cycle_Count        0x0032   001   001   000    Old_age   Always       -       722663
194 Temperature_Celsius     0x0022   104   092   000    Old_age   Always       -       46
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       1
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

Answer 1

请注意，WD Caviar Green SATA 磁盘需要 20 秒到几分钟（甚至更长时间）才能从读/写错误中恢复。WD 在其桌面驱动器上禁用了“限时错误恢复”(TLER)。WD 的企业级硬盘有 7 秒的读取时间限制和 0 秒的写入恢复时间限制。由于您正在创建一个带有 mkfs 调用的新文件系统，这当然会触及该驱动器上的许多扇区，有些可能需要恢复，这可能是超时的根源。

为了安全起见，请考虑badblocks在目标的支持磁盘上运行。

另请注意，即使磁盘已死，SMART 也可能显示 0 个错误。