设备:/dev/sda [SAT], 1 当前不可读(待处理)扇区

MSF*_*004 13 raid software-raid

我在 RHEL 上有一个软件 RAID 1 阵列。每天早上我都会收到此错误通过电子邮件发送给我:设备:/dev/sda [SAT], 1 当前不可读(待处理)扇区

当我在 sda(或 sdb)上运行测试时,一切似乎都通过了。我错过了什么吗?


mdstat 显示 RAID 阵列很好:

# cat /proc/mdstat 
Personalities : [raid1] 
md5 : active raid1 sdb5[1] sda5[0]
      108026816 blocks [2/2] [UU]

md1 : active raid1 sdb1[1] sda1[0]
      511936 blocks [2/2] [UU]

md2 : active raid1 sda2[0] sdb2[1]
      805306304 blocks [2/2] [UU]

md3 : active raid1 sda3[0] sdb3[1]
      62914496 blocks [2/2] [UU]

unused devices: <none>
Run Code Online (Sandbox Code Playgroud)

这是输出:# smartctl -q noserial -a /dev/sda

我也试过运行:# smartctl -t long /dev/sda


smartctl 5.43 2012-06-30 r3573 [x86_64-linux-2.6.32-279.9.1.el6.x86_64] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Model Family:     Hitachi Ultrastar A7K1000
Device Model:     Hitachi HUA721010KLA330
Firmware Version: GKAOAB0A
User Capacity:    1,000,204,886,016 bytes [1.00 TB]
Sector Size:      512 bytes logical/physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   7
ATA Standard is:  ATA/ATAPI-7 T13 1532D revision 1
Local Time is:    Sun May 21 17:51:42 2017 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x84) Offline data collection activity
                    was suspended by an interrupting command from host.
                    Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                    without error or no self-test has ever 
                    been run.
Total time to complete Offline 
data collection:        (15354) seconds.
Offline data collection
capabilities:            (0x5b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    No Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                    General Purpose Logging supported.
Short self-test routine 
recommended polling time:    (   1) minutes.
Extended self-test routine
recommended polling time:    ( 255) minutes.
SCT capabilities:          (0x003f) SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   098   098   016    Pre-fail  Always       -       4
  2 Throughput_Performance  0x0005   100   100   054    Pre-fail  Offline      -       0
  3 Spin_Up_Time            0x0007   122   122   024    Pre-fail  Always       -       550 (Average 591)
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       68
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000b   100   100   067    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0005   100   100   020    Pre-fail  Offline      -       0
  9 Power_On_Hours          0x0012   094   094   000    Old_age   Always       -       43202
 10 Spin_Retry_Count        0x0013   100   100   060    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       68
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       751
193 Load_Cycle_Count        0x0012   100   100   000    Old_age   Always       -       751
194 Temperature_Celsius     0x0002   090   090   000    Old_age   Always       -       66 (Min/Max 17/72)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       1
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     43186         -
# 2  Extended offline    Completed without error       00%     43170         -
# 3  Short offline       Completed without error       00%     43162         -
# 4  Short offline       Completed without error       00%     43138         -
# 5  Short offline       Completed without error       00%     43114         -
# 6  Short offline       Completed without error       00%     43090         -
# 7  Short offline       Completed without error       00%     43066         -
# 8  Short offline       Completed without error       00%     43042         -
# 9  Extended offline    Completed without error       00%     43031         -
#10  Short offline       Completed without error       00%     43024         -
#11  Short offline       Completed without error       00%     43018         -
#12  Extended offline    Completed without error       00%     43002         -
#13  Short offline       Completed without error       00%     42994         -
#14  Short offline       Completed without error       00%     42970         -
#15  Short offline       Completed without error       00%     42946         -
#16  Short offline       Completed without error       00%     42922         -
#17  Short offline       Completed without error       00%     42898         -
#18  Short offline       Completed without error       00%     42874         -
#19  Short offline       Completed without error       00%     42850         -
#20  Extended offline    Completed without error       00%     42833         -
#21  Short offline       Completed without error       00%     42826         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
Run Code Online (Sandbox Code Playgroud)

Ale*_*exD 10

您的 SMART Current Pending Sector 值为 1。这意味着磁盘上有一个坏扇区,驱动器固件无法重新分配它,但您的重新分配扇区数仍然为零,因此即使您的驱动器已运行了在不太健康的环境中使用 5 年 - 温度高达 72 C°。

您可以尝试找到这个坏扇区,dd if=/dev/sda of=/dev/null然后您需要通过覆盖它来重新映射这个扇区。

  • 我有点困惑我以为linux会自动重新分配扇区,为什么不呢?这似乎正是操作系统应该做的事情?另外,它没有告诉您邮件或 dmesg 中的阻止内容,这似乎有点糟糕。 (3认同)
  • badblocks 是更高级的实用程序,如果您无法在固件级别重新映射坏块,您将需要它,但它的某些模式具有破坏性,因此在使用它时需要小心。我会在第一个错误时更换这种老化的驱动器,而不会打扰自己进行恢复,但更换也会带来一些风险,尤其是如果它是由数据中心的一些过度劳累的技术人员完成的。由您来衡量所有风险并做出决定。 (2认同)
  • @MSF004 `badblocks` 命令适用于文件系统级别。这就是为什么你需要在 `/dev/md2` 而不是 `/dev/sda2` 上运行它。如果你在 `/dev/sda2` 上做你的系统会崩溃。 (2认同)

小智 6

我相信“197 Current_Pending_Sector”计数是读取失败的扇区计数。这确实意味着驱动器开始出现故障,但这并不一定意味着驱动器已损坏。如果这些扇区被重写,驱动器固件可以将它们重新映射,并且驱动器可能没问题。

搜索周围还发现讨论表明某些型号的 SSD 驱动器会定期将其切换到 1 并返回,这可能是 SMART 报告中的一个几乎无害的固件错误。

因此,您可以忽略它,并且只要您的文件系统可以处理偶尔的坏块读取(即,在幕后进行某种强大的raid/冗余),当文件系统覆盖这些块时,它可能会慢慢地自行清除。如果您的文件系统无法处理坏块读取,则每次尝试读取该块中的内容时,您都可能会收到 IO 错误。您仍然可以通过查找并删除该文件来恢复,并且文件系统最终将重写该扇区。

您还可以通过显式覆盖这些扇区来清除 Current_Pending_Sector 计数。这将破坏磁盘上的数据!这可能会损坏文件系统,从而丢失磁盘上的所有数据,而不仅仅是坏扇区中丢失的数据。因此,仅当您有能力丢失磁盘上的所有数据时才执行此操作。

您可以通过运行智能长测试来找到坏扇区;

# smartctl -t long /dev/sda
Run Code Online (Sandbox Code Playgroud)

然后,您可以检查长测试的状态以查看其是否完成,以及遇到的第一个错误的 LBA;

# smartctl -l selftest /dev/sda
smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.7.0-1-amd64] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     17711         12345
# 2  Short offline       Completed without error       00%     17709         -
# 3  Short captive       Interrupted (host reset)      10%       450         -
# 4  Short captive       Interrupted (host reset)      10%       228         -
Run Code Online (Sandbox Code Playgroud)

注意:如果您没有看到数字,LBA_of_first_error可以尝试使用-l xselftest

然后您可以使用 dd 覆盖坏扇区;

# dd if=/dev/zero of=/dev/sda seek=12345 count=1
Run Code Online (Sandbox Code Playgroud)

但请注意,默认的 bs=512 块大小可能小于驱动器上的物理扇区大小,因此您可能需要写入 count=8 才能完全擦除它。然后,您可以冲洗并重复测试/覆盖周期,直到所有坏扇区都被覆盖。最后,您可以检查计数是否已归零;

$ sudo smartctl -A /dev/sdc | grep Current_Pending_Sector
197 Current_Pending_Sector  0x0012   200   200   000    Old_age   Always       -       0
Run Code Online (Sandbox Code Playgroud)

我在旧的 WD 200G 硬盘上经历过这种情况,该硬盘出现某种故障,可能是由于热插拔连接不良导致 Current_Pending_Sector 计数为 26。当我运行长时间测试时,它通过了,没有发现任何坏扇区,但是count 仍然是 26。我可以通过使用 dd 将整个驱动器清零来将计数器清零;

dd if=/dev/zero of=/dev/sda bs=1M status=progress
Run Code Online (Sandbox Code Playgroud)

请注意,bs=1M 使其运行速度更快,但它不是 512 扇区大小,因此末尾会有部分块。随后的 smartctl 长和短测试均报告驱动器正常。