ZFS:尽管 zdb -DD 看起来不错,但重复数据删除的内存问题

6 zfs deduplication

我一直在一台装有 Ubuntu 12.10、32GB RAM(非 ECC,生产系统将具有 ECC)和一个 2x2TB Linux 管理的 RAID1(将移至 RAIDZ1 用于生产)的机器上试验 ZFS。我刚刚在 2TB soft-RAID1 设备上创建了 tank,启用了压缩和重复数据删除并存储了几个 100GB 的数据。

我得到了大约 3.5 倍的重复数据删除率(这对我的数据真的很有意义,这就是我想使用它的原因),但根本没有可用内存,系统变得无法使用。重新启动系统,一切正常,然后我写了几GB的数据,同样的事情。

然后我将 zfs_arc_max 设置为 12GB(因为显然我不是唯一一个内存消耗失控的人),这使系统免于无响应,但是写入几 GB 使内存限制达到最大值并且写入坦克变得非常缓慢,基本无法使用。

我知道重复数据删除需要 RAM,但据我所知,这

DDT-sha256-zap-duplicate: 615271 entries, size 463 on disk, 149 in core
DDT-sha256-zap-unique: 846070 entries, size 494 on disk, 159 in core

DDT histogram (aggregated over all DDTs):

bucket              allocated                       referenced          
______   ______________________________   ______________________________
refcnt   blocks   LSIZE   PSIZE   DSIZE   blocks   LSIZE   PSIZE   DSIZE
------   ------   -----   -----   -----   ------   -----   -----   -----
     1     826K   83.5G   51.7G   52.9G     826K   83.5G   51.7G   52.9G
     2     363K   34.6G   17.8G   18.5G     869K   81.9G   41.3G   43.0G
     4     138K   14.1G   8.89G   9.11G     654K   66.4G   41.0G   42.1G
     8    49.0K   3.94G   2.25G   2.34G     580K   44.3G   25.3G   26.4G
    16    37.2K   3.96G   3.06G   3.10G     865K   90.1G   69.9G   70.8G
    32    9.81K    854M    471M    488M     464K   40.5G   21.9G   22.7G
    64    1.84K    160M   80.8M   85.1M     148K   11.8G   5.99G   6.33G
   128    1.13K   60.4M   24.7M   27.7M     218K   11.2G   4.70G   5.26G
   256      545   52.9M   30.9M   32.1M     169K   15.5G   9.00G   9.36G
   512      120   7.17M   4.19M   4.51M    84.5K   5.09G   2.96G   3.18G
    1K      368   40.0M   19.0M   19.7M     480K   52.2G   24.8G   25.7G
    2K       16    401K     23K     76K    46.4K   1.31G   73.5M    226M
    4K        8      5K      4K     32K    39.9K   24.6M   20.0M    160M
 Total    1.39M    141G   84.3G   86.6G    5.32M    504G    299G    308G
Run Code Online (Sandbox Code Playgroud)

意味着该表应该只占用大约 90MB 的内存,所以我不明白发生了什么。我在没有重复数据删除的相同服务器上有相同的设置,而且那个似乎工作正常......

我真的很感激任何帮助(除了“禁用重复数据删除”,因为它确实对我的数据有意义;))!更多数据:

tank  type                  filesystem             -
tank  creation              Thu Jan 16 13:17 2014  -
tank  used                  342G                   -
tank  available             1.67T                  -
tank  referenced            341G                   -
tank  compressratio         1.64x                  -
tank  mounted               yes                    -
tank  quota                 none                   default
tank  reservation           none                   default
tank  recordsize            128K                   default
tank  mountpoint            /    tank                  default
tank  sharenfs              off                    default
tank  checksum              on                     default
tank  compression           lzjb                   local
tank  atime                 off                    local
tank  devices               on                     default
tank  exec                  on                     default
tank  setuid                on                     default
tank  readonly              off                    default
tank  zoned                 off                    default
tank  snapdir               hidden                 default
tank  aclinherit            restricted             default
tank  canmount              on                     default
tank  xattr                 sa                     local
tank  copies                1                      default
tank  version               5                      -
tank  utf8only              off                    -
tank  normalization         none                   -
tank  casesensitivity       sensitive              -
tank  vscan                 off                    default
tank  nbmand                off                    default
tank  sharesmb              off                    default
tank  refquota              none                   default
tank  refreservation        none                   default
tank  primarycache          all                    default
tank  secondarycache        all                    default
tank  usedbysnapshots       36.9M                  -
tank  usedbydataset         341G                   -
tank  usedbychildren        702M                   -
tank  usedbyrefreservation  0                      -
tank  logbias               latency                default
tank  dedup                 on                     local
tank  mlslabel              none                   default
tank  sync                  standard               default
tank  refcompressratio      1.64x                  -
tank  written               308K                   -
tank  snapdev               hidden                 default
Run Code Online (Sandbox Code Playgroud)

更新了更多数据。

好的,所以我进行了另一个测试 - 重新启动服务器,安装卷等:

             total       used       free     shared    buffers     cached
Mem:         32138        457      31680          0         19         66
-/+ buffers/cache:        372      31766
Swap:         7812          0       7812
Run Code Online (Sandbox Code Playgroud)

和 arcstats

4 1 0x01 84 4032 7898070146 560489175172
name                            type data
hits                            4    1059
misses                          4    185
demand_data_hits                4    0
demand_data_misses              4    0
demand_metadata_hits            4    971
demand_metadata_misses          4    49
prefetch_data_hits              4    0
prefetch_data_misses            4    7
prefetch_metadata_hits          4    88
prefetch_metadata_misses        4    129
mru_hits                        4    476
mru_ghost_hits                  4    0
mfu_hits                        4    495
mfu_ghost_hits                  4    0
deleted                         4    9
recycle_miss                    4    0
mutex_miss                      4    0
evict_skip                      4    0
evict_l2_cached                 4    0
evict_l2_eligible               4    0
evict_l2_ineligible             4    2048
hash_elements                   4    176
hash_elements_max               4    176
hash_collisions                 4    0
hash_chains                     4    0
hash_chain_max                  4    0
p                               4    6442450944
c                               4    12884901888
c_min                           4    1610612736
c_max                           4    12884901888
size                            4    1704536
hdr_size                        4    101424
data_size                       4    1448960
other_size                      4    154152
anon_size                       4    16384
anon_evict_data                 4    0
anon_evict_metadata             4    0
mru_size                        4    1231872
mru_evict_data                  4    206336
mru_evict_metadata              4    849408
mru_ghost_size                  4    0
mru_ghost_evict_data            4    0
mru_ghost_evict_metadata        4    0
mfu_size                        4    200704
mfu_evict_data                  4    0
mfu_evict_metadata              4    4096
mfu_ghost_size                  4    16384
mfu_ghost_evict_data            4    0
mfu_ghost_evict_metadata        4    16384
l2_hits                         4    0
l2_misses                       4    0
l2_feeds                        4    0
l2_rw_clash                     4    0
l2_read_bytes                   4    0
l2_write_bytes                  4    0
l2_writes_sent                  4    0
l2_writes_done                  4    0
l2_writes_error                 4    0
l2_writes_hdr_miss              4    0
l2_evict_lock_retry             4    0
l2_evict_reading                4    0
l2_free_on_write                4    0
l2_abort_lowmem                 4    0
l2_cksum_bad                    4    0
l2_io_error                     4    0
l2_size                         4    0
l2_asize                        4    0
l2_hdr_size                     4    0
l2_compress_successes           4    0
l2_compress_zeros               4    0
l2_compress_failures            4    0
memory_throttle_count           4    0
duplicate_buffers               4    0
duplicate_buffers_size          4    0
duplicate_reads                 4    0
memory_direct_count             4    0
memory_indirect_count           4    0
arc_no_grow                     4    0
arc_tempreserve                 4    0
arc_loaned_bytes                4    0
arc_prune                       4    0
arc_meta_used                   4    1498200
arc_meta_limit                  4    3221225472
arc_meta_max                    4    1449144
Run Code Online (Sandbox Code Playgroud)

玩了一会儿,直到 ARC 达到 vfs_arc_max (12GB):

4 1 0x01 84 4032 7898070146 1406380500230
name                            type data
hits                            4    7338384
misses                          4    117090
demand_data_hits                4    4841648
demand_data_misses              4    10072
demand_metadata_hits            4    2423640
demand_metadata_misses          4    35334
prefetch_data_hits              4    37879
prefetch_data_misses            4    65420
prefetch_metadata_hits          4    35217
prefetch_metadata_misses        4    6264
mru_hits                        4    2672085
mru_ghost_hits                  4    301
mfu_hits                        4    4615778
mfu_ghost_hits                  4    1183
deleted                         4    9
recycle_miss                    4    1022
mutex_miss                      4    17
evict_skip                      4    2
evict_l2_cached                 4    0
evict_l2_eligible               4    1977338368
evict_l2_ineligible             4    751589376
hash_elements                   4    166822
hash_elements_max               4    166828
hash_collisions                 4    59458
hash_chains                     4    21504
hash_chain_max                  4    4
p                               4    55022931
c                               4    12652319216
c_min                           4    1610612736
c_max                           4    12884901888
size                            4    12327222416
hdr_size                        4    55933440
data_size                       4    12149027328
other_size                      4    122261648
anon_size                       4    1056256
anon_evict_data                 4    0
anon_evict_metadata             4    0
mru_size                        4    6481734656
mru_evict_data                  4    6220393984
mru_evict_metadata              4    188646912
mru_ghost_size                  4    1902724096
mru_ghost_evict_data            4    1871710720
mru_ghost_evict_metadata        4    31013376
mfu_size                        4    5666236416
mfu_evict_data                  4    5643978240
mfu_evict_metadata              4    16081408
mfu_ghost_size                  4    708022272
mfu_ghost_evict_data            4    680676352
mfu_ghost_evict_metadata        4    27345920
l2_hits                         4    0
l2_misses                       4    0
l2_feeds                        4    0
l2_rw_clash                     4    0
l2_read_bytes                   4    0
l2_write_bytes                  4    0
l2_writes_sent                  4    0
l2_writes_done                  4    0
l2_writes_error                 4    0
l2_writes_hdr_miss              4    0
l2_evict_lock_retry             4    0
l2_evict_reading                4    0
l2_free_on_write                4    0
l2_abort_lowmem                 4    0
l2_cksum_bad                    4    0
l2_io_error                     4    0
l2_size                         4    0
l2_asize                        4    0
l2_hdr_size                     4    0
l2_compress_successes           4    0
l2_compress_zeros               4    0
l2_compress_failures            4    0
memory_throttle_count           4    0
duplicate_buffers               4    0
duplicate_buffers_size          4    0
duplicate_reads                 4    0
memory_direct_count             4    0
memory_indirect_count           4    1947
arc_no_grow                     4    0
arc_tempreserve                 4    0
arc_loaned_bytes                4    0
arc_prune                       4    0
arc_meta_used                   4    462466704
arc_meta_limit                  4    3221225472
arc_meta_max                    4    465357280
Run Code Online (Sandbox Code Playgroud)

和 free -m 显示了预期的内容,缓冲区/缓存和第一行同意使用/免费。但是,多玩一些会导致系统变得异常缓慢(复制 1GB 的时间)和

             total       used       free     shared    buffers     cached
Mem:         32138      31923        215          0          6      15442
-/+ buffers/cache:      16473      15665
Swap:         7812          0       7812

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 1  1    308 3774708  27204 9464052    0    0   386   271   72  348  1  2 83 15
Run Code Online (Sandbox Code Playgroud)

卸载 ZFS 卷并卸载内核模块会释放所有内存......所以对我来说,它真的看起来像是某种内存泄漏:设置了 zfs_arc_max,arcstats 说这个限制被观察到(见下文),但 ZFS 不知何故继续吃掉内存。呼...

    time  read  miss  miss%  dmis  dm%  pmis  pm%  mmis  mm%  arcsz     c  
14:08:08     0     0      0     0    0     0    0     0    0   9.8G   10G 
Run Code Online (Sandbox Code Playgroud)

b13*_*n1u 2

一个好的经验法则是为每 1 TB 磁盘规划大约 5 GB RAM。因此,如果您有 2TB 数据,则仅用于重复数据删除 + ARC + ZFS 元数据的数据就需要 10GB。这不是你想要的答案,但也不值得付出努力。启用压缩后您仍然可以节省一些费用。看看这篇文章

5GB 是一般规则,但不一定如此。我们假设您使用 64K 块,每 1TB 需要 5GB RAM。但 512b 和 128K 之间的块大小可能不同。解决方案可能是 L2ARC 和 SSD 驱动器,但价格昂贵。