那些遇到过的问题

Ruby 2.2中的垃圾收集器引发了意想不到的CoW

Tho*_*mus 10 ruby garbage-collection fork copy-on-write shared-memory

当我分叉我的进程时,如何防止GC激发写时复制？我最近一直在分析垃圾收集器在Ruby中的行为,因为我在程序中遇到了一些内存问题(即使对于相当小的任务,我的60核0.5Tb机器上的内存耗尽).对我来说,这确实限制了ruby在多核服务器上运行程序的实用性.我想在这里介绍我的实验和结果.

垃圾收集器在分叉期间运行时会出现问题.我调查了三个案例来说明这个问题.

情况1:我们使用数组在内存中分配了大量对象(字符串不超过20个字节).使用随机数和字符串格式创建字符串.当进程分叉并强制GC在子进程中运行时,所有共享内存都是私有的,导致初始内存重复.

情况2:我们使用数组在内存中分配了很多对象(字符串),但是使用rand.to_s函数创建了字符串,因此我们删除了与前一种情况相比的数据格式.我们最终使用的内存较少,可能是因为垃圾较少.当进程分叉并强制GC在子进程中运行时,只有部分内存变为私有.我们有重复的初始内存,但程度较小.

情况3:与之前相比,我们分配的对象更少,但对象更大,因此分配的内存量与之前的情况相同.当进程分叉并且我们强制GC在子进程中运行时,所有内存保持共享,即没有内存重复.

在这里,我粘贴用于这些实验的Ruby代码.要在不同情况之间切换,只需更改memory_object函数中的"option"值即可.在Ubuntu 14.04计算机上使用Ruby 2.2.2,2.2.1,2.1.3,2.1.5和1.9.3测试了代码.

案例1的示例输出:

ruby version 2.2.2 
 proces   pid log                   priv_dirty   shared_dirty 
 Parent  3897 post alloc                   38            0 
 Parent  3897 4 fork                        0           37 
 Child   3937 4 initial                     0           37 
 Child   3937 8 empty GC                   35            5
Run Code Online (Sandbox Code Playgroud)

完全相同的代码是用Python编写的,在所有情况下,CoW都可以正常工作.

案例1的示例输出:

python version 2.7.6 (default, Mar 22 2014, 22:59:56) 
[GCC 4.8.2] 
 proces   pid log                   priv_dirty shared_dirty 
 Parent  4308 post alloc                35             0 
 Parent  4308 4 fork                     0            35 
 Child   4309 4 initial                  0            35 
 Child   4309 10 empty GC                1            34
Run Code Online (Sandbox Code Playgroud)

Ruby代码

$start_time=Time.new

# Monitor use of Resident and Virtual memory.
class Memory

    shared_dirty = '.+?Shared_Dirty:\s+(\d+)'
    priv_dirty = '.+?Private_Dirty:\s+(\d+)'
    MEM_REGEXP = /#{shared_dirty}#{priv_dirty}/m

    # get memory usage
    def self.get_memory_map( pids)
        memory_map = {}
        memory_map[ :pids_found] = {}
        memory_map[ :shared_dirty] = 0
        memory_map[ :priv_dirty] = 0

        pids.each do |pid|
            begin
                lines = nil
                lines = File.read( "/proc/#{pid}/smaps")
            rescue
                lines = nil
            end
            if lines
                lines.scan(MEM_REGEXP) do |shared_dirty, priv_dirty|
                    memory_map[ :pids_found][pid] = true
                    memory_map[ :shared_dirty] += shared_dirty.to_i
                    memory_map[ :priv_dirty] += priv_dirty.to_i
                end
            end
        end
        memory_map[ :pids_found] = memory_map[ :pids_found].keys
        return memory_map
    end

    # get the processes and get the value of the memory usage
    def self.memory_usage( )
        pids   = [ $$]
        result = self.get_memory_map( pids)

        result[ :pids]   = pids
        return result
    end

    # print the values of the private and shared memories
    def self.log( process_name='', log_tag="")
        if process_name == "header"
            puts " %-6s %5s %-12s %10s %10s\n" % ["proces", "pid", "log", "priv_dirty", "shared_dirty"]
        else
            time = Time.new - $start_time
            mem = Memory.memory_usage( )
            puts " %-6s %5d %-12s %10d %10d\n" % [process_name, $$, log_tag, mem[:priv_dirty]/1000, mem[:shared_dirty]/1000]
        end
    end
end

# function to delay the processes a bit
def time_step( n)
    while Time.new - $start_time < n
        sleep( 0.01)
    end
end

# create an object of specified size. The option argument can be changed from 0 to 2 to visualize the behavior of the GC in various cases
#
# case 0 (default) : we make a huge array of small objects by formatting a string
# case 1 : we make a huge array of small objects without formatting a string (we use the to_s function)
# case 2 : we make a smaller array of big objects
def memory_object( size, option=1)
    result = []
    count = size/20

    if option > 3 or option < 1
        count.times do
            result << "%20.18f" % rand
        end
    elsif option == 1
        count.times do
            result << rand.to_s
        end
    elsif option == 2
        count = count/10
        count.times do
            result << ("%20.18f" % rand)*30
        end
    end

    return result
end

##### main #####

puts "ruby version #{RUBY_VERSION}"

GC.disable

# print the column headers and first line
Memory.log( "header")

# Allocation of memory
big_memory = memory_object( 1000 * 1000 * 10)

Memory.log( "Parent", "post alloc")

lab_time = Time.new - $start_time
if lab_time < 3.9
    lab_time = 0
end

# start the forking
pid = fork do
    time = 4
    time_step( time + lab_time)
    Memory.log( "Child", "#{time} initial")

    # force GC when nothing happened
    GC.enable; GC.start; GC.disable

    time = 8
    time_step( time + lab_time)
    Memory.log( "Child", "#{time} empty GC")

    sleep( 1)
    STDOUT.flush
    exit!
end

time = 4
time_step( time + lab_time)
Memory.log( "Parent", "#{time} fork")

# wait for the child to finish
Process.wait( pid)
Run Code Online (Sandbox Code Playgroud)

Python代码

import re
import time
import os
import random
import sys
import gc

start_time=time.time()

# Monitor use of Resident and Virtual memory.
class Memory:   

    def __init__(self):
        self.shared_dirty = '.+?Shared_Dirty:\s+(\d+)'
        self.priv_dirty = '.+?Private_Dirty:\s+(\d+)'
        self.MEM_REGEXP = re.compile("{shared_dirty}{priv_dirty}".format(shared_dirty=self.shared_dirty, priv_dirty=self.priv_dirty), re.DOTALL)

    # get memory usage
    def get_memory_map(self, pids):
        memory_map = {}
        memory_map[ "pids_found" ] = {}
        memory_map[ "shared_dirty" ] = 0
        memory_map[ "priv_dirty" ] = 0

        for pid in pids:
            try:
                lines = None

                with open( "/proc/{pid}/smaps".format(pid=pid), "r" ) as infile:
                    lines = infile.read()
            except:
                lines = None

            if lines:
                for shared_dirty, priv_dirty in re.findall( self.MEM_REGEXP, lines ):
                    memory_map[ "pids_found" ][pid] = True
                    memory_map[ "shared_dirty" ] += int( shared_dirty )
                    memory_map[ "priv_dirty" ] += int( priv_dirty )     

        memory_map[ "pids_found" ] = memory_map[ "pids_found" ].keys()
        return memory_map

    # get the processes and get the value of the memory usage   
    def memory_usage( self):
        pids   = [ os.getpid() ]
        result = self.get_memory_map( pids)

        result[ "pids" ]   = pids

        return result

    # print the values of the private and shared memories
    def log( self, process_name='', log_tag=""):
        if process_name == "header":
            print " %-6s %5s %-12s %10s %10s" % ("proces", "pid", "log", "priv_dirty", "shared_dirty")
        else:
            global start_time
            Time = time.time() - start_time
            mem = self.memory_usage( )
            print " %-6s %5d %-12s %10d %10d" % (process_name, os.getpid(), log_tag, mem["priv_dirty"]/1000, mem["shared_dirty"]/1000)

# function to delay the processes a bit
def time_step( n):
    global start_time
    while (time.time() - start_time) < n:
        time.sleep( 0.01)

# create an object of specified size. The option argument can be changed from 0 to 2 to visualize the behavior of the GC in various cases
#
# case 0 (default) : we make a huge array of small objects by formatting a string
# case 1 : we make a huge array of small objects without formatting a string (we use the to_s function)
# case 2 : we make a smaller array of big objects                                       
def memory_object( size, option=2):
    count = size/20

    if option > 3 or option < 1:
        result = [ "%20.18f"% random.random() for i in xrange(count) ]

    elif option == 1:
        result = [ str( random.random() ) for i in xrange(count) ]

    elif option == 2:
        count = count/10
        result = [ ("%20.18f"% random.random())*30 for i in xrange(count) ]

    return result

##### main #####

print "python version {version}".format(version=sys.version)

memory = Memory()

gc.disable()

# print the column headers and first line
memory.log( "header")   # Print the headers of the columns

# Allocation of memory
big_memory = memory_object( 1000 * 1000 * 10)   # Allocate memory

memory.log( "Parent", "post alloc")

lab_time = time.time() - start_time
if lab_time < 3.9:
    lab_time = 0

# start the forking
pid = os.fork()     # fork the process
if pid == 0:
    Time = 4
    time_step( Time + lab_time)
    memory.log( "Child", "{time} initial".format(time=Time))

    # force GC when nothing happened
    gc.enable(); gc.collect(); gc.disable();

    Time = 10
    time_step( Time + lab_time)
    memory.log( "Child", "{time} empty GC".format(time=Time))

    time.sleep( 1)

    sys.exit(0)

Time = 4
time_step( Time + lab_time)
memory.log( "Parent", "{time} fork".format(time=Time))

# Wait for child process to finish
os.waitpid( pid, 0)
Run Code Online (Sandbox Code Playgroud)

编辑

实际上,在分叉过程之前多次调用GC解决了这个问题,我感到非常惊讶.我也使用Ruby 2.0.0运行代码,问题甚至没有出现,因此它必须与您提到的这一代GC相关.但是,如果我调用memory_object函数而不将输出分配给任何变量(我只是创建垃圾),那么内存是重复的.复制的内存量取决于我创建的垃圾量 - 垃圾越多,内存变得越私有.

我有什么想法可以阻止这个吗？

这是一些结果

在2.0.0中运行GC

ruby version 2.0.0
 proces   pid log          priv_dirty shared_dirty
 Parent  3664 post alloc           67          0
 Parent  3664 4 fork                1         69
 Child   3700 4 initial             1         69
 Child   3700 8 empty GC            6         65
Run Code Online (Sandbox Code Playgroud)

在孩子中调用memory_object(1000*1000)

ruby version 2.0.0
 proces   pid log          priv_dirty shared_dirty
 Parent  3703 post alloc           67          0
 Parent  3703 4 fork                1         70
 Child   3739 4 initial             1         70
 Child   3739 8 empty GC           15         56
Run Code Online (Sandbox Code Playgroud)

调用memory_object(1000*1000*10)

ruby version 2.0.0
 proces   pid log          priv_dirty shared_dirty
 Parent  3743 post alloc           67          0
 Parent  3743 4 fork                1         69
 Child   3779 4 initial             1         69
 Child   3779 8 empty GC           89          5
Run Code Online (Sandbox Code Playgroud)

Eug*_*rov 2

UPD2

突然明白了为什么如果你格式化字符串，所有的内存都会变成私有的——你在格式化过程中会生成垃圾，禁用GC，然后启用GC，并且你生成的数据中会出现释放对象的漏洞。然后你分叉，新的垃圾开始占据这些洞，垃圾越多，私有页面就越多。

所以我添加了一个清理函数，每 2000 个周期运行 GC（仅启用惰性 GC 没有帮助）：

count.times do |i|
  cleanup(i)
  result << "%20.18f" % rand
end

#......snip........#

def cleanup(i)
      if ((i%2000).zero?)
        GC.enable; GC.start; GC.disable
      end
end   

##### main #####
Run Code Online (Sandbox Code Playgroud)

结果是（memory_object( 1000 * 1000 * 10)分叉后生成）：

RUBY_GC_HEAP_INIT_SLOTS=600000 ruby gc-test.rb 0
ruby version 2.2.0
 proces   pid log          priv_dirty shared_dirty
 Parent  2501 post alloc           35          0
 Parent  2501 4 fork                0         35
 Child   2503 4 initial             0         35
 Child   2503 8 empty GC           28         22
Run Code Online (Sandbox Code Playgroud)

是的，它会影响性能，但仅在分叉之前，即增加您的情况下的加载时间。

UPD1

刚刚发现ruby​​ 2.2 设置旧对象位的标准，它是 3 次 GC，所以如果您在分叉之前添加以下内容：

GC.enable; 3.times {GC.start}; GC.disable
# start the forking
Run Code Online (Sandbox Code Playgroud)

你会得到（该选项1在命令行中）：

$ RUBY_GC_HEAP_INIT_SLOTS=600000 ruby gc-test.rb 1
ruby version 2.2.0
 proces   pid log          priv_dirty shared_dirty
 Parent  2368 post alloc           31          0
 Parent  2368 4 fork                1         34
 Child   2370 4 initial             1         34
 Child   2370 8 empty GC            2         32
Run Code Online (Sandbox Code Playgroud)

但这需要进一步测试此类对象在未来 GC 上的行为，至少在 100 次 GC 后:old_objects保持不变，所以我认为应该没问题

登录GC.stat在这里

顺便说一句，还可以选择RGENGC_OLD_NEWOBJ_CHECK从一开始就创建旧对象，但我怀疑这是一个好主意，但对于特定情况可能有用。

第一个答案

我在上面评论中的主张是错误的，实际上位图表是救世主。

(option = 1)

ruby version 2.0.0
 proces   pid log          priv_dirty shared_dirty
 Parent 14807 post alloc           27          0
 Parent 14807 4 fork                0         27
 Child  14809 4 initial             0         27
 Child  14809 8 empty GC            6         25 # << almost everything stays shared <<
Run Code Online (Sandbox Code Playgroud)

还手工测试了 Ruby Enterprise Edition，它只比最坏情况好一半。

ruby version 1.8.7
 proces   pid log          priv_dirty shared_dirty
 Parent 15064 post alloc           86          0
 Parent 15064 4 fork                2         84
 Child  15065 4 initial             2         84
 Child  15065 8 empty GC           40         46
Run Code Online (Sandbox Code Playgroud)

（我让脚本严格运行 1 次 GC，增加到RUBY_GC_HEAP_INIT_SLOTS600k）

归档时间：

查看次数：

620 次

最近记录：

10 年，9 月 前
相关归档
使用Markdown获得带下划线的文字 75
使用paperclip使用Activeadmin Rails上传文件 41
Ruby on Rails-:符号,@ iVars和"字符串" - 哦,我的! 21
Rails递归地包含javascripts资源文件夹 20
是否可以在ruby中包含每个对象的模块？ 17
Intellij想法中的Ruby设置 13
Sidekiq和rails 4 actionmailer永远不会发送电子邮件 12
如何使用 ActionText 显示嵌入视频 12
Ruby - 测试数据库连接是否可行 9
包含实现IDisposable的项目的会话 7
难疑归档
'git pull'和'git fetch'有什么区别？ 11447
如何将新的本地分支推送到远程Git存储库并跟踪它？ 4154
如何在Java中调用另一个构造函数？ 2144
如何从Python字典中删除密钥？ 1539
JavaScript发布请求,如表单提交 1465
如何有效地计算JavaScript中对象的键/属性数？ 1452
每位程序员应阅读的最具影响力的单一书籍是什么？ 1439
什么是monad？ 1373
静态只读与const 1349
如何撤消'git reset'？ 1172