独角兽吃记忆

Question

我在亚马逊有一个m1.small实例,我的rails应用程序运行8GB硬盘空间.它运行平稳了2周,然后崩溃说内存已满.应用程序正在rails 3.1.1,unicorn和nginx上运行

我根本不明白13G是什么？
我杀死了独角兽,'free'命令显示了一些空闲空间,而df仍然说100%
我重新启动了实例,一切都开始正常.

免费(在杀死独角兽之前)

             total       used       free     shared    buffers     cached  
Mem:       1705192    1671580      33612          0     321816     405288  
-/+ buffers/cache:     944476     760716   
Swap:       917500      50812     866688 

df -l(在杀死独角兽之前)

Filesystem           1K-blocks      Used Available Use% Mounted on  
/dev/xvda1             8256952   7837520         4 100% /  
none                    847464       120    847344   1% /dev  
none                    852596         0    852596   0% /dev/shm  
none                    852596        56    852540   1% /var/run  
none                    852596         0    852596   0% /var/lock  
/dev/xvda2           153899044    192068 145889352   1% /mnt  
/dev/xvdf             51606140  10276704  38707996  21% /data  

sudo du -hc --max-depth = 1(在杀死独角兽之前)

28K ./root  
6.6M    ./etc  
4.0K    ./opt  
9.7G    ./data  
1.7G    ./usr  
4.0K    ./media  
du: cannot access `./proc/27220/task/27220/fd/4': No such file or directory  
du: cannot access `./proc/27220/task/27220/fdinfo/4': No such file or directory  
du: cannot access `./proc/27220/fd/4': No such file or directory  
du: cannot access `./proc/27220/fdinfo/4': No such file or directory  
0   ./proc  
14M ./boot  
120K    ./dev  
1.1G    ./home  
66M ./lib  
4.0K    ./selinux  
6.5M    ./sbin  
6.5M    ./bin  
4.0K    ./srv  
148K    ./tmp  
16K ./lost+found  
20K ./mnt  
0   ./sys  
253M    ./var  
13G .  
13G total   

免费(杀死独角兽后)

             total       used       free     shared    buffers     cached    
Mem:       1705192     985876     **719316**          0     365536     228576    
-/+ buffers/cache:     391764    1313428    
Swap:       917500      46176     871324  

df -l(杀死独角兽之后)

Filesystem           1K-blocks      Used Available Use% Mounted on  
/dev/xvda1             8256952   7837516         8 100% /  
none                    847464       120    847344   1% /dev  
none                    852596         0    852596   0% /dev/shm  
none                    852596        56    852540   1% /var/run  
none                    852596         0    852596   0% /var/lock  
/dev/xvda2           153899044    192068 145889352   1% /mnt  
/dev/xvdf             51606140  10276704  38707996  21% /data  

unicorn.rb

rails_env = 'production'  

working_directory "/home/user/app_name"  
worker_processes 5  
preload_app true  
timeout 60  

rails_root = "/home/user/app_name"  
listen "#{rails_root}/tmp/sockets/unicorn.sock", :backlog => 2048  
# listen 3000, :tcp_nopush => false  

pid "#{rails_root}/tmp/pids/unicorn.pid"  
stderr_path "#{rails_root}/log/unicorn/unicorn.err.log"  
stdout_path "#{rails_root}/log/unicorn/unicorn.out.log"  

GC.copy_on_write_friendly = true if GC.respond_to?(:copy_on_write_friendly=)  

before_fork do |server, worker|  
  ActiveRecord::Base.connection.disconnect!  

  ##  
  # When sent a USR2, Unicorn will suffix its pidfile with .oldbin and  
  # immediately start loading up a new version of itself (loaded with a new  
  # version of our app). When this new Unicorn is completely loaded  
  # it will begin spawning workers. The first worker spawned will check to  
  # see if an .oldbin pidfile exists. If so, this means we've just booted up  
  # a new Unicorn and need to tell the old one that it can now die. To do so  
  # we send it a QUIT.  
  #  
  # Using this method we get 0 downtime deploys.  

  old_pid = "#{rails_root}/tmp/pids/unicorn.pid.oldbin"  
  if File.exists?(old_pid) && server.pid != old_pid  
    begin  
      Process.kill("QUIT", File.read(old_pid).to_i)  
    rescue Errno::ENOENT, Errno::ESRCH  
      # someone else did our job for us  
    end  
  end  
end  


after_fork do |server, worker|  
  ActiveRecord::Base.establish_connection  
  worker.user('rails', 'rails') if Process.euid == 0 && rails_env == 'production'  
end  

Answer 1

我刚刚发布了' 独角兽工人杀手 '宝石.这使您可以根据1)最大请求数和2)进程内存大小(RSS)来杀死Unicorn worker,而不会影响请求.

它真的很容易使用.无需外部工具.首先,请将此行添加到您的Gemfile.

gem 'unicorn-worker-killer'

然后,请将以下行添加到您的config.ru.

# Unicorn self-process killer
require 'unicorn/worker_killer'

# Max requests per worker
use Unicorn::WorkerKiller::MaxRequests, 10240 + Random.rand(10240)

# Max memory size (RSS) per worker
use Unicorn::WorkerKiller::Oom, (96 + Random.rand(32)) * 1024**2

强烈建议随机化阈值以避免一次性杀死所有工人.

Answer 2

我认为你正在混淆内存使用和磁盘空间使用.它看起来像Unicorn和它的孩子使用大约500 MB的内存,你看第二个" - /+ buffers/cache:"数字来查看真正的空闲内存.就磁盘空间而言,我的赌注会出现在某种日志文件或类似的东西上.您应该在数据目录中执行du -h以找出使用如此多存储的确切内容.作为最后的建议,一个鲜为人知的事实是,如果Ruby分配它,它永远不会将内存返回给操作系统.它仍然在内部使用它,但是一旦Ruby抓住了一些内存,让它将未使用的内存返回给操作系统的唯一方法就是退出进程.例如,如果碰巧有一个进程将内存使用量增加到500 MB,即使请求完成且GC循环已运行,您也无法再次使用该500 MB.但是,Ruby将为未来的请求重用已分配的内存,因此不太可能进一步增长.

最后,谢尔盖提到上帝监视过程记忆.如果您有兴趣使用这个,已经有一个很好的配置文件在这里.请务必阅读相关文章,因为这个上帝配置假定您拥有的unicorn配置文件中有关键内容.

Answer 3

你可以设置上帝来监视你的独角兽工人，如果他们吃掉太多内存就杀死他们。然后，Unicorn master 进程将派生另一个工作进程来取代这个工作进程。问题解决了。:-)