可以避免Heroku上的ActiveRecord :: ConnectionTimeoutError？

Question

在Heroku上我有一个rails应用程序运行,有两个web dynos以及一个worker dyno.我在Sidekiq上全天运行数千个工作任务,但偶尔会引发ActiveRecord :: ConnectionTimeoutError(大约每天50次).我按如下方式设置了我的独角兽服务器

worker_processes 4
timeout 30
preload_app true

before_fork do |server, worker|
    # As suggested here: https://devcenter.heroku.com/articles/rails-unicorn
    Signal.trap 'TERM' do
        puts 'Unicorn master intercepting TERM and sending myself QUIT instead'
        Process.kill 'QUIT', Process.pid
    end

    if defined?(ActiveRecord::Base)
        ActiveRecord::Base.connection.disconnect!
    end
end

after_fork do |server,worker|
    if defined?(ActiveRecord::Base)
        config = Rails.application.config.database_configuration[Rails.env]
        config['reaping_frequency'] = ENV['DB_REAP_FREQ'] || 10 # seconds
        config['pool']            = ENV['DB_POOL'] || 10
        ActiveRecord::Base.establish_connection(config)
    end

    Sidekiq.configure_client do |config|
        config.redis = { :size => 1 }
    end

    Sidekiq.configure_server do |config|
        config = Rails.application.config.database_configuration[Rails.env]
        config['reaping_frequency'] = ENV['DB_REAP_FREQ'] || 10 # seconds
        config['pool']            = ENV['DB_POOL'] || 10
        ActiveRecord::Base.establish_connection(config)
    end
end

在heroku上我按照Heroku的建议将DB_POOL配置变量设置为2 .这些错误应该发生吗？似乎很奇怪,这是不可能避免这样的错误,不是吗？你会建议什么？

Answer 1

sidekiq服务器(在您的服务器上运行的实际执行延迟任务的进程)默认情况下最多可以拨打25个线程来处理其队列中的工作.如果您的任务需要,这些线程中的每一个都可以通过ActiveRecord请求与主数据库的连接.

如果你只有一个连接池有5个连接,但你有25个线程试图连接,5秒后线程将放弃,如果他们无法从池中获得可用连接,你将获得连接超时错误.

将Sidekiq服务器的池大小设置为更接近并发级别(-c在启动进程时使用标志设置)将有助于缓解此问题,但代价是打开更多数据库连接.例如,如果您使用Heroku并使用Postgres,他们的一些计划限制为20,而其他计划的连接限制为500(源).

如果您正在运行像Unicorn这样的多进程服务器环境,则还需要监视每个分叉进程所做的连接数.如果您有4个独角兽进程,并且默认连接池大小为5,那么任何给定时间的独角兽环境都可能有20个实时连接.您可以在Heroku的文档中阅读更多相关信息.另请注意,数据库池大小并不意味着每个dyno现在都有那么多打开的连接,但只有在需要新连接的情况下才会创建,直到最多创建了多个连接.

话虽如此,这就是我的所作所为.

# config/initializers/unicorn.rb

if ENV['RACK_ENV'] == 'development'
  worker_processes 1
  listen "#{ENV['BOXEN_SOCKET_DIR']}/rails_app"
  timeout 120
else
  worker_processes Integer(ENV["WEB_CONCURRENCY"] || 2)
  timeout 29
end

# The timeout mechanism in Unicorn is an extreme solution that should be avoided whenever possible. 
# It will help catch bugs in your application where and when your application forgets to use timeouts,
# but it is expensive as it kills and respawns a worker process.
# see http://unicorn.bogomips.org/Application_Timeouts.html

# Heroku recommends a timeout of 15 seconds. With a 15 second timeout, the master process will send a 
# SIGKILL to the worker process if processing a request takes longer than 15 seconds. This will 
# generate a H13 error code and you’ll see it in your logs. Note, this will not generate any stacktraces 
# to assist in debugging. Using Rack::Timeout, we can get a stacktrace in the logs that can be used for
# future debugging, so we set that value to something less than this one

preload_app true # for new relic

before_fork do |server, worker|
  Signal.trap 'TERM' do
    puts 'Unicorn master intercepting TERM and sending myself QUIT instead'
    Process.kill 'QUIT', Process.pid
  end

  if defined?(ActiveRecord::Base)
    ActiveRecord::Base.connection.disconnect!
  end

end

after_fork do |server, worker|
  Signal.trap 'TERM' do
    puts 'Unicorn worker intercepting TERM and doing nothing. Wait for master to sent QUIT'
  end

  Rails.logger.info("Done forking unicorn processes")

  #https://devcenter.heroku.com/articles/concurrency-and-database-connections
  if defined?(ActiveRecord::Base)

    db_pool_size = if ENV["DB_POOL"]
      ENV["DB_POOL"]
    else
      ENV["WEB_CONCURRENCY"] || 2
    end

    config = Rails.application.config.database_configuration[Rails.env]
    config['reaping_frequency'] = ENV['DB_REAP_FREQ'] || 10 # seconds
    config['pool']              = ENV['DB_POOL'] || 2
    ActiveRecord::Base.establish_connection(config)

    # Turning synchronous_commit off can be a useful alternative when performance is more important than exact certainty about the durability of a transaction
    ActiveRecord::Base.connection.execute "update pg_settings set setting='off' where name = 'synchronous_commit';"    

    Rails.logger.info("Connection pool size for unicorn is now: #{ActiveRecord::Base.connection.pool.instance_variable_get('@size')}")
  end

end

对于sidekiq:

# config/initializers/sidekiq.rb

Sidekiq.configure_server do |config|

  sidekiq_pool = ENV['SIDEKIQ_DB_POOL'] || 20

  if defined?(ActiveRecord::Base)
    Rails.logger.debug("Setting custom connection pool size of #{sidekiq_pool} for Sidekiq Server")
    db_config = Rails.application.config.database_configuration[Rails.env]
    db_config['reaping_frequency'] = ENV['DB_REAP_FREQ'] || 10 # seconds
    cb_config['pool']              = sidekiq_pool
    ActiveRecord::Base.establish_connection(db_config)

    Rails.logger.info("Connection pool size for Sidekiq Server is now: #{ActiveRecord::Base.connection.pool.instance_variable_get('@size')}")
  end
end

如果一切顺利,当您启动流程时,您会看到日志中的内容:

Setting custom connection pool size of 10 for Sidekiq Server
Connection pool size for Sidekiq Server is now: 20
Done forking unicorn processes
   (1.4ms)  update pg_settings set setting='off' where name = 'synchronous_commit';
Connection pool size for unicorn is now: 2

资料来源:

• https://devcenter.heroku.com/articles/concurrency-and-database-connections#connection-pool
• https://github.com/mperham/sidekiq/issues/503
• https://github.com/mperham/sidekiq/wiki/Advanced-Options