2014-03-06 Elasticbeanstalk Rails Config

Deploying a Ruby on Rails app to Amazon AWS Elastic Beanstalk, I had to make some changes to the default configuration. Luckily Elastic Beanstalk suppports a pretty powerful configuration file syntax. Nevertheless it took me some time to figure this all out hopefully I can save you some time:

Logentries

When using cloud services you need a central location for logs. I've used Logentries with good results so that's what I'm using but this setup should work for any service that supports syslog forwarding.

SSL Certificate

I'm not going to forward log entries in clear text. In order to support forwarding over ssl we need to install the SSL certificate on the server:

01-logentries-cert.config:

files:  
  "/etc/ssl/certs/logentries.all.crt":
    mode: "000644"
    group: root
    owner: root
    content: |
      [CERTIFICATE TEXT HERE]

This tells Elastic Beanstalk to create a file at /etc/ssl/certs/logentries.all.crt with the content of the certificate text. I've removed the actual certificate text since it's long. For logentries, I copied the the .logentries.com certificate, Intermediate certificate, and CA* certificate (available at https://logentries.com/doc/certificates) and pasted them in one after the other.

Forwarding to Logentries

Logentries has some documentation around integrating with Elastic Beanstalk but I found that it not only didn't suit my needs but didn't even work out of the box due to a syntax error!

02-logentries-forwarding.config:

packages:
  yum:
    rsyslog-gnutls: []
files:
  "/etc/rsyslog.d/00-files.conf":
    mode: "000644"
    owner: root
    group: root
    content: |
      # load file watcher module
      $Modload imfile

      # passenger
      $InputFileName /var/app/support/logs/passenger.log
      $InputFileTag passenger
      $InputFileStateFile passenger-state
      $InputFileSeverity info
      $InputFileFacility local6
      $InputRunFileMonitor

  "/etc/rsyslog.d/99-logentries.conf":
    mode: "000644"
    owner: root
    group: root
    content: |
      # set file poll seconds
      $InputFilePollInterval 10

      # enable ssl
      $DefaultNetstreamDriverCAFile /etc/ssl/certs/logentries.all.crt
      $ActionSendStreamDriver gtls
      $ActionSendStreamDriverMode 1
      $ActionSendStreamDriverAuthMode x509/name
      $ActionSendStreamDriverPermittedPeer *.logentries.com

      # log everything to logentries
      $template LogentriesFormat,"[TOKEN] %HOSTNAME% %syslogtag%%msg%\n"
      *.* @@api.logentries.com:20000;LogentriesFormat
commands:
  03restart-syslog:
    command: "service rsyslog restart"

There's a bit going on here.

First, in order for rsyslog to support forwarding over ssl we need the rsyslog-gnutls package. Then I specify 2 files:

/etc/rsyslog.d/00-files.conf configures rsyslog file monitoring ($Modload imfile) and then adds the Passenger log as a watched log file. /etc/rsyslog.d/99-logentries.conf sets the poll interval on all files to 10 (seconds), configures the ssl cert used to communicate with logentries (configured in the 01-logentries-cert.config), and adds the commands to forward everything to logentries. (note: [TOKEN] is should be replaced with your log token!).

Finally I set a command to run service rsyslog restart so that these changes will get picked up.

Sidekiq

I originally configured sidekiq according to these instructions posted by Aleksey Gureiev. However, I found that sidekiq errored out and I had to make a slight tweak - I removed line 17 (-e production). I also added a directive to forward the log file to logentries. Here's the full file I'm using:

03-sidekiq.config:

files:
  "/etc/rsyslog.d/11-sidekiq.conf":
    mode: '000644'
    content: |
      $InputFileName /var/app/support/logs/sidekiq.log
      $InputFileTag sidekiq
      $InputFileStateFile sidekiq-state
      $InputFileSeverity info
      $InputFileFacility local6
      $InputRunFileMonitor

  "/opt/elasticbeanstalk/hooks/appdeploy/post/50_restart_sidekiq":
    mode: "000777"
    owner: root
    group: root
    content: |
      cd /var/app/current

      if [ -f /var/app/support/pids/sidekiq.pid ]
      then
        kill -TERM `cat /var/app/support/pids/sidekiq.pid`
        rm -rf /var/app/support/pids/sidekiq.pid
      fi

      . /opt/elasticbeanstalk/support/envvars.d/sysenv

      sleep 10

      bundle exec sidekiq \
        -P /var/app/support/pids/sidekiq.pid \
        -C /var/app/current/config/sidekiq.yml \
        -L /var/app/support/logs/sidekiq.log \
        -d

  "/opt/elasticbeanstalk/hooks/appdeploy/pre/03_mute_sidekiq":
    mode: "000777"
    content: |
      if [ -f /var/app/support/pids/sidekiq.pid ]
      then
        kill -USR1 `cat /var/app/support/pids/sidekiq.pid`
      fi

Misc Packages

I needed to install some miscellaneous packages for the pg and rmagick gems:

10-misc-packages.config:

packages:
  yum:
    postgresql-libs: []
    postgresql-devel: []
    ImageMagick-devel: []
    ImageMagick: []