（超メモ）Elastic Beanstalk の Worker Tier について（cron っぽいことをやってみる）

tl;dr

Elastic Beanstalk の Worker Tier と Worker Tier で Cron っぽいことを試してみたメモ。

参考

AWS Black Belt Tech シリーズ 2015 - AWS Elastic Beanstalk from Amazon Web Services Japan

www.slideshare.net

Worker Tier について

その前に Elastic Beanstalk について

読み方的には「エラスティック・ビーン・ストーク」で「ビーンズ・トーク」では無い（重要）
PaaS（アプリケーション以下の環境は AWS がマネジメントしてくれる）
各種言語に対応
ELB と AutoScaling や RDS だって利用可能
Web Tier（今回は触れない）と Worker Tier がある

Worker Tier

SQS キューに書き込むことができる AWS サービスのバックグラウンド処理タスクを実行する
aws-sqsd が SQS のキューを監視、キューにメッセージが届いたらローカルホストのアプリケーションにアクセスする（下図参照）

（出典：Worker Environments）

Worker Tier を作成すると 2 つのキューが作成される（Worker Queue と DeadLetterr Queue）
DeadLetterr Queue を設けることで正常に処理出来なかったキューを後から確認することが出来る

トライ

IAM role を作る

Worker Tier を利用する場合には EC2 から SQS 等の各種 AWS リソースにアクセスする為の IAM role を作成する必要があるので、今回はマネジメントコンソールから Worker Tier を作成した際に自動で作成してくれる以下の IAM role を流用する。

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "QueueAccess",
      "Action": [
        "sqs:ChangeMessageVisibility",
        "sqs:DeleteMessage",
        "sqs:ReceiveMessage",
        "sqs:SendMessage"
      ],
      "Effect": "Allow",
      "Resource": "*"
    },
    {
      "Sid": "MetricsAccess",
      "Action": [
        "cloudwatch:PutMetricData"
      ],
      "Effect": "Allow",
      "Resource": "*"
    },
    {
      "Sid": "BucketAccess",
      "Action": [
        "s3:Get*",
        "s3:List*",
        "s3:PutObject"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::elasticbeanstalk-*-xxxxxxxxxxxx/*",
        "arn:aws:s3:::elasticbeanstalk-*-xxxxxxxxxxxx-*/*"
      ]
    },
    {
      "Sid": "DynamoPeriodicTasks",
      "Action": [
        "dynamodb:BatchGetItem",
        "dynamodb:BatchWriteItem",
        "dynamodb:DeleteItem",
        "dynamodb:GetItem",
        "dynamodb:PutItem",
        "dynamodb:Query",
        "dynamodb:Scan",
        "dynamodb:UpdateItem"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:dynamodb:*:xxxxxxxxxxxx:table/*-stack-AWSEBWorkerCronLeaderRegistry*"
      ]
    }
  ]
}

この IAM role を aws-elasticbeanstalk-ec2-worker-role という名前で保存しておく。必要に応じて他の AWS リソースへのアクセスポリシーも適宜追加する。

ebcli で新しいアプリケーション

ebcli を利用して作業を進める。ebcli についてはこちらのドキュメントを参考に進める。

% eb init

Select a default region
1) us-east-1 : US East (N. Virginia)
2) us-west-1 : US West (N. California)
3) us-west-2 : US West (Oregon)
4) eu-west-1 : EU (Ireland)
5) eu-central-1 : EU (Frankfurt)
6) ap-southeast-1 : Asia Pacific (Singapore)
7) ap-southeast-2 : Asia Pacific (Sydney)
8) ap-northeast-1 : Asia Pacific (Tokyo)
9) sa-east-1 : South America (Sao Paulo)
10) cn-north-1 : China (Beijing)
(default is 3): 8

Select an application to use
1) oreno-worker
2) eb-sinatra
3) [ Create new Application ]
(default is 3): 3

Enter Application Name
(default is "oreno-eb-sinatra"): oreno-eb-sinatra-worker
Application oreno-eb-sinatra-worker has been created.

It appears you are using Ruby. Is this correct?
(y/n): y

Select a platform version.
1) Ruby 2.2 (Puma)
2) Ruby 2.1 (Puma)
3) Ruby 2.0 (Puma)
4) Ruby 2.2 (Passenger Standalone)
5) Ruby 2.1 (Passenger Standalone)
6) Ruby 2.0 (Passenger Standalone)
7) Ruby 1.9.3
(default is 1): 1
Do you want to set up SSH for your instances?
(y/n): y

Select a keypair.
1) key1
2) key2
3) key3
4) key4
5) [ Create new KeyPair ]
(default is 5): 3

雑な Worker アプリケーション

Sinatra を使って / に HTTP アクセスすると /tmp/sinatra.log に Hello World! とリクエストヘッダをダンプするだけの超簡単な Worker アプリケーションを作ってみる。

Gemfile

source 'http://rubygems.org'
gem 'sinatra'

config.ru

require './helloworld'
run Sinatra::Application

helloworld.rb

require 'sinatra'
require 'logger'

logger = Logger.new('/tmp/sinatra.log')

post '/' do
  logger.info "Hello World!"
  http_headers = request.env.select { |k, v| k.start_with?('HTTP_')}
  logger.info http_headers
end

ebcli で Worker Tier を起動する

ebcli で Worker Tier を起動するには --tier worker オプションを利用する。また、合わせて --instance_profile で IAM role を指定する。

% eb create --tier worker --instance_profile aws-elasticbeanstalk-ec2-worker-role
Enter Environment Name
(default is oreno-eb-sinatra-worke): oreno-eb-sinatra-worker
WARNING: You have uncommitted changes.
Creating application version archive "app-52ef-151129_122719".
Uploading oreno-eb-sinatra-worker/app-52ef-151129_122719.zip to S3. This may take a while.
Upload Complete.
Environment details for: oreno-eb-sinatra-worker
  Application name: oreno-eb-sinatra-worker
  Region: ap-northeast-1
  Deployed Version: app-52ef-151129_122719
  Environment ID: e-xxxxxxxxxx
  Platform: 64bit Amazon Linux 2015.09 v2.0.4 running Ruby 2.2 (Puma)
  Tier: Worker-SQS/HTTP
  CNAME: UNKNOWN
  Updated: 2015-11-29 03:27:23.689000+00:00
Printing Status:
INFO: createEnvironment is starting.
INFO: Using elasticbeanstalk-ap-northeast-1-xxxxxxxxxxxx as Amazon S3 storage bucket for environment data.
INFO: Environment health has transitioned to Pending. There are no instances.
INFO: Created security group named: awseb-e-xxxxxxxxxx-stack-AWSEBSecurityGroup-XXXXXXXXXXXXXX
INFO: Created Auto Scaling launch configuration named: awseb-e-xxxxxxxxxx-stack-AWSEBAutoScalingLaunchConfiguration-XXXXXXXXXXXXXX
INFO: Added instance [i-xxxxxxx] to your environment.
INFO: Created Auto Scaling group named: awseb-e-xxxxxxxxxx-stack-AWSEBAutoScalingGroup-XXXXXXXXXXXXXX
INFO: Waiting for EC2 instances to launch. This may take a few minutes.
INFO: Created Auto Scaling group policy named: arn:aws:autoscaling:ap-northeast-1:xxxxxxxxxxxx:scalingPolicy:65f129b5-91f6-4ef7-afc6-ca0cc21cbad2:autoScalingGroupName/awseb-e-xxxxxxxxxx-stack-AWSEBAutoScalingGroup-XXXXXXXXXXXXXX:policyName/awseb-e-xxxxxxxxxx-stack-AWSEBAutoScalingScaleUpPolicy-XXXXXXXXXXXXXX
INFO: Created Auto Scaling group policy named: arn:aws:autoscaling:ap-northeast-1:xxxxxxxxxxxx:scalingPolicy:118024ed-e434-45a3-9aa2-cdda3d60aa11:autoScalingGroupName/awseb-e-xxxxxxxxxx-stack-AWSEBAutoScalingGroup-XXXXXXXXXXXXXX:policyName/awseb-e-xxxxxxxxxx-stack-AWSEBAutoScalingScaleDownPolicy-XXXXXXXXXXXXXX
INFO: Created CloudWatch alarm named: awseb-e-xxxxxxxxxx-stack-AWSEBCloudwatchAlarmHigh-XXXXXXXXXXXXXX
INFO: Created CloudWatch alarm named: awseb-e-xxxxxxxxxx-stack-AWSEBCloudwatchAlarmLow-XXXXXXXXXXXXXX
INFO: Successfully loaded 1 scheduled tasks from cron.yaml.
INFO: Environment health has transitioned from Pending to Ok.
INFO: Successfully launched environment: oreno-eb-sinatra-worker

Ctrl + c でプロンプトに戻ることが出来る。その間もインスタンスの作成等の環境構築が進行している。その確認は eb statsu で確認することが出来る。（以下の状態はインスタンスの起動中の為 status: Launching で Health: Grey となっている。また、Tier: Worker-SQS/HTTP となっており Worker として環境が起動していることが判る。）

% eb status
Environment details for: oreno-eb-sinatra-worker
  Application name: oreno-eb-sinatra-worker
  Region: ap-northeast-1
  Deployed Version: None
  Environment ID: e-wruk4dmtq8
  Platform: 64bit Amazon Linux 2015.09 v2.0.4 running Ruby 2.2 (Puma)
  Tier: Worker-SQS/HTTP
  CNAME: UNKNOWN
  Updated: 2015-11-29 13:54:04.618000+00:00
  Status: Launching
  Health: Grey

暫くすると...以下のように Status と Health の状態が変わって必要な環境の構築が終了となる。

% eb status
Environment details for: oreno-eb-sinatra-worker
  Application name: oreno-eb-sinatra-worker
  Region: ap-northeast-1
  Deployed Version: app-52ef-151129_225352
  Environment ID: e-wruk4dmtq8
  Platform: 64bit Amazon Linux 2015.09 v2.0.4 running Ruby 2.2 (Puma)
  Tier: Worker-SQS/HTTP
  CNAME: UNKNOWN
  Updated: 2015-11-29 13:56:58.315000+00:00
  Status: Launching
  Health: Green

以下のように SQS のキューも作成されている。

f:id:inokara:20151129221251p:plain

おお。

eb ssh でインスタンスにログイン

% eb ssh
INFO: Attempting to open port 22.
INFO: SSH port 22 open.
Last login: Sun Nov 29 03:33:32 2015 from xxx.xxx.xxx.jp
 _____ _           _   _      ____                       _        _ _
| ____| | __ _ ___| |_(_) ___| __ )  ___  __ _ _ __  ___| |_ __ _| | | __
|  _| | |/ _` / __| __| |/ __|  _ \ / _ \/ _` | '_ \/ __| __/ _` | | |/ /
| |___| | (_| \__ \ |_| | (__| |_) |  __/ (_| | | | \__ \ || (_| | |   <
|_____|_|\__,_|___/\__|_|\___|____/ \___|\__,_|_| |_|___/\__\__,_|_|_|\_\
                                       Amazon Linux AMI

This EC2 instance is managed by AWS Elastic Beanstalk. Changes made via SSH 
WILL BE LOST if the instance is replaced by auto-scaling. For more information 
on customizing your Elastic Beanstalk environment, see our documentation here: 
http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers-ec2.html
-bash: warning: setlocale: LC_CTYPE: cannot change locale (UTF-8): No such file or directory
[ec2-user@ip-xx-xxx-xxx-xxx ~]$

手動でキューにメッセージを送る

f:id:inokara:20151129220325p:plain

以下のようにアプリケーションのログが出力されている。

$ cat /tmp/sinatra.log 
# Logfile created on 2015-11-29 03:31:05 +0000 by logger.rb/47272
I, [2015-11-29T03:34:25.483217 #2023]  INFO -- : Hello World!
I, [2015-11-29T03:34:25.483344 #2023]  INFO -- : {"HTTP_VERSION"=>"HTTP/1.0", "HTTP_HOST"=>"localhost", "HTTP_X_FORWARDED_FOR"=>"127.0.0.1", "HTTP_CONNECTION"=>"close", "HTTP_USER_AGENT"=>"aws-sqsd/2.0", "HTTP_X_AWS_SQSD_MSGID"=>"ff89eb93-b3cc-46e2-860f-2b6a500b32b3", "HTTP_X_AWS_SQSD_RECEIVE_COUNT"=>"1", "HTTP_X_AWS_SQSD_FIRST_RECEIVED_AT"=>"2015-11-29T03:34:25Z", "HTTP_X_AWS_SQSD_SENT_AT"=>"2015-11-29T03:34:25Z", "HTTP_X_AWS_SQSD_QUEUE"=>"awseb-e-xxxxxxxxxx-stack-AWSEBWorkerQueue-XXXXXXXXXXXXXX", "HTTP_X_AWS_SQSD_PATH"=>"", "HTTP_X_AWS_SQSD_SENDER_ID"=>"xxxxxxxxxxxx"}

cron っぽいことをする

こちらやこちらを参考にして cron っぽいことをやってみたいので、アプリケーションのルートディレクトリに cron.yaml を以下のように作成する。

version: 1
cron:
 - name: "test"
   url: "/"
   schedule: "*/5 * * * *"

上記の設定で aws-sqsd が 5 分毎に / に HTTP アクセスする。

作成したらアプリケーションをデプロイする。

% ls -l
total 32
-rw-r--r--  1 kappa  staff   43 Nov 28 19:30 Gemfile
-rw-r--r--  1 kappa  staff   48 Nov 28 19:29 config.ru
-rw-r--r--  1 kappa  staff   72 Nov 29 11:42 cron.yaml
drwxr-xr-x  3 kappa  staff  102 Nov 29 11:43 elasticbeanstalk
-rw-r--r--  1 kappa  staff  218 Nov 29 01:13 helloworld.rb
% git add .
% git commit -m "add cron.yaml"
% eb deploy
WARNING: You have uncommitted changes.
Creating application version archive "app-52ef-151129_221803".
Uploading oreno-eb-sinatra-worker/app-52ef-151129_221803.zip to S3. This may take a while.
Upload Complete.
INFO: Environment update is starting.                               
INFO: Deploying new version to instance(s).                         
INFO: Successfully loaded 1 scheduled tasks from cron.yaml.         
INFO: New application version was deployed to running EC2 instances.
INFO: Environment update completed successfully.

インスタンスにログインして確認する。

$ tail -f /tmp/sinatra.log
I, [2015-11-29T02:49:59.908826 #2708]  INFO -- : Hello World!
I, [2015-11-29T02:49:59.908955 #2708]  INFO -- : {"HTTP_VERSION"=>"HTTP/1.0", "HTTP_HOST"=>"localhost", "HTTP_X_FORWARDED_FOR"=>"127.0.0.1", "HTTP_CONNECTION"=>"close", "HTTP_USER_AGENT"=>"aws-sqsd/2.0", "HTTP_X_AWS_SQSD_MSGID"=>"3422e8f4-7023-4bf0-89e7-db882fdf9618", "HTTP_X_AWS_SQSD_RECEIVE_COUNT"=>"1", "HTTP_X_AWS_SQSD_FIRST_RECEIVED_AT"=>"2015-11-29T02:49:59Z", "HTTP_X_AWS_SQSD_SENT_AT"=>"2015-11-29T02:49:59Z", "HTTP_X_AWS_SQSD_QUEUE"=>"awseb-e-qmjp6upame-stack-AWSEBWorkerQueue-XXXXXXXXXXXXX", "HTTP_X_AWS_SQSD_PATH"=>"/", "HTTP_X_AWS_SQSD_SENDER_ID"=>"XXXXXXXXXXXXXXXXXXXXXX:i-xxxxxxxx", "HTTP_X_AWS_SQSD_SCHEDULED_AT"=>"2015-11-29T02:50:00Z", "HTTP_X_AWS_SQSD_TASKNAME"=>"test"}
I, [2015-11-29T02:54:59.878897 #2708]  INFO -- : Hello World!
I, [2015-11-29T02:54:59.879049 #2708]  INFO -- : {"HTTP_VERSION"=>"HTTP/1.0", "HTTP_HOST"=>"localhost", "HTTP_X_FORWARDED_FOR"=>"127.0.0.1", "HTTP_CONNECTION"=>"close", "HTTP_USER_AGENT"=>"aws-sqsd/2.0", "HTTP_X_AWS_SQSD_MSGID"=>"3fde7c59-7535-4ec0-9567-374a90a24eb7", "HTTP_X_AWS_SQSD_RECEIVE_COUNT"=>"1", "HTTP_X_AWS_SQSD_FIRST_RECEIVED_AT"=>"2015-11-29T02:54:59Z", "HTTP_X_AWS_SQSD_SENT_AT"=>"2015-11-29T02:54:59Z", "HTTP_X_AWS_SQSD_QUEUE"=>"awseb-e-qmjp6upame-stack-AWSEBWorkerQueue-XXXXXXXXXXXXX", "HTTP_X_AWS_SQSD_PATH"=>"/", "HTTP_X_AWS_SQSD_SENDER_ID"=>"XXXXXXXXXXXXXXXXXXXXXX:i-xxxxxxxx", "HTTP_X_AWS_SQSD_SCHEDULED_AT"=>"2015-11-29T02:55:00Z", "HTTP_X_AWS_SQSD_TASKNAME"=>"test"}

タイムスタンプを見ると 5 分ごとにアクセスが発生していることが判る。

あとかたづけ

EC2 や SQS もまとめてターミネート。

% eb terminate
The environment "oreno-eb-sinatra-worker" and all associated instances will be terminated.
To confirm, type the environment name: oreno-eb-sinatra-worker
INFO: terminateEnvironment is starting.
INFO: Deleted CloudWatch alarm named: awseb-e-xxxxxxxxxx-stack-AWSEBCloudwatchAlarmLow-XXXXXXXXXXXXX 
INFO: Deleted CloudWatch alarm named: awseb-e-xxxxxxxxxx-stack-AWSEBCloudwatchAlarmHigh-XXXXXXXXXXXXX 
INFO: Deleted Auto Scaling group policy named: arn:aws:autoscaling:ap-northeast-1:XXXXXXXXXXXX:scalingPolicy:65f129b5-91f6-4ef7-afc6-ca0cc21cbad2:autoScalingGroupName/awseb-e-xxxxxxxxxx-stack-AWSEBAutoScalingGroup-XXXXXXXXXXXXX:policyName/awseb-e-xxxxxxxxxx-stack-AWSEBAutoScalingScaleUpPolicy-XXXXXXXXXXXXX
INFO: Deleted Auto Scaling group policy named: arn:aws:autoscaling:ap-northeast-1:XXXXXXXXXXXX:scalingPolicy:118024ed-e434-45a3-9aa2-cdda3d60aa11:autoScalingGroupName/awseb-e-xxxxxxxxxx-stack-AWSEBAutoScalingGroup-XXXXXXXXXXXXX:policyName/awseb-e-xxxxxxxxxx-stack-AWSEBAutoScalingScaleDownPolicy-XXXXXXXXXXXXXINFO: Waiting for EC2 instances to terminate. This may take a few minutes.
INFO: Deleted Auto Scaling group named: awseb-e-xxxxxxxxxx-stack-AWSEBAutoScalingGroup-XXXXXXXXXXXXXINFO: Deleted Auto Scaling launch configuration named: awseb-e-xxxxxxxxxx-stack-AWSEBAutoScalingLaunchConfiguration-XXXXXXXXXXXXX
INFO: Deleted security group named: awseb-e-xxxxxxxxxx-stack-AWSEBSecurityGroup-XXXXXXXXXXXXX
INFO: Removed instance [i-xxxxxxxx] from your environment.
INFO: Deleting SNS topic for environment oreno-eb-sinatra-worker.
INFO: terminateEnvironment completed successfully.

以上

Elastic Beanstalk を使って Cron っぽい動作は cron.yaml を置くだけで簡単に実装出来ることが解った。Lambda と単純に比較出来るわけでもなく、サーバレスアーキテクチャでも無いが、既存のアプリケーションを手軽にバッチ処理として利用する面では Elastic Beanstalk の Worker Tier も選択肢の一つになると考えている。（※実際にプロダクション環境等で利用する場合にはアプリケーションのエラー処理、タイムアウト処理等を実装する必要がある）

以上。

ようへいの日々精進XP

よかろうもん