- Notifications
You must be signed in to change notification settings - Fork1
miketheman/fullstack
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
Full-stack DevOps demo
The purpose of this is to demonstrate some great automation tools in orchestra.
(in alphabetic order)
- Apache2 (withmod_wsgi)
- bluepill
- Bottle.py
- Chef (chef-client, ohai, knife)
- Datadog
- EC2 (Amazon Linux AMI)
- HAProxy
- MongoDB (server,ruby &python drivers)
- Python
- Ruby
- Siege
- Spiceweasel
- bluepill is a process manager, similar to SysV init, Upstart, supervisord, runit, etc.
- Bottle.py is a web micro-framework written in python.
- Siege creates web requests based on an input file for load testing.
- Spiceweasel generates Chef's knife commands from a config file
Thanks todiagrammr.
The Bottle.py application is a simplistic word counter, acts like a REST interface, where/insert/<someword> will add the word to the database and increment its counter.
The/get/<someword> will retrieve the word, the unique object ID, and the count of times this word was hit.
A call to/toplist will bring back the top 10 words that have been hit.
All cookbooks used are released on the Chef Community site.The only cookbooks here are a personalizedmongodb cookbook, due to a currently faulty community cookbook, and thefullstack cookbook, that performs the customized functions this stack needs.
Some EC2 security group work:
ec2-create-group fullstack -d "Full Stack Demo"# Allow pingsec2-authorize fullstack --protocol icmp --icmp-type-code=-1:-1 --source-or-dest-group fullstack# Could be shorter: ec2-authorize fullstack -P icmp -t=-1:-1 -o fullstackec2-authorize fullstack -P tcp -p 0-65535 -o fullstackec2-authorize fullstack -P udp -p 0-65535 -o fullstackec2-authorize fullstack -P tcp -p 22 # SSHec2-authorize fullstack -P tcp -p 80 # HTTP# Optional, don't use in a production environment unless neededec2-authorize fullstack -P tcp -p 22002 # HAProxy Statsec2-authorize fullstack -P tcp -p 8080 # Webapp nodeA chef server (open source or hosted) must exist, andknife.rb must be set up correctly with AWS credentials.My personal one is excluded from the repo.
I recommend using a dedicated server/organization since the cleanup actions are destructive.
current_dir = File.dirname(__FILE__)log_level :infolog_location STDOUTnode_name "<my username>"client_key "#{current_dir}/<my username>.pem"validation_client_name "<organization-name>-validator"validation_key "#{current_dir}/<organization-name>-validator.pem"chef_server_url "https://api.opscode.com/organizations/<organization-name>"cache_type 'BasicFile'cache_options( :path => "#{ENV['HOME']}/.chef/checksums" )cookbook_path ["#{current_dir}/../site-cookbooks"]# AWS credentialsknife[:ssh_user] = "ec2-user"knife[:ssh_identity_file] = "#{current_dir}/../.aws/<key pair cert>.pem"knife[:aws_access_key_id] = "<some key id>"knife[:aws_secret_access_key] = "<some secret string>"### END ###A Users Databag item must be placed indata_bags/users/<username>.json. An example is:
{ "id": "bofh", "ssh_keys": "ssh-rsa AAAAB3Nz...yhCw== bofh", "groups": "sysadmin", "uid": 2001, "shell": "\/bin\/bash", "comment": "BOFH",}See theusers cookbook for more help.
A Credentials databag item for Datadog monitoring is also excluded from the repo, since it contains API keys.Construct your own atdata_bags/credentials/datadog.json so:
{ "id": "datadog", "api_key": "YOURAPIKEY", "application_key": "APPLICATIONKEYFORCHEF"}You can obtain your API key and Application Key from the Integrations => API page, or from the Agent Setup => Chef page, you will find a "Generate Application Key" button.
spiceweasel fullspice.yml | bashGet the top list of words:
open http://`knife search node 'role:load_balancer' -a ec2.public_hostname |grep ec2.public_hostname | cut -f4 -d" "`/toplistHAProxy web console:
open http://`knife search node 'role:load_balancer' -a ec2.public_hostname |grep ec2.public_hostname | cut -f4 -d" "`:22002/Find the mongodb replset primary:
knife search node "fqdn:`knife ssh 'role:mongodb-replset-member' -a ec2.public_hostname 'curl http://localhost:28017/replSetGetStatus?text=1' | grep -B4 PRIMARY | grep name | awk '{print $4}' |cut -f1 -d":" | sed 's/^.\{1\}//' | uniq`" -iNOTE: This is probably overly complicated, but awesome. Probably better to have chef-client update the node record with the current state.
Kill the primary:
knife ec2 server delete --purge -y <instance-id from previous command># or:knife ec2 server delete --purge -y `knife search node "fqdn:`knife ssh 'role:mongodb-replset-member' -a ec2.public_hostname 'curl http://localhost:28017/replSetGetStatus?text=1' | grep -B4 PRIMARY | grep name | awk '{print $4}' |cut -f1 -d":" | sed 's/^.\{1\}//' | uniq`" -a ec2.instance_id | grep instance_id | cut -f2 -d":"`Launch a new mongodb replica:
spiceweasel fullspice.yml | grep --color=none rolemongodb-replset-member | uniq | bashAdd a webserver:
spiceweasel fullspice.yml | grep --color=none rolewebserver | uniq | bashShow counts of servers:
knife status -r | awk '{print $9}' | sort | uniq -cDo something on all nodes:
knife ssh '*:*' -a ec2.public_hostname 'hostname -f'Spiceweasel, in reverse:
spiceweasel -d fullspice.yml | bashknife client bulk delete i-.*That's all, folks!
- Mike Fiedler
- Daniel Crosta - huge help getting this going
