Sunday, September 8, 2013

Acme Air NetflixOSS AMI's - For your consumption pleasure

TL;DR - Go here for AMI's for the core of the Netflix OSS platform using Acme Air as a sample.

Some time ago, we showed a working Acme Air recoded to use the Netflix OSS running on EC2 with Apache Tomcat and WebSphere eXtreme Scale at load.  Later, we showed the same thing with the WebSphere Liberty Profile and WebSphere eXtreme Scale.

While these were great demonstrations, they weren't the easiest for someone to replicate themselves.  To replicate the results, you have to build the Acme Air application from source and create two server images (back end auth service and front end web app), deploy and configure a WebSphere eXtreme Scale cluster, build Asgard from source and configure/deploy it as a server, build Eureka from source and configure/deploy it as a server, build Hystrix (for the console) and run it locally.  While the standardized github and gradle based builds help make the code really easy, for the most part, easy to compile, the steps to configure and deploy the various servers are not trivial.  It took us a few weeks of studying wiki's, reading code, and learning from the Netflix RSS sample to devise a "production" setup.  It could take much longer than two weeks, especially if you are not a cloud platform engineer.  The NetflixOSS platform has deep and broad value, but being able to bootstrap yourself on the platform is currently pretty tough.

We decided to make this easier for the masses by creating Amazing Machine Images (AMI's) of the six servers required to run a pretty basic setup of the platform.  We're not the first to create AMI's of Netflix server components.  Netflix themselves created an Asgard AMI and released it at the last meetup.  Other Netflix Cloud Prize participants are creating Netflix server AMI's as part of their submissions.  Some of those include Nicholas Whittier's work to create them with Puppet and Peter Sankauskas's work to create them with Ansible playbooks.  While these AMI's are valuable, they lack two important aspects - a sample application and the information of how each of the servers should be tied together in a way that showcases the entire platform.  Specifically, of the six AMI's required to run the end to end application of Acme Air, these AMI's would provide for two of the servers (Asgard and Eureka) but wouldn't help you immediately configure you application to access Eureka.  This is not to say these other AMI's aren't valuable.  They are likely built in a way that others can improve upon and learn from (something lacking in the AMI's we've built just to demonstrate the end to end configuration of the NetflixOSS platform).  Eventually it would be nice to see Peter's work and/or Netflix themselves have not only Netflix server AMI's but AMI's for a sample application such as Acme Air and instructions of how to startup the entire platform end to end.

We decided to freeze dry our running images into AMI's and provide instructions of how to re-bootstrap the entire platform in a minimalist configuration.  We figured that such an end to end view would be helpful to the community.  While it will be important to enhance such an effort later to show further configuration for enhanced HA, security, and operational excellence, we decided it would be interesting to show the basic configuration first.  You can get started using these AMI's using instructions we posted on the Acme Air Netflix wiki.  All of the contained technology for the AMI's come from Apache 2.0 licensed github projects (Tomcat, NetflixOSS and Acme Air github projects) and for development use only WebSphere products (WebSphere Liberty Profile and WebSphere eXtreme Scale).  I'd encourage you to review and agree to comply with the contained technologies' licenses before proceeding to work with the AMI's, but the main points are the use of the WebSphere technologies is for development use only and there is no support offered or implied by IBM or Apache or Netflix.  These AMI's are just samples to be learned from.

If you find the AMI's useful or have questions, drop a comment on this blog or visit the Acme Air user group mailing list (please note that you're using the Netflix version of Acme Air).