Web Scale with Acme Air (Project Scale) - Part One

This blog post is part one of a post about the work we've been doing in the performance space around applications scalability.  While the second part will talk to the results we can share on Acme Air, this first part will cover some of the history and context.

Over the last year, I have been looking at the benchmarking space and how current benchmarks are useful to show scaling of the technologies.  If you look at some of the most popular standardized benchmarks, the scaling aspects are less relevant than they used to be.  In the past it was interesting to prove that certain middleware and data technologies could scale up to larger SMP systems and across clusters of SMP systems.  However, as the hardware systems have gotten larger some of the current approaches to scaling applications have become less relevant (pinning processes to network cards, CPU sockets, memory subsystems, etc.) as most single applications don't need to scale to the size of modern large scale systems and most users can't afford to replicate such complex tuning.  These complex tunings are very error prone for general users and if not applied properly can lead to worse performance than not using them at all.

Also, from a middleware perspective high availability isn't usually tested fully.  Many of the benchmarks do not require realistic high availability in the middle tier, which allows for stateful implementations that wouldn't work under failure.  Meanwhile the data tier remains monolithic and non-distributed.  This means the benchmark competition isn't based on technical merit, but who can buy and optimize the largest database.  In fact some of the most popular standardized benchmarks have documented costs ranging from a few million dollars to in one case thirty million dollars.

For these reasons, we have chosen for existing middleware benchmarks to focus on number of transactions per processing processor core as (a) this reflects true engineering performance work in our products regardless of scale and (b) this is directly related to how people pay for products - any improvement in this metric means cost savings for end users.  Keep this history in mind as we discuss the performance of Acme Air and most specifically the scaling work on Acme Air.  We call the scale up work using Acme Air "Project Scale".

When I looked externally to see who was gauging scale in the world, I found the programmable web's "Billionaire's Club".  This study documents the trend of supporting Web API calls of what used to be browser facing clients and now more and more is not only browsers, but also mobile and business to business Web API's.  If you look at the web and cloud today, the list of use cases and billionaires presented in this presentation better document the scaling that matters today than standardized benchmarks.  I agree with the motivating factors listed in the presentation for why this modern scaling is occurring - mobile and partner web API enablement many times driven by consumer facing applications.  I would add to that the sensors and the internet of things based use cases.  Consider the number of sensors in automobiles, those tracking your bags during airline flights, traffic monitoring, streams of point of sales purchases, etc.  When you put mobile together with the internet of things, the scale of applications growth with new bounds not yet known in typical enterprise applications.  Also, you will note that almost every current billionaire is a "born on the cloud" application.  Cloud is the more typical approach to deployment these days in large scale systems.  Deployment on the cloud changes the approach to running and tuning workloads dramatically over approaches in standardized benchmarks.  Finally, if you look at some of the top billionaires they are documenting through blogs and open source how they are getting there (NetflixOSS and Linkedin for example).

As I looked to see what it takes to scale middleware to the levels shown by this billionaire's club my experience told me it would be important to consider typical enterprise qualities of service - high availability, security, transactionality (when required).  There are many real world examples and benchmarks that did not consider those aspects and when the world discovered the end result was usually a black eye to any existing performance claims.  First on the "billionaire's club" side, many of the companies born on the cloud found success greater than their initial expectations and failed under increased load at some point in time.  On the technology side, a great example was some of the initial MongoDB benchmarking which were not run with more advanced durability and replication configurations.  A funny (and NSFW) video that documents this is the MongoDB is Web Scale video.  To be fair to MongoDB, since those times the benchmarks have been re-run with better levels of durability and availability.  In any benchmark of large scale, the benchmark and technologies employed must consider high availability, security, and transactionality.

Given all of this my team set out to create a workload (that is embodied in the OSS released recently called Acme Air) and run it to levels of scale that would put it in the billionaire's club reaching a few billion mobile and browser requests per day.  We decided early on to do the scaling work in the cloud.  We also made sure we continuously looked at the requirements for security, high availability, and transactionality.  We also decided to do the work in the open (through open sources, discussion forums and blogs) to get the same level of worldwide understanding that the similar billionaire's club members have.

In the next post, I'll document our scaling results of Project Scale.  I hope this context will give you enough background to understand why we are doing this work in the way we are.

Northwest Raleigh Wifi Speeds

Likely not much interest to others, but if you're a remote worker in the NW Raleigh area, feel free to contribute speedtest results.  I'm personally doing this to prove that the Brier Creek Caribou is getting poor service from their ISP.

Brier Creek Caribou Coffee

• 6/10/2013 - 9:45 AM
• Test 1 - Ping 299, Download Speed - 0.44 Mbps, Upload Speed - 0.23 Mbps
• Test 2 - Ping 183, Download Speed - 0.51 Mbps, Upload Speed - 0.36 Mbps
• Test 3 - Ping 193, Download Speed - 0.44 Mbps, Upload Speed - 0.27 Mbps
• 6 other laptops, using www.speedtest.net website

• 6/11/2013 - 10:56 AM
• Test 1 - Ping 53, Download Speed - 0.74 Mbps, Upload Speed - 0.56 Mbps
• Test 2 - Ping 55, Download Speed - 0.75 Mbps, Upload Speed - 0.48 Mbps
• Test 3 - Ping 53, Download Speed - 0.74 Mbps, Upload Speed - 0.67 Mbps
• 3 other laptops, using www.speedtest.net website

• 6/14/2013 - 12:21 PM
• Test 1 - Ping 80, Download Speed - 0.13 Mbps, Upload Speed - 0.52 Mbps
• Test 2 - Ping 102, Download Speed - 0.76 Mbps, Upload Speed - 0.55 Mbps
• Test 3 - Ping 67, Download Speed - 0.51 Mbps, Upload Speed - 0.56 Mbps
• 6 other laptops, using www.speedtest.net website

• 6/14/2013 - 12:41 PM (these tests failed and had to be restarted)
• Test 1 - Ping 380, Download Speed - 0.10 Mbps, Upload Speed - 0.39 Mbps
• Test 2 - Ping 152, Download Speed - 0.09 Mbps, Upload Speed - 0.25 Mbps
• Test 3 - Ping 106, Download Speed - 0.09 Mbps, Upload Speed - 0.59 Mbps
• 6 other laptops, using www.speedtest.net website

• 6/18/2013 - 10:20 AM
• Test 1 - Ping 100, Download Speed - 0.16 Mbps, Upload Speed - 0.63 Mbps
• Test 2 - Ping 93, Download Speed - 0.16 Mbps, Upload Speed - 0.60 Mbps
• Test 3 - Ping 97, Download Speed - 0.30 Mbps, Upload Speed - 0.36 Mbps
• 6 other laptops, using www.speedtest.net website

• 6/19/2013 - 9:15 AM
• Test 1 - Ping 44, Download Speed - 0.69 Mbps, Upload Speed - 0.50 Mbps
• Test 2 - Ping 44, Download Speed - 0.47 Mbps, Upload Speed - 0.43 Mbps
• Test 3 - Ping 45, Download Speed - 0.51 Mbps, Upload Speed - 0.54 Mbps
• 0 other laptops first test, 3 by last test, using www.speedtest.net website

• 6/20/2013 - 1:51 PM
• Test 1 - Ping 52, Download Speed - 0.51 Mbps, Upload Speed - 0.60 Mbps
• Test 2 - Ping 44, Download Speed - 0.49 Mbps, Upload Speed - 0.58 Mbps
• Test 3 - Ping 45, Download Speed - 0.44 Mbps, Upload Speed - 0.37 Mbps
• 3 other laptops, using www.speedtest.net website

• 9/8/2013 - 7:37 AM
• Test 1 - Ping 52, Download Speed - 0.74 Mbps, Upload Speed - 0.66 Mbps
• Test 2 - Ping 52, Download Speed - 0.77 Mbps, Upload Speed - 0.61 Mbps
• Test 3 - Ping 52, Download Speed - 0.53 Mbps, Upload Speed - 0.67 Mbps
• 0 other laptops, using www.speedtest.net website

At Earthfare Brier Creek

• 6/18/2013 - 2:04 PM
• Test 1 - Ping 62, Download Speed - 2.79 Mbps, Upload Speed - 0.65 Mbps
• Test 2 - Ping 64, Download Speed - 2.82 Mbps, Upload Speed - 0.66 Mbps
• Test 3 - Ping 64, Download Speed - 2.86 Mbps, Upload Speed - 0.66 Mbps
• 1 other laptops, using www.speedtest.net website

• 6/20/2013 - 10:22 PM
• Test 1 - Ping 55, Download Speed - 2.63 Mbps, Upload Speed - 0.69 Mbps
• Test 2 - Ping 56, Download Speed - 2.85 Mbps, Upload Speed - 0.68 Mbps
• Test 3 - Ping 56, Download Speed - 2.84 Mbps, Upload Speed - 0.69 Mbps
• 0 other laptops, using www.speedtest.net website

At Bruegers Bagels

• Test 1 - Ping 203, Download Speed - 2.84 Mbps, Upload Speed - 0.12 Mbps
• Test 2 - Ping 297, Download Speed - 2.56 Mbps, Upload Speed - 0.10 Mbps
• Test 3 - Ping 163, Download Speed - 1.85 Mbps, Upload Speed - 0.08 Mbps
• Test 4 - Ping 163, Download Speed - 1.85 Mbps, Upload Speed - 0.08 Mbps
• Test 5 - Ping 1533, Download Speed - 2.66 Mbps, Upload Speed - 0.07 Mbps
• Test 6 - Ping 1433, Download Speed - 0.81 Mbps, Upload Speed - 0.09 Mbps
• Test 7 - Ping 190, Download Speed - 2.45 Mbps, Upload Speed - 0.07 Mbps
• Not sure how devices were connected, using speedtest app on friend's phone

At home (Time Warner Cable)

• 6/10/2013 - 8:22 PM
• Test 1 - Ping 59, Download Speed - 13.62 Mbps, Upload Speed - 0.70 Mbps
• Test 1 - Ping 57, Download Speed - 5.44 Mbps, Upload Speed - 0.77 Mbps
• Test 1 - Ping 45, Download Speed - 8.76 Mbps, Upload Speed - 0.96 Mbps
• Many other devices connected, using www.speedtest.net website

• 6/14/2013 - 2:50 PM
• Test 1 - Ping 28, Download Speed - 13.11 Mbps, Upload Speed - 0.97 Mbps
• Test 1 - Ping 28, Download Speed -  17.18 Mbps, Upload Speed - 0.99 Mbps
• Test 1 - Ping 28, Download Speed -  17.02 Mbps, Upload Speed - 0.98 Mbps
• Many other devices connected, using www.speedtest.net website

My phone (3G)

• 6/14/2013 - 12:31 PM
• Test 1 - Ping 205, Download Speed - 0.3 Mbps (386 kbps), Upload Speed - 0.029 Mbps (29 kbps)
• Test 2 - Ping 203, Download Speed - 0.3 Mbps (325 kbps), Upload Speed - 0.041 Mbps (41 kbps)
• Test 3 - Ping 243, Download Speed - 0.1 Mbps (116 kbps), Upload Speed - 0.071 Mbps (71 kbps)
• Sprint network using speedtest.net android application

My phone (4G)

• 6/14/2013 - 12:35 PM
• Test 1 - Ping 108, Download Speed - 1.182 Mbps (1182 kbps), Upload Speed - 0.745 Mbps (745 kbps)
• Test 2 - Ping 102, Download Speed - 1.716 Mbps (1716 kbps), Upload Speed - 0.796 Mbps (796 kbps)
• Test 3 - Ping 103, Download Speed - 1.555 Mbps (1555 kbps), Upload Speed - 0.733 Mbps (733 kbps)
• Sprint network using speedtest.net android application