As I mentioned in my last blog post, I had a multi-faceted project over the holidays to try out continuous delivery, Cassandra, and Docker in the context of Acme Air. In this post, I wanted to focus on the Cassandra aspect.
First, why is this called part two if there is no part one? There is a part one of this story, but I never blogged about it. When Jonathan Bond and I were working on the Netflix Cloud Prize sample work, Jonathan implemented a Cassandra set of services for Acme Air using the NetflixOSS Astyanax Cassandra client and Astyanax's JPA like entity persister support. You can see his cassandra-cli DDL script for the column families, some samples of how he did queries and what the entities looked like by reading through github.
I started by looking at what Jonathan did and decided to recode for the following reasons. Jonathan was working under a time crunch for the cloud prize work and we had decided to keep the application as portable as possible across the tested data services. In the past we have implemented back end implementations for WebSphere eXtreme Scale and MongoDB. The reality is each different back end required specific additions to the entity model of the data returned to the web applications. WebSphere eXtreme Scale requires the use of PartitionableKey interfaces on primary key classes to co-locate related data in the grid (Users and User Bookings for example). Mongo required, when using the Spring Data MongoDB support, specific annotations to help the mapping and, when using Morphia, specific deserializers for things like BigDecimal. Jonathan's implementation tried to come up with a common interface for entities and then implementations specific to each back end, but I think the entity code became the least common denominator vs. the best demonstration of how to use Astyanax/Cassandra. Next Jonathan worked the problem top down starting with an entity model allowing the Astyanax persister support to create the columns dynamically. It wasn't clear to us, especially since we never performance or scale tested the final code, if this top down approach functioned well under load. Finally, I'm the kind of guy who wants to start with the most code possible and then move to higher level abstractions proving that they add value without sacrificing things like performance or maintainability of the tables in production. In the relational world, I have seen problems come from pure top down mapping approaches of relational to Java objects that were solved with more JDBC like approaches or more sophisticated meet in the middle mappings.
This all said, I started to back up and look for a way to start with a more (in the relational world) JDBC/SQL like approach. Given I was new to Cassandra, I got myself bootstrapped two ways. First, I grabbed Nicolas Favre-Felix's Docker Cassandra automation (more on that in my last blog post). Second, I went through the Datastax Java Developer online course in order to learn Cassandra and how to code to Cassandra in Java. I highly recommend the course but be aware that it will lead you to quickly embrace Cassandra 2.0.x and CQL3 (vs. Cassandra 1.2.x and Thrift) which eventually caused me problems. I didn't actually work through the sample application that accompanies the course, but instead planned to apply my learning to Acme Air. The course took three days off and on and I ended up passing the final exam to get my certification (yeah!).
My education complete, I took the current Acme Air NetflixOSS enabled codebase and branched it to a "Astyanax" branch. I coded up a static CQL3 DDL script and a bit of the data loader program and ran it. I quickly ran into the dreaded "InvalidRequestException(why:Not enough bytes to read value of component 0)" error. I found others confused by this on the Netflix Astyanax Google Group and found that I could add "COMPACT STORAGE" to my table definition and get limping along again. Later Michael Oczkowski responded to the forum post with great links I should have found that explained why COMPACT STORAGE was needed and what issues I was likely up against using Astyanax (which is based on the Thrift client to Cassandra) against a data model created with the assumption of using the CQL3 protocol.
As I wanted to use Astyanax, I decided that I wanted to move back to Thrift and cassandra-cli created tables. You can see the "new" Thrift based DDL in github that I created at that point. You can see the Astyanax code I used to persist to these column families in older code on github. Note that both of these older versions of code aren't complete or up to date with the final version of the DataStax Java Driver code as I gave up. I was able to get some of the less interesting entities and queries working, but started to run into issues when I needed the equivalent of "composite keys" in CQL3. An example was Flight which in CQL3 I define with a primary key of (text flight_segment_id, Date scheduled_departure_time) with a secondary index of text flight_id. I believe either CQL3 is easier for SQL-historical guys like me to understand or there is less documentation for Thrift for these composite type of keys (or both). I played around with various Thrift definitions looking at the resulting CQL3 view but could never replicate such a composite key. I tried to simplify to a case where I was trying to do a primary key that was a partition key of text and then two clustering columns. I could force it to be something like PRIMARY KEY((String, String) String) or PRIMARY KEY (String, (String, String)), but never PRIMARY KEY(String, String, String). For what it's worth I gave up vs. pushing my way through total understanding here due to lack of clear documentation on these scenarios for Thrift. I also later realized I was somewhat confused on how composite key'ed rows were stored. I assumed that the partitioning and the clustering columns defined the hash of the node for storage. After reading this article, I realized that my key/value + SQL model was wrong. Also, I tried to define annotated composite keys in Astyanax as you can see commented in the latest code. Finally, I really wanted to use the CQL3 syntactic sugar for collections. I understand such collections are possible with coding approaches with Thrift, but the sugar seemed more palatable with CQL3.
All of the above was really more of a commentary on Thrift vs. CQL3. However, I did have a few issues unique to Astyanax.
First, there is very little end to end documentation on Astyanax. The wiki quotes both the Netflix RSS Recipe and Flux Capacitor. Unfortunately both are very simple single column family examples of Cassandra with little complexity to the primary keys and data modeling. If you look at the source code both are really simple keyed rows with at most 2 columns with add, query by primary key and remove. Additionally for composite keys, there is a wiki example of how to query, but not how to store or update. There is a slightly better example in the Astyanax serializer tests, but the table definition isn't very easy to understand as it was written as a test case vs. a sample application. I'm guessing (but haven't verified) that the example code in the sample that goes along with the DataStax Java education course would be far more end to end as it would include more complex data modeling with composite keys and secondary indexes in already working code.
Next the Astyanax API is more to abstract the query language to something familiar to Java programmers. I found, with my previous JDBC/SQL knowledge, the DataStax to be less to learn as I was already pretty familiar with the concepts of getSession, prepareStatement (against a known QL string), bindStatement, execute, and walk rows. Being able to work more directly with the queries as Strings vs. build them through a Java API fit better with my experience. I need to get back to Astyanax to see if this sort of pattern is supported (the entity persister has a query language that can be executed with find).
Finally, two parts of the Astyanax API left me with unanswered questions. I couldn't see how to pass BigDecimal to ColumnListMutations which was easy to do with the DataStax Java Driver via bind(BigDecimal) and Row.getDecimal(). Also, it seemed like I needed to use MutationBatch to update (or initially create) multiple columns in the same row. It wasn't clear to me if this would perform poorly (the DataStax online education mentioned batch and how it was slower) and if the simpler "INSERT INTO table (col1, col2, col3) VALUES (?, ?, ?)" DataStax Java Driver was more efficient for this scenario.
The above is more of a state in time level of understanding of Thrift and Astyanax vs. CQL3 and the DataStax driver and Cassandra in general. As mentioned in Netflix's blog, they are working on Astyanax over the DataStax Java Driver and CQL3. I wonder how much of what I've been focusing on will be handled by this Astyanax update. Also, as I start to performance test, scale, and operationalize the tables I created, it is very likely that I'll learn there are still some concepts in Cassandra I don't fully understand. I certainly know I'm not yet using Cassandra enough in the dynamic column and sparse row aspects so my understanding of de-normalization and optimal Cassandra data modeling might not yet be complete.
Based on skills I gathered during the DataStax Java online education and due to some of the fore mentioned issues, I decided to switch horses to use CQL3 fully and the DataStax Java driver. You can see in the latest github code the CQL3 DDL and the service implementation that makes calls through the DataStax driver. I have the application fully functional and tested against a three node Cassandra cluster. I have not yet scaled the data up nor performance or scale tested. With my experience with Java/JDBC plus the DataStax Java education, I was able to code this in about two days or focused effort.
I think it would be good to have a part three of this adventure into Cassandra with Acme Air. It would be good to take the new (or even old) Astyanax driver and complete a Thrift as well as CQL3 version based on this completed CQL3/DataStax Java Driver implementation. I would additionally like to see usability and performance/scale comparisons. I think the Netflix tech blog does a good job of showing primitive performance comparisons, but it would be interesting to see how the more complex data and query model of Acme Air compares.
As is likely obvious, I was new to Cassandra in this work. Therefore comments, as always, are welcome. I'm not sure when I'll get back to this work, but if I do, pointers on Thrift/Astyanax documentation would be welcomed. Finally, if you have the time and knowledge of Astyanax, I would welcome a OSS port of the working code.
