Cassandra Triggers for Indexing and RDBMS Synchronization

We love Cassandra as a data store, but unfortunately it doesn't support all of our use cases natively.  We need to support ad hoc structural queries, online analytics, and full-text searches of the data stored in Cassandra.    These use cases are best supported by other storage mechanisms: indexes for search and RDBMS + BI for reporting and analysis.

Initially we took a batch approach to the problem, relying on Hadoop and Map/Reduce jobs to keep the external systems up to date.  We would perform map/reduce jobs over column families to bulk update the external systems.  This had obvious draw backs.  Until the batch process completes, the index and the RDBMS are out of synch with Cassandra.  Additionally, we would run over large portions of the column family even though only a small number of records had changed.

To keep the other systems synchronized, we could have complicated the cassandra clients, embedding the logic to orchestrate updates to all of the relevant systems, but that seemed like a nightmare. In the end, we decided to go for real-time trigger-like functionality.  This removes the burden off of the client and allows us to keep other systems in synch in near real-time.

Maxim Grinev came to the same conclusion and submitted a patch to Cassandra, which triggered a lengthy discussion. (pun intended)

In the end, we decided to implement our own trigger mechanism using Aspect-Oriented Programming (AOP). Our mechanism is roughly based on Jonathan Ellis's Crack-Smoking Commit Log (CSCL). For each column family mutation, we write to a commit log. The log entries are then processed asynchronously by the triggers. Upon successful execution, the log entry is removed. We've released the project at github: https://github.com/hmsonline/cassandra-triggers

The design is certainly heavy and the documentation is still a bit rough around the edges, but its small amount of code and it is working like a champ. We've setup installation and configuration instructions. Let us know if you have any trouble getting started.

Code Coverage: Cobertura, and Testing Load-time Weaving with Maven

Our cassandra-triggers implementation uses AOP. Virgil also uses it for connection pooling and retries. More specifically, each of these projects uses load-time weaving, where the byte code of the classes is changed at runtime. For testing purposes, we have the following added to our pom file:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>2.9</version>
    <configuration>
    <argLine>-javaagent:lib/aspectjweaver.jar</argLine>
    </configuration>
</plugin>

This adds the aspectjweaver agent to the jvm runtime, which enables and performs the load-time weaving. When we went to gather code coverage information on the projects, we ended up with the following exception out of aspectj.

[IsolatedClassLoader@1342ba4] warning register definition failed -- (BCException) malformed class file
malformed class file
org.aspectj.weaver.BCException: malformed class file
 at org.aspectj.weaver.bcel.Utility.makeJavaClass(Utility.java:466)
 at org.aspectj.weaver.bcel.BcelWeaver.addLibraryAspect(BcelWeaver.java:189)

Evidently, aspectj doesn't like cobertura instrumented classes. After much googling, I couldn't find a good solution. There appeared to be a lot of back and forth between cobertura and aspectj. To avoid the situation entirely, I simply decided to maintain to pom files. I swapped out the plugin above that adds the agent jvm, for the snippet below that enables compile-time weaving:

<plugin>
    <groupId>org.codehaus.mojo</groupId>
    <artifactId>aspectj-maven-plugin</artifactId>
    <version>1.4</version>
    <configuration>
        <complianceLevel>1.5</complianceLevel>
        <weaveDependencies>
            <weaveDependency>
                <groupId>org.apache.cassandra</groupId>
                <artifactId>cassandra-thrift</artifactId>
            </weaveDependency>
            <weaveDependency>
                <groupId>org.apache.cassandra</groupId>
                <artifactId>cassandra-all</artifactId>
            </weaveDependency>
        </weaveDependencies>
    </configuration>
    <executions>
        <execution>
            <goals>
                <goal>compile</goal>
            </goals>
        </execution>
    </executions>
</plugin>

This swaps from load-time weaving during testing to the compile-time weaving, which allows cobertura and aspectj to play well together, which means you'll get accurate code coverage numbers in Sonar.