Virgil: Remote Hadoop Job Deployment via REST

In the latest release, Virgil has taken its Hadoop support one step further allowing you to deploy jobs to remote Hadoop clusters via REST. 

We've taken the Ruby example and added the ability to remotely deploy those ruby scripts.

In our deployment model, we run Virgil, Cassandra and Hadoop on each node in the cluster, with an HTTP load balancer in front of the Virgil instances. 

With this latest release of Virgil, we've added a new shell script that allows you to start Virgil with the Hadoop configuration on the classpath.

bin/virgil-hadoop -host localhost


This starts Virgil, remotely accessing the Cassandra on the localhost.  Additionally, we alter the Hadoop configuration (in $VIRGIL_HOME/mapreduce/conf/) set to use the collocated Hadoop cluster.  The Hadoop configuration is described in detail on the Virgil Wiki.

Then, when you sling ruby at the REST API, that ruby is deployed remotely to the entire cluster.   Since we started Virgil using "localhost" that is used as the initial "seed address" in the Map/Reduce job, which means each it will its local Cassandra instance to access the Cassandra cluster minimizing the data passed around the network.

Give it a try and let us know what you think.

Next up: Cassandra Triggers.

Virgil goes DropWizard... who needs war files?

For years now I've been saying that J2EE may be dead (or dying).  Wherever possible, people are trying to cut down on code and configuration.  The lighter the better.  These days, even Tomcat seems heavy for the light-weight services we want to deploy.  Enter DropWizard.

Thanks goes to Coda Hale for putting together DropWizard, a rock-solid lightweight REST stack.  Its got everything you could want: configuration, metrics, parsing, health checks, etc.  It can even serve up static content.

With all those features, who needs war files?

Recently we contributed to DropWizard, adding the ability to serve up static content from the root url, which allowed us to port Virgil over.  Since Virgil has an entirely ExtJS based UI, static content is all we needed for the UI.

We just released a new version Virgil with DropWizard under the covers.

Using DropWizard, Virgil now provides an Admin UI.  Additionally, there are additional configuration parameters for logging and the http server.

If you get a chance, take a look at DropWizard.   It might just have you replacing your old servlet container.