Brian ONeill's Blog

Virgil: Remote Hadoop Job Deployment via REST

2012-01-24T04:58:00.000-08:00

In the latest release, Virgil has taken its Hadoop support one step further allowing you to deploy jobs to remote Hadoop clusters via REST.

We've taken the Ruby example and added the ability to remotely deploy those ruby scripts.

In our deployment model, we run Virgil, Cassandra and Hadoop on each node in the cluster, with an HTTP load balancer in front of the Virgil instances.

With this latest release of Virgil, we've added a new shell script that allows you to start Virgil with the Hadoop configuration on the classpath.

bin/virgil-hadoop -host localhost

This starts Virgil, remotely accessing the Cassandra on the localhost. Additionally, we alter the Hadoop configuration (in $VIRGIL_HOME/mapreduce/conf/) set to use the collocated Hadoop cluster. The Hadoop configuration is described in detail on the Virgil Wiki.

Then, when you sling ruby at the REST API, that ruby is deployed remotely to the entire cluster. Since we started Virgil using "localhost" that is used as the initial "seed address" in the Map/Reduce job, which means each it will its local Cassandra instance to access the Cassandra cluster minimizing the data passed around the network.

Give it a try and let us know what you think.

Next up: Cassandra Triggers.

Virgil goes DropWizard... who needs war files?

2012-01-24T04:28:00.000-08:00

For years now I've been saying that J2EE may be dead (or dying). Wherever possible, people are trying to cut down on code and configuration. The lighter the better. These days, even Tomcat seems heavy for the light-weight services we want to deploy. Enter DropWizard.

Thanks goes to Coda Hale for putting together DropWizard, a rock-solid lightweight REST stack. Its got everything you could want: configuration, metrics, parsing, health checks, etc. It can even serve up static content.

With all those features, who needs war files?

Recently we contributed to DropWizard, adding the ability to serve up static content from the root url, which allowed us to port Virgil over. Since Virgil has an entirely ExtJS based UI, static content is all we needed for the UI.

We just released a new version Virgil with DropWizard under the covers.

Using DropWizard, Virgil now provides an Admin UI. Additionally, there are additional configuration parameters for logging and the http server.

If you get a chance, take a look at DropWizard. It might just have you replacing your old servlet container.

Programmatically submitting jobs to a remote Hadoop Cluster

2011-12-19T19:17:00.000-08:00

I'm adding the ability to deploy a Map/Reduce job to a remote Hadoop cluster in Virgil. With this, Virgil allows users to make a REST POST to schedule a Hadoop job. (pretty handy)

To get this to work properly, Virgil needed to be able to remotely deploy a job. Ordinarily, to run a job against a remote cluster you issue a command from the shell:

hadoop jar $JAR_FILE $CLASS_NAME

We wanted to do the same thing, but from within the Virgil runtime. It was easy enough to find the class we needed to use: RunJar. RunJar's main() method stages the jar and submits the job. Thus, to achieve the same functionality as the command line, we used the following:

List args = new ArrayList();
args.add(locationOfJarFile);
args.add(className);
RunJar.main(args.toArray(new String[0]));

That worked just fine, but would result in a local job deployment. To get it to deploy to a remote cluster, we needed Hadoop to load the cluster configuration. For Hadoop, cluster configuration is spread across three files: core-site.xml, hdfs-site.xml, and mapred-site.xml. To get the Hadoop runtime to load the configuration, you need to include these files on your classpath. The key line is found in the configuration Hadoop Javadoc.

"Unless explicitly turned off, Hadoop by default specifies two resources, loaded in-order from the classpath:"

Once we dropped the cluster configuration onto the classpath, everything worked like a charm.

Binaries available for download from Virgil

2011-12-12T18:47:00.000-08:00

In response to a few requests for a binary distribution, we just posted artifacts for Virgil.

http://code.google.com/a/apache-extras.org/p/virgil/downloads/list

For simplicity, we're keeping the version number aligned with the version of Cassandra.

(which is important when you are running with an embedded Cassandra ;)

Also, we changed it so you can simply specify the Cassandra instance you want to run against as a command line parameter:

http://code.google.com/a/apache-extras.org/p/virgil/wiki/runmode

This makes it easy to point the GUI at different Cassandra instances.

Now, all you need to do is download the binary distribution, untar/unzip and type:

bin/virgil -h CASSANDRA_HOST

Let me know if anyone has any trouble.

Dependencies and Repositories in SBT 0.11 (vs 0.7) for Kestrel

2011-12-09T10:47:00.001-08:00

I was trying to build Kestrel last night, which is written in Scala.

After installing scala and sbt on my mac using homebrew, Kestrel wouldn't build because it requires an older version of sbt. Instead of installing the old version of sbt, I took it as a good opportunity to learn what sbt was about. In migrating the Kestrel build file to sbt 0.11, I found the documentation somewhat lacking. (especially when compared to 0.7) Thus, here are two tidbits that I had a hard time finding... (I actually had to dig through other github projects to see how they did it)

I learned that sbt uses maven repositories as one source for dependencies. Just like maven, you need to declare the dependencies and the available repositories. Here is how you do it.

I forked the Kestrel project and posted my new build.sbt here:
https://github.com/boneill42/kestrel/blob/master/build.sbt

To add a dependency to your project add the following line to the build.sbt.

libraryDependencies += "com.twitter" % "util-core" % "1.12.4"

In maven speak, the first string is the group identifier.The second string is the artifact identifier.And the third string is the version identifier.

In this case, Kestrel required additional repositories.

To add a repository to your project add the following line to the build.sbt.


  resolvers += "twitter.com" at "http://maven.twttr.com/"

The first string is the name of the repository. The second string is the url for the repo.

Hope that helps some people.

Hadoop/MapReduce on Cassandra using Ruby and REST!

2011-12-01T10:16:00.000-08:00

In an effort to make Hadoop/MapReduce on Cassandra more accessible, we added a REST layer to Virgil that allows you to run map reduce jobs written in Ruby against column families in Cassandra by simply posting the ruby script to a URL. This greatly reduces the skill set required to write and deploy the jobs, and allows users to rapidly develop analytics for data store in Cassandra.

To get started, just write a map/reduce job in Ruby like the example included in Virgil:
http://code.google.com/a/apache-extras.org/p/virgil/source/browse/trunk/server/src/test/resources/wordcount.rb

Then throw that script at Virgil with a curl:


curl -X POST http://localhost:8080/virgil/job?jobName=wordcount\&inputKeyspace=dummy\&inputColumnFamily=book\&outputKeyspace=stats\&outputColumnFamily=word_counts --data-binary @src/test/resources/wordcount.rb

In the POST, you specify the input keyspace and column family and the output keyspace and column family. Each row is fed to the ruby map function as a Map, each entry in the map is column in the row. The map function must return tuples (key/value pairs), which are fed back into Hadoop for sorting.

Then, the reduce method is called with the keys and values from Hadoop. The reduce function must return a map of maps, which represent the rows and columns that need to be written back to Cassandra. (keys are the rowkeys, sub maps are the columns)

Presently, the actual job runs inside the Virgil JVM and the HTTP connection is left open until the job completes. Over the next week or two, we'll fix that. We intend to implement the ability to distribute that job across an existing Hadoop cluster. Stay tuned.

For more information see the Virgil wiki:
http://code.google.com/a/apache-extras.org/p/virgil/wiki/mapreduce

Concurrent use of embedded Ruby in Java (using JRuby)

2011-11-30T11:05:00.000-08:00

Last night I was finishing up the map/reduce capabilities within Virgil. We hope to allow people to post ruby scripts that will then get executed over a column family in Cassandra using map/reduce. To do that, we needed concurrent use of a ScriptEngine that could evaluate the ruby script. In the below code snippets, script is a String that contains the contents of a ruby file with a method definition for foo.

First, I started with JSR 223 and the ScriptEngine with the following code:


public static final ScriptEngine ENGINE = new ScriptEngineManager().getEngineByName("jruby");
ScriptContext context = new SimpleScriptContext();
Bindings bindings = context.getBindings(ScriptContext.ENGINE_SCOPE);
bindings.put("variable", "value");
ENGINE.eval(script, context);

That worked fine in unit testing, but when used within map/reduce I encountered a dead-lock of sorts. After some googling, I landed in the Redbridge documentation. There I found that jruby exposes a lower-level API (beneath JSR223) that exposes concurrent processing features. I swapped the above code, for the following:


this.rubyContainer = new ScriptingContainer(LocalContextScope.CONCURRENT);
this.rubyReceiver = rubyContainer.runScriptlet(script);
container.callMethod(rubyReceiver, "foo", "value");

That let me leverage a single engine for multiple concurrent invocations of the method foo, which is defined in the ruby script.

This worked like a charm.

Introducing run-modes to Virgil: support for embedded or remote cassandra instances

2011-11-28T19:20:00.000-08:00

Since Virgil was originally developed as an embedded REST layer for the Cassandra Server, it ran as a daemon inside the server and performed operations directly against the CassandraServer classes. Running in a single JVM had some performance gains over a separate server that communicated over Thrift (either directly or via Hector) since operations didn't have to take a second hop across the network (with the associated marshalling/unmarshalling)

We had a request come in to add the ability to run Virgil against a remote Cassandra:

http://groups.google.com/group/virgil-users/browse_thread/thread/d8a99e026567b101

That seemed reasonable since there are a lot of existing cassandra clusters and users may just want to add a REST layer to support webapp/gui access or SOLR integration.

To support those cases, we added run-modes to the configuration:

http://code.google.com/a/apache-extras.org/p/virgil/wiki/runmode

Let us know what you think.

Virgil: GUI for Cassandra now included in Virgil

2011-11-21T19:27:00.000-08:00

Sure, its read-only.

Sure, its focused on Strings.

But it was written in only 100 lines of code using Virgil's REST layer for Cassandra and includes all of ExtJS's goodness. (if you are into that kind of thing)

You can see the entire the GUI is contained in a single javascript class:

http://code.google.com/a/apache-extras.org/p/virgil/source/browse/trunk/src/main/webapp/main.js

That javascript uses two GridPanel's: one to display column families grouped by keyspaces (on the east region panel), and another to display columns grouped by rowkeys (in the center panel). Each of the GridPanel's uses a store backed by an ExtJS model.

To accomodate the GUI, we added fetch capabilities the REST layer for both schema information and rows using key ranges. I'll detail those capabilities in a follow up post.

For instructions on how to access the GUI and to see what it looks like check out the wiki page.

Even in its existing state, this a useful GUI to quickly inspect the contents of a Cassandra node. It is also a good demonstration of how you might include a javascript component for visualization into your own application with very little effort.

...

Virgil now includes an elementary REST interface. (thanks to Dave Strauss @ Pantheon for his help defining the interface) It also includes simple SOLR integration and a GUI. Next up, map/reduce for the masses via REST. Stay tuned.

As always, comments and contributions welcome and appreciated.

PATCH methods on JAX-RS

2011-11-10T11:56:00.000-08:00

We added PATCH semantics for Virgil.

This was fairly straight forward, except we need to add support for a @PATCH annotation and PatchMethod for HttpClient.

To do this, we created a PATCH annotation. Take a look at PATCH.java. The contents of which are shown below:


@Target({ElementType.METHOD})
@Retention(RetentionPolicy.RUNTIME)
@HttpMethod("PATCH")
public @interface PATCH {
}

This then allows us to use @PATCH on an annotation on a REST service.


   @PATCH
   @Path("/data/{keyspace}/{columnFamily}/{key}")
   @Produces({ "application/json" })
   public void patchRow(@PathParam("keyspace") String keyspace,
           @PathParam("columnFamily") String columnFamily, @PathParam("key") String key,
           @QueryParam("index") boolean index, String body) throws Exception

That worked like a charm. Then we needed to call it using HttpClient. To that, we created a PatchMethod class that extended PostMethod. You can see that here.

Then we could use that just like any other HTTP method.


PatchMethod patch = new PatchMethod(BASE_URL + KEYSPACE + "/" + COLUMN_FAMILY + "/" + KEY);
 requestEntity = new StringRequestEntity("{\"ADDR1\":\"1235 Fun St.\",\"COUNTY\":\"Montgomery\"}",
   "appication/json", "UTF8");
 patch.setRequestEntity(requestEntity);

Hope that helps people.

Virgil: PATCH semantics added to REST layer for Cassandra

2011-11-10T11:18:00.000-08:00

Virgil now supports PATCH semantics for row updates in Cassandra via REST.

In REST, when a resource is modified rather than fully replaced by an HTTP operation, the IETF is proposing a new HTTP method, PATCH.

http://tools.ietf.org/html/rfc5789

Virgil now allows users to use this HTTP method to add and modify columns in a single post (without reposting the entire row). We've included an example in the Getting Started instructions.

Likewise, PUT operations will now replace the entire row, per HTTP semantics.

(Thanks to David Strauss for suggesting this)

Cassandra integration w/ SOLR using Virgil

2011-11-04T13:37:00.001-07:00

Up front, I'd like to say this is still pretty raw. We'd love to get feedback and contributions.

That said, Virgil now has the ability to integrate SOLR and Cassandra. When you add and delete rows and columns via the REST interface, an index is updated in SOLR.

For more information check out:
http://code.google.com/a/apache-extras.org/p/virgil/wiki/solr

Let us know what we can do better.

Virgil gets a command-line interface (virgil-cli)

2011-10-24T19:05:00.000-07:00

Tonight I bundled the cassandra command-line interface (CLI) into virgil. Since the CLI uses the thrift-based CassandraDaemon, the main method now starts a thrift server along side the REST server.

Now, when you (or your application) issues commands through the REST interface, you can verify that they worked through the command-line interface. For more information, check out the wiki.

Specifically, if you use the curl commands in the Getting Started section. You should see the following in the command-line interface.


bone@zen:~/dev/code.google.com/virgil/trunk-> bin/virgil-cli -h localhost
Connected to: "Test Cluster" on localhost/9160
Welcome to the Cassandra CLI.

Type 'help;' or '?' for help.
Type 'quit;' or 'exit;' to quit.

[default@unknown] use playground;
Authenticated to keyspace: playground
[default@playground] list toys;
Using default limit of 100
-------------------
RowKey: swingset
=> (column=bar, value=33, timestamp=1319508065134)
=> (column=foo, value=1, timestamp=1319508065126)

1 Row Returned.
[default@playground] quit

Virgil: a GUI and REST layer for Cassandra

2011-10-20T18:14:00.001-07:00

Love Cassandra? Love REST?
Wish you could have both at the same time?
Now you can.

After much discussion, I'm happy to announce the birth of a new project, Virgil. The project will provide a GUI and a services layer on top of Cassandra, exposing data and services via REST.

Virgil already has a REST layer for CRUD operations against keyspaces, column families, and data. We hope to quickly add Pig/Hadoop support via REST as well as a thin, javascript-based GUI that uses the REST services.

How can you help nurture the baby?
Head over to Apache Extras,
http://code.google.com/a/apache-extras.org/p/virgil/

Star the project, and then get involved.
Grab the source code and give it a try.

Pig / Cassandra: binary operator expected

2011-10-05T13:11:00.000-07:00

If you are trying to run Pig on Cassandra and you encounter: "binary operator expected"

You are most likely running pig_cassandra against the latest release of Pig, which has two jar files in it one with hadoop and one without hadoop. Your PIG_HOME is set to the root directory of your pig installation, which contains those two jar files. The existence of TWO jar files breaks the pig_cassandra shell script.

I've submitted a patch for this:
https://issues.apache.org/jira/browse/CASSANDRA-3320
(Please vote to get it included)

Until that is committed, you can simply remove the jar file you don't want to use:


rm -fr $PIG_HOME/pig-0.9.1-withouthadoop.jar

That should fix you.
Happy pigging.

Cassandra / Hadoop : Getting the row key (when iterating over all rows)

2011-10-03T06:48:00.000-07:00

I thought I would save some people some time...

The word count example is fantastic, and is enough to get you going. But, you it may leave you wondering how to get at the row key since the "key" passed into the map is the name of the column and not the key of the row. Instead the key is in the context. Take a look at the snippet below.

public void map(ByteBuffer key, SortedMap<ByteBuffer, IColumn> columns, Context context) 
   throws IOException, InterruptedException {
  for (ByteBuffer columnKey : columns.keySet()){
     String name = ByteBufferUtil.string(columns.get(columnKey).name());
     String value = ByteBufferUtil.string(columns.get(columnKey).value());            
     logger.info("[" + ByteBufferUtil.string(columnKey) + "]->[" + name + "]:[" + value + "]");
      logger.info("Context [" + ByteBufferUtil.string(context.getCurrentKey()) + "]);             
  }

Cassandra / Hadoop : No local connection available

2011-09-14T13:09:00.000-07:00

If you are trying to run the word_count example in Cassandra Hadoop, and you encounter the following exception:

java.lang.RuntimeException: java.lang.UnsupportedOperationException: no local connection available
at org.apache.cassandra.hadoop.ColumnFamilyRecordReader.initialize(ColumnFamilyRecordReader.java:132)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:418)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:620)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
Caused by: java.lang.UnsupportedOperationException: no local connection available
at org.apache.cassandra.hadoop.ColumnFamilyRecordReader.getLocation(ColumnFamilyRecordReader.java:176)
at org.apache.cassandra.hadoop.ColumnFamilyRecordReader.initialize(ColumnFamilyRecordReader.java:113)
... 4 more

Then you have hit a problem with local IP resolution in Java.

Cassandra currently uses the following line to resolve IP addresses;

localAddresses = InetAddress.getAllByName(InetAddress.getLocalHost().getHostAddress());

There are better ways to do this using the NetworkInterface,

http://stackoverflow.com/questions/901755/how-to-get-the-ip-of-the-computer-on-linux-through-java

http://www.jguru.com/faq/view.jsp?EID=790132

But until Cassandra uses that you'll need to make sure that the bit of code above resolves properly by manipulating your /etc/hosts to resolve localhost to match the configuration in Cassandra, which by default is looking for localhost bound to 127.0.0.1.

I submitted a patch for this. If you are having issues, please vote to get the patch accepted.
https://issues.apache.org/jira/browse/CASSANDRA-3211

Master Data Management : Open Source Solutions???

2011-08-31T18:03:00.000-07:00

The movement towards digital records is generating exponential amounts of data, tremendously valuable data. But building a system to manage that data and extract value from it requires acceptance of a paradox; the system needs to be flexible and tolerant, while simultaneously enforcing structure and standards. This is hard.

Many of us have felt the pain of Master Data Management (MDM). In large enough enterprises, even something as simple as keeping addresses current across multiple lines of business is a herculean task. Furthermore, it’s a problem that’s difficult to ignore because poor data management can drive substantial expense and leave invaluable opportunities on the table.

Since joining Health Market Science (HMS), a company focused on MDM for the healthcare space, that sentiment has been reinforced tenfold. Two things became immediately apparent: the complexity of a complete solution and the value of the same.

The problem is complex because there is a temporal aspect to the data. A complete MDM solution doesn't just provide a current view of entities, but a historical perspective as well. It provides that perspective across all the entities and the all the relationships between those entities. Then, add in the fact that each source system may have a different schema representing each entity and that the schema itself may evolve over time. Then, pile on all the necessary processing to analyze the data. Entities need to be consolidated based on precedence rules, standardized, and matched using fuzzy matching. What a fantastic recipe for a fun problem to solve.

But given the complexity, it isn't difficult to see why companies shy away from an "in-house" MDM solution. Solving these problems isn't easy. You may think that if you get yourself a good Data Architect and a massive Oracle instance, you could crank something out. You’ll soon find that standard relational structures quickly become unwieldy and you end up in “meta-meta” world building out schemas to manage schemas. This can be extremely painful.

I'm wondering if people have seen any open source solutions capable of tackling this problem. From what I can find, the open source "MDM" solutions only tackle the Extract-Transform-Load (ETL) portion of the problem. Unfortunately, that is the easy part. Does anyone know of any open-source communities focused on delivering a complete solution, ETL through down into storage?

Rails vs. Grails

2010-12-02T11:56:00.000-08:00

I was recently asked the question: Rails or Grails? I needed to summarize the key differences and industry sentiment. This was my response.

Before I make any subjective comments, let me start with some objective metrics I found:
http://www.expectationgap.com/posts/13

Now for my sentiments, I believe there are three key-dimensions: momentum, cloud-support, and people.

Momentum

Rails has the momentum as evidenced by the number of committers and plugins available. Many of the rockstar developers I know are all now developing in Rails, supporting that community, etc. The underlying Ruby language is incredibly expressive and allows dramatic productivity improvements. (write less code that can do more)

Only a handful of rockstars I know are developing on Grails. Typically, those developing on Grails are in enterprises or markets that are comfortable with Java and they want to continue leveraging that investment. (infrastructure, app servers, production support, etc.) Grails/Groovy does have the advantage that It can continue to leverage the output of the Java community (e.g. Spring). The better projects within the Java community are quickly integrated into Grails. For example, Grails uses Hibernate and Spring Security under the hood. As those projects improve, so will Grails.

Cloud Support

Rails and the cloud go hand in hand. Again, of the rockstar Rails developers I know, nearly all of them are running on virtual hardware owned and managed by others. Some of those let others manage the entire application server / platform (e.g. EngineYard). In contrast, all of the Grails development I know of is running on hardware (virtualized) owned and managed by the company itself in their own data centers (or rented space). As mentioned before, this may actually have been the motivating factor to select Grails. That is not to say that there isn’t cloud support for Grails. And now that VMWare owns SpringSource, this may change rapidly now that enterprises will have a trusted name (in VMWare) to host their Grails applications.

People

Rails has the momentum, more committers and potentially a more active community, but lets not fool ourselves -- there is an army of labor out there to sling Java code. Offshore resources are readily available. The market for Rails is developing, but demand isn’t yet high enough to drive consulting companies to substantially increase their supply. As one of my good friends put it, “I’d love to develop in Ruby, but Java pays the bills.”. It is that sentiment, that has may limit the availability of resources.

However, even with readily available Java resources, that doesn’t guarantee the availability of Groovy resources. Both Grails and Rails require a slightly different mindset. If you take a Java developer and tell them to code in Groovy or Ruby, you’ll end up with Java code using Ruby/Groovy syntax. (not good) But at least the tooling, debugging practices, and libraries will all be familiar.

And just to stoke the fire...

Here are my entirely subjective scores:
Momentum: Rails (9) vs. Groovy (4)
Cloud Support: Rails (8) vs. Groovy( 5)
People: Rails (6) vs. Groovy (8)
---------------------------------------------------
Rails (23) vs. Groovy (17)

XML Validation with Eclipse

2010-08-03T12:24:00.000-07:00

When I'm doing serious schema work, I typically use XMLSpy, but for simple schema validation it isn't worth leaving the IDE. As of late, I've been using the SpringSource IDE, which comes with XML plugins pre-installed. With the plugins, XML validation is simple enough. You just need to make sure you have the schemaLocation defined in your XML.

Let's say I have a simple document:

<zoo>
   <animal>cat</animal>
</zoo>

If you open the document, you can right click and Validate the document. If there is no associated schema, you'll receive a warning that says:

No grammar constraints (DTD or XML schema) detected for the document.

To rectify that, you'll need to specify the schema location in the XML. That will require namespaces. Let's assume we have the following simple schema with namespaces:

<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema"
   targetNamespace="http://www.skookle.com/philly"
   xmlns:philly="http://www.skookle.com/philly" >
 <xsd:complexType name="zoo">
  <xsd:sequence>
   <xsd:element name="animal" type="xsd:string"/>
  </xsd:sequence>
 </xsd:complexType>
 
 <xsd:element name="zoo" type="philly:zoo"/>
</xsd:schema>

Now, lets revisit the simple XML to add namespaces and a schemaLocation element that references the above xsd. (zoo.xsd)



<philly:zoo xmlns:philly="http://www.skookle.com/philly"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.skookle.com/philly zoo.xsd">
    <animal>cat</animal>
</philly:zoo>

Note: To get the schemaLocation attribute you'll need the XMLSchema-instance namespace.

Now, you can right-click on the XML and it will validate. Also, you'll get the fancy element completion while you type. Rock on, even less reason to leave the IDE. =)

SSH Port Forwarding Example

2010-03-27T19:54:00.000-07:00

Like everyone, it takes me a few tries before I get a port forwarding command line right. Here is yet another example for everyone, just in case you happen to have this specific example.

Lets say you want to get to your database, but it isn't exposed:
CLIENT --> DMZ --> DB_HOST

You want to forward some port on DMZ to the DB. Lets say it is Oracle running on 1521. Then, from the DMZ host you should issue the following command:


  ssh -g -L 5000:DB_HOST:1521 boneill@DB_HOST

At this point, you should be able to connect to DMZ:5000 as if it were DB:1521.

REALLY REALLY IMPORTANT is the "-g", that allows remote connections to 5000. Otherwise, it will just bind to the local interface and you won't be able to connect from CLIENT.

SOAP Testing over HTTPS, using Siege instead of Ab

2009-11-25T08:17:00.000-08:00

I've been using ab for a while now to perform load testing. Recently, I ran into trouble however because ab doesn't work with HTTPS out of the box (at least on my Ubuntu install). That triggered an investigation into other tools. Siege fit the bill nicely. Basically, I needed a way to perform an HTTP Post with a SOAP body.

On ubuntu, it is a simple "apt-get install siege".

Then, you create a siegerc file:


verbose = true
logging = true
protocol = HTTP/1.1
connection = keep-alive
concurrent = 50 #Number of concurrent requests
file = ./urls.txt
delay = 0 #If you are using for benchmarking value should be 0
benchmark = true

The settings are fairly obvious. Note that the file parameter points to an additional file you'll need, which lists out all of the urls you want included in your test. Below is an example:


https://foo.com/ws/Blah POST < soapMessage.xml

Notice after the url we specify "POST", which makes it a POST method instead of GET. Additionally, we pipe in the XML file which contains the SOAP message.

Then we can simply invoke siege with:


siege --rc=./siegerc --header="Authorization:Basic BASGHAJSG78236ds"

Notice, we are supplying an extra header for Basic Auth.

MySQL: LEFT Join with Constraints

2009-09-22T11:56:00.001-07:00

I just fixed a bug in a colleague's select statement. And I think it makes a good example. So here it is:

You have two tables foo and bar. You are performing different types of analysis on the items in foo, and writing that analysis to bar. Foo has a primary key "id". And bar has three columns: "foo_id", "analysis" and "version". The bar table contains the following entries.

1, blah, 3
1, erg, 2
2, erg, 2

This means that item 1 in foo has been analyzed for blah and erg versions 3 and 2 respectively. Item 2 has been analyzed for only erg version 2.

Now, lets say there are three items in foo and you want a query to pull back all of the items that need analysis for blah version 3. The right answer is items 2, and 3.

You may start with:

select * from foo LEFT JOIN bar.foo_id = foo.id

This returns 4 rows:
1, blah, 3
1, erg, 2
2, erg, 2
3, null, null

From this resultant table there is no good way to select foo.id's that have not been processed by the right version of blah. However, if we add a join constraint to our select we can. The select becomes:

select * from foo LEFT JOIN bar.foo_id = foo.id AND bar.analysis = 'blah'

This returns a beautiful table:
1, blah, 3
2, null, null
3, null, null

Essentially the left join takes place with only a subset of bar that meets the join constraint.

NOTE: This is entirely different from the following:

select * from foo LEFT JOIN bar.foo_id = foo.id WHERE bar.analysis = 'blah'

That query will only return the single row. To manipulate the original join you would probably end up using a "NOT EXISTS IN" kind of construct. Yuck.

The JOIN with a constraint presents a nice clean simple solution that lets us figure out which foo items still need analysis.

ERROR 1005 (HY000): Rails Migrations for Unsigned Integers

2009-09-10T11:36:00.000-07:00


ERROR 1005 (HY000): Can't create table './*/#sql-3e29_1209.frm' (errno: 150)

If you are getting this when trying to add a foreign key constraint make sure the columns are of the same type. This includes attributes! Specifically pay attention to signed vs. unsigned columns. Just because two columns are integer (or int(10) or int(11)), you won't be able to create a foreign key constraint between them unless they are have the same sign.

Furthermore, in a Rails Migration, to get a signed/unsigned column use the following:


    create_table :tblFoo, :primary_key => :UID  do |t|
      t.column 'Bar', 'int(10) unsigned', :null => false
      t.timestamps
    end

Access denied for user 'debian-sys-maint'@'localhost'

2009-09-09T13:46:00.000-07:00

I wanted to get replication setup to do backups offline as recommended by this article. I did the backup and loaded it onto the slave. When I went to restart, I received:


ERROR 1045 (28000): Access denied for user 'debian-sys-maint'@'localhost'

It turns out I backed up the users table as well so it blew away the debian user. I found the fix here .

The trick is to reset the debian-sys-maint users password by looking in /etc/mysql/debian.cnf and performing the following with whatever password you find in there:


GRANT ALL PRIVILEGES ON *.* TO 'debian-sys-maint'@'localhost' IDENTIFIED BY '' WITH GRANT OPTION;

Global Subversion Ignore Settings (Used for Eclipse Project Files)

2009-07-28T18:49:00.000-07:00

I hate having to run a global svn propset ignore command. Instead it is much easier to open up .subversion/config and edit the following in the file.


global-ignores = .project .target .classpath .settings *.o *.lo *.la #*# .*.rej *.rej .*~ *~ .#* .DS_Store

Notice, I added all of the eclipse files to the ignore statement. Once you update that file, that should be it. Next time you run svn status (or anything) it will take those ignore patterns into account.

Java Wordnet Library (JWNL) Jar file Repo Entry

2009-07-28T07:09:00.001-07:00

Here is the maven pom entry for Java Wordnet Library (JWNL)


        <dependency>
            <groupId>net.didion</groupId>
            <artifactId>jwnl</artifactId>
            <version>1.4</version>
        </dependency>

CSV (Comma Separated Values) Processing in Ruby

2009-07-27T21:05:00.000-07:00

FasterCSV rocks. You can find it here:
http://fastercsv.rubyforge.org/

Start by installing it using the gem.


sudo gem install fastercsv

After that, you are all set. Just make sure you require rubygems first.


require 'rubygems'
require 'fastercsv'

i=0
FasterCSV.foreach('blogs.csv') do |row|
   i=i+1
   puts("#{row[2]}")
end

As you can see from above, the row is an array that contains the values from the CSV file.

ActiveRecord outside of Rails (even with ODBC)

2009-07-27T20:32:00.000-07:00

There are three quick lines that you need in order to use ActiveRecord outside of Rails. First, you need to load gems, then you can load ActiveRecord. Then, you can pick and choose which of your models to use.


require 'rubygems'
require 'activerecord'
require @@RAILS_APP_HOME + '/app/models/foo.rb'

Then, you'll need this little snippet to establish the connection to the database for ActiveRecord:


require 'rubygems'
require 'activerecord'
require 'yaml'

@@DATABASE_CONFIGURATION = YAML::load(File.open(File.dirname(__FILE__) + '/config/databases.yml'))

def establish_connection(database)
  dbconfig = @@DATABASE_CONFIGURATION
  ActiveRecord::Base.establish_connection(dbconfig[database])
#  ActiveRecord::Base.logger = Logger.new(STDERR)
  if (dbconfig['mode'] == 'odbc')
    puts("Connecting to [#{database}]: ODBC,"+
        " DSN=#{dbconfig[database]['dsn']}/#{dbconfig[database]['adapter']}"+
        " [user=#{dbconfig[database]['username']}]")
  else
    puts("Connecting to [#{database}]: #{dbconfig[database]['adapter']}, "+
        "#{dbconfig[database]['database']}@#{dbconfig[database]['host']}"+
        " [user=#{dbconfig[database]['username']}]")
  end
end

def remove_connection
  ActiveRecord::Base.remove_connection
end
~

I put the above snippet in a central ruby file, then require that file anywhere I need to use the ActiveRecord objects. After a call to establish_connection, you can start using any model you've imported. Note, you'll see a slightly different URL constructed for ODBC.

Hadoop: java.io.IOException: Type mismatch in key from map

2009-07-22T07:06:00.000-07:00

We've been working with hadoop for a while now, and inevitably newbies run into this error the first time they go to create their own Hadoop job. If you are running into this error, it is most likely a mismatch between your Map and/or Reduce implementation and the job configuration.

Your Map implementation probably looks something like this:


   public static class MapClass extends MapReduceBase
            implements Mapper {
        private Text word = new Text();
        public void map(LongWritable key, Text value,
               OutputCollector output,
               Reporter reporter) throws IOException {
               ...
        }

Now, your map and reduce phases can have different output types and that is what sometimes causes the problems. If your phases are producing different types, be sure to set those types in the JobConf. You do this as follows....

Then when configuring your job you need to declare the appropriate output classes.


        // Set the outputs for the Map
        conf.setMapOutputKeyClass(Text.class);
        conf.setMapOutputValueClass(IntWritable.class);

        // Set the outputs for the Job
        conf.setOutputKeyClass(Text.class);
        conf.setOutputValueClass(ArrayWritable.class);

Hope that saves people some time.

Finding a java class in a Jar file (or a set of files)

2009-07-22T06:26:00.000-07:00

Often classpath problems are hard to diagnose. Sometimes you pick up an errant class on the classpath that conflicts with a version of the class that you need. (Very evil people sometimes rip apart jars and package all of their dependent classes together in a single jar)

Any who, however it happens, it is sometimes necessary to get a list of all classes everywhere, in all jar files. Then you can search that list for duplicate instances of a class.

I can't tell you how many times I've used this trick, especially with the sometimes unclear world of what is packaged into the JDK, application server, and what is in the actual application.

Use this:


find . -name '*.jar' -exec unzip -l {} \; > all_classes.txt

That line finds all the jar files recursively from the current working directory, lists the contents of the archive, and pipes that output to a text file that can be searched.

handy voodoo.

UnsatisfiedLinkError with Surefire (on Mac OS X)

2009-07-19T22:03:00.000-07:00

Wow, this was a needle in a hay stack. I recently needed to use LinkGrammar on my Mac OS X, and I wanted to use it via the Java Native Interfaces (JNI). I had been using LinkGrammar on my linux (ubuntu) boxes for some time. So, I was no stranger to compiling it with the java support. I even tested the compilation and install with:


nm /usr/local/lib/liblink-grammar-java.a | grep Java

However, when I was running maven, I received an UnsatisfiedLinkError. After MUCH googling I found:
http://www.nabble.com/Trouble-with-Java-Native-Libraries-td20293666.html

Simply setting the system property in maven for surefire is not sufficient, because maven changes the system property at runtime, which is too late for the VM to link to the library. Thus you need to use the following in your build section of your pom:


          <plugin>
              <groupid>org.apache.maven.plugins</groupid>
              <artifactid>maven-surefire-plugin</artifactid>
              <configuration>
                  <forkmode>once</forkmode>
                  <workingdirectory>target</workingdirectory>
                  <argline>-Djava.library.path=/usr/local/lib</argline>
              </configuration>
           </plugin>

After that, the VM will link to your libraries and surefire and all dependent tests should be able to see the java interfaces and access the necessary libraries.

Rails Error: no such file to load -- application

2009-07-16T21:21:00.000-07:00

I recently deployed an application I was building to production and received the following error:


no such file to load -- application

It turns out that Rails changed the name of the application controller between versions. My production environment was 2.2.2 and my development environment was 2.3.2. Renaming the application_controller file fixed everything.


mv app/controllers/application_controller.rb app/controllers/application.rb

Rake Production (Specifying an Environment to Rake)

2009-07-16T21:15:00.000-07:00

I always forget how to specify an environment for rake commands. So I thought I would capture it here because, as it turns out, it is a hard thing to google.


rake RAILS_ENV=production db:migrate

Exec format error (in cron)

2009-07-16T07:28:00.001-07:00

If you are like me, you sometimes get lazy and forget to include the shell command at the top of your scripts. This usually isn't a problem, but in certain cases (where the shell isn't set, or doesn't exist yet) it will cause problems. So, even if your script is set to be executable (chmod +x), you'll receive an error like:


Exec format error

Just such a case manifests itself when using a script through cron (and run-parts). To remedy this problem put the following at the beginning of the script:


#!/bin/sh

That way, the system will know what to use when executing the script.

An Elegant Matching Algorithm In Ruby

2009-07-15T23:16:00.000-07:00

Recently, I wanted to sit down a learn ruby (independent of rails), so I grabbed a fairly standard hacking problem and went to town on it. I now love ruby more than ever.

The Problem:
Given a set of people and a set of jobs, where each person may take a different amount of time to do each job, optimally match people to jobs (1:1) to minimize the amount of time it will take to complete all jobs.

Many people will recognize this as a standard matching problem for which the Hungarian Algorithm is a known solution. But for those that have implemented the Hungarian Algorithm (or seen implementations of it), you know well enough to steer clear. It is error prone, and a very specific algorithm. So, I sought to implement a more elegant (and generally applicable) solution using graphs.

I found this article over on topcoder describing max-flow algorithms and the beauty of such. I fell in love and decided that I needed to solve this with max-flow.

The formulation:
Lets convert our problem to a bipartite graph. Let one set of nodes be the people, and the other set of nodes be the jobs. Create an edge from each person to each job, with a weight (NOT capacity!) equal to the time it will take that person to do that job.

In our situation, the capacity for each line is one since only one person can do a job. Flow is ofcourse initialized to zero for each edge (no one is doing any of the jobs). Lastly, we connect every job to a SINK node in the graph.

The philosophy and algorithm: (the important part)
Essentially, we'll be finding shortest paths in the graph, from each person to the SINK (via jobs) iterating through each of the people. On each iteration, we augment the graph with the path.

To recap the topcoder article, augmenting the graph consists of incrementing the flow for each edge in the augmenting path and adjusting edges to represent the new flow/capacity. There is an edge that represents "residual capacity" with capacity == capacity - flow, and there is an edge in the reverse direction that represents "upstream flow" with capacity == flow.

REMEMBER, in our case:
Capacity is ALWAYS 1.
Flow is either 1 or 0.
This is ENTIRELY independent of the cost/value/weight of an edge.

What does that mean to us you say? Well, in our case there are only two situations, a person is assigned to a job (flow == 1), or a person is not assigned ot a job (flow == 0). In the first case, where a person is assigned a job, there is an edge from the job to the person. During the algorithm, this edge essentially represents the path to UNDO the assignment. In the second case, the edge simply represents the
making that assignment.

String Array Initialization in Java

2009-07-15T12:46:00.000-07:00

Just for all those out there that also find string array initialization in Java counter intuitive (especially if you are also a ruby enthusiast)...

To initialize an array in Java, the syntax is

String[] foo = {"bar", "serf", "doodle"};

The key is the curlies. =)

initializationError0 in JUNit

2009-07-15T08:02:00.000-07:00

If you end up with a initializationError0 error coming out of JUnit, it is because the JUnit engine can't invoke your test method. Most likely this is because the method you annotated as a test takes a parameter. Simply remove the parameter, and you should be all set.

RSync for Backup over SSH using Different Port Number and Bandwidth Limit

2009-07-13T09:23:00.000-07:00

Over the years, I've come to love rsync for offsite backups. It is incredibly flexible and can run over SSH. Here is the most flexible one-line backup you'll ever see:


rsync --bwlimit=100 --partial --progress --size-only -av "/Volumes/Shed/stuff/" --rsh='ssh -p 2828' "foo@offsitebackup.com:/home/stuff/"

This backups my local stuff (in /Volumes/Shed/stuff) and puts it in /home/stuff on offsitebackup.com. It also keeps partially transferred files (--partial) and shows progress (--progress). When comparing two files, it only considers the sizes (--size-only). I do this because dates could be different. Furthermore, it transfers using ssh, but over a different port (2828 in this case). Finally, I limit the bandwidth that the rsync consumers (--bwlimit) to 100 kb/s.

Very handy.

JDBC to MySQL Datetime (Time truncated)

2009-07-01T13:10:00.000-07:00

Be careful when accessing Datetime fields in a MySQL database through JDBC. You might think that the java.sql.Date would work, but that actually truncates the Date back to midnight. In order to access the actual time, use getTimestamp() instead. Here is the code:


        Result rs = ...
        while (rs.next()) {
          return rs.getTimestamp(1).getTime();
        }

Sorting list of FIles in Java (returned from listFiles)

2009-07-01T08:20:00.000-07:00

Here is some handy code to get files in order...


        File[] files = dir.listFiles();
        Arrays.sort(files, new Comparator() {
            public int compare(File f1, File f2) {
                return f1.getName().compareTo(f2.getName());
            }
        });

Vim syntax highlighting on Ubuntu

2009-06-30T11:32:00.001-07:00

To get vim syntax highlighting:

First, make sure you have vim installed in addition to plain old vi.


sudo apt-get install vim

Then, create a vimrc dot file.


vim ~/.vimrc

In that file, add the following:


:syntax on

Thats it! Next time you edit a file in vim, you should have syntax highlighting.

Pidgin (and Adium) not connecting to Yahoo

2009-06-24T10:48:00.000-07:00

OK, I entered a phase where my yahoo login started failing in Pidgin with Ubuntu. I first went in through my.yahoo.com and reactivated my account. It turns out that yahoo turns off your account if you don't login for a few years. =)

After that, it was a problem with the underlying libraries underneath both Pidgin and Adium. On Mac, I updated to the latest version of Adium (1.3.5) and that fixed it.

On ubuntu, it turns out that ubuntu won't update to new versions unless their is a security problem. In order to get functional enhancements, you need to upgrade using pidgin's instructions here:
http://www.pidgin.im/download/ubuntu/

After you enter those two commands, start update manager and you should be able to upgrade pidgin.

attachment_fu: No thumbnail generation (make sure you have a parent_id!)

2009-06-18T19:00:00.000-07:00

I've been using attachment_fu to upload images in a rails application. Everything was working well except the thumbnails weren't generating. I think everyone follow's Mike Clark's excellent blog:
http://clarkware.com/cgi/blosxom/2007/02/24

He doesn't mention that the parent_id is required, so I thought it was simply the id of the model to which the photo/upload was attached.

After some digging, I found a conversation here:
http://www.ruby-forum.com/topic/104213

Turns out attachment_fu, creates a new entry for each thumbnail, then strings them together with the parent_id. So, make sure you keep the parent_id column otherwise attachment_fu silently fails to generate the thumbnails.

LDAP/Active Directory Authentication with Apache

2009-06-17T07:43:00.000-07:00

Alright, it took a few hours to get all the parameters correct, but we finally achieve centralized authentication by linking apache authentication to our ActiveDirectory. The critical concept to keep in mind when doing it is that there are two things you need to specify. First, you need to specify the user that apache will connect as, known as the "BindDN". Second, you need to specify the query string that allows apache to locate a user in the directory. This is the LDAP url.

In the end, this is the element we needed to add to our apache config. On ubuntu, we dropped this into the site-specific configuration file in /etc/apache2/sites-available.


 <Location />
        AuthBasicProvider ldap
        AuthType Basic
        AuthzLDAPAuthoritative off
        AuthName "Portal"
        AuthLDAPURL "ldap://actrivedirectorymachine/DC=foo,DC=com?sAMAccountName?sub?" 
        AuthLDAPBindDN "CN=apache,CN=Users,DC=foo,DC=com"
        AuthLDAPBindPassword "PASSWORD"
        require valid-user
  </Location>

In the above example, I created a user specifically for apache, with password PASSWORD. I highly recommend using JXplorer to verify your bind credentials.

When configuring JXplorer, if you are using ActiveDirectory, most likely you'll need a Base DN as well. This was "DC=foo,DC=com". The user name is the exact string from above, same with password.

The AuthLDAPURL is a query that will be used to grab the entry associated with the username that the user types in when prompted by the browser. In the example above, it will search within DC=foo,DC=com against the attribute "sAMAccountName".

I hope this helps people out.

git installation on Mac OSX (to fix undefined method acts_as_mappable)

2009-06-14T19:38:00.000-07:00

I recently started playing around with geokit within rails. I kept getting an "undefined method `acts_as_mappable'" error. It turns out that the script/plugin command for the installation of the geokit-rails plugin was silently failing. When I entered the script/plugin install command, the command would run and not output anything to the screen. I ended up with an empty vendor/plugins directory.

The root cause ended up being:

I didn't have git installed and on my path. So, I installed it first using this:
http://code.google.com/p/git-osx-installer/

And added /usr/local/git/bin to my path.

(Then later I installed git via ports.)

When I returned to the rails app and re-entered:
script/plugin install git://github.com/andre/geokit-rails.git

All worked properly.

WMV to FLV Conversion and Thumbnail Generation

2009-06-05T08:55:00.000-07:00

Here are two handy command lines for converting WMV to FLV (which is a better format for hosted video), and generating thumbnails for those videos. Both use ffmpeg.

To convert from wmv to flv:
ffmpeg -i test_video_clip.wmv -ab 48 -ar 22050 -s 512x384 -g 50 -qblur 1 -pass 1 -b 800 -r 25 -y encodedvideo.flv

To generate the thumbnail:
ffmpeg -i test_video_clip.flv -an -ss 00:00:03 -an -r 1 -vframes 1 -y %d.jpg

Eclipse on Mac OS X

2009-05-12T19:52:00.000-07:00

I don't know why Java support is so bad on Mac OS X. If you want to run Eclipse on Mac you may need to finagle it a bit. When I first tried to run Eclipse, I go the following error:
_NSJVMLoadLibrary: NSAddLibrary failed for /System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK/Libraries/libjvm.dylib

To fix this, I first followed this blog post:
http://blog.kischuk.com/2008/05/08/running-eclipse-on-macbooks-with-java-6/

But, unfortunately I didn't have 1.5.0 installed. After a lot of googling, I found that Update 4 (available to registered developers through ADC) includes JDK 1.5.0

So, if you are having trouble getting Eclipse running, install Java for Mac OS X Update 4.

Invalid Arguments 501 5.5.4 Error when sending SMTP email through exchange

2009-04-29T06:49:00.000-07:00

OK, I recently needed to do a quick and dirty mail send from java using an exchange server. I kept getting a 501 5.5.4 error back, which is Invalid Arguments to the RSET method. It turns out, when looking at the example:
http://snippets.dzone.com/posts/show/3328

I had neglected to set the submitter properly. When googling, all of the discussions were around trailing dots or spaces. After wiresharking it, I finally saw the problem.

Word to the wise, make sure when you are using AUTH with SMTP to set the submitter properly.

Vanilla Discussion Forum : Initial Login Problems

2009-04-21T09:40:00.000-07:00

In the past I've been a big phpBB fan, but recently I've wanted a leaner meaner discussion forum for integration into an existing site/application. I stumbled upon Vanilla.

http://getvanilla.com/

So far, I'm a big fan. I did have some trouble getting setup though. When installing, you set the COOKIE_DOMAIN, which php then uses to store your cookie. This needs to match the url through which you are accessing the site. If it does not, the authentication will work but you will not be signed in. In other words, if you type in the wrong password you will be rejected. If you type in the correct password you will be forwarded to the main page, but you will still see the option to sign in up in the top right hand corner.

Specifically this will happen if you have it deployed locally and access it using "localhost". In this case, simply map your deployment domain to your local IP address in your /etc/hosts file.

Also make sure your conf/settings.php contains the correct domain. I searched for localhost in that file and made sure all were changed to be the deployment domain.

I hope this helps anyone trying to get started with Vanilla.

Building a REST Service in PHP

2009-04-15T08:38:00.000-07:00

OK, I'm a big fan of using the right tool for the job. Daily I switch between Rails, Java and PHP. Rails is probably the easiest to use for developing a REST Service, but sometimes you don't want the overhead of a rails server and you want that feel good all over sense of stability that LAMP gives you.

First, I created an alias to send all requests to a single php file (service.php).


 AliasMatch /Beers/.* /var/www/www.liquidmirth.com/service.php

This sends all http://www.liquidmirth.com/Beers/* requests to service.php. Then, I created a generic service class. This grabs the method, id and the target object from the url, and uses reflection to invoke the appropriate method on a service class. The code for the service.php class is below.


   include 'beer.php';
   header('Content-Type: text/xml');
   $method = strtolower($_SERVER["REQUEST_METHOD"]);
   $object = $_SERVER["REQUEST_URI"];
   $object = substr($object, 1, strlen($object)-1);
   $vars = explode("/", $object);
   $object = $vars[0];
   $id = $vars[1];
   // echo "method ==> (".$method.")
";
   // echo "object ==> (".$object.")
";
   // echo "id ==> (".$id.")";
   $method = new ReflectionMethod($object, $method);
   echo $method->invoke(NULL, $id);

In the above code, the request is received, the Beer object is created and the get() method is invoked on that object with the id supplied as a parameter. So, when http://www.liquidmirth.com/Beer/1234 is requested. Beer->get(1234) is invoked.

Its that simple.

Fixing Sendmail (dead.letter issue)

2009-04-08T08:26:00.000-07:00

OK, we have lots of machines that we host out on slicehost. We want them to be part of our domain (foo.ourdomain.com). So, we configure their hostname to be that. Unfortunately, we want them to be able to send mail to people in our domain (john@ourdomain.com). Since the machine thinks it is part of the domain, it looks for that user, can't find them and consequently fails to send the mail out.

Testing sendmail, it always drops the mail into dead.letter immediately.

To fix this, I simply edited /etc/mail/sendmail.cf and uncommented the domain, setting it to some fictitious name (since the machine won't be the destination for any mail.

I uncommented this line:
Dj$w.Foo.COM

Linux System Information

2009-01-05T09:03:00.000-08:00

Presently we are building an inhouse compute cloud, which means admining a bunch of machines. To get quick information on the compute power of a machine look in:

/proc/meminfo
/proc/cpuinfo
/etc/issue (since these are ubuntu machines)
and a df-k for disk space.

-brian

Installing Rails on Linux (Ubuntu)

2008-11-03T06:36:00.001-08:00

I intend to start using this blog more for notes that might be useful for other people.

Recipe for RoR on Ubuntu:


sudo apt-get install ruby
sudo apt-get install rubygems
sudo apt-get install irb
sudo apt-get install ri rdoc ruby1.8-dev build-essential
sudo gem install rails --include-dependencies --remote
sudo apt-get install libmysql-ruby
sudo gem install activerecord-sqlserver-adapter --source=http://gems.rubyonrails.org

Is J2EE Dead?

2008-03-03T09:35:00.003-08:00

I recently presented at the Philly JUG. In my presentation , I asked the question, "Is J2EE dead?", half-kidding, but mostly serious.

I've spent the last six months developing in Ruby (some on Rails). It quickly went from just a past time, where I was developing a silly little beer review site called liquidMirth to serious business applications (at the day job: rVooz). And now I have to admit, the compile, build, deploy cycle of a standard J2EE application seems unbelievably daunting. Combine that with the cumbersome and seemingly never ending twiddling of meta-data and deployment descriptor files required by the libraries and frameworks (e.g. Spring & Hibernate) that are supposedly helping the problem and you have an incredibly unproductive environment for web application development.

I'd emphasize that this is unproductive for web applications, because web applications need to be especially responsive to user requirements. J2EE simply isn't agile enough. There is a
fantastic video by someone out at NASA JPL that does a fantastic job comparing the latest web application development frameworks. He does a great job emphasizing this point.

Now, all that said, J2EE != Java and Java's present sweet spot is in SOA, and business process integration, where things need to be transactional, and message-oriented/event-driven. Web frameworks are having trouble in that area, where things don't fit into a nice request/response interaction. On the "back-end" you need a bit more rigor, and typically you need to integrate with systems that won't conform to convention.

So, IMHO -- let more flexible languages accommodate the fickle nature of users. Use Java, with specifications like JBI and SCA, to implement an ESB on the back-end to do the heavy business process lifting, systems integration, and B2B style interactions.

And if you do that, you'll quickly realize that you don't need much (if any) of the J2EE spec to get-r-done. Instead, OpenESB in J2SE, ServiceMix, or Mule fill the bill quite nicely.

just two cents.

Philly Emerging Technologies Event: Call for Papers

2007-12-06T07:11:00.000-08:00

I'm Philly born and bred. And aside from cheesesteaks and soft pretzels, the Emerging Technologies Event (ETE) is one of the best things associated with the city of brotherly love. Chariot Solutions organizes this event annually and this year looks to be better than ever.

ETE looks to have a block buster lineup covering the latest lightweight/agile development frameworks, web 2.0 concepts, java and ruby/rails. ETE has both business and technical tracks. So, there is something for everyone. If you would like to be part of this event, the call for papers just went out.

Come hang out and while your here grab a cheesesteak and soft pretzel. ;)

ESBs: the value of indoor plumbing, public water and sewer

2007-09-06T10:24:00.000-07:00

I just saw Joe McKendrick's entry on the value of an ESB. Although I appreciate his comments, and even agree with some of his points, I don't come to the same conclusions.

There are two things that drive common understanding: shared experiences, and common problems. The wide acceptance of ESBs is driven by common experiences trying to implement SOA on a large scale with disparate organizations involved.

Without an ESB, the product of such situations is often a fragile, tightly coupled, tangled web of services that is nearly impossible to change, or even reason about -- this causes a lot of pain that motivates people to consider alternative approaches. (This is often the architectural equivalent of burning your hand on the stove -- something you don't do twice =)

Now, this is not to say that ESBs for the sake of ESBs is a good idea. Often, organizational boundaries can be used as a good litmus test when deciding whether or not to use an ESB, but consider "organizational boundaries" in the more broad sense of the term. This is more than just the end-consumers/producers involved (e.g. trading partners). It takes into account the vendors supplying and integrating the functional components (infrastructure) of the system. For example, if you're authentication & authorization components are coming from a different vendor than your service stack; you have an organizational boundary you probably want to consider.

Thus, if your scope is small enough, and you are getting all of your infrastructure from the same vendor (and plan to continue to do so), ESBs may not be worth it.

Joe was actually just reacting to an article by Bobby Woolf, which has some good insights in it...
"The problem is this: An ESB by itself produces no business value. An ESB is a means to an end, not the end itself. An ESB is like the electrical wiring or plumbing of an SOA. Plumbing does not produce value; sinks with water coming out of their faucets do."

I completely agree with Bobby's statement and just (IMHO) believe Joe made an incorrect inference.

Faucets and toilets deliver the value, but the pipes are not to be overlooked. Lets all face it, if it wasn't for plumbing we'd all still be sh*tting in outhouses. There *is* value in being able to connect a Moen faucet to any well pump, and being able to inject a UV-filter and water softener into the line without disturbing either the faucet or the pump; plumbing enables that. If we take another step back, there is value in being able to then hook into a public sewer system eliminating the need for everyone to install their own septic system.

A real world example of this can be found in Chad's SIP to XMPP tutorial. He was able to build that capability without coding because he used a JBI-based ESB approach. In this example, the two networks (SIP & XMPP) had "plumbing interfaces", and BPEL could be used as the manifold to connect them. Consider that Chad could have quickly and easily also added security "in-line" as well that ask the question "Is Alice authorized to talk to Bob?". These kinds of things are difficult to do without a standards-based, orchestratable messaging and transformation infrastructure (i.e. an ESB).

So, although plumbing might not be worth it for everyone, I for one don't like having to walk outside to relieve myself -- especially in the dead of winter. =)

An Open Technology Manifesto

2007-08-24T07:57:00.000-07:00

Many people have read Stevey's post. where he contrasts agile with Agile. Stevey does a great job of separating out the important "agile" value system from the methodologies and processes of "Agile" that were created to help the masses implement it.

Because in fact, it is in the implementation of agile that the values sometimes get lost. Ive' seen Agile often bundled with Extreme Programming tactics (Test-Driven Development, Paired Programming, etc.). which muddied the waters and resulted in teams claiming "agility" simply because they have junit tests. When people focus on processes, tools and methodologies too much, they can't see the forest for the trees.

Likewise, I believe we need a similar separation between values and implementation in "open development". In the details of the implementation, there are decisions about licensing (GPL or Apache?), and community (Java.Net or Sourceforge?). These can be argued just like people argue over the details of Scrum vs. Lean Devlopment vs. RUP. But in the end, "open development" manifests itself more so within the attitudes of the team members, than in the implementation decisions -- just like "agile".

Thus, humbly, I'd like to propose just such a manifesto:

We are uncovering better ways of developing software by doing it and helping others do it.

Through this work we have come to value:

Open discussion over closed "white-paper" analysis.
Collaborative contributions to existing efforts over proprietary (re)invention.
Open sharing of experiences over knowledge hording.

These combine with an overarching desire to commoditize the lower-level functions of the software stack, allowing all software development (and service-delivery) to focus on ever harder more valuable problems.

Enlightened developers often embrace the above values without knowing it, just like many of the same are inherently agile (in the "move fast and react fast sense of the term"). Others operate with a very closed mentality, despite the fact that sometimes they've "open sourced" their code.

For example, we often see "white-paper" analysis performed by the evil genius architects that lock themselves in a room with their crystal ball, carefully considering all options, finally emerging with an architecture that accounts for all the requirements, *and* all of the unforeseen changes to come (notice the paradox). Meanwhile, the enlightened developer has spent the same time discussing the options on a mailing list, effectively forcing some of the "unforeseen" changes forward in time by soliciting input from others and adopting suggestions (even if they don't directly address existing requirements).

Likewise, once an enlightened developer accomplishes a feat, he/she will blog about it to advance the productivity of the community at large, whilst the less enlightened will add this to their little secret bag of tricks enhancing their "job security". (the same developer goes on to prove black is white and gets himself killed in a crosswalk)

I was in diapers when the pioneers were doing the real work on open source. So I feel presumptuous even suggesting such a manifesto and I certainly don't intend to even hint that these are novel thoughts. I'm sure the likes of Eric Raymond and others have been espousing such values for years. My only point (if any) is that the Agile Manifesto did a great job of distilling agile into four statements that could be internalized by all. Open Development might be served well to do the same, separating it away from the "noisy" licensing debates discussing freedoms, and capitalism.

FWIW.

JBI: Catch the second wave...

2007-08-20T08:49:00.000-07:00

If people remember back to the early days of Java EE (J2EE), there was a lot of promise, but a lot of frustration. I still have mental scars (and a slight eye twinge) from the countless hours tracking down typos in deployment descriptors. But, we were willing to endure the pain because there was a lot of power in the platform.

And as a model, J2EE was solid. Sure, there were squabbles over the value of different parts of the model (e.g. Entity Beans), but the general abstractions and approach were incredibly valuable, as evidenced by its success. JBI is similarly positioned.

However, mainstream adoption requires not just a solid design model . Mainstream demands ease of use, immediate gratification, and passionate community support. This is the fundamental appeal of Ruby/Rails and why J2EE struggled in the beginning. In the beginning of J2EE, it was a week long effort to get an EJB deployed. Meanwhile, Rails has caught on like wild fire because it takes less than 10 minutes to get an entire n-tier application up and running.

In an effort to learn from this, independent of the actual implementation communities (Apache - ServiceMix, Java.Net - OpenESB, etc.), we've created a JBI community site jBIZint, where the various implementation communities can come together to discuss common needs that will improve a user's over all experience. If this sounds like something you would like to be involved in, please join the mailing lists.

As an example, there are two movements afoot that will improve JBI adoption and ease of use. The first is a movement to standardize the diagramming technique for service assemblies, and the communication between them. Much like SCA has done, a standard diagramming technique will allow system designers to communicate effectively, the components, meta-data and communication paths of the services in the system. (Again, please join the discussion on the mailing lists to get more info)

The second movement is to get an "easy-install" package management system in place akin to Debian's apt-get and maven's repositories. (credit Bruce Snyder for the idea). This would make it simple to install components through tooling (e.g. Eclipse, NetBeans, or command-line). The package management system would pull down the component and all of its dependencies (e.g. shared libraries), perhaps even a sample service unit.

Combining standardized diagramming with an easy-to-use package management system should make it even easier to pick up one of the numerous "10-minute" tutorials on JBI, like Chad's XMPP-RSS Demo, and get it up and running in 2 minutes with an understanding of how and what you did. =)

How JBI can improve your love life. (JBI for Social Networking)

2007-07-12T11:47:00.000-07:00

Our team just released four JBI Components, a SIP BC, an XMPP BC, an RSS BC, and a UDDI BC . Having developed those components, we're now reaping the rewards; we're able to combine these low-level collaboration "services" to produce new and cool collaboration capabilities. Essentially, the four components together create a collaboration platform. One use for just such a platform might be a social networking application. hmmmm...

Lets pretend I'm looking for love (in reality, i'm happily married, but we can pretend). I've joined an online singles dating site and I want to write a quick application that will let me find a date. Lets call this SaaS app the lovinator.

First, things first --- I don't want to have to continually go to the dating site to get the information. So, I quickly write a a python script to crawl the website looking for profiles of interest. Once the script finds a profile, it posts the profile to the lovinator.

Fortunately, the lovinator is built on a JBI platform. An HTTP BC receives the request, which contains a raw XML document that has the profile. That XML is routed to the RSS BC via a simple BPEL process. On my desktop, I'm subscribed to the lovinator feed, and am able to view new profiles.

I see a particular profile that is interesting. Now, I want to be able to see when he/she comes online. And since I'm so hip (a.k.a geekish), I have an iPhone with a jabber client installed. Lets assume the prospect's XMPP and SIP addresses are in their profile and I'm headed out of the office. Leveraging the XMPP BC, the lovinator can subscribe to the prospect's presence information and send me an update when he/she comes online; directly to my iphone. Nice.

OK, so now I can peruse prospectives from my desk via RSS, and stalk them from my iphone using XMPP. The only thing left to do is call them out of the blue. (believe me -- they love that) Fortunately, the lovinator can do that for me; automagically. Using the SIP BC to perform a bit of third-party call control (3PCC), the lovinator can setup a call between us. (saves me the time required to look at their number and dial the phone)

Now we're cooking, but where is the social networking aspect? Well, as I said -- prospects are disseminated over RSS. So, there is nothing to prevent others from subscribing to the same feeds. Additionally, it would be trivial to incorporate a Java EE service engine that would allow me (via a webapp) to submit feedback about prospectives.

Lets say that voice conversation didn't go so well. (e.g. I found out he/she has a dozen cats and lives with his/her mother at the age of 39... and I'm not feline fan) I could feed that information back into the prospect's profile feed via the webapp. That would get disseminated to all those subscribed, saving other people time and energy -- or making that "special someone's" day.

One can imagine a ton of extensions to the lovinator. For example once I do find that special someone, the lovinator could send me messages(via the XMPP BC) when their birthday rolls around. (anniversaries, etc)

A bit contrived? perhaps. But the point is that JBI creates a great, extensible, platform with new loosely coupled capabilities being added everyday that we can "wire in" easily.

So, the point is; we save time.

Estimated time to develop the lovinator? Only a few weeks. I just checked and the domain lovinator.com is available. Any takers? =)

For some more realistic use cases you can check out jBIZint where we're trying to outline the "tougher" use cases, that the spec may or may not accommodate.

JBI: Application Development w/o Coding, are we there yet?

2007-07-12T11:46:00.001-07:00

Many people, including myself, approached JBI as a standards-based means of achieving ESB capabilities. That was the big value proposition. In the early stages of using it, we treated JBI like JMS on steroids. It wasn't until later in the game, we realized the full potential of JBI.

And, it really boils down to knowing the Fundamentals of JBI. For developers like myself that have a tad bit of an attention deficit disorder, it is easy to overlook the implications of those fundamentals. Specifically, the JBI specification had plans within plans, and containers within containers.

The JBI run-time acts as the first level container, consuming service assemblies for deployment. Those service assemblies contain service units which get deployed to service engines and binding components. So, JBI components themselves act as a second-level containers for the artifacts within service units.

This is often misconstrued by people that believe Service Engines *implement* the business logic in an application. This is not the case. Instead, business logic gets *deployed* to Service Engines as part of a service unit (within an assembly). Service Engines contain and execute the business logic, but they by themselves do not provide it. This is tremendously important because it allows JBI to deliver, "Application Development without coding".

Lets look at an example, the BPEL Service Engine lets me wire together services mapping the results of one service to the inputs of another. So, without any coding I can deploy a bpel process to the BPEL Service Engine. Furthermore, I can connect that process to specific transports, like XMPP and SIP by simply deploying service units to the XMPP BC and SIP BC respectively.

In this situation, the BPEL SE, XMPP BC and SIP BC are acting as containers for specific business logic and "instances" of capability that could have been defined graphically in an IDE like NetBeans.

Thus in our never ending quest to make our lives easier and leave the world of semi-colons, I believe JBI has brought us one step closer.

SCA & JBI: Made for each other. (SOA Gone Wild)

2007-06-27T13:56:00.000-07:00

I agree with Shaun Connolly's
observation that "While they [JBI and SCI] may share some similar patterns, those who pit SCA vs. JBI only demonstrate their inability to distinguish between the two perspectives".

In fact, we set out to prove the assertion that:
- JBI and SCA are entirely complementary.

And we've made quite a bit of progress by taking an example SCA composite application / contribution and figuring out how it translates into a JBI runtime environment. But translation is only one example of how JBI & SCA may fit together.

There are actually three different relationships that make sense. First, SCA run-times that already exist (e.g. Tuscany) could be deployed as containers within JBI by packaging them as Service Engines. JBI would leverage the SCA Service Engine as a means of integrating SCA contributions into a JBI runtime. In this respect, JBI provides an integration backbone that would improve the utility of SCA contributions by making them available to networks, capabilities and systems for which there is no SCA component implementation. (e.g. SIP and XMPP) Essentially, this would attach an SCA contribution to an bus architecture, with all of the benefits that comes with that.

Secondly, I don't see any reason why there wouldn't be a JBI Component Implementation added to SCA's list of implementations. This would allow an SCA contribution to declare a JBI service unit (or service assembly) as a component implementation within an SCA contribution.

Lastly and most interesting, JBI can be the run-time for SCA contributions. In this capacity, a translation layer would translate an SCA contribution into a set of service units deployed to binding components and service engines. In this example, each binding in the SCA composite application the would result in a service unit bound to a binding component. Likewise, for each service implementation in the SCA composite application, a service unit would be deployed to a service engine. (e.g. bpel, pojo, etc.)

The translation is a tremendously powerful construct. It lets JBI deliver all of the bus-like features, life-cycle management, and administrative controls, while allowing architects to define their composite applications in SCA.

We are working all of this through on jBIZint . If all goes well, we hope to deliver a Grand Unified Platform by the end of the year that when combined with Eclipse and/or NetBeans will directly interpret developers brainwaves into SOA. =)

JBI on your cell phone?

2007-06-27T13:55:00.000-07:00

Despite the fact that many early JBI implementations are reusing enterprise infrastructure for their implementations (e.g. JMS messaging backbones and J2EE containers), my prediction is that JBI will start invading the mobile market.

Already, people are looking to leverage JBI implementations on small footprint platforms as a means of realizing the benefits of an "enterprise" service bus, but without the overhead of the "enterprise". In this capacity, JBI is a great means of achieving service re-use , while maintaining loosely coupled composition.

For example, JBI may be the best means to inject security into a mobile application. Since JBI focuses on data flow, , a developer can just "drop in" security components, rewire the messages to first pass through an authentication and authorization engines. Without changing the app at all, it would be capable of invoking services that require digital signatures. I go through this use case here.

It is a common misconception that OpenESB is wed to NetBeans and Glassfish. This isn't the case at all. In fact, you can follow these instructions to get OpenESB up and running in a J2SE environment, with XMPP capabilities and RSS feeds!

Lots of fun. =)

JBI as a Convergent Communication Platform

2007-06-27T13:51:00.000-07:00

I have a hard enough time keeping my mind straight as it is, maintaining multiple internet persona's only makes my life more difficult: a skype account, IRC, AOL, XMPP, Yahoo, email, PSTN, mobile, etc. Some clients have been doing a better job of bridging the networks from a user perspective, but this does nothing for capabilities development on the server side -- and the disparity between the networks still causes user headache.

For example, even using one of the multi-protocol chat clients, to coordinate a chat-room I need to get everyone to agree on a common protocol, everyone needs accounts, and everyone needs to be on the same "type of network". (ie. firewalls need to permit traffic to the agreed upon network)

Why?

Because there isn't a server out there capable of maintaining presence, and location information across the multiple protocols. Sure, there are lots of gateways being built (e.g. XMPP -> SIP and back), but this results in the need for nxn gateways. Any architect will tell you that you need a common normalized schema to reduce the number of necessary conversions. Then you only need to develop n gateways, each converting from the protocol specific format into the normalized message format.

Hmmmm, interesting - this sounds awfully familiar. Enter JBI.

The devil is in the details, but if we ignore them for a moment, this is exactly the concept behind binding components (BCs) and the normalized message router (NMR). Over the past six month, we've developed binding components for XMPP , SIP , UDDI , and RSS . The purpose of each of these is to convert between normalized messages and their protocol specific equivalents.

Since the BCs expose those disparate communications networks over a single "bus", this appears to be the perfect environment for convergent applications development. Thus, you could implement a single presence service, a single locations repository, etc. Finally, I'll be able to join a chat-room over SIP, while friends join over XMPP, with the entire conversation broadcast over RSS.

Now, back to the devilish details. In a previous life, I spent long hours working on SIP (registrar, presence and proxy) servers. More specifically a commercial derivative of the NIST JAIN-SIP-Presence-Proxy server. Recently we've formed a new project, Marlin to attempt the very idea described in this blog, implementing the same capabilities as the JAIN-SIP-Presence-Proxy project, but independent of the protocol.

We're putting use cases together here , and meeting to determine how all of this relates to Sailfin .

Its an exciting time to be in communications infrastructure. =)

SOA, ESB, JBI & SCA: A caveman's perspective

2007-06-27T13:50:00.002-07:00

Software development techniques and processes are changing at ever faster rates as open collaboration development spreads globally (thanks to the adoption and promotion of open source software). As the systems we architect and implement become increasingly complex, the mechanisms, infrastructure and patterns with which we build the complex systems constantly improve to keep the problems tractable.

The Java Business Integration (JBI) specification and Service Component Architecture (SCA) are the latest additions to our tool set.

Now, I'm a simple man, a caveman really. And for the last three years, I've been working to construct a multi-channel service oriented architecture for the federal government, with pluggable enterprise services (pub/sub, security/encryption, messaging, VoIP, etc.)

As you can imagine, this is a hard problem for a caveman to solve, especially when you layer in a complicated political landscape filled with software infrastructure vendors, and their professional services divisions. We needed an industry standard on which to hang our hat.

Now, most people would hang their hats on "Web Services". Those same people would probably define web services as WSDL + SOAP + HTTP + (UDDI). They would then proceed to convert all RMI, RPC, and potentially database communication into web services: then call that a SOA. Unfortunately, that only results in a rat's nest of invocations and derived dependencies that can bring down an entire enterprise. (we have the battle scars to prove it)

So, we evolved.

We realized that we needed a common communications backbone to unravel the mess. Enter Enterprise Service Bus (ESB) concepts, but again we were disappointed. We looked around for one of these new fangled ESBs, but all we found were re-applied messaging systems that required proprietary extensions to achieve a services layer. (and we knew enough to stay away from those, and the vendor lock-in that came with them)

SO, we looked long and hard at the writing on the (cave) wall; and found a promising acronym: JBI. We set about downloading JBI implementations. Immediately, we were hooked. JBI provided a standard interface through which we could deliver our enterprise service "lollipops". Literally, we could drop our lollipops into existing JBI containers, rewire the connections and provide service consumers and producers with security, compression, discovery and other capabilities. Wow, nirvana.

In fact, as cavemen - we were more productive than ever. We could develop applications without writing a single line of code. Applications development became a matter of drawing pretty pictures. Using a palette of components, and drawing lines between them, in minutes we could create new convergent capabilities - e.g. take messages from XMPP, and post them to RSS feeds.

But alas, nothing gold can stay. The political landscape shifted, and SCA came down from the heavens like a meteor crashing into the earth. We ran for cover, trying to figure out how SCA fit into our new world view. What we had heard about SCA was promising, but we had trouble putting our fingers on it. SCA seemed like a great language with which to communicate with other cave people, a common way to describe what we were building, and compose larger services from smaller, recursively. This was much needed, and completely complimentary to our new found love - JBI. As cavemen, we couldn't see where SCA dictated a run-time, but we didn't need it to. We had JBI already.

Us cavemen proceeded forward with a new world understanding - determined to build a bridge demonstrating an SCA composite application, running in a JBI runtime.

You can follow along at jBIZint .

Better than Java.net?

2007-06-27T12:15:00.000-07:00

I'm just wondering if blogspot is better than java.net. So, this is my test drive.
Ordinarily, I have a blog over on java.net, but I noticed some people had moved over here and wanted to see what all the hubbub was about.