Friday, January 25, 2013

Zen and the Art of Collaborative Software Development

Conway's law suggests that designs are constrained by organizational communication structures.  I've seen that law manifest itself over and over again and I'd assert that it is impossible to develop a cohesive software platform unless the proper collaborative dynamics exist.  Specifically, to develop a software platform that can satisfy the needs of many different product-lines, consumers, and/or dependent projects, you want those dependent projects to be able contribute back and co-develop the platform.  This approach shares ownership, shortens the development life-cycle, and enables innovation across the organization.

It follows that the dynamics required to develop a platform are different from normal silo'd team dynamics.  The dynamics you need mimic that of the open-source community.  Developers need to be good citizens in a larger community.  Here is what I think that means:

First.  Be Self-Aware. 

There are four stages to mastery: Unconscious Incompetence, Conscious Incompetence, Conscious Competence, and Unconscious Competence.   It is very important to know where you are on that progression before you interact with a community. 

If you aren’t self-aware, you run the risk of making an unfounded assertion when a question may have been more appropriate.  (We all know the A-hole that emails a discussion list making claims before doing his/her homework)   Thus, I’d recommend always starting from the conscious incompetence perspective and communicate with that tone.  If you are new to a project, communicate via questions to confirm assumptions before making assertions.

Once you’ve achieved conscious competence, help others out!  Take questions from others, and propose solutions to them politely and in an open audience.  Everyone will benefit from the ensuing discussion and it will enable innovation.  You may have a solution that others can improve upon, but the tone should remain propositional.

As you progress to Unconscious Competence, swtich from proposing solutions to delivering them.  Instead of simply proposing solutions in email, you should be submitting pull requests with working code.

Second. Be aware of a project’s maturity.

Early on, projects are trying to pickup momentum.   They may be throwing stuff at a wall to see what sticks.  It is important to recognize that. Often, in the early stages of a project the participants are trying to demonstrate the most amount of value in the shortest amount of time, which is one way to get a project funded / off-the-ground.   If a project is in that state, complaining about configurability and elegance of interface might not be the best idea.

Third.  Be aware of others.

(IMHO) Passionate rock-star developers are often arrogant and obsessive-compulsive.  Those great developers want things their way, and they believe they have the best solution.  (Myself included, I must have been an asshole to work with early in my career)

As you start collaborating with larger communities of developers, you realize that beauty is in the eye of the beholder.  You can appreciate other’s perspectives with an increased tolerance for other ways of doing things. (other coding styles, languages, and best practices)

Finally, I think you get to a place where you can listen to others’ ideas without feeling an immediate compulsion to improve upon them.  This is powerful, especially for seedling ideas.  Passion for an idea is a fickle thing.  Sometimes its more important to keep your mouth shut, and let a peer evolve an idea before suggesting improvements and vocalizing all the nuances, edge cases, and counter examples that might make it difficult.  You never know what might grow out of any random thought.

As a corollary, it’s important that people feel welcome to bring ideas out into the open.   If other people don’t feel that they can bring ideas to you, or you feel you cannot bring ideas to them, it is YOUR fault and no-one else's  Its important for each citizen to own that dynamic and ensure the atmosphere is conducive to innovation.

IMHO, these dynamics are essential in any successful collaborative community.  Furthermore, such dynamics are cultivated by successful benevolent dictators. (shout-out to @zznate and @spyced, two of the better dictators I’ve met.)  

We had a great discussion on this topic at Health Market Science (HMS), where we do a lot of open source work.   Incase anyone is interested, I posted the slides that drove the conversation.

Hopefully people find it useful.

(tnx to @jstogdill for the mentoring over the years)

Sunday, January 20, 2013

Webinar on Event Processing on Cassandra w/ Storm

Thanks again to all those that made it to the webinar on Thursday.  It was a lot of fun tag teaming with Taylor Goetz.  Storm-cassandra has come a long way.  The slides and video are now available.



As always, please shout if you have any additional questions, or if we got something wrong.

Wednesday, January 2, 2013

Native support for collections in Cassandra 1.2! (no more JSON blobs?)

In case you haven't seen it yet,  Apache released Cassandra 1.2:

I'm stoked.   Presently we write maps/lists as text blobs in Cassandra using JSON.  This has obvious limitations.  In order to add something to the map/list, we need to read the list, append the data then write the data back to C*.  Read before write is not good, especially in a large distributed system. Two clients could read at the same time, append, and the second write would effectively remove the element added by the first.  Not good.

Although I think the Thrift support is a bit clunky (via JSON), CQL3 supports native operations on collections. 

Now, we just need to figure out how to migrate all of our data. =)

Creating Your Frist Java Application w/ Cassandra

Looking to kick off 2013 with a fun project to get familiar with Cassandra?  Why don't you build a globally scalable naughty and nice list?  At least, that's what we did in the webinar a few weeks ago. 

If you would like to relive the webinar, Datastax posted the slides and video.

We went through two quick examples using Astyanax and CQL3 to manage Santa's naughty list.  As much as it was a demo of the APIs, it also shows how you can use cqlsh and cassandra-cli to get two perspectives on the data: a logical view via cqlsh and a physical view via cli.   Its important to keep both perspectives in mind if you want to build a scalable app.

You can find the code here:

Please let me know if you have any questions, or if I flubbed up on anything.

BTW, Jonathan has a webinar coming up on "What's new in 1.2?"  It should be a good one.