With all the activity in the Big Data arena, there are tons of great technical presentations out there. (kudos to Acunu, Druid, Titan, Priam, Spark, etc.) With all of the bright, shiny and new toys out there, its easy to overlook the system-level innovations around processes and organizational dynamics enabling all that birght, shiny, and new. The system-level innovations enable us to stitch the toys together into powerful, yet tenable systems that can evolve over time.
Two presentations in particular have stuck out in my mind:
- Adrian Cockcroft's : Netflix: Building up and Scaling out on Open Source (slides + video TBD)
- Eric Lubow's: The Big Data Revolution is an Evolution
These presentations resonated with me, because we are beginning to roll our Big Data platform out across all of our product lines, and across the organization. The platform is now impacting/enabling all aspects of the business, including: Analytics, Information Management, Infrastructure(IT), and Development Operations (DevOps).
That roll-out has made it crystal clear that decisions made by software development have a ripple effect throughout the organization. As that ripple flows through the organization, and we apply the platform to an increasing number of product lines, we are constantly finding new ways to apply the technologies. Often however, that means changes to the system. (e.g. new data models, new interfaces, components, etc.) One key to innovation is creating the processes and organizational dynamics that support those changes, some of which may be dramatic.
And when you partner that with the new and shiny technologies, that also enable innovations...
See Billy Bosworth's quote from USA Today:
"There is a new breed of software developer, and he could not care less about relational databases," Bosworth says. "This technology frees a developer to think differently."
You get a powerful combination that allows developers to integrate those new technologies and rapidly evolve their use of those technologies. (boo yah!)
What's this mean? Shift architectural focus to the things that will result in an Evolutionary Architecture, one capable of evolving with the business.
It means don't spend a lot of time trying to divine the perfect architecture upfront. Chances are you'll be wrong. BUT -
- Spend time creating the proper abstraction layers that will allow you to change your decisions down the line, BUT do this at a systems level!
- (IMHO -- you don't need to wrap every java module with interfaces and impls, abstractions you'll never use)
- Spend time enabling automation & continuous delivery
- Spend time enabling monitoring and metrics gathering so you can rapidly react to operations
In a nutshell, build the software from the ground up so:
- You don't have to spend time supporting it. (Traditional non-functional reqs: H/A, etc.)
- You don't have to spend time deploying it. (Automation)
- You can anticipate problems *before* they occur. (Metrics / Monitoring)
- When problems do occur, you can remediate them quickly.
- Soft-Deployments / Roll-back, etc. (Infrastructure / Continuous Delivery)
Yes, these are traditional concepts, but Evolutionary Architecture pulls them forward in the life-cycle and makes them fundamental parts of the application design. That is one of the reasons we like Dropwizard. It comes out of the box with patterns for metrics, health-checks, and support for diagnosing issues. (REST call for thread dumps)
Anyway, incorporating the concept of an evolving architecture into the actual applications themselves enables the organization to focus on the new and shiny.... speaking of which...
Anyway, incorporating the concept of an evolving architecture into the actual applications themselves enables the organization to focus on the new and shiny.... speaking of which...
