Saturday October 07, 2006

It's broadly appreciated how scaling up is usually driven by business demand, but the requirements for scaling down are rarely as appreciated. Questions about how web 2.0 business scale up abound these days. As the challenges of service growth and business plans stress technical infrastructure, startups try to squeeze everything they can out of their architecture with a number of widely accepted practices. However, scaling considerations for the other direction are oft neglected.

1. Isolated functional testing to mitigate the riskiness of change

   End-to-end testing that doesn't require duplication of production infrastructure is a strategic advantage. I know of a financial analytics system run by a large institution that is untestable. This system has cron jobs, data feeds and query systems built on top of Perl code going back at least a decade. The inputs and outputs are so convoluted, that the system is untestable. So if this code is making the bank that owns it tens of millions of dollars every day (it is!), what's wrong with that? Well, it could be probably be more profitable if it could be changed and optimized safely. As it stands, the folks maintaining the code don't really know what modifications might break the system and with income produced at that scale, who wants to risk it? So look at the systems you're working on now, think about the "scaling up" considerations you've made and ask yourself: Is a system testable in a developer's environment? Can they unit test? Can they perform functional tests? Do the tests require access to resources only available at the data center? Is "now" hardcoded to the present in your code? Using scaled down database, messaging, caching and application runtimes that have no dependencies on a connected network and production infrastructure should be considered up front in your design consideration.

2. Operational costs of vertical vs horizontal scaling

   If a system makes assumptions about the process space it runs in that allows for functionality to be accessed from other runtimes, bravo: you may be headed in the right direction of service oriented architecture and horizontal scaling. But can the application stack be collapsed? This is like the OMG-moment when folks first started running J2EE application tiers over remote interfaces and realized that they've ended up with so much complexity and overhead, they have no choice but to scale up. That complexity can have all kinds of expensive side effects with how effectively systems can be triaged when they ail.

3. Business agility or just changing your mind

   Businesses are run be people. People make mistakes. Wetware is imperfect. When you buy a long term commitment to a data center, you may be assuming liabilities that will outlive the business proof. Make sure the hardware footprint you're signing up for is one you can sustain it or you can get out of it. When you build gratuitous tiers, the costs of taking them out when it's time to consolidate functionality can be stifling. So ask yourself: If systems scaled up to meet business objective that aren't met, can you "retreat" from the scale-up offensive?

Every time I see a system that's hard to test, has sysadmins overwhelmed or are not meeting business objectives and has to be reeled in, I'm reminded of the importance of thinking about scaling in both directions. No, I haven't read the book yet but as someone burdened with too much stuff at home, I've got it on my list.

web 2.0   unit testing   functional testing   technical operations   system architecture   software