fromMarch 2011

Release Management

”On the Truck”

There is a management school exercise to complete a project and ship it to headquarters by tomorrow morning; the truck leaves at 5pm. The point of the lesson is that your project is either on the truck or it's not, and if it's not on the truck, it really doesn't matter how well you complete the project.

Your customer, client, consumer or boss, doesn't really care how you get there. They care that you gave them what they think they want. Our job is to make the right compromises to deliver good software that is just what our users need when they need it. This is what Release Management is all about.

I am a Release Manager. I am the human coder.module. I check for architectural soundness, for performance and scalability, for ease of maintenance, for refactoring or simplification. I ask questions, leave suggestions, teach and mentor, sometimes by example sometimes by comment. I coordinate our development, product, and QA teams. I manage risk by controlling releases of code and configuration to staging and production environments.

I work at, a Quantified top 100 North American website, probably the highest traffic Drupal 7 website in the world. Most minutes on most days we deliver customized non-cached content to over a thousand people, every page built to the individual based on location or preference. In the time it takes you to count the number of people near you, we'll probably touch more people. Developing, testing, and deploying code under these circumstances is challenging. Screen shots of Facebook and Twitter fail whale pages remind me that we are not alone. Keeping a high volume rapidly growing production website up 100% of the time is impossible. Yet, since redeveloped on Drupal 7, our daily traffic has increased and our monthly availability has gone from 99% uptime to 99.97% uptime; hours of downtime are now measured in minutes. Just today we hit another record day, breaking the record from just yesterday, breaking the record from last week. That's all significant!

Here's how we do it...

Style and standards review

Drupal has coding standards. Standards don't exist for their own sake. Standards don't exist so that someone can be a pedantic or syntactic tyrant. Standards ensure that the cacophony of code commits assemble into a beautiful orchestra. Standards create a coherent look and feel to the code that make it easier to read, maintain, learn and enhance.

For DrupalCons Szeged and Boston, I reviewed all of Drupal's contrib modules using coder.module's drush features and presented the top 10 common coding standards mistakes. All of these might seem petty and breaking any of these rules will not make your code any less functional. But adhering to these standards will guard that your code is maintainable long after you abandon the module or move to another project. Standards also make it easier for a person or machine to scan and correct problems.

I believe software is functional art. Good code isn't just functional, it's creative. It has a solid logic that is beautiful in the same way that the fabric of our universe is logical and beautiful. Software as art follows the maxim “beauty is in the eye of the beholder”. While I believe this, I suggest that you resist the urge to violate standards just because you think it looks better. When it comes to style, conform. Show your artistry architecturally.

Architectural review

Design for extensibility. Learn the Drupal API's. Use alter hooks to extend core functionality, but also add alter hooks to your own code so that others can extend what you build. Use Drupal variables instead of constants. Using variables allows you to change values more easily than pushing code, even if you never implement an admin page. Only change core to fix bugs, enhance performance, or extend it with new hooks. Period. If you do change core, submit a Drupal issue, and save the patch file with your code. We version control patch files named after the issue number.

Design for speed. Understand databases, queries and indexes. MySQL joins are bad. At we solved the field join problem with a NoSQL mongodb_field_storage.module solution, made possible by the new Drupal 7 field API. Minimize full table locks to rows. Use only indexed queries, but not too many indexes. Learn how btree indexes work. I'll share more details at my Chicago presentation.

Never call a third party site from your web page, always do this from cron and save the results locally. Beware third party javascript widgets, most are poorly written. Multiple third party widgets can interact in unexpected ways and cause problems.

When there's a Drupal way, try that first. Use drupal_http_request instead of curl.

When releasing code to servers, stage it first, then move it. Copying code from central repositories directly to the Apache directories is too slow and will cause inconsistencies in the code that will throw occasional php errors.

Remember, the truck is coming

We use time boxes. The time box is our truck. We start and end releases on Thursday mornings. QA is completed Wednesday night. Code freeze starts on Tuesday, code thaw on Monday, merge requests should be in early on Monday. Thursday and Friday are development days, but also time for hot-fixes. While that's a tight schedule, that's just my schedule. Each developer has their own schedule that simply overlaps with the release schedule. What's important is that developers have their tasks ready to get on my truck before it starts down the road.

This is how it's supposed to work, but it rarely does. Someone signs a new contract, the boss wants a new feature, we find some critical bug, the site almost burns down... You have to be flexible and make sane choices. Prioritize. You might delay the truck a bit, we sometimes slip to Fridays, but anything past that changes next week's schedule. Force management to make choices.

My job starts with a code review. Reviews start with a diff. Once you read the diff, you can apply it to a branch, look at the code within context, make changes, test, merge, revert, or check-in. While there are Drupal standards and project standards, I have my own standards too.

I make sure every function has a header comment. I make sure comments use sentence case. I look for common coding violations such as spacing. I look for consistent variable names, so the code is readable and maintainable. By consistent variable names I mean, call a tid a tid, not an id a tax or a term. If the name needs further clarification, call it foo_tid. If a term is a taxonomy object, it should always refer to an object and all variables named foo_term should also refer to objects. If edition is a term it should always be an object and never a tid.

Within css, I look for commas between selectors and left curly brackets before definitions. Our coding standards help scanning for these easier.

I use and to check function calls.

I check that all queries use indexes, and have a comment about which index they use. I balance the need for fast queries with having too many indexes and sometimes look for ways to reduce indexes by using multiple queries or sorting or filtering in php. I check that we only call other sites from cron. I make sure we cache as much as possible. I often refactor code. I often ask others to do secondary reviews. I'll ask themers to review other themers or a javascript expert to review third party beacons. Delegate.

Remember the truck. A couple thorough reviews will teach most developers what you want. After that, the code is either good enough or it's not. If it's good enough, and won't crash the site, and won't make things worse than it was before, let it pass to QA.

Trust the meritocracy. Spend your time reviewing people who need it. Don't bother your best developers with cosmetic changes, do those yourself. Ask questions. Don't tell. Explain the concerns. Collaborate. Resist the urge to change other peoples code. When the time arises, teach by example, write code, but do it in the developers branch, in a single revision, tell them you did it, and ask for a review.

Remember the truck. Prioritize. Question. Manage people, risk, and code. Teach. But get it on the truck. If you miss this week's truck, there's always next week's truck. The trick is to keep traffic moving and make sure the most important stuff is on the earlier trucks.