Kicking the data line to the curb

September 25, 2007

After 3 years as a Crackberry addict that demonstrated a Pavlovian response whenever the damn thing vibrated, I have decided to go data free with my new phone. Reasons:

  1. The new company no longer pays for the phone and data line. They give us a re-imbursement on monthly fees instead. Translation: I’m a cheap bastard.
  2. The thrill of feeling important enough to check my messages in the middle of dinner, conversation, etc has diminished in equal proportion to the amount of irritation demonstrated by my wife.
  3. I ruined a perfectly good vacation by getting stressed out after reading the damn thing.

I’m on the computer 8-10 hours a day anyways, and am usually on it first thing in the AM and last thing at night, so it’s not like I’m actually offline at all. And even when I am, and someone needs to get hold of me, they could do the old fashioned thing and actually call.

This could be the early stage of recovery, which is usually followed by a relapse if Lindsay Lohan is a good indicator, but I’m feeling good about this, I really am.

When I tell people I’m dropping the data line on my phone, they have two reactions: (1) “best thing you’ll ever do”, and (2) “are you f**king kidding me? How will you ever survive?”. Stay tuned, same Bat time, same Bat channel…

What I like about Scrum

September 23, 2007

After about a year of working with an ‘agile by committee’ development methodology that was going along quite well, a couple of things happened at the startup I work with.

(1) We found a CEO who is one of those rare people blessed with considerable technical, business, and managerial ability.

(2) We acquired the IP and developers of a very exciting technology that could leverage our existing technology to make a killer app. That technology was being developed for an enterprise product, and their process was much more waterfall because they had to completely understand and respond to customer feature requests that were very detailed and tied to a lot of money.

After about a month of thrashing around, the CEO declared that we would all be using Scrum. There was some hysteria and some drama from the other team as they flat out declared that moving in such small iterations (1 month) would not allow them to think their problems through. Another CEO would have thrown in the towel, but ours had seen Scrum work, at scale, across multiple teams, all doing invention based work. So he held his ground. I for one was skeptical about Scrum as well, mainly because I thought our process was just fine. However, I thought about our process some more, and found the following flaws:

  • we weren’t moving from well defined requirements, they were being made up on the fly.
  • we weren’t having the right discussions with the right people in the room. High level discussions were being had while developers who really didn’t care about details they considered ‘lipstick’ were in the room. Low level discussions were being had while program managers, who didn’t have the technical chops to keep pace with the conversation, kept interrupting and throwing everyone off.
  • Our iterations, which were 2 weeks long, were too short to show visible progress.

I also did some reading here and here for starters, and finally found a great pdf that answered a lot of my more granular questions.

So, long story short, this is what I’m liking about Scrum:

  1. A month of development with a demo at the end is a really nice rhythm to follow. 2 weeks was too short. Anything longer than a month is too hard to plan for.
  2. Product Owners own the vision and the final output of the product.
  3. Dev teams own the implementation of the product.
  4. Product Owners are not allowed to inform, change, or participate in product development after the sprint planning meeting — the “Chickens and Pigs” rule. This separates concerns quite nicely. Product owners care about the final product, developers care about the day to day details of making that final product, and the iteration deliverables are what everyone agrees on.
  5. The Sprint Planning meeting requires that there is a Product Backlog in place that is actually understood by everyone on the team. This is a forcing function on Product Management to get their shit together and hammer out concise, complete requirements.
  6. Estimating tasks for the next month forces devs to think through initial implementation much more deeply than if they were following a waterfall methodology. This leads to more accurate estimates.
  7. By the same token, they don’t have to know everything up front. The process allows for tasks to be added by the development team if the agreed upon deliverables require more work.
  8. The Product Owner cannot change the iteration deliverables once the iteration has started. This provides the stability that allows a dev team to make rapid forward progress without thrashing.
  9. Tracking progress and updating progress is extremely low tech — a board with stickies on it is easy to look at and easy to update.
  10. The board is an information radiator. If the iteration is going badly, it is immediately obvious from the board. Same if it is going well.

A week into adopting the scrum process, we were sitting in a room with a product owner and 4 devs, trying to hammer out a solution. This is actually a violation of (4) above…and it showed. The product owner didn’t have the technical depth to add value to the discussion, yet she felt the need to comment frequently, and her comments, which were not very relevant to the technical details of the discussion, kept derailing everyone else. After the initial meeting we realized our mistake and had a dev only meeting, and resolved in 10 minutes what was taking us over an hour. She wasn’t too happy about the fact that we met without her, but we were doing the right thing, and the rate of progress showed it.

Scrum is the right separation of concerns. I know from writing code that when I separate my concerns just right, the design and coding go much better. I’m looking forward to seeing if Scrum has the same effect on our development organization.

the right tools for the job

September 1, 2007

I’ve been working on a little page scraping project , nothing to write home about, nothing as hip and cool as MapReduce, but still a good learning experience. Today I learned a couple of things that I already thought I knew:

(1) Always pick the right tools for the job.

I was trying to extract chunks of text from an HTML page where the text spanned several DOM nodes that could be on different tree branches in an arbitrary DOM. The chunks were delimited by known words, that was my only real hint. Because I had been using HPricot to get down to the nodes, I continued to use it to try and pick out the nodes I wanted to scrape text from. This was really, really painful, basically I was writing an algorithm to traverse part of a tree, track the depth in the tree, and grab text under a set of nodes that were at the same level but on different tree branches. I was so wrapped up in getting this algorithm correct that I ignored the little voice inside my head that kept saying “this is too hard…”. Finally I grabbed Phil and said “I know I’m working too hard at this. The algorithm is correct but it’s a complete clusterf*ck to maintain. Got any ideas?”.

Phil looked at it for about 5 seconds and said “why dont you just use a regex to split the text by the known words?”. Duh. A regex is a much better way to get text that cuts across the DOM, especially when I have known markers I’m separating the text by. Where my original algorithm took 30 painful minutes to code, the regex call took about 5 to code up, and it worked the first time. Which brings me to my second lesson of the day:

(2) Less is more

If I can only code what is absolutely necessary, and no more, my code is by definition elegant. I find that TDD helps me stay away from building code cathedrals. Some people do this naturally. I find that it takes a lot of discipline to not overdesign. It also takes a village:

(3) A second (and sometimes third) pair of eyes saves hours

I used to believe that coding was a solitary, metaphysical exercise, and I was like some kind of enlightened being dispensing my wisdom in elegant logic capsules. My code as a result was ornate, baroque, and buggy. I firmly believe that the best code I have written in my life was the code that received frequent reviews, or at least code that was written in close collaboration with others. A lot of pretty screwed up design can be avoided just by having someone to talk to. Later on in the day I was trying to track down a bizarre error centered around a module variable that the interpreter said was uninitialized. I stared it the screen for a while, then actually listened to the little voice inside my head that said “go ask someone else to take a look”. Alex came along and in short order we found a place where I had a circular ‘require’ statement, i.e. I required class_1 from class_2, and required file_2 from file_1. The issue never manifested in unit testing because class_2 and class_1 where tested in isolation. But in a unit test of class_3, which required both class_2 and class_1, the bizarre error showed up. Just having someone there to bounce ideas off of accelerated the debugging process and got me unstuck.

(3) despite my rabid assertions to the contrary, I’m becoming a Ruby Fanboy.

I am currently in rapture with the active support gem, especially the CoreExtensions::NumericSupport::Time class, because it saved a ton of time for me today. I had to write a quick script to allow someone from another team to request a resource from our web service and save it to file. The download part was two lines of code, but they needed to request that the resource be generated from all new content that had been ingested X minutes/hours/days ago. No sweat, right? Not so about those boundary conditions? For instance, if I request a file from 3 minutes ago and it’s 12:01 AM, that flips the hour and the day back.

Yes, there is a very elegant recursive way to do this, and I will code it out when I am resting on the beach after a killer dawn patrol, but like they say “time is money”, and today I had neither. Taking heed of the lessons I had already learned in the day, I asked around, and was pointed to the active support documentation on The Time class extends Fixnum with methods like minutes(), seconds(), etc that return Fixnums. It also extends Fixnum with the ago() method, which allows you to write code like this:

y = 10.minutes.ago

Talk about coding by intention!