Howto: Find Regression Windows Through Manual Binary Search

Thunderbird 3 Beta 2 has had its code frozen for a couple of days, and Standard8 discovered a crasher during this QA period. (Bug 479206) The natural way to QA this was to test each previous release to see whether the bug was occurring… (with a fresh profile every time, of course)

… Thunderbird 3 Beta 1 has this bug …

… Shredder Alpha 3 has this bug …

… Shredder Alpha 2 has this bug …

… Shredder Alpha 1 has this bug …

… Thunderbird 2.0.0.19 does not have this bug!

Hmm, this still leaves quite a large regression window, from the point of branching of 1.8 and 1.9, till the current time…

I went back further, trying trunk builds from Mozilla FTP using nightlies every year (backwards) …

Tried the build from 2007-12-01 (Shredder Alpha 1 was released in the early part of 2008)… No luck.

2006-12-01 does not have this bug. Now I’m getting somewhere…

2007-06-30 has this bug. Ah, a 6 month regression window, finally… but still a tad too large.

(The key point is to take intervals that divide themselves by half everytime – from 1 year, to 6 months, and now I’m going down to 3 months…)

2007-03-31 does not have this issue! More happy thoughts…

Nor does 2007-05-15!

Nor 2007-06-07! Really close now…

2007-06-17 has this bug! And I continue to chop up the regression window…

… chop chop …

And I end up with:

2007-06-11 does not crash.

2007-06-12 does crash!

So bingo! By making use of this technique, you have now found a suitably short regression window!

Some things you can further do to track this down – searching in hg annotate (for newer builds), or for my example above, Bonsai and inputting the date and time in a slightly large range (I used 2007-06-11 02:00:00 as the start time and 2007-06-12 04:00:00 as the end time, as Thunderbird nightlies start building from 03:00:00 PDT or PST). Search for probably-guilty checkins then comment in the bug. CC’ing the developer who wrote that initial patch does help too.

Developers from that point onwards will most likely be able to jump in quickly to debug the issue, and this is where QA usually finishes its work.

(Endnote: I was usually most bogged up by Mozilla FTP nightly download times – 10-15 minutes every time – but YMMV)