QA in an open source community

As my internship with Mozilla Messaging draws to a close in a couple of days’ time, I will only be sporadically online for the next few weeks due to vacations prior to my resumption of school in mid-August. This presents a QA shortage for Thunderbird, especially since the code freeze date for Shredder Alpha 2 is still scheduled for July 8, 2008. Wayne Mery, known on IRC as wsmwk, has kindly stepped up to volunteer his time to lead the QA efforts for Shredder Alpha 2. David Ascher has also blogged about the incredible aspects and virtually "impossible" job of a QA lead for Thunderbird.

Perhaps I should start off by stating the steps taken to improve the QA process of Thunderbird:

  • - Over the months, the documentation on the Mozilla Wiki was overhauled and now presents a more comprehensive and structured approach in an effort to be much clearer and more precise to the reader, compared to the old layout.
  • - Bugdays were implemented with invaluable input from various community members and developers, and together we worked towards a significant drop in the number of unconfirmed bugs.
  • - We reached out to the community through many channels, including but not limited to newsgroups, Bugzilla, email, IRC etc.
  • - It is imperative that we continue these efforts in the months ahead, as anybody who steps up will no doubt prove his or her worth in one way or another, be it as sophisticated as providing a patch for a new functionality or as simple as triaging / verifying bugs.

We would like to maintain a certain level of quality even for an alpha release, and certainly having a crash on startup on a platform that is officially supported but never had any QA to detect the crash, is definitely not an ideal scenario. The challenge here is as such, how can we maintain and guarantee the quality of an open source email client that is widely used by thousands of people around the world, leveraging almost purely on the contributions of its community in all parts of the world?

Such previously mentioned ways are merely the start of a gradual QA process, something to push a train with rusty wheels along a track that hasn’t been maintained for a long time. A comprehensive QA for a release such as an alpha or beta, requires slightly more efforts. Once the build team gives the word "Go!" from the moment the builds are completed, these efforts run parallel to the ones already listed above:

  • - Smoketests on all platforms, as seen in the test plan here and which are based on Litmus, are absolutely vital to being able to have community input on the alpha release. It has to be made sure that all supported platforms are tested.
  • - The estimated timeframe given the manpower / schedule constraints within a reasonable amount of time. e.g. I spent a couple of days planning out the way I would test the Alpha, and approximately three days to execute it, though with much of the hardware and time already prepared and accounted for in advance.
  • - Much time was also spent on gathering input and feedback on the plan itself, i.e. whether it was comprehensive enough, or whether it ensured that all platforms / functionalities were adequately covered.
  • - Previous QA efforts, i.e. before I hopped on, for Thunderbird were small, and I adapted the QA plan for testing Thunderbird from Firefox’s plan. It wasn’t exactly similar in that one was a browser and the other a email client. Some test approaches were similar, such as whether there was a crash on startup / shutdown, while others were obviously different, in that a email client requires testing of the transfer messages to and from an IMAP account much more than the proper rendering of a HTML document with special edge-case and rarely used markup code.
  • - Every bug found is reported in Bugzilla with the corresponding screenshot and listed on the testplan for archival purposes. While every effort is made to ensure that there are no duplicates filed, having dupes is definitely better than having a bug that was not filed at all due to someone mistaking it for a dupe. Duplicates can also be reopened in the future, as deemed appropriate.
  • - Having a testday for the release and timing QA efforts around it are crucial. Regular triagers encourage new folks to test out the soon-to-be released product and help out in the IRC channels, such as #testday, whenever they can. There will always be a couple of people interested in contributing their valuable time in testing out an open source product.
  • - Obviously, ensuring that the QA folks have comprehensively executed the testplan is a very important step in ensuring the level of quality in the product.

The challenge goes into ensuring that at least all of the above needs are met, in order to adequately ensure a reputable QA sign-off for a product, with much assortment of testers around the world. The following ways are suggested:

  • - Continue the smoketests and assign one platform of each smoketest to a reputable and trustworthy community member / developer. He / she must ensure that the smoketests are executed in full, and the corresponding bugs reported as such.
  • - Plan the entire process based on the above assumption that each smoketester would take a certain number of hours to execute the smoketest, and come up with an estimated timeframe of completion, on the condition that no blocker bugs are found. The overall lead should then consolidate the testing results, and after discussion with the developers, the drivers will then decide whether the tested build is ready for release.
  • - Guide new folks who are ad-hoc testing the release using the IRC channels.
  • - All these with the time and efforts of people in all parts of the world, different timezones and different languages, with different needs and different wants for testing of an open source product.

It is a collaborative QA effort for Thunderbird. In the event that blocker bugs are found and brought to the relevant developer’s attention, the process is stopped, new builds respun, and the process resumed from the beginning. There is no issue about who gets the blame for delaying the release here, quality is what sets the bar for a Mozilla release. We will do what is necessary to ensure that the quality bar remains as appropriate for an alpha / beta, and especially so for a final release in the future.