Written on 18 Nov 07, and last updated on 23 Nov 07. (3,000+ words and counting)
Edits till 21 Nov 07: Completed main text.
Edit on 22 Nov 07 till 24 Nov 07: Added Condorcet / other voting algorithms using OpenSTV to determine most preferred choices as suggested by an anonymous commenter. (Two tries; original calculations were borked)
Edit on 10 Aug 08: Migrate images over to DreamHost, update links.
What is this?
The Rumbling Edge Thunderbird 2 survey (version 1) ran for three weeks, from 29 October 2007 till 18 November 2007, and there were just under seven hundred complete responses, a thousand if partial ones are to be included. A similar number (1,022 to be exact) were abandoned. I made use of SurveyGizmo to gather the results from the questions which numbered ten in total though some were optional.
Where do the respondents come from?

Demographics of respondents (click for a full size version)
The above screenshot capture was taken from SurveyGizmo, just after I closed the survey. As you can see from the image, most respondents (and hence people who know about, are using or have used Thunderbird) are from North America and Europe, though there are others from South America, India, East Asia and Oceania. You can even see responses from Iceland.
The raw reports compiled within SurveyGizmo have been saved into PDF format, both around 50 pages with rounded percentages. (Without partial responses / With partial responses)
Which report are you referring to?
I am referring to the report without partial responses, as I feel that the information is more likely to be more accurate than if they were included. Anyone is free to come up with their own analysis on the questions on any report though; I’d be glad to link to them from here. I activated randomization of options within the questions so as to reduce the effect of "first option preference" as well. There was no limit on the number of times people could take the survey, though the number who did so is considered to be negligible.
What is the purpose of the survey?
I have been running The Rumbling Edge for more than three years, and with the pending formation of MailCo, I thought it would be a good idea to gather ideas for improvement from the community, especially regarding the future development of Thunderbird 3. No, I am not doing this under the Mozilla Community Surveys Project, it is just something that I do in my free time, so please take the analysis (and maybe the results as well) with an overdose of sodium chloride =) .
Enough of all this. I want to read the analysis now!
Question 1: Please choose the level of expertise you are at for Thunderbird.
Question 1 results (click for larger image)
|
Number |
Percentage |
Power user |
413 |
60% |
Average |
216 |
31% |
Developer / tester |
53 |
8% |
Beginner |
6 |
1% |
Total |
688 |
100% |
The sample size of 688 were mostly taken from readers of The Rumbling Edge, which may not necessarily be a true reflection of the views of millions of Thunderbird users in the world. This question merely shows the breakdown of the survey respondents, who are mostly power users and average users. Whether the percentages apply to the whole Thunderbird population is anyone’s guess at this point in time.
Question 2: How satisfied are you with the current state of Thunderbird? (If unsure, just select "Neutral")
Question 2 results (click for larger image)
|
Extremely dissatisfied |
Quite dissatisfied |
Neutral |
Quite satisfied |
Extremely satisfied |
Total |
Adherence to open standards |
0% (2) |
1% (8) |
14% (94) |
51% (353) |
34% (231) |
688 |
Stability |
1% (4) |
5% (33) |
11% (73) |
51% (352) |
33% (226) |
User interface |
2% (11) |
10% (71) |
19% (133) |
59% (403) |
10% (70) |
Level of functionality |
1% (5) |
10% (70) |
24% (163) |
57% (395) |
8% (55) |
General state of Thunderbird |
1% (6) |
11% (78) |
23% (155) |
57% (395) |
8% (54) |
Total |
28 |
260 |
618 |
1,898 |
636 |
|
Sorting according the satisfaction amongst the respondents, it can quite clearly be seen that Thunderbird is popular due to its adherence to open standards and stability (~85% satisfaction for both). People were not as sure about the user interface (~70% satisfaction), while a larger proportion of users were dissatisfied with the level of functionality and the general state of Thunderbird. Though not a large percentage (just over 10% dissatisfied in each), these areas could be where Thunderbird are in need of improvement to gain more users, or even to retain the current ones.
Question 3: Currently, where do you go to obtain Thunderbird support? Please tick those that apply.
Question 3 results (click for larger image)
|
Number |
Percentage |
MozillaZine forums |
383 |
56% |
MozillaZine knowledgebase |
328 |
48% |
Google * |
138 |
20% |
Mozilla newsgroups |
128 |
19% |
Others (open-ended) |
51 |
7% |
No support needed * |
45 |
7% |
IRC via irc.mozilla.org |
41 |
6% |
Mozilla feedback form (Hendrix) |
14 |
2% |
Bugzilla * |
10 |
1% |
Note: * denotes notable answers that were entered in the "others" field.
Most headed to MozillaZine for Thunderbird issues, though a sizable proportion headed to search engines, such as Google, and Mozilla newsgroups. IRC and other places, such as thunderbird-mail.de and geckozone forums made up ~7% each, as did respondents who did not require support. The Hendrix feedback form is virtually unused, while Bugzilla may only appeal to developers, so they bring up the rear.
An underlying indication could be the lack of proper Thunderbird documentation along the lines of an official knowledgebase similar to that for Firefox that is currently in beta form. Negligible numbers made use of the current help documentation found at Thunderbird Help, which can be accessed via the help function within Thunderbird.
Question 4: How do you usually keep track of the development of Thunderbird? (Optional) Please tick those that apply.
Question 4 results (click for larger image)
|
Number |
Percentage |
Planet Mozilla blogs |
402 |
58% |
MozillaZine forums |
257 |
37% |
Bugzilla |
140 |
20% |
The Rumbling Edge * |
90 |
13% |
Others (open-ended) |
78 |
11% |
Slashdot * |
32 |
5% |
IRC via irc.mozilla.org |
23 |
3% |
heise.de * |
9 |
1% |
Digg * |
7 |
1% |
Bonsai * |
4 |
1% |
Note: * denotes notable answers that were entered in the "others" field.
A vast majority proceed to Planet Mozilla for their dose of Thunderbird development news, of which The Rumbling Edge is a part of (~70% if combined). MozillaZine forums come in runners-up, followed by Bugzilla. IRC accounts for only 3%, while news sites, such as Slashdot and heise.de, and other sites make up the last ~20%.
Question 5: Rank the following suggested improvements to functionalities that already exist in Thunderbird.
Options to be ranked: (removed raw average scores since they may not tell the whole tale)
- Do not preview spam
- Icon refresh throughout the default UI
- Improved global search (across all types of accounts)
- Improved spam filtering
- More localizations
- More useful homepage
- Other default webmail settings, besides Gmail
- Others
The OpenSTV .blt file I used is available here. The various options were used with their default settings.
a) Instant runoff voting algorithm ("If you are electing one person and simplicity is important, then we recommend instant runoff voting. Instant runoff voting is easy to explain and gives good results." – from OpenSTV FAQ)
Output from OpenSTV:
Election: Rank the following suggested improvements to functionalities that already exist in Thunderbird.
Method: IRV
Number of Ballots: 689
8 candidates running for 1 seats.
Round |
Improved global search |
Do not preview spam |
Improved spam filtering |
Icon refresh throughout default UI |
Others |
More useful homepage |
More localizations |
Other default webmail settings |
Exhausted |
1 |
163 |
114 |
80 |
71 |
79 |
62 |
65 |
55 |
0 |
2 |
171 |
124 |
90 |
75 |
88 |
67 |
74 |
|
0 |
3 |
191 |
139 |
95 |
82 |
95 |
|
87 |
|
0 |
4 |
219 |
154 |
105 |
|
112 |
|
99 |
|
0 |
5 |
251 |
181 |
128 |
|
129 |
|
|
|
0 |
6 |
295 |
227 |
|
|
167 |
|
|
|
0 |
7 |
375 |
314 |
|
|
|
|
|
|
0 |
Round 1: Count of first place rankings.
Round 2: Eliminating candidate Other default webmail settings, besides Gmail.
Round 3: Eliminating candidate More useful homepage.
Round 4: Eliminating candidate Icon refresh throughout the default UI.
Round 5: Eliminating candidate More localizations.
Round 6: Eliminating candidate Improved spam filtering.
Round 7: Eliminating candidate Others (Please suggest at last question of the survey).
Candidate Improved global search (across all types of accounts) is elected.
Winner is Improved global search (across all types of accounts).
b) Condorcet voting algorithm ("If you are electing one person and simplicity is not important, then we recommend Condorcet voting. Most people agree that Condorcet is the best method for electing one person, but it is more difficult to explain." – from OpenSTV FAQ)
Output from OpenSTV:
Election: Rank the following suggested improvements to functionalities that already exist in Thunderbird.
Method: Condorcet
Completion Method: Schwartz Sequential Dropping
Number of Ballots: 689
8 candidates running for 1 seat.
Pairwise Comparison Matrix:
|
Improved global search |
Do not preview spam |
Improved spam filtering |
Icon refresh throughout default UI |
Others |
More useful homepage |
More localizations |
Other default webmail settings |
Improved global search |
0 |
375 |
423 |
440 |
433 |
439 |
444 |
466 |
Do not preview spam |
314 |
0 |
411 |
437 |
419 |
438 |
454 |
459 |
Improved spam filtering |
266 |
278 |
0 |
408 |
414 |
427 |
422 |
448 |
Icon refresh throughout default UI |
249 |
252 |
281 |
0 |
398 |
419 |
427 |
448 |
Others |
256 |
270 |
275 |
291 |
0 |
413 |
432 |
455 |
More useful homepage |
250 |
251 |
262 |
270 |
276 |
0 |
410 |
446 |
More localizations |
245 |
235 |
267 |
262 |
257 |
279 |
0 |
443 |
Other default webmail settings |
223 |
230 |
241 |
241 |
234 |
243 |
246 |
0 |
Smith Set: Improved global search (across all types of accounts)
No completion necessary since the Smith set has just one candidate.
Winner is Improved global search (across all types of accounts).
c) Scottish STV voting algorithm ("If you are electing multiple people and simplicity is important, then we recommend Scottish STV, as it is the simplest of all the STV methods." – from OpenSTV FAQ)
Output from OpenSTV:
Election: Rank the following suggested improvements to functionalities that already exist in Thunderbird.
Method: Scottish STV
Number of Ballots: 689
Threshold Name: Droop Static Whole
Threshold Value: 173.0
Delayed Transfer of Surplus: Not Enabled.
Batch Elimination: None
8 candidates running for 3 seats.
Round |
Improved global search |
Do not preview spam |
Improved spam filtering |
Icon refresh throughout default UI |
Others |
More useful homepage |
More localizations |
Other default webmail settings |
Exhausted |
Surplus |
1 |
163 |
114 |
80 |
71 |
79 |
62 |
65 |
55 |
0 |
0 |
2 |
171 |
124 |
90 |
75 |
88 |
67 |
74 |
|
0 |
0 |
3 |
191 |
139 |
95 |
82 |
95 |
|
87 |
|
0 |
18 |
4 |
173 |
145.6 |
98.8 |
85.2 |
97.2 |
|
89.3 |
|
0 |
0 |
5 |
173 |
171.7 |
117.6 |
|
121.2 |
|
105.5 |
|
0 |
0 |
6 |
173 |
216.7 |
152.6 |
|
146.8 |
|
|
|
0 |
43.7 |
7 |
173 |
173 |
182.8 |
|
160.2 |
|
|
|
0 |
9.8 |
Round 1: Count of first place rankings.
Round 2: Eliminating candidate Other default webmail settings, besides Gmail.
Round 3: Eliminating candidate More useful homepage.
Candidate Improved global search (across all types of accounts) is elected.
Round 4: Transferring surplus votes from candidate Improved global search (across all types of accounts).
Round 5: Eliminating candidate Icon refresh throughout the default UI.
Round 6: Eliminating candidate More localizations.
Candidate Do not preview spam is elected.
Round 7: Transferring surplus votes from candidate Do not preview spam.
Candidate Improved spam filtering is elected.
Winners are Improved global search (across all types of accounts), Do not preview spam, Improved spam filtering.
d) Meek STV voting algorithm ("If you are electing multiple people and simplicity is not important, then we recommend Meek STV. Most people agree that Meek STV is the best variant of STV, but it can only be implemented with a computer program." – from OpenSTV FAQ)
Output from OpenSTV:
Election: Rank the following suggested improvements to functionalities that already exist in Thunderbird.
Method: Meek STV
Number of Ballots: 689
Threshold Name: Droop Dynamic Fractional
8 candidates running for 3 seats.
Round |
Improved global search |
Do not preview spam |
Improved spam filtering |
Icon refresh throughout default UI |
Others |
More useful homepage |
More localizations |
Other default webmail settings |
Exhausted |
Surplus |
Threshold |
1 |
163 |
114 |
80 |
71 |
79 |
62 |
65 |
55 |
0 |
0 |
172.3 |
2 |
171 |
124 |
90 |
75 |
88 |
67 |
74 |
|
0 |
0 |
172.3 |
3 |
191 |
139 |
95 |
82 |
95 |
|
87 |
|
0 |
18.7 |
172.3 |
4 |
172.3 |
145.9 |
98.9 |
85.3 |
97.3 |
|
89.4 |
|
0 |
0 |
172.3 |
5 |
197.5 |
163 |
110.6 |
|
115.9 |
|
101.9 |
|
0 |
25.3 |
172.3 |
6 |
172.3 |
173.6 |
117.2 |
|
120.5 |
|
105.4 |
|
0 |
1.4 |
172.3 |
7 |
197.9 |
204.8 |
145.4 |
|
140.9 |
|
|
|
0 |
58.2 |
172.3 |
8 |
180.8 |
182.4 |
171.4 |
|
154.5 |
|
|
|
0 |
18.6 |
172.3 |
9 |
174.6 |
175.3 |
180.1 |
|
159 |
|
|
|
0 |
13.3 |
172.3 |
Round 1: Count of first place rankings.
Round 2: Eliminating candidate Other default webmail settings, besides Gmail.
Round 3: Eliminating candidate More useful homepage.
Candidate Improved global search (across all types of accounts) is elected.
Round 4: Transferring surplus votes.
Round 5: Eliminating candidate Icon refresh throughout the default UI.
Round 6: Transferring surplus votes.
Candidate Do not preview spam is elected.
Round 7: Transferring surplus votes.
Eliminating candidate More localizations.
Round 8: Transferring surplus votes.
Round 9: Transferring surplus votes.
Candidate Improved spam filtering is elected.
Winners are Improved global search (across all types of accounts), Do not preview spam, Improved spam filtering.
Top 3 among all voting methods: 1 – Improved global search, 2 – Do not preview spam, 3 – Improved spam filtering.
Here, we see that improved global search and improved spam functionalities are the most requested improvements. Having more default webmail settings, icon refreshes and a more useful homepage are statistically tied as "next-in-line". If the raw report data is to be scrutinized, there are 123 (18%) respondents voting for "Others" as the top priority; there are other improvements that they feel are more important but which are not listed here. Finally, including more localizations seems to be the least priority, but then again, those who make frequent use of localized builds seldom come around The Rumbling Edge anyway, since it is in English.
Continued in part two..