Survey results out! – part 1

Written on 18 Nov 07, and last updated on 23 Nov 07. (3,000+ words and counting)

Edits till 21 Nov 07: Completed main text.

Edit on 22 Nov 07 till 24 Nov 07: Added Condorcet / other voting algorithms using OpenSTV to determine most preferred choices as suggested by an anonymous commenter. (Two tries; original calculations were borked)

Edit on 10 Aug 08: Migrate images over to DreamHost, update links.

What is this?

The Rumbling Edge Thunderbird 2 survey (version 1) ran for three weeks, from 29 October 2007 till 18 November 2007, and there were just under seven hundred complete responses, a thousand if partial ones are to be included. A similar number (1,022 to be exact) were abandoned. I made use of SurveyGizmo to gather the results from the questions which numbered ten in total though some were optional.

Where do the respondents come from?

Demographics
Demographics of respondents (click for a full size version)

The above screenshot capture was taken from SurveyGizmo, just after I closed the survey. As you can see from the image, most respondents (and hence people who know about, are using or have used Thunderbird) are from North America and Europe, though there are others from South America, India, East Asia and Oceania. You can even see responses from Iceland.

The raw reports compiled within SurveyGizmo have been saved into PDF format, both around 50 pages with rounded percentages. (Without partial responses / With partial responses)

Which report are you referring to?

I am referring to the report without partial responses, as I feel that the information is more likely to be more accurate than if they were included. Anyone is free to come up with their own analysis on the questions on any report though; I’d be glad to link to them from here. I activated randomization of options within the questions so as to reduce the effect of "first option preference" as well. There was no limit on the number of times people could take the survey, though the number who did so is considered to be negligible.

What is the purpose of the survey?

I have been running The Rumbling Edge for more than three years, and with the pending formation of MailCo, I thought it would be a good idea to gather ideas for improvement from the community, especially regarding the future development of Thunderbird 3. No, I am not doing this under the Mozilla Community Surveys Project, it is just something that I do in my free time, so please take the analysis (and maybe the results as well) with an overdose of sodium chloride =) .

Enough of all this. I want to read the analysis now!

Question 1: Please choose the level of expertise you are at for Thunderbird.

Q1
Question 1 results (click for larger image)
 
Number
Percentage
Power user
413
60%
Average
216
31%
Developer / tester
53
8%
Beginner
6
1%
Total
688
100%

The sample size of 688 were mostly taken from readers of The Rumbling Edge, which may not necessarily be a true reflection of the views of millions of Thunderbird users in the world. This question merely shows the breakdown of the survey respondents, who are mostly power users and average users. Whether the percentages apply to the whole Thunderbird population is anyone’s guess at this point in time.

Question 2: How satisfied are you with the current state of Thunderbird? (If unsure, just select "Neutral")

Q2
Question 2 results (click for larger image)
 
Extremely dissatisfied
Quite dissatisfied Neutral Quite satisfied Extremely satisfied
Total
Adherence to open standards
0% (2)
1% (8)
14% (94)
51% (353)
34% (231)
688
Stability
1% (4)
5% (33)
11% (73)
51% (352)
33% (226)
User interface
2% (11)
10% (71)
19% (133)
59% (403)
10% (70)
Level of functionality
1% (5)
10% (70)
24% (163)
57% (395)
8% (55)
General state of Thunderbird
1% (6)
11% (78)
23% (155)
57% (395)
8% (54)
Total
28
260
618
1,898
636
 

Sorting according the satisfaction amongst the respondents, it can quite clearly be seen that Thunderbird is popular due to its adherence to open standards and stability (~85% satisfaction for both). People were not as sure about the user interface (~70% satisfaction), while a larger proportion of users were dissatisfied with the level of functionality and the general state of Thunderbird. Though not a large percentage (just over 10% dissatisfied in each), these areas could be where Thunderbird are in need of improvement to gain more users, or even to retain the current ones.

Question 3: Currently, where do you go to obtain Thunderbird support? Please tick those that apply.

Q3
Question 3 results (click for larger image)
 
Number
Percentage
MozillaZine forums
383
56%
MozillaZine knowledgebase
328
48%
Google *
138
20%
Mozilla newsgroups
128
19%
Others (open-ended)
51
7%
No support needed *
45
7%
IRC via irc.mozilla.org
41
6%
Mozilla feedback form (Hendrix)
14
2%
Bugzilla *
10
1%

Note: * denotes notable answers that were entered in the "others" field.

Most headed to MozillaZine for Thunderbird issues, though a sizable proportion headed to search engines, such as Google, and Mozilla newsgroups. IRC and other places, such as thunderbird-mail.de and geckozone forums made up ~7% each, as did respondents who did not require support. The Hendrix feedback form is virtually unused, while Bugzilla may only appeal to developers, so they bring up the rear.

An underlying indication could be the lack of proper Thunderbird documentation along the lines of an official knowledgebase similar to that for Firefox that is currently in beta form. Negligible numbers made use of the current help documentation found at Thunderbird Help, which can be accessed via the help function within Thunderbird.

Question 4: How do you usually keep track of the development of Thunderbird? (Optional) Please tick those that apply.

Q4
Question 4 results (click for larger image)
 
Number
Percentage
Planet Mozilla blogs
402
58%
MozillaZine forums
257
37%
Bugzilla
140
20%
The Rumbling Edge *
90
13%
Others (open-ended)
78
11%
Slashdot *
32
5%
IRC via irc.mozilla.org
23
3%
heise.de *
9
1%
Digg *
7
1%
Bonsai *
4
1%

Note: * denotes notable answers that were entered in the "others" field.

A vast majority proceed to Planet Mozilla for their dose of Thunderbird development news, of which The Rumbling Edge is a part of (~70% if combined). MozillaZine forums come in runners-up, followed by Bugzilla. IRC accounts for only 3%, while news sites, such as Slashdot and heise.de, and other sites make up the last ~20%.

Question 5: Rank the following suggested improvements to functionalities that already exist in Thunderbird.

Options to be ranked: (removed raw average scores since they may not tell the whole tale)

  • Do not preview spam
  • Icon refresh throughout the default UI
  • Improved global search (across all types of accounts)
  • Improved spam filtering
  • More localizations
  • More useful homepage
  • Other default webmail settings, besides Gmail
  • Others

The OpenSTV .blt file I used is available here. The various options were used with their default settings.

 

a) Instant runoff voting algorithm ("If you are electing one person and simplicity is important, then we recommend instant runoff voting. Instant runoff voting is easy to explain and gives good results." – from OpenSTV FAQ)

Output from OpenSTV:

Election: Rank the following suggested improvements to functionalities that already exist in Thunderbird.
Method: IRV
Number of Ballots: 689
8 candidates running for 1 seats.

Round
Improved global search
Do not preview spam
Improved spam filtering
Icon refresh throughout default UI
Others
More useful homepage
More localizations
Other default webmail settings
Exhausted
1
163
114
80
71
79
62
65
55
0
2
171
124
90
75
88
67
74
 
0
3
191
139
95
82
95
 
87
 
0
4
219
154
105
 
112
 
99
 
0
5
251
181
128
 
129
     
0
6
295
227
   
167
     
0
7
375
314
           
0

Round 1: Count of first place rankings.
Round 2: Eliminating candidate Other default webmail settings, besides Gmail.
Round 3: Eliminating candidate More useful homepage.
Round 4: Eliminating candidate Icon refresh throughout the default UI.
Round 5: Eliminating candidate More localizations.
Round 6: Eliminating candidate Improved spam filtering.
Round 7: Eliminating candidate Others (Please suggest at last question of the survey).
Candidate Improved global search (across all types of accounts) is elected.

Winner is Improved global search (across all types of accounts).

b) Condorcet voting algorithm ("If you are electing one person and simplicity is not important, then we recommend Condorcet voting. Most people agree that Condorcet is the best method for electing one person, but it is more difficult to explain." – from OpenSTV FAQ)

Output from OpenSTV:

Election: Rank the following suggested improvements to functionalities that already exist in Thunderbird.
Method: Condorcet
Completion Method: Schwartz Sequential Dropping
Number of Ballots: 689
8 candidates running for 1 seat.

Pairwise Comparison Matrix:

 
Improved global search
Do not preview spam
Improved spam filtering
Icon refresh throughout default UI
Others
More useful homepage
More localizations
Other default webmail settings
Improved global search
0
375
423
440
433
439
444
466
Do not preview spam
314
0
411
437
419
438
454
459
Improved spam filtering
266
278
0
408
414
427
422
448
Icon refresh throughout default UI
249
252
281
0
398
419
427
448
Others
256
270
275
291
0
413
432
455
More useful homepage
250
251
262
270
276
0
410
446
More localizations
245
235
267
262
257
279
0
443
Other default webmail settings
223
230
241
241
234
243
246
0

Smith Set: Improved global search (across all types of accounts)
No completion necessary since the Smith set has just one candidate.

Winner is Improved global search (across all types of accounts).

c) Scottish STV voting algorithm ("If you are electing multiple people and simplicity is important, then we recommend Scottish STV, as it is the simplest of all the STV methods." – from OpenSTV FAQ)

Output from OpenSTV:

Election: Rank the following suggested improvements to functionalities that already exist in Thunderbird.
Method: Scottish STV
Number of Ballots: 689
Threshold Name: Droop Static Whole
Threshold Value: 173.0
Delayed Transfer of Surplus: Not Enabled.
Batch Elimination: None
8 candidates running for 3 seats.

Round
Improved global search
Do not preview spam
Improved spam filtering
Icon refresh throughout default UI
Others
More useful homepage
More localizations
Other default webmail settings
Exhausted
Surplus
1
163
114
80
71
79
62
65
55
0
0
2
171
124
90
75
88
67
74
 
0
0
3
191
139
95
82
95
 
87
 
0
18
4
173
145.6
98.8
85.2
97.2
 
89.3
 
0
0
5
173
171.7
117.6
 
121.2
 
105.5
 
0
0
6
173
216.7
152.6
 
146.8
     
0
43.7
7
173
173
182.8
 
160.2
     
0
9.8

Round 1: Count of first place rankings.
Round 2: Eliminating candidate Other default webmail settings, besides Gmail.
Round 3: Eliminating candidate More useful homepage.
Candidate Improved global search (across all types of accounts) is elected.
Round 4: Transferring surplus votes from candidate Improved global search (across all types of accounts).
Round 5: Eliminating candidate Icon refresh throughout the default UI.
Round 6: Eliminating candidate More localizations.
Candidate Do not preview spam is elected.
Round 7: Transferring surplus votes from candidate Do not preview spam.
Candidate Improved spam filtering is elected.

Winners are Improved global search (across all types of accounts), Do not preview spam, Improved spam filtering.

d) Meek STV voting algorithm ("If you are electing multiple people and simplicity is not important, then we recommend Meek STV. Most people agree that Meek STV is the best variant of STV, but it can only be implemented with a computer program." – from OpenSTV FAQ)

Output from OpenSTV:

Election: Rank the following suggested improvements to functionalities that already exist in Thunderbird.
Method: Meek STV
Number of Ballots: 689
Threshold Name: Droop Dynamic Fractional
8 candidates running for 3 seats.

Round
Improved global search
Do not preview spam
Improved spam filtering
Icon refresh throughout default UI
Others
More useful homepage
More localizations
Other default webmail settings
Exhausted
Surplus
Threshold
1
163
114
80
71
79
62
65
55
0
0
172.3
2
171
124
90
75
88
67
74
 
0
0
172.3
3
191
139
95
82
95
 
87
 
0
18.7
172.3
4
172.3
145.9
98.9
85.3
97.3
 
89.4
 
0
0
172.3
5
197.5
163
110.6
 
115.9
 
101.9
 
0
25.3
172.3
6
172.3
173.6
117.2
 
120.5
 
105.4
 
0
1.4
172.3
7
197.9
204.8
145.4
 
140.9
     
0
58.2
172.3
8
180.8
182.4
171.4
 
154.5
     
0
18.6
172.3
9
174.6
175.3
180.1
 
159
     
0
13.3
172.3

Round 1: Count of first place rankings.
Round 2: Eliminating candidate Other default webmail settings, besides Gmail.
Round 3: Eliminating candidate More useful homepage.
Candidate Improved global search (across all types of accounts) is elected.
Round 4: Transferring surplus votes.
Round 5: Eliminating candidate Icon refresh throughout the default UI.
Round 6: Transferring surplus votes.
Candidate Do not preview spam is elected.
Round 7: Transferring surplus votes.
Eliminating candidate More localizations.
Round 8: Transferring surplus votes.
Round 9: Transferring surplus votes.
Candidate Improved spam filtering is elected.

Winners are Improved global search (across all types of accounts), Do not preview spam, Improved spam filtering.

 

Top 3 among all voting methods: 1 – Improved global search, 2 – Do not preview spam, 3 – Improved spam filtering.

Here, we see that improved global search and improved spam functionalities are the most requested improvements. Having more default webmail settings, icon refreshes and a more useful homepage are statistically tied as "next-in-line". If the raw report data is to be scrutinized, there are 123 (18%) respondents voting for "Others" as the top priority; there are other improvements that they feel are more important but which are not listed here. Finally, including more localizations seems to be the least priority, but then again, those who make frequent use of localized builds seldom come around The Rumbling Edge anyway, since it is in English.

Continued in part two..