Use and Trust in Official Statistics Survey A Colmar Brunton Report July 2015

© Colmar Brunton 2015

1

Background, objectives and method

This report details the survey findings from a survey of potential users of ‘official’ statistics to understand the extent to which they use them and their perceptions of the statistics. Statistics NZ commissioned Colmar Brunton to undertake a survey into how different audiences perceive official statistics, including the extent to which they trust them. The survey was conducted online. Two email reminders were used to maximise the response rate. The following table details the three audiences including the sample frame used, the number of interviews achieved and the response rate. Audience

Sample source

Sampling details

No. of interviews

Response rate

Central government workers

Statistics NZ

The sample included members of one of the three following groups: the Official Statistics System Reference Group, the Policy Network and the Planners Network. The list included 584 active email addresses.

182

31%

Local government workers

Society of Local Government Managers (SOLGM)

The sample was mainly comprised of strategic and corporate planners, who are involved in long term planning and RMA planning. Due to restrictions on data sharing, SOLGM sent an open survey link to its relevant members on behalf of Colmar Brunton. The sample included 257 members with representation from every local authority except Selwyn District Council and the Chatham Islands.

47

18%

Businesses

Colmar Brunton online panel

The sample frame includes over 20,000 decision makers across businesses in New Zealand. We drew the sample to be nationally representative by industry group and region (using data published by Statistics New Zealand). Larger businesses with 50+ employees were over-sampled to enable analysis of this group. Quotas were set by employee size within industry group. The data was weighted to ensure the sample was nationally representative and took account of the oversampling of larger businesses.

329

35%

© Colmar Brunton 2015

2

Analysis and reading this report

Findings for all three audiences have been combined to present an overall score. The business sample was weighted to ensure it was representative of the national profile, and consequently we weighted each central and local government respondent to a weight of ‘1’ to ensure their responses are included in the overall sample. The effective sample size (which takes account of the impact of the weighting) for respondents overall is 381, with a margin of error of +/- 5%. The report presents the findings for respondents overall, as well as the three audiences. The charts and commentary highlight where differences in the findings between the three audiences and the respondents overall are statistically significant (at the 95% confidence level). This means we can be 95% certain that the observed difference occurs in the population, rather than being a consequence of surveying a sample of clients. Sub-group analysis has also been conducted on other key demographic information and attitudinal information. Differences of note, which are statistically significant are included in the report commentary. It should be noted that sometimes percentages for questions where a respondent can only choose one response do not add up to 100%. This is due to rounding. Similarly percentages which sum one or more categories together (such as ‘satisfied’ for those who are very or fairly satisfied) are sometimes one percentage point different from the number achieved through a straightforward addition of each single category, again this is because of rounding. The term ‘official statistics’ is used in the report to refer to statistics published by Statistics New Zealand.

© Colmar Brunton 2015

3

Key findings © Colmar Brunton 2015

4

Key findings Use of official statistics Three in five respondents (who have heard of Statistics New Zealand) have used official statistics in the last 12 months. Local government respondents use official statistics most frequently (74% do so every few months) compared to 51% of central government respondents and 12% of businesses. Satisfaction with official statistics Satisfaction with official statistics is relatively high; 76% of users are satisfied, and 6% are dissatisfied. Satisfaction is higher amongst central government (87%) and local government respondents (80%) compared to businesses (64%). Satisfaction is also higher amongst those who completely trust official statistics (92%) and those who feel they are completely free from political interference (85%). This implies that in order to maximise satisfaction it is important to reinforce perceptions of them as trustworthy and independent. Perceptions of official statistics are generally positive, and are strongest for those factors which users feel are most important: trustworthiness (86% of users rate this as good), unbiased (83%) and accuracy (80%). The survey findings suggest that is important to maintain these factors. However, there is also scope to improve ratings around timeliness and being easy to interpret; just over half rate them as good. Timeliness appears to be a particular issue for local government users (31% rate is as good). There is also scope to improve the accessibility of statistics, with just over half of users (51%) reporting it is easy to find them. Trust in official statistics Trust in official statistics is high; 82% who have heard of Statistics New Zealand say they trust official statistics (completely or mostly). It is higher amongst central government respondents (97%) and local government respondents (96%) than businesses (72%). Trust is also correlated with a perception that statistics are free from political interference; 57% of respondents who feel statistics are completely free from political interference also completely trust official statistics compared to 25% overall. This suggests that in order to build trust it is important to emphasise their independence. Data sharing

Respondents feel increased data sharing could be beneficial. 95% of central government and 98% of local government respondents can see benefits in increased data sharing across government, while 55% of businesses can see benefits in increased sharing between the public and private sector. © Colmar Brunton 2015

5

Main findings © Colmar Brunton 2015

6

Awareness of Statistics New Zealand and use of official statistics

© Colmar Brunton 2015

7

Nearly all of the respondents surveyed have heard of Statistics New Zealand, although familiarity varies by the public and private sector. The majority of central government (72%) and local government (85%) respondents feel they know a great deal or fair amount about Statistics New Zealand compared to one in four businesses (25%). However, larger businesses (with over 50 employees) are more likely to feel familiar than smaller business (with up to 50 employees) (43% compared to 25%).

Q: How much do you know about Statistics New Zealand?

Total

8%

Central Government

Local Government

Business 1%

A great deal / fair amount 38%

16%

45%

55%

19%

24%

9% 1%

72%

28%

66%

58%

45%

85%

15%

15%

2%

25%

A great deal A fair amount A little I have heard of Statistics New Zealand but know nothing about it I have not heard of Statistics New Zealand = Significantly higher/lower than Overall

Base: All respondents - Overall (n=558); Central Government (n=182); Local Government (n=47); Business (n=329)

© Colmar Brunton 2015

8

Three in five respondents (who have heard of Statistics New Zealand) have used official statistics in the last 12 months. Once again, this varies by audience. The local government respondents use official statistics most frequently (74% do so at least every few months) compared to 51% of central government respondents and 12% of businesses. Usage does not vary significantly by business size.

Q: How often, if at all, have you used statistics published by Statistics New Zealand in the last 12 months?

Total

12%

Central Government

Local Government

Business 1% 11%

18%

31%

25%

35%

26%

38%

4%

32%

36%

32%

At least every 2-3 months

16%

21%

50%

1%

4%

6%

30%

51%

74%

12%

Every week / every 2-3 weeks (high user) Every month / every 2-3 months (medium user) A few times in the last 12 months / I have used them once in the last 12 month (low user) I have not used them in the last 12 months (non-user) Don't know / can't remember = Significantly higher/lower than Overall

Base: All who have heard of Statistics NZ - Overall (n=555); Central Government (n=182); Local Government (n=47); Business (n=326)

© Colmar Brunton 2015

9

Respondents who use official statistics do so for a variety of reasons, but primarily research (61% overall) and providing context to reports or advice (56%). The different audiences make different uses of official statistics, with businesses in general less likely to use them for multiple reasons. Local government respondents, in particular, are more likely to use official statistics to develop policy advice which reflects the fact that the local government sample is much more likely to include respondents in a policy role (57% compared to 16% in central government). Finally, larger businesses (with over 50 employees) are more likely to use official statistics for providing context to reports or advice than smaller business (with up to 50 employees) (47% compared to 28%).

Q: For which of the following reasons did you use the statistics for? Overall

Research

61%

Providing context to reports or advice

56%

Developing policy advice

20%

Business plan

20%

A publication

18%

Evaluation

17%

Other Don't know / can't remember

4% 9% 5%

Local Government workers

50%

93%

38%

78%

24%

44%

11%

28% 7% 10%

24%

22%

44%

18%

27%

5%

16%

7% 3%

Businesses 82%

75%

29%

Performance monitoring

Media release

Central Government workers 67%

11% -

27% 7%

14% 1% 11% 9% = Significantly higher/lower than Overall

Base: All who have used statistics published by Statistics NZ in last 12 months - Overall (n=364); Central Government (n=152); Local Government (n=45); Business (n=167)

© Colmar Brunton 2015

10

There is a clear perception that official statistics help respondents understand New Zealand. Two thirds of those who have heard of Statistics New Zealand feel official statistics are very or fairly important in helping them understand New Zealand. Local government (96%) and central government (86%) respondents are much more likely to feel they are important compared to just under half of businesses (49%). This perception is consistent across larger and smaller businesses.

Q: How important or not are statistics published by Statistics New Zealand in helping us understand our country? Critical / very important Total

26%

Central Government

46%

Local Government

Business

Critical

39%

40%

51%

12%

Very important

29%

14%

45%

38%

Fairly important

42%

Not very important

3%2%

Not at all important

1%

65%

86%

4%

96%

5% 1%4%

49%

Don't know = Significantly higher/lower than Overall

Base: All who have heard of Statistics NZ - Overall (n=555); Central Government (n=182); Local Government (n=47); Business (n=326)

© Colmar Brunton 2015

11

Awareness of Tier 1 statistics varies across the three audiences. Two in three central government respondents (67%) have heard of them, compared to less than one in four local government respondents (23%) and one in twenty businesses (6%). Awareness also varies by the frequency of use of official statistics. High users (every week / every 2-3 weeks) are more likely to be aware of Tier 1 statistics (74%) than medium users (every month / 2-3 months) (42%) and lower users (a few times / once in the last year) (23%).

Q: Have you heard of Tier 1 statistics?

Central Government

Overall 4%

1% 27%

69%

Local Government 6%

23%

Business 5% 6%

32% 67% 70%

89%

Yes No Don’t know

= Significantly higher/lower than Overall

Base: All respondents - Overall (n=558); Central Government (n=182); Local Government (n=47); Business (n=329)

© Colmar Brunton 2015

12

Satisfaction with official statistics © Colmar Brunton 2015

13

Users of official statistics are typically satisfied with them. Overall 76% of respondents are satisfied, and just 6% are dissatisfied. Satisfaction is higher in the public sector, with 87% of central government respondents and 80% of local government respondents satisfied, compared to 64% of businesses. However, satisfaction is higher for larger businesses (81% satisfied) than smaller businesses (64%). Other key sub-groups which are more likely to be satisfied include those who completely trust official statistics (92%), high users (87%) and those who feel official statistics are completely free from political interference (85%). The implication is that in order to maximise satisfaction with official statistics it is important to reinforce perceptions of them as trustworthy and independent. Q: How satisfied or dissatisfied are you with the statistics produced by Statistics New Zealand?

Total

20%

Central Government

Business

55%

26%

Local Government

14%

7%

58%

50%

11%

22%

76%

6% 4%

61%

22%

14%

Very / fairly satisfied

7%

5%1%

87%

7% 2%

80%

8%

64%

Very satisfied

Fairly satisfied

Neither satisfied nor dissatisfied

Fairly dissatisfied

Very dissatisfied

Don't know = Significantly higher/lower than Overall

Base: All who have used statistics published by Statistics NZ in last 12 months - Overall (n=364); Central Government (n=152); Local Government (n=45); Business (n=167)

© Colmar Brunton 2015

14

There is scope for official statistics to better meet the needs of customers. Less than half (46%) say they meet them very or fairly well. There is some variation in perceptions around this, with central government respondents most likely to feel the statistics meet their needs well (57%), and businesses least likely (35%). The findings also differ by size of business with larger businesses (with more than 50 employees) more likely to feel the statistics meet their needs well (53%), and smaller businesses (with up to 50 employees) less likely to feel this (35%).

Q: To what extent do the statistics produced by Statistics New Zealand meet your information needs? Very well / well

Total

18%

Central Government

19%

Local Government

18%

Business

28%

31%

38%

26%

33%

16%

19%

Very well

Well

18%

16%

22%

Somewhat

1% 1%

17%

Poorly

46%

57%

51%

27%

38%

Adequately

1%4%

2% 8%

35%

Don't know = Significantly higher/lower than Overall

Base: All who have used statistics published by Statistics NZ in last 12 months - Overall (n=364); Central Government (n=152); Local Government (n=45); Business (n=167)

© Colmar Brunton 2015

15

The most important attributes for users of statistics is that they are accurate, trustworthy and unbiased. They are less likely to prioritise relevance, being easy to interpret and consistency. This reinforces the earlier finding that in order to drive satisfaction, users need to be reassured official statistics are trustworthy and free from political interference. The different survey audiences are relatively consistent in rating what is important to them when using statistics. However, trustworthiness is as important for central government respondents as accuracy (51% say each one is critical). Central government workers place more emphasis on statistics being relevant (24% critical) than businesses (9%) but less so on them being easy to interpret (14% vs. 23% for businesses). Q: When using statistics, how important or not are each of the following to you personally? Accuracy

47%

Trustworthiness

39%

Unbiased

38%

Consistency

18%

Easy To Interpret

19%

Timely

Critical

45% 50% 49%

23%

Relevancy

17%

53%

Fairly important

Base: All who have used statistics published by Statistics NZ in last 12 months - Overall (n=364)

25% 31%

Not at all important

91%

1% 9% 1%

90%

16%

62%

49%

7% 1%

10% 2%1%

59%

15%

Very important

Critical / very important

Not very important

87%

2%

82%

1%2

79%

1% 2%

72%

1% 4%

64%

Don't know

© Colmar Brunton 2015

16

Users of official statistics (produced by Statistics New Zealand) rate them positively across the different attributes. Positively for Statistics New Zealand, the factors which are most important to users, are the factors they score highest on, including being trustworthy, unbiased and accurate. There is scope to improve ratings around timeliness and being easy to interpret, with just over half rating these as good. Government respondents are more positive than businesses in their ratings of the most important factors: being trustworthy, unbiased and accurate. They are much more likely to rate each one as very good. However, local government respondents are less likely to feel the statistics are timely, with 31% rating this as very or fairly good compared to 57% of central government respondents and 56% of businesses. Very / fairly good

Q: How would you rate the statistics produced by Statistics New Zealand on each characteristic? Trustworthiness

47%

39%

9% 1%5%

86%

Unbiased

48%

35%

10% 1% 6%

83%

Accuracy

34%

Consistency

Relevancy

47%

27%

53%

19%

Easy To Interpret

14%

Timely

15% Very good

10% 14%

50%

25%

44%

35%

38% Fairly good

Base: All who have used statistics published by Statistics NZ in last 12 months - Overall (n=364)

36% OK

Fairly poor

Very poor

9%

80%

1%5%

80%

2%4%

69%

4% 3%

57%

7% 4%

53%

Don't know

© Colmar Brunton 2015

17

The Strategic Priority Analysis (below) suggests emphasis should be placed on maintaining and building accuracy, trustworthiness and the independence of the figures. The following chart plots the average score for each attribute in terms of importance vs. performance. The red lines represent the average scores for importance and performance. The resulting four quadrants form a Strategic Priority Analysis. The top left quandrant would normally represent priorities for improvement (factors which are relatively important to users but where performance is not as strong as other areas), however, none of the factors fall into this quandrant. As such we believe the emphasis should be placed on maintaining and building key strengths in the top right quandrant (factors which are relatively important to users but where performance is also relatively strong). Q: When using statistics, how important or not are each of the following to you personally? – IMPORTANCE AXIS Q: How would you rate the statistics produced by Statistics New Zealand on each characteristic? – PERFORMANCE AXIS 4.5

Accuracy

4.4 4.3

Trustworthiness

4.2 IMPORTANCE (average score 4.1 out of 5)

Unbiased

Relevancy

4 3.9

Easy To Interpret

3.8

Timely

Consistency

3.7 3.6 3.5 3.5

3.6

3.7

3.8

Base: All who have used statistics published by Statistics NZ in last 12 months - Overall (n=364)

3.9

4

4.1

4.2

4.3

4.4

4.5

PERFORMANCE (average score out of 5)

© Colmar Brunton 2015

18

There is scope to improve the accessibility of statistics, with just over half of users (51%) reporting it is easy to find them. The findings on accessibility are relatively consistent across the three audiences. Concerns around navigation of the website are also raised in the open ended question.

Q: How easy or difficult is it to find statistics produced by Statistics New Zealand? Very / fairly easy

Total

6%

Central Government

7%

Local Government

4%

Business

5%

Very easy

45%

27%

49%

22%

49%

40%

Fairly easy

Neither easy nor difficult

16%

19%

29%

32%

Fairly difficult

16%

14%

Very difficult

1%4%

51%

2%1%

56%

2%

53%

8%

46%

Don't know = Significantly higher/lower than Overall

Base: All who have used statistics published by Statistics NZ in last 12 months - Overall (n=364); Central Government (n=152); Local Government (n=45); Business (n=167)

© Colmar Brunton 2015

19

Trust in official statistics © Colmar Brunton 2015

20

Trust in official statistics is high. Over four in five respondents who have heard of Statistics New Zealand say they trust official statistics (completely or mostly). Trust is almost universal in the public sector with 97% of central government respondents and 96% of local government respondents saying they trust official statistics completely or mostly. Businesses are more cynical, although over seven in ten still trust them. Trust is somewhat better in larger businesses (83%) than smaller ones (72%). Trust is also correlated with a perception that statistics are free from political interference; 57% of respondents who feel statistics are completely free from political interference also completely trust official statistics compared to 25% overall. This suggests that in order to build trust it is important to show the statistics are independent. Q: To what extent do you trust the statistics produced by Statistics New Zealand?

Total

25%

Completely/ mostly

57%

12%

2%04%

82%

Central Government

40%

57%

0 2% 1% 1%

97%

Local Government

40%

55%

4%0

96%

Business

15%

Completely

57%

Mostly

Somewhat

18%

A little bit

Not at all

3%1% 6%

72%

Don't know = Significantly higher/lower than Overall

Base: All who have heard of Statistics NZ - Overall (n=555); Central Government (n=182); Local Government (n=47); Business (n=326)

© Colmar Brunton 2015

21

Seven in ten of those who have heard of Statistics New Zealand feel official statistics are completely, or mainly, free from political interference. This perception is important as earlier findings highlight the importance to users of having statistics which are free from bias, and the correlation between being free from political interference and overall satisfaction with official statistics. Central government (88%) and local government (85%) respondents are much more likely than businesses (56%) to think official statistics are free from political interference. The difference between larger (67%) and smaller businesses (56%) is not statistically significant.

Q: To what extent do you think statistics published by Statistics New Zealand are free from political interference? Completely/ mainly free Total

30%

Central Government

42%

Local Government

Business

39%

46%

53%

19%

Completely free

15%

Mainly free

20%

Partly free

11%

9%

32%

37%

5%

Not at all free

69%

3%

6% 2% 6%

7%

17%

88%

85%

56%

Don't know = Significantly higher/lower than Overall

Base: All who have heard of Statistics NZ - Overall (n=555); Central Government (n=182); Local Government (n=47); Business (n=326)

© Colmar Brunton 2015

22

Data sharing © Colmar Brunton 2015

23

There is a recognition of the benefits of increased data sharing across the public sector, and between the public and private sector. Almost all public sector respondents (96%) think there are benefits to them from increased data sharing across government. The majority also feel there are benefits from increased data sharing between the public, private and NGO sectors. The majority of businesses (55%) also feel they might benefit but are more likely to be uncertain of this (26% say don’t know). Larger businesses are more likely to see benefits in data sharing (76%) than smaller ones (55%).

Q: Do you think there are benefits for your agency from increased data sharing between governmental agencies for operational purposes?

Central Government

Overall 3% 2%

Local Government

3% 2%

96%

2%

95%

Yes No Don’t know

98%

Base: All Central and Local Government workers - Overall (n=229); Central Government (n=182); Local Government (n=47)

Q: Do you think there are benefits for your agency / business from increased data sharing between governmental agencies, and with NGOs and the private sector for statistical purposes? Central Government

Overall 19%

5%

Local Government

8%

Business

11% 2%

26%

13%

55%

68%

19% 87%

87%

Yes No Don’t know

Base: All respondents - Overall (n=558); Central Government (n=182); Local Government (n=47); Business (n=329) = Significantly higher/lower than Overall © Colmar Brunton 2015

24

Most government agencies feel they have capability for data sharing, although local government is more likely to need support. Two in three central and local government respondents believe they have a great deal or a fair amount of capability for data sharing. It is noticeable that central government respondents are more likely to feel they have a great deal of capability (34%) compared to local government respondents (15%). This suggests local government might require more support.

Q: What capability does your agency have regarding data sharing? Great deal / fair amount

Total

30%

Central Government

Local Government

37%

34%

35%

15%

A great deal

18%

18%

45%

A fair amount

19%

A little

None

2%

4%

10%

67%

9%

69%

4%

19%

60%

Don't know / refused = Significantly higher/lower than Overall

Base: All Central and Local Government workers - Overall (n=229); Central Government (n=182); Local Government (n=47)

© Colmar Brunton 2015

25

Additional comments © Colmar Brunton 2015

26

Feedback from the open ended question raises a number of concerns and areas for improvements, as well as referencing the value of official statistics and praising customer service from Statistics New Zealand. The key concerns or areas for improvement include being able to find relevant data on the website; the accuracy of the statistics (this principally relates to the quality of the data provided by respondents but also includes some comments on question design); the need for more granularity for statistics; increased support in interpreting statistics and reducing the burden placed on those providing statistics (this comment mostly comes from businesses). Q: What if, anything else, would you like to say about Statistics New Zealand or the statistics they produce? You guys do an outstanding job. I am constantly impressed with the calibre of the staff and the quality of the work you do. However, I suggest that you engage an evaluator or two to balance the strong research focus and provide the so what evaluative thinking that would add value and help SNZ identify and track success more effectively. Central Government

Timeliness of producing statistics is an issue. Statistics are often years out of date when we use them and that is of concern. Local Government

More disaggregation to the regional/local level would be highly desirable, especially in areas where Local Government have the principal policy role (e.g. environmental statistics and regional councils).

They are very helpful for data collection when doing a business plan to start up a new business. Business, 1-50 employees

Too many statistics return required from companies. Business, 51+ employees

Local Government

For us, there are a few gaps in the data that is actually collected (e.g. robustness/frequency of services data), however I think the bigger issue/opportunity is for Statistics NZ to consider the products it produces and whether they can be better fit for purpose for users. It would also be great to see more analysis provided alongside the data. Central Government

Respected agency that offers huge value to NZ in terms of the information that it produces. A real challenge for Stats is how it turns itself from a bulk producer of statistics to an enabler of information/ data that is easily and widely available to all. Experience to date suggests good intentions, but if anything increasing process/ bureaucracy, allied with shortages of key staff - creating bottlenecks and frustrated users. Central Government

It is generally quite difficult to find your own statistics from the website but the staff are always extremely helpful at assisting me to navigate around and find the information I need. Local Government

Statistics are as good as the questions asked, or the answers given. The census is a good example of flawed data. eg. Do you ride a bike to work. You may only ride it one day but actioned as if you do it every day Business, sole trader

© Colmar Brunton 2015

27

For further information please contact:

Edward Langley Colmar Brunton, a Millward Brown Company Level 9, Legal House 101 Lambton Quay Wellington 6011

PO Box 3622, Wellington 6140 Phone (04) 913 3007 E: [email protected] www.colmarbrunton.co.nz

© Colmar Brunton 2015

28