CEB Workforce Surveys & Analytics
Using Normative Data and Benchmarks Making Insightful Internal and External Comparisons Introduction Benchmark data, or norms, are a valuable tool for interpreting survey data. To use norms to the best advantage, it is important to understand the type of norms available, how they are sourced, and which norm can help you gain the greatest insight on employee perspectives. But be warned, overreliance on norms and normative comparisons can distract from the story of the survey data and the true value of the survey process. The average employee opinion survey can produce hundreds of data points. Results for each question cut by job type, length of service, function, country, region, manager, gender, and many other demographics can create a mountain of data, even for a relatively short survey. Faced with so much data for many different groups, it can be challenging to sort through the findings to determine strengths and opportunities for improvement. How will we know a good score when we see one? How low can scores go before we should worry about the consequences for the company? Scores for employees in the United States are very different from those in the United Kingdom; is that a problem? Absolute survey results—the favorable, unfavorable, or neutral scores on a given question or category for a specific group of employees—are important to the survey data analysis process but very difficult to interpret on their own. Without a reference point, it is difficult to separate the successes from the concerns. Norms, or normative data, provide that reference point, lending valuable assistance in interpreting survey data in the context of what is typical or even best in class.
It’s All About Context Is a score of 72% favorable high or low? It all depends on context. The context may refer to the country in which the employees reside, the respondents’ job level(s), previous scores on this question, or the topic addressed in the survey. The graph below highlights the value of norms in understanding survey data, as well as the wide range of scores that a single question can generate around the world.
USING NORMATIVE DATA AND BENCHMARKS © 2013 The Corporate Executive Board Company. All Rights Reserved. FWE6360013SYN
1
www.executiveboard.com
This study may not be reproduced or redistributed without the expressed permission of The Corporate Executive Board Company.
Figure 1: Compared to others doing similar work in your company, how would you rate your performance? Top 25%
Middle 26%–74%
Bottom 25%
79%
US
20%
70%
Canada
The Netherlands
44%
Germany
52%
45%
Spain
52%
43%
48%
47%
45%
France
50%
37%
Japan 0%
1%
33%
54%
Italy
2%
28%
66%
UK
1%
52% 50%
2% 3% 5% 5% 5% 11% 100%
Source: CEB analysis.
Is it possible that the phrase “top 25% of performers”—despite being well defined mathematically—incites a different response around the world? Or is it that Americans tend to have a higher opinion of their own performance than the Japanese? Imagine your company included this question on the employee survey, and 75% of your company’s US employees indicated they were in the top 25% of performers. Without a normative comparison you might think the survey question was unclear or the data coding was incorrect. How could 75% of employees be in the top 25%? Normative comparisons provide the context necessary to evaluate survey results in a meaningful and insightful way. Using a variety of norms will provide maximum insight on survey data. Many types of comparisons are available, both internally and externally sourced: ■■ Internal comparisons—These comparisons allow managers to interpret their survey results in the context of what is typical in their company. Comparisons to the next level in the company, division, or the total company enable the manager to determine if their scores are higher or lower than what is typical in that company. Care must be taken to compare similar results. For example, a manager of a group of employees in production may not be well served to compare results to an internal group that includes large numbers of finance or other professionals. ■■ History from previous surveys—A score of 72% can be interpreted positively if it has improved from the previous survey, or it can be viewed as bad news if it has declined significantly in a year or shows a successive pattern of declines over time that in aggregate are meaningful. Comparing the trend in one group to the trend for the whole provides additional insight.
USING NORMATIVE DATA AND BENCHMARKS © 2013 The Corporate Executive Board Company. All Rights Reserved. FWE6360013SYN
2
www.executiveboard.com
This study may not be reproduced or redistributed without the expressed permission of The Corporate Executive Board Company.
There is cause for concern if a group’s results declined from the previous survey while the rest of the company’s improved. Once a company’s baseline results are established, history becomes an extremely valuable reference for assessing progress and change across the company. Company restructurings are common, and history data must be interpreted with caution if there has been a significant company restructuring. Extensive restructurings, particularly those involving acquisitions or divestitures, may render history data irrelevant and unuseful. ■■
External benchmarks or norms—These norms allow for a comparison with what is typical in an industry, region of the world, country, job type, average company, or leading company.
Three General Approaches to Sourcing External Benchmarks Panel data are known for their completeness; all questions are administered to the same sample, and the data are collected at the same time. As a result, there is a consistently sized set of data across all subgroups surveyed, including regions, countries, and job types.
■■
Panel-based norms: Data are collected via surveys administered to a sample of employed adults. Panel data are known for their completeness; all questions are administered to the same sample, and the data are collected at the same time. As a result, there is a consistently sized set of data across all subgroups surveyed, including regions, countries, and job types. Because panel surveys are administered broadly among residents in a given country, the database will include many of the multiple industries, job types, and employers in that country. However, it is important to keep in mind that the norm may not be perfectly representative of all industries, job types, and other cuts within a country or region. Assuming the norm is large enough, panel norms may be sliced by demographics, providing an even more specific comparison source. As panel data are collected on a regular basis in a defined time window, it is possible to track question trends over time. These trends provide additional useful context for interpreting survey results through their ability to compare to trends in one’s own company data and understand how economic and business shifts impact employee viewpoints. This approach to norms also allows survey providers to continuously innovate and test new items at their own expense and then make them available for clients’ benefit.
■■
Client-based norms: Consulting firms may choose to create norms by accumulating clients’ data into a common database. As data are collected on common questions, the data are added to the normative database along with demographic data. The composition of the normative database represents the composition of that firm’s clients; it may be heavy on financial services companies, light on non-US results, and average when it comes to representing certain job types. Not every company included in the norm chooses to use exactly the same survey questions, and in fact there may be slight variations in the wording used by each included company. These slight variations in question wording can lead to not so slight differences in the meaning and intent of the question. Not every client of a given firm does business in the same set of countries, and as a result client-based norms do not have a consistent amount of data for every question and every demographic in the same way that panel norms do. Since data are added to the database as they are collected, it could have been collected over a two- or three- or five-year period or even longer. The process of adding individual companies to the norm or dropping companies from the normative database may significantly shift the results. Therefore, trends in client-based norms need to be interpreted with caution. The quality of client-based norms is highly dependent on the consulting firm that is building the norm. Very large consulting firms, or those that specialize in a particular industry, may have enough survey volume to produce a solid, representative benchmark.
USING NORMATIVE DATA AND BENCHMARKS © 2013 The Corporate Executive Board Company. All Rights Reserved. FWE6360013SYN
3
www.executiveboard.com
This study may not be reproduced or redistributed without the expressed permission of The Corporate Executive Board Company.
■■
Consortium norms: Companies that routinely survey their employees may choose to join a survey consortium: a group of companies that have joined together to share survey benchmarks on a defined set of questions. Consortium members have to meet a specified set of membership criteria specific to each consortium. Naturally this means that many companies might not meet the eligibility criteria for consortium membership or have access to these norms. Consortium members agree to include a specific number of consortium-designed questions in their employee survey and submit the data on a regular basis to develop norms. These data are pooled by a third party, and confidential normative data reports are generated and distributed to members of the consortium. One value of consortium norms is that their source is clearly known; member companies know exactly what companies are represented in the database, even though they cannot identify which data was contributed by which company. Although consortium norms typically provide data on a relatively limited set of questions, the quality of the data are high given the quality of the companies that join such groups and the stringent guidelines they adhere to in the submission of their survey data. Leading survey consortia include Mayflower Group and the Information Technology Survey Group (ITSG) in the United States, and the RACER Benchmark Group in Germany.
Figure 2: Use of Norms Yes
No
Possibly true; some companies may be similar or relevant, others not; data collected and compiled over an extended period of time
Internal Normative Comparisons Comparison to total company or parent group
History from previous survey
External Normative Comparisons Clientbased norms
Panelbased norms
Consortium norms
Source of data clearly known Data represent companies, not individuals Identical questions used across respondents Results from a relevant population Data collected during a defined period of time Data can be cut by demo-graphics Can track trends over time
USING NORMATIVE DATA AND BENCHMARKS © 2013 The Corporate Executive Board Company. All Rights Reserved. FWE6360013SYN
4
www.executiveboard.com
This study may not be reproduced or redistributed without the expressed permission of The Corporate Executive Board Company.
Predictable Patterns in Survey Data Demographics have varying levels of influence on survey results, and many demographics produce predictable patterns in survey data.
All three sources of normative data can provide detailed comparisons for different employee subgroups, whether that is country, job type, level, gender, or industry. Demographics have varying levels of influence on survey results, and many demographics produce predictable patterns in survey data. Understanding and anticipating these patterns provides a useful reference point for interpreting survey data, both when the expected patterns appear, and perhaps more importantly, expected patterns do not appear in the data. So what is the connection between demographic patterns in data and normative comparisons? Norms help the user understand when differences in the data follow typical patterns and when they don’t. Providing norms for some of these demographics may allow the user to understand that what may seem like a crisis on the surface (e.g., scores for employees in Japan are shockingly low) is a predictable pattern in the results (scores for Japan are nearly always lower than in other countries). Although it is not always necessary to provide norms at all demographic levels, it is necessary to highlight when a normative comparison might be valuable. The following normative demographic comparisons are the most commonly used to aid interpretational, although not all are commonly used in manager level reports: ■■
Country—Because country culture significantly influences employee survey results, country norms are commonly used in both manager-level reports and preparing executive briefings and/or distilling key insights for the whole organization. Multiple research studies over years have identified predictable patterns in survey data based on country and/or region of the world. Emerging economies (Brazil, Russia, China, and India) tend to have more positive survey results on virtually all questions than results from countries with more mature economies. US employees tend to be more positive than those in Western Europe, which in turn are more positive than scores in many countries in Southern and Northern Europe. Scores in Latin America tend to be very positive, but scores in Japan are among the lowest in the world. These predictable differences in scores—many of which can be quite large—highlight the danger of companies directly comparing scores for the countries in which they do business. Leaders can erroneously deduce that there are significant engagement issues in Japan, whereas the US has top scores. Had those same leaders compared their US scores to a US norm and their Japan scores to a Japanese norm, they may have learned exactly the opposite. Although US scores may indeed be higher than those in Japan, they could be significantly lower than a US benchmark, and those low scores from Japan, may be higher than the Japanese external benchmark. Accurate interpretation depends on comparing survey results to the appropriate norm.
■■
Job Type/Job Level—Survey scores are also influenced by job level and job type. Survey scores typically increase as jobs progress up the organizational hierarchy; leaders and managers tend to have more favorable responses to survey questions than nonmanagers, and professional positions report more favorably than hourly or blue– collar positions. Normative comparisons for very specific job types aren’t needed, but comparing results for hourly or production employees to a comparable norm can be a valuable way to determine if pronounced differences from scores for executives reflect issues or are typical.
USING NORMATIVE DATA AND BENCHMARKS © 2013 The Corporate Executive Board Company. All Rights Reserved. FWE6360013SYN
5
www.executiveboard.com
This study may not be reproduced or redistributed without the expressed permission of The Corporate Executive Board Company.
■■
■■
■■
Length of Service—Employees are rarely more engaged than they are in their first 12 months in a company. Survey scores drop after the first year, typically hitting their low point mid-career before rising again with more years of service. Many companies see results in this U-shaped pattern, while others find that survey scores decline after the first year and remain fairly consistent across subsequent years of service. Including a length of service norm in reports isn’t common, but it is helpful to educate data users on this pattern via training materials. Industry Type—Many companies are interested in comparing their results to companies’ in the same industry, as they reason that these companies have similar workforces and compete in the same market space. Industry, however, tends to have less of an influence on survey scores than other demographics. In fact, much of the difference between companies in different industries may reflect a company’s job type composition rather than the industry itself. For example, a manufacturing organization may have lower scores than a financial services organization because the organization has a large percentage of employees in production jobs relative to a higher percentage of professionals in the financial services company. This difference is due less to industry and more to the makeup of the employee population. High-Performing Companies—By definition, norms represent the average or what is typical in companies and employee groups around the world. Many companies prefer to compare themselves to a higher standard. High-performance norms or high-performing companies norms provide a comparison to leading companies rather than the average. Criteria for identifying a company as high performing could include business performance that has exceeded recognized averages for an extended time period, designation as being highly admired or a top employer, or inclusion as part of a top 50 or top 100 list. High-performing companies norms provide an aspirational comparison point and give leaders and managers insight on what issues to address to drive organization performance. A comparison of results between high-performing companies and the average indicate that factors such as high employee engagement, clear understanding of strategy, confidence in senior leaders, and a focus on the company’s future success may best distinguish the top companies from the rest.
Using Norms Effectively Normative comparisons can be a highly useful tool for survey data analysis, providing an important context for interpreting results. But given what seems to be an endless list of benchmark options, what is the best way to use them? ■■
■■
Choose the right norm. Be it a global norm, country-specific or linked to a particular employee type, make sure the norm represents a meaningful comparison point. It is very easy to draw the wrong conclusion by comparing hourly employees to a norm that consists primarily of results from professional employees or by comparing results from Japan with a global norm. Keep in mind that what is meaningful should be considered with reference to who you compete with for talent. A high-performance norm, for example, might be more appropriate than an industry-specific one for your context. Not every question on the survey needs a normative comparison. Most companies include some number of custom questions on their surveys in addition to those from a normative database. These custom questions usually assess topics specific to the company, such as their mission, vision, or unique business environment. A balance of normed and non-normed questions works well for most companies as long as the survey includes some normative questions, if not individual questions, for each major topic to show how the company stands on a topic in general.
USING NORMATIVE DATA AND BENCHMARKS © 2013 The Corporate Executive Board Company. All Rights Reserved. FWE6360013SYN
6
www.executiveboard.com
This study may not be reproduced or redistributed without the expressed permission of The Corporate Executive Board Company.
■■
Take a broader perspective and use normative comparisons as one of many data points. ■■
■■
■■
■■
Remember that external or internal context may trump a norm. For example, if your company has recently closed several sites, certain questions may not score well relative to the norm. These actions had a strong influence on results, rendering the norm less valueable and an understanding of recent internal actions more valuable. Limit the number of norms used in reports. Too many norms can yield significant confusion for managers. It is better to use fewer norms than to provide managers with so many different normative comparisons that they are unable to determine which to use and how to interpret different profiles versus the norm. Include other data to round out the analysis. Marry the normative comparison with other data. On its own, scoring higher than the norm might be good news, but perhaps less so if a comparison to history is showing sharp declines in scores from survey to survey. Take a broader perspective and use normative comparisons as one of many data points. Use the norms to help set appropriate goals. Normative data help define what is possible when it comes to survey question responses. Using the norm as a reference point helps define whether a given score has significant room for improvement (significantly below the norm) or is near the upper boundaries of survey responses (significantly above the norm). Remember that the goal of the survey is not to beat the norm. The goal of the survey is to collect information that will help define opportunities for continuous improvement. Normative data are best used as a reference point to identify these opportunities. When beating the norm becomes the primary focus of the survey program, the focus on driving change suffers.
Overusing Norms: A Cautionary Tale Too often leaders and managers become overly concerned with “beating the norm” or driving scores to achieve a specific number relative to the benchmark.
Normative data clearly provide significant value in interpreting survey data, and identifying when patterns in survey data are “as expected” and when they deviate significantly. It is difficult to accurately interpret survey data without access to these benchmarks. However, it is possible to overuse normative data. Benchmarks provide valuable context for analysis, but they should not become the primary focus of the survey. Too often leaders and managers become overly concerned with “beating the norm” or driving scores to achieve a specific number relative to the benchmark. Other leaders request many different, highly customized norms against which to compare. These customized norms represent “fast-moving,” “mature,” or some combination of multiple adjectives that are meant to very specifically define the companies in the norm. Some consulting firms tout these highly specialized norms as invaluable, yet the data may represent only bits and pieces of a handful of companies that meet an unusual set of criteria. Further, the data may have been collected during significantly different time periods using different question sets. It is quite difficult to create normative databases that are focused exclusively on companies that fit a very specific set of criteria, and there is little to suggest that these norms provide greater value or insight than standard norms or high-performing companies norms. The truth is that comparing results against multiple different norms can confuse managers more than inform them. The risk of such a situation is that leaders may discredit the survey process because it is difficult to draw insight from multiple norms drawn from multiple sources representing multiple organization types and demographics. In these cases, the survey can devolve into an exercise of normative comparisons, rather than the more useful and productive use of survey data to identify issues that require action at multiple levels of the organization. Normative comparisons, despite their value, need to play a minor role in the survey process relative to the company’s own data. An overemphasis on multiple, highly specific norms can derail the survey process by encouraging managers to expend too much energy and focus trying to beat a number that may be dubious at best.
USING NORMATIVE DATA AND BENCHMARKS © 2013 The Corporate Executive Board Company. All Rights Reserved. FWE6360013SYN
7
www.executiveboard.com
This study may not be reproduced or redistributed without the expressed permission of The Corporate Executive Board Company.
Moderation Is the Key Normative comparisons are a valuable tool for interpreting survey data. Used in moderation, they can inform the user by identifying patterns of data that are typical, atypical, leading, and lagging. The key is to use these reference points as a jumping off point for additional analysis and ultimately taking focused action to drive positive change in the organization.
About CEB CEB is the world’s leading member-based advisory company. We have a unique view into what matters—and what works—when capitalizing on drivers of business performance. With 30 years of experience working with top companies to share, analyze, and apply proven practices, we begin with great outcomes and reverse engineer to help you unlock your full potential. Every year we equip more than 16,000 senior leaders from more than 6,000 organizations across 60 countries with the insights and actionable solutions they need to respond quickly to evolving business conditions and transform operations. We do this by combining our advanced research and analytics with best practices from clients across more than 87% of the Fortune 500, 62% of the Dow Jones Asian Titans, and 80% of the FTSE 100. As a result, our members achieve outsized returns by more effectively optimizing talent investments, creating new sources of efficiency, reducing risk, and enabling and accelerating growth.
About CEB’s Workforce Surveys & Analytics
Contact Us to Learn More www.executiveboard.com
survey-solutions @executiveboard.com
CEB’s Workforce Surveys & Analytics offerings focus on improving our clients’ organizational performance by better aligning talent management with business strategy. We understand that every successful organization is powered by a unified and passionate workforce, and to this end, we work in close partnership with some of the world’s largest and best-known corporations, providing customized solutions that help drive organizational change and advancement. We bring an unrivaled ability to flawlessly execute the most complex surveys and to meet the diverse needs of our global clients. Our clients are uniquely positioned to drive change through guidance provided by our experienced consultants and our frontline-focused action planning tools and improvement resources.
USING NORMATIVE DATA AND BENCHMARKS © 2013 The Corporate Executive Board Company. All Rights Reserved. FWE6360013SYN
8
www.executiveboard.com
This study may not be reproduced or redistributed without the expressed permission of The Corporate Executive Board Company.
