A National Job Analysis of the Professional Counselor December 2010
Conducted for the
National Board for Certified Counselors and Affiliates
Copyright © 2010. PROPRIETARY. All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy or recording, or any information and retrieval system, without permission in writing from the National Board for Certified Counselors and Affiliates, Inc.
Table of Contents
Page Executive Summary .............................................................................. 1 Introduction ........................................................................................ 4 Methodology ......................................................................................... 6 Findings Survey Response Results .............................................................. 9 Rating Reliabilities ........................................................................... 9 Task List Completeness ............................................................... 10 Tasks/Activities ............................................................................ 10 Factor Analysis .............................................................................. 12 Tasks ......................................................................................... 12 NCE Test Composition .................................................................. 14 Summary ............................................................................................ 15 Appendix ............................................................................................. 17 A. NCE Practice Analysis Survey ................................................. 17
EXECUTIVE SUMMARY This report describes the procedures followed in the job analysis conducted for the National Board for Certified Counselors by The Center for Credentialing & Education, Inc. (CCE) in 2010. The purpose of this analysis was to describe the job tasks performed by professional counselors. The resulting examination framework will be used for the continued development of a professional, job-related licensure certification examination. The NBCC Job Analysis Advisory Committee (AC) conducted the activities necessary to identify the job tasks which were the basis for the job analysis survey and the subsequent test framework for the National Counselor Examination for Licensure and Certification (NCE). The AC members were identified through nomination by the NBCC from current and past NBCC development committee members. All AC members were experts in the duties and activities associated with the profession. The AC was responsible for the following activities regarding job analysis survey development: a. b. c. d.
identifying job tasks, developing a sampling plan, establishing the survey rating scales, and determining the demographic variables.
The draft job analysis survey was distributed to all AC members via the use of an internet based web form. Based on comments from the AC the survey instrument was modified and the final survey was created for distribution to stratified sample of NBCC certificant holders and licensed professional counselors; all surveyed were National Certified Counselors both licensed and non licensed. Three thousand two hundred eighty seven (3,287) surveys were disseminated to potential respondents. Individuals responding could do so via the enclosed paper survey form or they could submit their responses via the internet. There were eight hundred eighty (880) usable surveys completed, and most were completed using the online form. One hundred seven (107) surveys were returned either incomplete, past the extended deadline, or not deliverable. Therefore, the corrected return rate was 27.7%, which is high for a survey of this type and length. Responses to the demographic questions indicated that there were sufficient numbers from relevant groups for subsequent analyses. The respondents were asked to provide demographic data and to evaluate 196 counselor tasks. More than 97% of the respondents felt the job task list adequately or completely addressed the responsibilities of the professional counselor. Summary statistics were computed for the demographic data. The task ratings were summarized and a factor analysis performed. All of the counselor tasks were rated as important and frequently performed by the professional counselor. Therefore, the examination content outline was constructed
Copyright © 2010. NBCC Job Analysis Report
Page 1
from the 196 items. The AC decided that 160 examination items would continue to sufficiently sample the content domain to render a pass or fail decision based on examination scores. The resulting examination matrix and detailed content outline will be used by the NBCC Examination Development Committee to assemble future examination forms. The eight hundred eighty (880) respondents were asked to list the state regulated professional licenses or certifications they held. The respondents had the choice of fourteen (14) professional credential categories. Although each category was represented (including licensed physician) the largest categories were licensed professional counselor (N=534 – 60.7%), licensed clinical mental health counselor (N=136 – 15.5%), and licensed or certified school counselor (N=127- 14.4%). We assume that the LPC and LCMHC categories are mutually inclusive, therefore a total of at least 670 (76.1%) of the respondents were state regulated counseling professionals. Comparison of licensed/certified versus non-state licensed/certified counselors reveals no differences in answering items related to both importance and frequency of the 196 defined tasks. Although we are certain that there are no differences between National Certified and licensed/certified counselors; NBCC will continue in future job analyses to partition the two groups for comparison.
Copyright © 2010. NBCC Job Analysis Report
Page 2
Copyright © 2010. NBCC Job Analysis Report
Page 3
INTRODUCTION The purpose of conducting this job analysis was to provide a basis for the continued development of the National Counselor Examination for Licensure & Certification (NCE) and ensuring that the content of the examination is job-related. The National Board for Certified Counselors requested the Center for Credentialing and Education (CCE) to facilitate a study which would identify the relevant tasks required to conduct safe and effective practice. The NBCC appointed an Advisory Committee (AC) to assist in the development of the job analysis survey instrument. The AC reviewed professional materials to develop a thorough list of tasks required for professional counselors to perform the job. The job analysis survey was distributed to a representative sample of professional counselors according to the sampling plan developed by the AC. The returned surveys were analyzed to determine the importance of each to the professional counselor, and how frequently each was performed.
Copyright © 2010. NBCC Job Analysis Report
Page 4
Copyright © 2010. NBCC Job Analysis Report
Page 5
METHODOLOGY Creating the Advisory Committee The AC provided expert subject matter consultation throughout the stages of the job analysis process to ensure that expert judgment was available to the Center for Credentialing & Education (CCE) staff. The members of the AC were selected to reflect diverse backgrounds within the profession. Advisory Committee Responsibilities 1. 2.
3. 4.
Provide CCE current information about the job. Develop the Job Analysis Survey: a. develop a sampling plan for the survey, b. identify tasks for the survey instrument, c. determine the survey rating scales, d. determine the relevant demographic variables of interest, and e. integrate the definition, tasks, rating scales, and demographics into a survey instrument. Review the final form of the Job Analysis Survey for completeness, relevance to the profession, appropriate language and clear instructions. Interpret and review survey results, determine task exclusion criteria, and create the final detailed content outline.
Copyright © 2010. NBCC Job Analysis Report
Page 6
Designing the Job Analysis Survey Task/Activities List The NBCC AC developed a comprehensive list of job tasks needed to perform the job. The draft document consisted of 196 task statements. Survey respondents were allowed to suggest additional tasks. The complete Job Analysis Survey and cover letter are shown in Appendix A. Rating Scales The AC also developed the rating scales used in the job analysis survey. importance scale, and a frequency scale were included as part of the survey.
An
Two rating scales were used for the Task items. Both scales are Likert-Type 5-point scales. The scales were designed to identify the tasks considered most important, and the most frequently performed by professional counselors. This data was used to ensure that the examination assesses tasks that are relevant to the safe and effective performance of counselors as define by counselors. The following scales were used: Scale
Definition
Importance
Tasks/Activities How important is the task to your safe and effective performance as a practicing counselor?
Frequency
As a practicing counselor, how often do you perform the activity?
Copyright © 2010. NBCC Job Analysis Report
Response Options 1. Not important 2. Somewhat important 3. Moderately important 4. Important 5. Extremely important 1. Never 2. Rarely 3. Occasionally 4. Frequently 5. Almost Always
Page 7
Background Information Questions The demographic section of the survey was designed to gather information about the counselor survey respondent. The AC reviewed and modified survey questions that were presented to the AC. Additional demographic questions were added by the AC to help qualify the respondents, and to assist in establishing the validity of the data. The following characteristics about the counselor survey respondent were available: • • • • • • • • • • • • • • • • • • • • •
Location of employment Work activity type Years of experience Employment status Primary work setting Secondary work setting Time spent in direct counseling service Time spent in administrative counseling activities Annual pro bono counseling hours Age groups served Racial/ethnic groups served Theoretical approaches used Highest academic degree Education program Council for Accreditation and Related Educational Programs (CACREP) or Non-CACREP Professional voluntary certifications held State regulated licenses or certifications held Professional association memberships held Service provided Client populations Gender of respondents Ethnic identity of respondents
Draft Survey The survey sections were incorporated into a draft survey. The draft survey was reviewed by the AC via an internet based web form. This on-line review allowed the AC to comment on all the survey components. A report on suggested changes, additions, and time spend to complete the survey was reviewed by the AC Chair. Appropriate changes were made to the survey prior to distribution to the sample population. Survey Distribution The survey and cover letter (Appendix A) were sent to 3,287 professional counselors throughout the United States. The sample was stratified across geographic region to ensure representative proportions within and across nine regions. Regions consisted of New England, Middle Atlantic, East North Central, West North Central, South Atlantic, East South Central, West South Central, Mountain and Pacific. Stratifying by state would eliminate many states with small populations from proper scrutiny. Inclusion of regions allows for typical practice across state lines.
Copyright © 2010. NBCC Job Analysis Report
Page 8
FINDINGS Return Rate and Sample Size Of the 3,287 surveys originally disseminated, 880 useable surveys were returned. Ninety-five surveys were returned as non-deliverable, and an additional 12 surveys were considered non useable because they were not delivered in time for data analysis. Therefore, the corrected return rate was 27.7%. Task and Respondent Rating Reliabilities Table 1 presents the coefficient alpha internal consistency reliability estimate and the intraclass correlation for the tasks statements. These statistics were used to determine the extent to which tasks were consistently rated within each survey section, and to determine the extent to which the respondents were consistent in their ratings. The ratings were considered highly reliable.
Scale
Table 1: Task Reliabilities Reliability Tasks Intraclass Correlation Coefficient Alpha
Importance
196
.99
.96
Frequency
196
.99
.98
.
Copyright © 2010. NBCC Job Analysis Report
Page 9
Coverage of Tasks Inadequately 2%
Completely 31%
Adequately 67%
Figure 1. Task Coverage
The respondents were requested to indicate how well the survey covered the important activities of the counselor. They could select one of the following responses: completely, adequately, or inadequately. Figure 1 shows that of the 790 respondents who answered this question, 98% thought the survey completely or adequately described the professional counselor. Tasks/Activities Central tendency was computed for the Importance and Frequency ratings of each item. The purpose of these data was to determine which tasks will remain on the final content outline and be used in the development of examination items and examinations. The data was normed, the t-scores (mean of 50 and standard deviation of 10) and the means and standard deviations are presented in Appendix C. The means, standard deviations and t-scores were computed from the items comprising each category. The Frequency rating was given priority and the rankings determined by the t-scores. The largest t-score received a rank of 1 and the smallest the rank of 10. This data is presented in Table 1.
Copyright © 2010. NBCC Job Analysis Report
Page 10
Table 1: Categories as Presented on Survey Frequency Mean SD Overall Survey Id Content Counsel clients with issues I related to: Structure the counseling process for individuals, couples, family, or groups II by: Conducting initial III assessment of client by: Conduct the on-going IV assessment by: Administering IV.1 assessments/tests Interpreting IV.2 assessments/tests Provide professional V service by: Supervision and VI consultation: Practice counseling techniques by VII implementing: VIII Provide self care by:
t-score
3.12 0.95
Importance Mean SD t-score 3.58 0.74
Rank 7
2.88 0.69
47
3.55 0.55
50
3
3.64 0.60
55
3.93 0.42
55
5
3.30 1.03
52
3.59 0.81
50
4
3.47 0.74
54
3.72 0.63
52
10
2.01 0.45
38
2.64 0.34
37
9
2.17 0.36
40
2.78 0.29
39
6
2.97 1.06
48
3.42 0.86
48
8
2.69 0.65
46
3.49 0.49
49
2 1
3.94 0.90 4.07 0.51
59 60
4.10 0.70 4.41 0.25
57 61
Table 3 outlines the rule approved by the AC to ensure that the tasks included on the final examination content design were performed frequently enough or found important enough to be performed by a minimally competent counselor. Table 2: Decision Rule 1. Keep only tasks rated by respondents and with an overall t-score at or above 30 for either frequency or importance
Factor Analysis Copyright © 2010. NBCC Job Analysis Report
Page 11
Factor analysis was performed to identify item clustering and item relationships.
Tasks Factor analysis identified five Task factors. Each factor is composed of related items. With few exceptions, the individual items are highly rated for either or both importance and frequency. Special consideration should be given to items which have a significant difference between the importance and the frequency ratings. After reviewing the factor analysis and based on subject matter expert opinion relating to the counseling profession the AC decided to structure the National Counselor Examination content outline on the original task categories. The AC concluded that the ratings obtained from the job analysis survey were consistent with their evaluation of the profession. Subsequently, the AC agreed that the results of the job analysis survey were adequate for the purpose of establishing the content and developing the examination specifications for a national counseling examination. Examination Item Classification and Development Examination items must represent the tasks required for the safe and effective performance of counseling. Each examination item must be categorized by a category and task. The categorization is used to compile the items, according to the Examination Content directions, for an examination. Each examination item must be given a Content (Category) and Sub content (Job Analysis item) code. The Content code is the number of the Category and the Sub content is the number of the Item from the examination framework in Appendix D. For example: II-23 would be used to categorize items pertaining to “Helping clients develop support systems” which is the twenty-third item in the second category. The format of an examination item (multiple choice, etc.) is the responsibility of the item writers and is not a topic addressed by this job analysis. However, professional standards for item writing must be followed. The specific items to be included in an examination must be coded as indicated above and must be selected as specified for the Examination Content.
Examination Content The content of an examination must represent the Tasks required for the safe and Copyright © 2010. NBCC Job Analysis Report
Page 12
effective performance of counseling and must ensure adequate and equitable coverage of the categories. Coverage is the inclusion of an appropriate percentage of items from each category. The percentage of items from each category is derived from the percentage of the total number of tasks comprising each category, the responses to this job analysis and the professional opinion of experts in the counseling profession. The suggested percentage of items to be included from each category is the product of: 1. The total number of tasks comprising each category 2. The responses to this job analysis Categories and Percentage of Examination Items After reviewing all the data the AC decided to consolidate original categories IV, V, and VIII, and categories VI and VIII. The percentage of examination items to be drawn from each category (domain) is presented in Table 3
Table 3: Category representation on an examination Number of items on the Suggested number of the items content outline on an examination
Factor 1. Fundamental Counseling Issues 40
32
2. Counseling Process 55 3. Diagnostic and Assessment services 32 4. Professional Practice 45
45 25 38
5. Professional Development, Supervision, and Consultation 24
20
Examination Composition In consultation with CCE, the NBCC AC evaluated the results of the job analysis to develop the test specifications. It was determined that the examination should be Copyright © 2010. NBCC Job Analysis Report
Page 13
composed of items from each of the categories according to the “Suggested number of the items on an examination” column of the above table. The distribution of items on each form of the NCE, including field test, linking, and scored items will be developed by the NCE Examination Committee as part of the NCE Test Framework.
Copyright © 2010. NBCC Job Analysis Report
Page 14
SUMMARY A job analysis can be effective only to the extent to which the items actually evaluated are meaningful for and pertinent to the profession. The iterative, multiphase process used in this study was implemented successfully with this criterion in mind. Numerous professionals provided input about the final form of the survey distributed. Thus, the survey distributed was one that had both items and a format that readily could be interpreted by professional counselors. Respondents in this study were all NBCC credential holders and/or licensed professional counselors who effectively represented a cross section of those who identify themselves as professional counselors. More specifically, the sample encompassed appropriately proportionate counselors in regard to their varying professional credentials, personal characteristics, and employment settings. Because the respondents represented the major facets of the counseling profession, it is reasonable to assume that their responses also effectively represented the nature of the work in the counseling profession. The comprehensiveness and diversity of the sample has several important implications when considering the results of this study. For example, the sample was of sufficient size so that the statistical analyses applied could be used with confidence; the basic theoretical assumptions underlying them were not violated. More important, however, the nature of the sample implies that the results were derived primarily from practicing professionals. The results reflect the realities of counseling practice, rather than esoteric opinions of what counseling practice is supposed to be. The major goal of this study, to develop a base of counseling job activities for the continued development of a criterion-referenced examination, was accomplished. Each form of the examination will follow the test specifications as outlined in this report, thus providing evidence to support decisions regarding candidates’ abilities to perform the counselors’ job.
Copyright © 2010. NBCC Job Analysis Report
Page 15
Copyright © 2010. NBCC Job Analysis Report
Page 16
Appendix A
The National Certified Counselors Job Analysis Survey
Copyright © 2010. NBCC Job Analysis Report
Page 17
Copyright © 2010. NBCC Job Analysis Report
Page 18
Dear National Certified Counselor: The National Board for Certified Counselors (NBCC) is requesting your participation in a national job analysis survey to determine the important activities and issues relevant to the work of a professional counselor. The survey will be used to review and revise the National Counselor Examination for Licensure and Certification (NCE). A job analysis study is an essential step in keeping examinations current and job related. By completing the enclosed survey, you will be participating in the unique effort to improve the NCE. Please take the time to complete the survey and return it to NBCC in the enclosed self-addressed, postage-paid envelope. If you have any questions or need assistance in completing the survey, please contact Shawn W. O’Brien at NBCC (336) 547-0607 or
[email protected]. The survey can also be completed on-line at http://www.ResourcesForCounseling.com/ja_nce.aspx. The survey is best completed with Internet Explorer 6.0 or later or fully compatible browser. The survey will be unavailable daily from 12:00 (midnight) to 1:00 a.m. ET. On behalf of NBCC, I would like to thank you for taking the time to complete this important survey. By your participation, you are helping to undertake one of the most important projects in establishing evidence for the validity of the NCE, and the continued development of professional counseling.
Sincerely,
Thomas W. Clawson, Ed.D, NCC President and CEO National Board for Certified Counselors, Inc and Affiliates
Copyright © 2010. NBCC Job Analysis Report
Page 19
Copyright © 2010. NBCC Job Analysis Report
Page 20
Copyright © 2010. NBCC Job Analysis Report
Page 21
Copyright © 2010. NBCC Job Analysis Report
Page 22
Copyright © 2010. NBCC Job Analysis Report
Page 23
Copyright © 2010. NBCC Job Analysis Report
Page 24
Copyright © 2010. NBCC Job Analysis Report
Page 25
Copyright © 2010. NBCC Job Analysis Report
Page 26
Copyright © 2010. NBCC Job Analysis Report
Page 27
Copyright © 2010. NBCC Job Analysis Report
Page 28
Copyright © 2010. NBCC Job Analysis Report
Page 29
Copyright © 2010. NBCC Job Analysis Report
Page 30
