FSL PROFICIENCY TEST A Discussion Concerning the Possibility of a National Proficiency Certificate Based on a Proficiency Test for Core French

June 2003 Prepared for: CASLT Prepared by: Alina MacFarlane

This document presents a discussion concerning the possibility of a national proficiency certificate based on a proficiency test for core French. It is divided into four sections: Section 1 examines preliminary questions that need to be considered. · · Section 2 explores what is required for the construction of a national test to be used for certification. Section 3 examines existing provincial and other tests and their suitability for the · purposes of a proficiency certificate for core French. · Section 4 outlines recommendations and conclusions for implementation.

2

1.0 Preliminary Questions This section will explore questions which need to be considered before undertaking this project. These questions deal primarily with the general and specific purposes of the test since these purposes affect the format of the test, levels of reliability and validity needed, and implementation. 1.1 General Purpose of the Test Tests are samples of behaviour intended to reflect whether the candidate possesses certain knowledge or to predict whether s/he can perform certain acts. They usually have one of three categories of purpose : 1. evaluation of language proficiency (for placement in language courses, admission to programs, certification etc.) 2. diagnosis of particular areas of strength or weakness in language proficiency 3. evaluation of achievement relevant to a particular instructional unit or program It is assumed that the purpose of the FSL test is to evaluate grade 12 French students’ (grade 11 in Québec) language proficiency to provide information on his/her language skills for a proficiency certificate. A secondary purpose might also be evaluation of achievement in provincial/territorial core French programs. A test is usually constructed with a single purpose in mind. If both purposes (i.e. evaluation of grade 12 French students’ language proficiency and evaluation of achievement in provincial/territorial core French programs) were to be addressed, a dual purpose test (or tests) would need to be constructed. However, Brindley (1998) reports that the distinction between assessing proficiency and assessing achievement is becoming increasingly blurred in a number of ways. Some methods used to assess ongoing achievement are virtually indistinguishable from those used to assess proficiency. This applies particularly to criterion-referenced forms of assessment in which domains of language ability and standards of performance are well-defined, often involving an investigation of the communicative context in question.

1.2 Specific Purposes In addition to the general purpose, purpose must be also defined as a function of a particular set of candidates and score users (decision makers), the uses of the test results and the consequences of these uses. McNamara (1996:92) defines the purpose of a test as “who wants to know what about whom and for what purpose?” The following section will discuss: · candidates for the test; · the uses of the test results; · the consequences of these uses; · decision makers;

Candidates 3

Two questions need to be addressed concerning candidates for the test: 1. Will candidates for this test be only core French students or students from all FSL program-types (core French, extended French, immersion) in Canada ? This test is initially being considered for core French, but there are several different FSL programs in Canada including various forms of immersion, extended French, and core French. The proficiency levels of graduates for each of these programs vary as a function of program differences, particularly the number of hours devoted to language instruction. These proficiency levels would represent various points on a continuum. Given that core French students make up more than 90% of the FSL students in Canada, it may be efficient to test students from all FSL programs rather than excluding immersion. Including all FSL students would provide decision makers and parents, with useful information to determine which program to choose given the time commitment required versus the outcomes attained in each program and provide accurate information to universities/colleges/employers on the skills of FSL students. Testing all FSL students may also reduce misunderstandings and competition between FSL programs by illustrating, for example, that enhancing core French by introducing intensive French at grade 6 will not bring core French students to the same level as immersion students. 2.

Will candidates for this test be all FSL students in Canada or a representative sample of students from FSL programs in all parts of Canada?

There are currently more than 30,000 graduates from FSL programs annually. Implementation will necessitate financial and temporal considerations. If the purpose of the test is to demonstrate the exact proficiency level of each candidate, all students will need to be tested. If the purpose of the test is to demonstrate the approximate proficiency levels attained by students in various FSL programs across Canada, a representative sample will provide that information. Uses of the test results What types of decisions will be made with the results of this test? These uses dictate the type of information, the degree of detail, and the process of implementation. 1.

Will the test be used to determine the proficiency level of individual students?

As mentioned previously, this will determine whether the test is given to a sample of students or to every student. 2.

Will the test be used to compare the proficiency levels of students graduating from different FSL programs?

If students from all FSL programs are to be tested, the test will need to be able to discriminate among a greater number of proficiency levels or two versions (i.e. an easier one for core French students and a more difficult one for immersion students) will need to be prepared.

4

3.

Will the test be used to compare outcomes in different provinces/territories?

It would not be possible to compare outcomes at the present time because of provincial/territorial variations in FSL formats. The provinces/territories will have to agree on performance standards and test format/components if comparisons were to be made. Consequences of these uses What will be the consequences of the uses of the test results? These consequences dictate the type of information and the level of reliability and validity1 needed. 1.

Could a students fail his/her FSL course as a result of this test? Will the test represent part of the student’s final course mark in French?

If so, the test will need to be directly linked to curriculum. Otherwise, it would be difficult to claim that it was a valid test of what had been taught. Depending on the test, it might also have little face validity for students and parents. 2.

Will the test be used to certify the French proficiency level of graduating FSL students for academic and employment purposes?

If this is the case, the certificate will need to have currency for university/college admission and employers and be accurate for predictions of effectiveness in post-secondary study and/or bilingual jobs.

1

Reliability deals with the consistency of measurement. The reliability of an assessment is a measure of the consistency with which the test produces the same result under different but comparable conditions (i.e. Similar students in different provinces obtain similar scores.) Validity is concerned with sample-population representativeness.( i.e. The knowledge and skills covered by the test items should be representative to the larger domain of knowledge and skills language proficiency.)

5

3.

Will the results of this test be used to make decisions about existing FSL programs? Will the test be used for quality control? Could poor results lead to the elimination of a program? Will national funding to various provinces be affected by the results of this test?

If test results are used for politically-charged decisions, validity and reliability will need to be very high to justify these decisions. Decision Makers In addition to the end users of the test results (universities/colleges, employers, government officials, etc.) outlined in the sections above, two additional questions concerning decision makers need to be addressed. 1.

Who will make the decisions about the test?

Due to funding constraints, the initiative for the development of this proficiency test is national. However, education is a provincial/territorial affair and responsibility for program design and program outcomes rests with the provinces/territories. Undoubtedly, decisions will be made by the funders of the project but it is unlikely that the provinces will cooperate if all decisions are made at the national level. 2.

Who will administer the test and decide the proficiency levels?

The FSL test will evaluate language proficiency. Validity and reliability requirements depend on whether the consequences of the test are low stakes decision (e.g. passing a high school language course), or high stakes decisions (e.g. demonstrating that candidates have sufficient ability to be able to use the language in specific areas such as academic study, medicine, or technology). This affects both the time and the cost involved: the higher the stakes involved, the greater the need for a valid and reliable instrument, and trained professional assessors and scorers. 1.3

Summary

As illustrated in the previous sections, the purposes of a test affect its format, levels of reliability and validity needed, and implementation considerations. A project such as an FSL proficiency test must be preceded by consensus building discussions with all stakeholders involved.

6

2.0 Framework for FSL Proficiency Test Once decisions are made about the general and specific purposes of a test, test specifications need to be developed. These are generally based on a curriculum framework (also called benchmarks, bandscales, or language standards) which is composed of: 1. content standards (i.e. an analysis of a target language use situations and the performance(s) which the test is intended to predict), and 2. performance standards (i.e. a descriptive taxonomy of how well candidates know and are able to do what is set out in the content standards.) Such a framework already partially exists for core French. The following sections will describe what already exists for core French and outline several models for what remains to be developed. 2.1

Provincial/Territorial Content Standards A needs analysis was conducted in the preparation of the curriculum framework for the National Core French Study (NCFS). It outlines the overall conceptual framework or “content standards” but this framework was generic. Resulting provincial curricula vary from province to province (CASLT, 1994). Vandergrift (1999) examined programs of study for core French and compared the organization of and commonalities in existing outcomes for all provinces with the idea of proposing national standards for core French. He found that most provincial core French programs are solidly grounded in the NCFS framework (see Table 1). · Six provinces (Alberta, Saskatchewan, New Brunswick, Nova Scotia, Prince Edward Island and Newfoundland) organize their outcomes by the four syllabi of a multidimensional curriculum as advocated by the NCFS. · Ontario organizes its core French outcomes by skill. · Manitoba combines both, incorporating more specific outcomes by skill within some of the syllabi advocated by the NCFS. · British Columbia maintains that its core French program has been inspired by the NCFS but outcomes have been organized by some similar but other different syllabi than those advocated by the NCFS. · Québec has chosen a cognitive model, organizing its new core French program by intellectual operations and grouping the outcomes by skill (Vandergrift, 1999).

7

Table 1. Curriculum Framework by Province* BC/ YK based on NCFS syllabi based on other syllabi

AB/ NWT

SK

MB

x

x

x

ON

NS

PE

NF

x

x

x

x

x

x

x

x

x

based on skills

x

x

based on functions general outcomes

QC NB

x x

x

x

x

detailed outcomes

x

x

?

(*adapted from Vandergrift, 1999) The overall specified outcomes do not differ greatly. Essentially, the programs agree on the competencies that a language learner must demonstrate to be considered proficient in the target language. All programs consider communication to be meaningful and purposeful and that second language programs must prepare students to use the target language for real communication purposes. The extent of this common base would provide a strong foundation for interprovincial cooperation. However, Vandergrift (1999) points out that these common outcomes do not state very precisely what students are able to do with their second language skills nor stipulate how well students can perform the given outcomes. What is missing is the “how well” (the accompanying performance standards). If the proficiency certificate is to be useful to employers, post-secondary institutions and other interested parties, the description of a language learner’s ability (or proficiency level) must be precisely specified and include the performance conditions, and situational constraints in a format which has credibility for everyone involved. 2.2 Models of Performance Standards As outlined above, there is a distinction between content standards (which define “what” a student should be able to do) and performance standards (which define “how well” a should should be able to do them). While existing provincial/territorial curriculum documents describe content standards, none of them describes performance standards (a language learner’s ability to accomplish certain tasks at different levels under specific performance and situational conditions). For example, the Atlantic Provinces Education Foundation describes FSL communication outcomes as follows:

8

Table 2. Description of Atlantic Provinces Education Foundation Key Stage Curriculum Outcomes for Communication. (Nova Scotia Department of Education (1999?) By the end of grade 12, students will have achieved the outcomes for the previous cycles and should also be able to · interact in a classroom where French is the language spoken · participate spontaneously in a conversation · prepare detailed descriptions · express and justify opinions and points of view · engage in a variety of interactive activities · create a variety of questions · summarize the main ideas in a text · process information so as to meet their needs · interpret and respond to texts critically and creatively · produce a variety of texts according to a set of given criteria · narrate events that they have experienced There are however, several examples of performance standards that may be considered as models for Canadian FSL programs. This section will examine three models of performance standards for oral communication which approximate the level which could be expected of grade 12 core French graduates. They have been developed by: · the Public Service Commission · the Centre for Canadian Language Benchmarks · ACTFL Public Service Commission The Public Service Commission has established three second language proficiency levels which describe the language skills necessary to perform the duties and responsibilities required for particular bilingual positions within the Public Service. These levels or standards are ordered from A (lowest) to C (highest) and are organized by skill (oral interaction, reading, writing). The specific outcomes and sample tasks describe the ability level in the context of tasks that a candidate is expected to perform. Table 3 below outlines the ability level for Level B (the middle level of 3) for oral interaction (Public Service Commission, 1993). This is the minimum level in oral interaction for positions that involve departure from routine or repetitive second language use.

9

Table 3. Description of Public Service Commission Oral Interaction Ability for Level B. Level

Skill

Specific Outcomes

Sample Tasks

B

Oral interaction

Can sustain a conversation on concrete topics

Can handle requests for routine information from other employees or members of the public (either by telephone or in face-to-face conversations) about such things as services, publications, or competitions.

May make many errors and deficiencies in grammar, pronunciation, vocabulary, and fluency, but these do not seriously interfere with communication

Can participate in informal meetings or work sessions dealing with such matters as the assignment of work, steps to be followed, priorities, or deadlines.

Cannot be expected to cope with situations that are sensitive or that require the understanding or expression of subtle or abstract ideas

Can give factual accounts to colleagues or investigators of actions taken or events that have occurred.

Limited ability to deal with situations involving hypothetical ideas and the use of persuasion.

Can give and follow instructions or explanations about how work is to be done, what information is needed and what steps or alternatives are to be followed.

The Public Service Commission proficiency levels account for factors affecting language performance such as context (i.e. work-related), performance conditions (i.e. face-to-face), and constraints (i.e. may make many errors ...). However, these proficiency levels are highly contextualized according to specific work tasks performed by public servants which high school students would not need to know, are not relevant to their lives, and are not at all interesting for them. Core French standards would need to be more generic describing competencies required for daily living and leisure activities or pursuing post-secondary studies. Centre for Canadian Language Benchmarks Canadian Language Benchmarks are language standards developed specifically for adult immigrant learners by Citizenship and Immigration Canada. The benchmarks are grounded in the communicative competence framework (Canale & Swain, 1980), with the goal of describing linguistic, discourse, sociolinguistic and strategic competence for three skill areas: 10

speaking/listening, reading and writing. These benchmarks are grouped into three levels or stages of proficiency: basic, intermediate and advanced. Each of these stages of proficiency is further divided into four sub-levels of developing competence; initial, developing, adequate and fluent resulting in twelve benchmarks for each skill. The following general description and Table 4 below outlines the ability level for Benchmark 6, the developing competence level in intermediate oral communication (Ministry of Supply and Services Canada,1996). General description: · Learner can communicate in less predictable contexts in common daily situations related to needs (within familiar topics but with some unfamiliar vocabulary). · Follows main ideas in simple formal and informal conversations face-to-face, understands and uses everyday vocabulary and a limited number of idioms. May avoid topics (gaps in vocabulary). · Uses a variety of structures. Accuracy may be reduced; grammar and pronunciation errors are frequent and may sometimes impede communication. Discourse is reasonably fluent, of moderate length. · Information content is complex, with detail and requires complex questioning and careful listening. · Discourse and social skills are required for successful communication. · Uses phone on familiar matters. Table 4. Listening/Speaking Benchmark 6: Developing competence in intermediate oral communication What The Person Can Do

Performance Conditions

Situational Conditions

1. Instructions Give spoken instructions.

Gives clear spoken instructions as required. Uses imperatives, statements, requests; gives a sequence of steps in order, uses sequence markers.

Context strongly supports the utterance.

2. Social interaction Apologize, make excuse. Express and respond to disappointment. Accept offer.

Responds appropriately to and initiates apology, excuse. Expresses and responds to disappointment. Accepts offer. (I am sorry about..., not able, cannot, could not, thank you, great, would)

3. Information Relate a short familiar event. Give information about own educational background. Describe characteristics/strengths required for people in different social roles. Give information and ask for information to make or cancel appointment/express obligation, ability, inability, certainty. Explain what isn’t/wasn’t known.

Describes clearly/intelligibly a sequence of events in the past on a topic related to personal life which includes obligation, ability, certainty; explains details. Uses past tense inflection on many familiar verbs, conjunctive links (sequence markers), adverbs of time, frequency and duration, correct simple structures with some omission/reduction of elements (e.g. articles, past tense morphemes). Hypothetical future condition and time clauses appear but are often incorrect. Uses comparative structures and ellipsis. Describes personal characteristics/strengths. Gives and aks information on the phone as required in clear predictable discourse.

Indicate problems in communication.

11

Interaction is face to face or on the phone. Speech is slow, context is clear and predictable. Psychological stress is usually moderate in face to face exchanges (friends, counsellors, teachers, receptionists);learner can partially prepare the exchange.. Phone exchanges with strangers are stressful for the learner. Topics may include health, education, careers and occupations, job search, banking, and other consumer services. * May not know spelling conventions on the phone (e.g “A”as in Alpha, “B”as in Bravo).

(Have to able, sure, sorry, I don’t know, if, she doesn’t like it but he does).

4. Suasion (getting things done) Give suggestions/advice. Predict consequences.

Clarifies meaning, paraphrases. (Did you say “...? Can you repeat/explain this? Do you mean ...?) (Should, shouldn’t, had better, you can...)

Sample tasks: · C (Community) 1.Explain how to make/do something properly, give a short set of instructions (i.e. change the light, make soup, register a car). 3. Call to make or cancel a medical appointment. Give information about medical history (e.g. surgery, illness, allergies, medication). Ask and answer questions in a parent/teacher interview. Tell about past experience. · S (Study) 3. Describe a good parent, student, friend, employee, supervisor. Call to report absence, illness. Lead a simple classroom discussion, e.g. on similarities and differences between people, places, jobs, events, etc. · W (Work) 3. Make an appointment for a job interview. Prepare a speech about your personal qualities and work experience for a job interview.

The Canadian Language Benchmarks’ (comparable document recently compiled for French) descriptor for each area of competence includes communication skills, sociolinguistic skills and strategy use. The performance conditions state the minimum requirements in terms of success of communication and quality of communication (effort, accuracy, and appropriateness). The situational conditions provide the contextual limitations and any required referential information. Finally, sample generic tasks provide an even clearer picture of the competencies that the language learner can perform. The Canadian Language Benchmarks would provide a good model for national standards for FSL although sample tasks may not be age-appropriate.

The ACTFL Performance Guidelines for K-12 Four national US language organizations (ACTFL, AATF, AATG, AATSP) developed standards for foreign language education, grades K-12 in 1996. In 1999, these generic standards were republished and complemented by 9 language-specific standards including French. They answer the question of “what” should be taught in American foreign language classrooms in the same way the NCFS answered the question of “what” should be taught in core French classrooms. In addition to the language standards , ACTFL Performance Guidelines for K-12 (November 1999) were written primarily to describe “how well” should students be expected to do the “what” of the content standards. They describe the language of students who begin language study in kindergarten and continue through grade 12 (elementary: 3-5 days per week, 30-40 min. per day; middle school: daily, 40-50 min. per day; high school: 4 units of credit). Following the organizing principle of Standards for Foreign Language Learning, they are divided into three Modes of Communication (Interpersonal, Interpretive, and Presentational) for three Benchmark Levels (Novice Learner (K-4, 5-8, 9-10), Intermediate Learner (K-8, 7-12), and Pre-Advanced Learner (K-12) for five Domains of Performance: · Comprehensibility (How well is the student understood?) · Comprehension (How well does the student understand?) · Language Control (How accurate is the student's language?) · Vocabulary Usage (How extensive and applicable is the student's language?) 12

· ·

Communication Strategies (How do they maintain communication?) Cultural Awareness (How is their cultural understanding reflected in their communication?) Developers of the standards viewed the use of language “modes”as a richer and more natural way of envisioning communication than the traditional four skills of listening, speaking, reading and writing. Looking at these language modes places primary emphasis on the context and purpose of the communication rather than concentrating on any one skill in isolation. Division into these modes of communication together with the five domains of performance makes the US approach very similar to the multidimensional syllabi of the NCFS (see Table 5). Although the three modes of communication (interpersonal, interpretive, and presentational) make it more complicated, what is assessed covers all dimensions of a multidimensional curriculum grounded in the NCFS.

Table 5. Comparison of NCFS Syllabi and ACTFL Performance Guidelines K-12 NCFS syllabi

ACTFL

Modes

communication

comprehensibility (How well are they understood?) comprehension (How well do they understand?)

Interpersonal Presentational

culture

cultural awareness (How is their cultural awareness reflected in their communication?)

Interpersonal Interpretive Presentational

general language education

communication strategies (How do they maintain communication?)

Interpersonal Interpretive Presentational

language

language control (How accurate is their language?)

Interpersonal Interpretive Presentational

vocabulary use (How extensive and applicable is their vocabulary?)

Interpersonal Interpretive Presentational

Interpersonal Interpretive

Table 6 presents the performance guidelines for the pre-advanced learner who has studied the 13

second language from K to grade 12 for comprehensibility (oral communication)in the interpersonal and presentational modes. Table 6. Performance Guidelines for the Pre-Advanced Learner (K -12) for Comprehensibility PRE-ADVANCED LEARNER Grade K-12 COMPREHENSIBILITY

Interpersonal

How well are they understood? ·

·

·

· ·

Presentational

·

·

· ·

narrate and describe using connected sentences and paragraphs in present and other time frames when interacting on topics of personal, school and community interest; are understood by those with whom they interact, although there may still be a range of linguistic inaccuracies, and on occasion, the communication partner may need to make special effort to understand the message; use pronunciation and intonation patterns that are understandable to a native speaker unaccustomed to interacting with language learners; use language confidently and with ease, with few pauses; are able to meet practical writing needs such as letters and summaries by writing descriptions and narrations of paragraph length and organization, showing sustained control of basic structures and partial control of more complex structures and time frames. report, narrate and describe, using connected sentences, paragraphlength and longer forms of discourse, in oral and written presentations on topics of personal, school, and community interest; use pronunciation and intonation patterns that are understood by native users of the language, although the listener/reader may on occasion need to make a special effort to understand the message; use language confidently and with ease, with few pauses; communicate with a fairly high degree of facility when making oral and written presentations about familiar and well-researched topics.

The ACTFL proficiency guidelines go further than Canadian provincial outcomes in describing “how well” students are expected to execute the expected outcomes. Although the descriptions are not as detailed as those of the Canadian Language Benchmarks, quality of performance is considered. Given that the criteria for language performance on the ACTFL proficiency guidelines are congruent with most Canadian core French programs, these guidelines would supply the needed “performance standards” to complete the content standards of the 14

NCFS and would thus serve as a framework for test construction.

In addition, ACTFL has prepared a visual representation which graphically illustrates the influence of time on language performance ability and shows what ability is reasonable to expect of students who begin foreign language study at various points in the K-12 spectrum. This chart also indicates the relationship between the ACTFL Performance Guidelines for K-12 Learners and the regular ACTFL Proficiency Guidelines which have currency throughout the world. Preadvanced is considered the threshold (Intermediate High) level. After this level the regular ACTFL guidelines can be used through to Superior. (see Figure 1).

15