6/18/2014

The Garden Analogy

Developing Rubrics to Support a Differentiated Classroom Catherine Brighton, Ph.D. University of Virginia [email protected]

Place logo or logotype here, otherwise delete this.

If we think of our children as plants …   Formative assessment is the equivalent of feeding and watering  the plants appropriate to their needs ‐ directly affecting their  growth. Summative assessment, on the other hand, of the plants is the  process of simply measuring them. It might be interesting to  compare and analyze measurements but, in themselves, these do  not affect the growth of the plants.

1

6/18/2014

Summative Assessment Summative Assessments are given periodically  to determine at a particular point in time what  students know and do not know. end of several lessons; end of unit

Some Examples of Summative  Assessments: * State assessments (e.g., ISAT) * District benchmark or interim       assessments  (e.g., MAP) * End‐of‐unit or chapter tests * End‐of‐term or semester exams * Scores that are used for accountability  for schools (AYP) and students  (report card grades).

2

6/18/2014

Key to Summative Assessment Think of summative assessment as a means to  measure, at a particular point in time, student  learning relative to determined  content  standards. Although the information that is  gathered from this type of assessment is  important, it can only help in evaluating  certain aspects of the learning process.

Disadvantages of Summative  Assessment Because they are spread out and occur after  instruction every few weeks, months, or once a  year, summative assessments are tools to help  evaluate the effectiveness of programs, school  improvement goals, alignment of curriculum, or  student placement in specific programs.

3

6/18/2014

Summative Assessment: A Coarse  Sieve Summative assessments happen too far  down the learning path to provide  information at the classroom level and to  make instructional adjustments and  interventions during the learning  process….whereas Formative assessment  is a fine sieve

A Balanced Act When assessment at the classroom level  balances formative and summative  assessment, a clear picture emerges of where  a student is relative to learning targets and  standards. 

4

6/18/2014

Clapping Institute The Clapping Institute

5

6/18/2014

I Need Volunteers…  • 5 Clappers • 3 Judges • 1 Recorder

RUBRICS Can be used  formatively and summatively

6

6/18/2014

Scales vs. Rubrics Below Meets Exceeds Expectation Expectation Expectation Citations

X

7

6/18/2014

Scales vs. Rubrics

Refrigerator – Rubric example 4

Entire refrigerator is sparkling and smells clean. All items are fresh, in proper containers (original or Tupperware, with lids), and organized into categories

3

Refrigerator is generally wiped clean. All items are relatively fresh, in some type of container (some Tupperware lids are missing or don’t fit) and are sitting upright

Below Meets Exceeds Expectation Expectation Expectation Citations

The paper fails to cite sources using a consistent, formal, citation style

Most follow a consistent style, although some contain minor errors or incomplete information

All citations are complete, accurate, and consistently conform to a formal style

8

6/18/2014

2

1

Some of the shelves are wiped clean, although there are some crusty spots. There are some suspicious smells. Items are in containers, but there seems to be some green stuff growing in some of the Tupperware Items stick to the shelves when they are picked up. The smells linger long after the refrigerator door is closed. Several items need to be thrown out— Tupperware and all

Questions Answered by Rubrics • By what criteria will the work be judged? • What is the difference between exemplary  work, good work, and weaker work? • How can we ensure judgments are valid and  reliable? • How can both students and teachers focus  their preparation on excellence?

9

6/18/2014

Advantages using Rubrics • Teachers can increase the quality of their  instruction by providing focus, emphasis and  attention to particular details as a model for  students • Students have explicit guidelines regarding  expectations • Students can use rubrics as tools to develop their  abilities • Consistency of grading across multiple activities

Analytic and Holistic Rubrics • An analytic rubric articulates levels of performance  for each criterion so the teacher can assess student  performance on each criterion.  • In contrast to an analytic rubric, a holistic rubric does  not list separate levels of performance for each  criterion. Instead, a holistic rubric assigns a level of  performance by assessing performance across  multiple criteria as a whole. 

10

6/18/2014

Analytic Rubric has 4 Important  Elements • • • •

Dimension Definition and example of dimension (criteria) Scale (Expert to Novice continuum) Standards of Excellence (what makes quality  rubrics)

Holistic Or Analytic? HOLISTIC—pros and cons +Takes less time to create. Well…  +Effectively determines a “not fully developed” performance as  a whole +Efficient for large group scoring; less time to assess ‐ Not diagnostic ‐ Student may exhibit traits at two or more levels at the same  time.

11

6/18/2014

Holistic Or Analytic? Analytic=Separate facets of performance are  defined, independently valued, and scored. Example: Music Facets scored separately: melody; harmonics;  rhythm; bowing & backup; confidence

Create an Analytic Rubric for a  Child’s Bedroom 1. With your group, determine the  purpose of evaluating children’s  bedrooms 2. Identify relevant domains for this  purpose 3. Create levels of each domain you  identified 4. Be prepared to share with the larger  group

12

6/18/2014

Alex, 9, Rio de Janeiro, Brazil

Rhiannon, 14, Darvel, Scotland

13

6/18/2014

Dong, 9, Yunnan, China

Risa, 15, Kyoto, Japan

14

6/18/2014

Lamine, 12, Bounkiling Village, Senegal

Tzvika, 9, Beitar Illit, The West Bank

15

6/18/2014

Joey, 11, Kentucky, United States

Netu, 11, Kathmandu, Nepal

16

6/18/2014

Indira, 7, Kathmandu, Nepal

Think about something you teach: Identify the purpose of the rubric Identify needed domains to consider Sketch out one domain’s levels

17

6/18/2014

Tip #1 • Don’t make task‐specific rubrics – Efficiency issue – “Generalizable” or template rubric

Tip #2 • Don’t use generic or “canned” rubrics without  careful consideration of their quality and  appropriateness for the assessment. • These are your students, not someone else’s. • Your students have received your instruction.

18

6/18/2014

Tip #3 • Avoid dysfunctional detail. – “…in most instances, lengthy rubrics probably can  be reduced to succinct…more useful versions for  classroom instruction.  Such abbreviated rubrics  can still capture the key evaluative criteria needed  to judge students’ responses.  Lengthy rubrics, in  contrast, will gather dust” (Benjamin 23).

Tip #4 • Limit the number of criteria – Well… – Don’t combine independent criteria.   • “very clear” and “very organized” (may be clear but not  organized or vice versa).

‐‐Includes wordiness, jargon, negativity

19

6/18/2014

Tips #5 • Use key, teachable “criteria” (What counts) – Don’t vaguely define levels of quality.  – Concrete versus abstract • “poorly organized” (Organization: sharply focused  thesis, topic sentences clearly connected to thesis,  logical ordering of paragraphs, conclusion ends with  clincher) • “inventive” “creative” “imaginative” UNLESS… Key Question to ask yourself:  What does it look like?

Tips #6 • Use measurable criteria. ‐‐Specify what quality or absence looks like vs. comparatives (“not as thorough as”) or value language (“excellent content”) ‐‐‐Highlight the impact of the performance  ‐‐Was the paper persuasive or problem solved?  (Note importance of PURPOSE) ‐‐What are the traits of effective persuasion? ‐‐Be sure that the descriptor is not the criterion and vice  versa

20

6/18/2014

Tip #7 • Aim for an even number of levels – Create continuum between least and most – Define poles and work inward – List skills and traits consistently across levels

Tip #8 • Include students in creating or adapting  rubrics • Consider using “I” in the descriptors • I followed precisely—consistently—inconsistently— MLA documentation format. • I did not follow MLA documentation format.

21

6/18/2014

Tip #9 • Motivate students to use rubric. Instructional rubric (“Buy one, get one…”) “At their very best, rubrics are also  teaching tools that support student  learning…”  (Andrade 13). • Do they understand the criteria and descriptors? How do you  know? • When do you give the rubric to your students?

Tip #10 • Provide models of the different performance  levels.

22

6/18/2014

It’s hard work… • Expect to revise…and revise… – One problem is that the rubric must cover all potential performances;  each should fit somewhere on the rubric.

• “There are no final versions, only drafts and deadlines.”

Step 1 in developing a rubric 1.  Clearly define the assignment.    What is the student expected to produce?   What are the common expectations across  instructors?  

• When you’ve got a good one, SHARE IT!

23

6/18/2014

Step 2 in developing a Rubric

Step 3 in developing a Rubric

2.  Consider what student learning outcomes will be  assessed.

3. Determine the key criteria that you are interested in‐

 Often with a culminating project, students are  expected to demonstrate several of the  department/program outcomes.   For example, for a senior seminar paper, outcomes  related to writing and critical thinking may be  assessed, as well and more discipline‐specific  outcomes.

‐ for example, for the senior seminar paper, what  aspects of writing will be assessed?

 Coherence  Organization  Mechanics

24

6/18/2014

Step 4 in developing a Rubric 4. Clearly define those key criteria.   What do you mean by organization?  What does organized writing look like?

Step 5 in developing a Rubric 5. Establish clear and detailed descriptors for each  performance level for each criteria    

Determine what the different levels of performance look like  within each criteria Use sample papers of high, mid and low performers to help It is usually easiest to begin by describing the highest level of  performance Using specific language for the descriptors of performance  levels increases the chances that multiple faculty members  will apply the rubric in a similar manner. 

25

6/18/2014

Step 6 in developing a Rubric 6. Try out the rubric on a few students with several  raters to see if the rubric works and gets  consistent scoring from multiple raters.

First Step: Identifying the Enduring Understanding

• Students will understand that the interactions  at plate boundaries influence the likelihood,  strength, and extent of volcanic eruptions

26

6/18/2014

Goal: Why is this Important? Step 2: Connecting the Understanding to Standards

• The purpose behind the performance • Students will apply an understanding of a  volcanic eruption related to a plate  boundary to predict the likelihood and  extent of other similar volcanic eruptions

– To reduce the loss of life and damage to property by  accurately predicting when and to what extent, a volcanic  eruption might affect Mt. Shasta.

27

6/18/2014

Step 3: Identifying who does this  in real life?

Ideas for Professional Roles and  Situations Through Which Students  Can “Perform with Knowledge”

28

6/18/2014

Museum Curator • Examples:  – Design museum exhibits – Create museum exhibit brochure – Compete for grant money with other designers

• Ad agency director Design advertising campaign or book jackets

• Psychologist/sociologist Conduct surveys, perform  statistical analyses, graph  results, write newspaper  articles on the meaning of results

29

6/18/2014

• Policy Analyst • Archaeologist Determine the culture or time frame of a mystery artifact  or person Create documentary script on given time period based on  artifacts

• Newspaper editor or writer Write articles and editorials set in the studied historical  time

Predict future in a country being studied

• Product Designer/Campaign Manager Conduct research, design ad campaign,  present proposal to panel

• Expert Witness to Congress Testify on behalf of or against advertising  claims, regulations, children’s TV, current  policy issue

30

6/18/2014

Role • The job undertaken in the scenario

Step 4: Identifying the Audience

• The group to which the performance or  product is targeted

• In our scenario…. • In our scenario Students are put in the position of an urban  planner.

To present their plan to city council

31

6/18/2014

Step 5: Creating the Situation

Step 6: Product or Performance

• The context of the scenario – You have been asked to develop and  present a risk assessment and  mitigation plan for the city of Mt.  Shasta.

• What is produced as a result of the project

CREATIVE PROBLEM GENERATION to ENGAGE STUDENTS in the  PROCESS of LEARNING.

ACID TEST: Would students want to do this even if it were not tied to standards and part of the curriculum?

Acid Test: Would students want to do this even if it  were not tied to standards?

In our scenario Prepare a PowerPoint presentation that includes a  map and other visuals. Generate slide pages for  the audience.

32

6/18/2014

Task Vignette You are an urban planner with the city of Mt. Shasta in California. You have been asked to develop and present a risk assessment and mitigation plan in the event that Mt. Shasta should erupt. You hope to reduce the loss of life and property damage by accurately predicting when, and to what extent, a volcanic eruption might affect Mt. Shasta. You will need to prepare a plan that you present to the city council. Present your plan in the form of a PowerPoint presentation. Be sure to inform the public with critical background information by telling the geologic story of Mt. Shasta. Include a research-based prediction of when the next eruption might occur. Prepare a map that shows the areas most likely to be affected by an eruption. Describe your emergency response plan based on the predicted flow of the path of the lava. In addition to maps, it is fine to use additional visuals as needed, as long as they support the presentation. Your presentation should be clear, compelling, and scientifically accurate.

Step 7: Standards for Success • Criteria for assessing the product (Developing  Rubrics) In our scenario: • Be sure your presentation: – Informs the public by telling the geologic story of Mt. Shasta – Includes a research‐based prediction of when the next eruption  might occur – Has a map of the areas most likely to be affected by an eruption  along with other visuals that support the presentation – Includes an emergency response plan based on the predicted flow  path of the lava – Is clear, compelling, and scientifically accurate

33

6/18/2014

The Equalizer 1. Foundational

Transformational Information, Ideas, Materials, Applications

2. Concrete

Abstract Representations, Ideas, Applications, Materials

3. Simple

Complex Resources, Research, Issues, Problems, Skills, Goals

4. Single Facet

Multiple Facets

A Differentiated Authentic Assessment

Directions, Problems, Application, Solutions, Approaches, Disciplinary Connections

5. Small Leap

Great Leap Application, Insight, Transfer

6. More Structured

More Open Solutions, Decisions, Approaches

7. Less Independence

Greater Independence Planning, Designing, Monitoring

8. Slow

Pace of Study, Pace of Thought

Quick

34

6/18/2014

The Renovation Guide Purpose/Rationale The purpose of this task is to:  • assess students’ understanding of chemical  and physical changes  • help students make connections between  scientific concepts and real‐world products  and processes 

Objectives Students will demonstrate their ability to… • research products, chemicals, and tools to determine their intended use. • classify products as physical or chemical change agents. • explain the processes of physical and chemical changes in the context of common products, chemicals, and tools.

35

6/18/2014

Objectives • analyze the pros and cons of using particular products (in terms of individual safety, product efficiency, environmental risks, etc.). • communicate scientific information to a specific audience in visual, written, and/or verbal formats.

Overview  (Prompt 1) Students work as product designers for a company that  specializes in developing products to aid in household  renovations. The task is to design a kit that contains an effective and  efficient combination of products and tools‐‐ some that  perform their functions by causing chemical changes in  matter, and some that perform their functions by  causing physical changes in matter. 

36

6/18/2014

Overview  (Prompt 3) Students work as the director of the city’s  beautification department.  The task is to complete a cost‐benefit analysis of the  chemicals & tools. In the analysis, the student  must consider not only material costs and  benefits, but also costs and benefits to the  environment and to the individual.

Openendedness Prompt 3 is more open ended than  prompt 1 or 2 because it does not  specify the problems/situations that  the student should investigate. It gives  the student the freedom to choose  issues of his/her interest.

37

6/18/2014

Structure

Information

Prompt 3 also gives fewer directions.  The student has to identify ideas of  where to look for help or information,  while Prompt 1 directs the student to  the information he/she needs to  complete the task.

Prompt 3 gives less information to  students.  Prompt 1 tells students which part of the  product causes the changes and what  specific things to look for.

38

6/18/2014

Multifaceted/Complexity Prompt 3 is multifaceted. It requires that the  student research more areas (like public  health, environment, economics, etc.) and  make more disciplinary connections among  them  than prompt 1 or 2. Prompt 3 may  also involve more complex data collection  by involving interviews.

Transformational & Abstract Prompt 3 is more transformational and abstract than  prompts 1 and 2 because it requires that students use  their knowledge and ideas in a more unfamiliar setting;  it involves them in a decision making process; it  demands that it is presented in a professional way. It  also requires a more sophisticated product that  involves not only good writing but also  speaking/oratory skills to convey and convince others  of their message. Prompt 1 is more concrete because  the final product is mostly a summary of their findings.

39

6/18/2014

The Renovation Guide

The Renovation Guide • Summary

• The Equalizer: – Abstract: requires mental manipulation and is  idea based. – Great Mental Leap: requires insight, many  unknowns, flexible thought processes. – Complex: combines ideas, emphasizes  elegance and originality.

– Interdisciplinary – Challenging Objectives – Equalizer: Abstract, Great Leap, Complex – Makes real‐life connections – Professional products

40

6/18/2014

Clapping Rubrics Volume

5 – Clapper carefully controls the volume of his/her clapping, taking all or many relevant factors into consideration

4 – Clapper does a good job of controlling his/her clapping volume, taking a few relevant factors into consideration

3 – Clapper does an average job of controlling his/her volume, taking at least one relevant factor into consideration

2 – Clapper appears to minimally control his/her volume, taking no relevant factors into consideration

1 - Clapper appears to be completely oblivious to the importance of his/her volume

Clapping Rubrics Appropriateness

5 – Clapper’s style is excellent and completely appropriate, based on the described setting.

4 – Clapper’s style is mostly appropriate, based on the described setting.

3 – Clapper’s style is moderately appropriate, based on the described setting.

2 – Clapper’s style is inappropriate for the described setting.

1 - Clapper’s style is completely inappropriate for any setting.

41

6/18/2014

Clapping Rubrics Creativity

5 – Clapper demonstrates a style that, based on the described setting, is so creative it influences or engages other observers in some way.

4 – Clapper demonstrates a style that, based on the described setting, is fairly creative, but make no visible impact on the observers.

3 – Clapper demonstrates some creativity in his/her clapping style, though it is not based on the described setting.

2 – Clapper demonstrates very little creativity in his/her clapping style.

1 - Clapper demonstrates absolutely no creativity in his/her clapping style.

42