INSIGHT

Feedback Information   Measuring what pupils know and can do

Contents Feedback that you receive via the Secondary+ secure website will include: Assessment Feedback..............................................................................page 3 Standardised Scores for each pupil Band Profile Graphs Individual Pupil Record Sheets Box Plot Profile of What Children Know and Can Do Cohort Achievement in terms of Key Stage 3 Levels Individual Key Stage 3 Equivalences Attitudes to Science

Contents

Attitudinal Survey Responses

Predictions and Chances Graph Software..............................................page 12 Predictions to GCSE based on each pupil’s INSIGHT performance Subject Chances Graphs, for each subject Pupil Chances Graphs giving a guide to an individual pupil’s likely attainment at GCSE

Value-added Software.............................................................................page 15 Value-added Feedback from MidYIS to INSIGHT (where data are available) Value-added Feedback from INSIGHT to GCSE

Standards Referencing...........................................................................page 21

2

Assessment Feedback This section includes examples of the type of assessment feedback that schools will receive following the INSIGHT Assessments. Assessment feedback contains: • Feedback standardised on the INSIGHT sample • Band Profile Graphs • Individual Pupil Record sheets • Individual Key Stage 3 Equivalences • Cohort Achievement in terms of Key Stage 3 Levels • Attitudes to Science • Attitudinal Survey Responses As INSIGHT uses innovative computer based assessments, your feedback will be available within a short period following the end of the testing window. All your feedback can be downloaded from our secure Secondary+ website using your username and password. The website holds all previous years’ feedback for each school and thus allows schools easily and quickly to replace any mislaid files or to obtain updated files. It also provides full documentation. INSIGHT is part of a suite of monitoring systems established by CEM. Part of Durham University, CEM is recognised around the world for its pioneering educational research and school monitoring. It is a trusted supplier of educational assessments delivering over 1 million tests to pupils each year across 37 nations.

Feedback standardised on the INSIGHT sample INSIGHT assessment results are standardised against a sample of schools who participated in the current year’s assessments. For ease of use, feedback is presented as Microsoft Excel workbooks or PDF documents and are made promptly available from the Secondary+ website. For each pupil there are standardised scores for each assessment area: Reading, Maths, Science and Developed Ability.

What are standardised scores? The results for each INSIGHT assessment are standardised to have a mean of 100 and a standard deviation of 15. That is, the average pupil in the sample will score 100 and the majority of pupils will score between 70 and 130. These scores allow you to judge how well your pupils have performed on the INSIGHT assessments relative to one another, and also how they have performed relative to the sample average.

Assessment Feedback

• Box Plot Profile of What Children Know and Can Do

3

What about Key Stage 3 levels?

Assessment Feedback

Scores are also available as Key Stage 3 levels equivalents by using a simple toggle button on the software.

Why are they useful? Comparison between the Reading, Maths and Science achievement scores and the measure of Developed Ability can highlight pupils who are: • able but who are under performing • lower ability pupils that work hard to achieve above their potential.

Band Profile Graphs Band Profile Graphs are included with the INSIGHT Feedback, for each subassessment plus the overall section score for Reading, Maths, Science and Developed Ability*. How these are calculated • The pupils’ scores for each of the assessments are given a band—A to D (where A is the highest and D is the lowest). • The bands are constructed using quartiles, each band containing 25% of the pupils in the sample. • A pupil scoring in band A has scored in the top 25% of the sample. What these graphs tell us... The graphs show the proportion of pupils falling into each of the four performance bands A – D. They allow schools to see how the pupils in their school performed compared to the INSIGHT sample of pupils and thus provide an overview of the attainments of their own pupils. A school with a profile like the standardisation sample, in terms of average attainment and range of performance, would have 25% of pupils in each band. This is shown by the red broken line on the graphs.

4 * Not included in INSIGHT Core

Monitoring your intake over a number of years helps you to anticipate future expected performance in external examinations. Comparisons between the graphs for achievement and developed ability can show up marked differences and as such will lead to discussion about teaching and learning within your school.

Individual Pupil Records (Iprs) Individual Pupil Records display a personalised spreadsheet, for each pupil, summarising all the INSIGHT assessment information for that pupil.

What the IPR tells us… For each sub assessment and overall score in Reading, Maths, Science and Developed Ability*: • The INSIGHT band (A to D) into which the pupil’s performance falls. • The stanine (one of nine divisions, with 9 being highest) into which the pupil’s performance falls.

Assessment Feedback

Why are they useful?

• The percentile (the percentage of the sample that this pupil performed better than) corresponding to the pupil’s performance. • The pupil’s standardised score. • The pupil’s standardised score in each subject, together with 95% confidence bands.

* Not included in INSIGHT Core

5

Assessment Feedback

An example of an INSIGHT pupil’s Individual Pupil Record is given below.

Why is this useful? The graphical representation of a pupil’s performance on the INSIGHT Assessments allows you to quickly and easily identify any areas for concern, particularly between Reading, Maths and Science but also importantly between these measures and developed ability*. The IPR provides an easy visual way to identify discrepancies in performance which could highlight the able pupil that does not try hard in lessons or conversely the low ability pupil that goes the extra mile.

6

Where the confidence bands do not overlap between assessment sections this indicates that the pupil has performed significantly better or worse in that area * Not included in INSIGHT Core

compared to other areas. A significantly low score on a particular section may mean that intervention is needed in order that the pupil achieves his or her full potential and could have implications for teaching and learning.

Box Plot Profile: What Your Cohort Know and Can Do

Why is it useful? This graph allows you to easily identify any discrepancies in performance and developed ability at the end of Key Stage 3 for your entire cohort. It may highlight areas of concern where results from one assessment appear to be out of kilter with other assessments or developed ability.

Individual Key Stage 3 Level Equivalences Although mandatory Key Stage 3 testing is no longer with us, we appreciate that teachers will still want to gain a feeling for how INSIGHT scores relate to Key Stage 3 levels. For that reason we have developed the two pieces of feedback in terms of Key Stage levels.

Assessment Feedback

The box plot is a simple way of summarising all your students’ scores from your INSIGHT assessments on one graph. It shows the distribution of your students’ scores and allows easy, cohort level comparison between the different test sections. The box plot also shows the median ability within the school and illustrates any outliers (exceptional pupils).

7

Pupil Level Key Stage 3 Level Equivalencies

Assessment Feedback

The first of these graphs shows an individual pupil’s performance on each of the Reading, Maths and Science assessments in terms of Key Stage 3 levels. To aid you further we have broken this down into sub-level, e.g. 5a, 5b, 5c.

8

Why is this graph useful? This information: will support you when arriving at teacher assessed levels for the end of Key Stage 3; will help you in writing reports to parents; and will aid continuity in the change from Key Stage 3 tests to INSIGHT scores.

Cohort-Achievement in terms of Key Stage 3 Levels The second graph shows, for your entire cohort of pupils, their performance on each of the Reading, Maths and Science assessments in terms of Key Stage 3 levels.

Why is this useful? This stacked bar chart, a useful management tool, allows you to easily see the percentage of your pupils who on the INSIGHT Reading, Maths and Science assessments, have achieved in line with each key stage 3 level.

Attitudes to Science

Cohort Attitude to Science Looking across all the questions for a school we are able to provide cohort-level attitudes to Science. The results are calculated as a percentage of the maximum marks available. 50% is equivalent to a pupil giving the middle response for every question (e.g. neither agree or disagree).

Subsection Summary by Pupil This report gives a summary of each pupils’ attitudes to science. The subsection is graded from 1 to 5 where 5 is an extremely positive attitude.

Assessment Feedback

As part of the Science Assessment we collect pupils’ attitudes towards Science. The responses are categorised into four subsections: Perception of Science, Relevance of Science to Self, Enjoyment of Science and Environmental Empathy. The responses are reported at a cohort level against the entire INSIGHT sample (Cohort Attitude to Science) and also at an individual level (Subsection Summary by Pupil). This report is provided as a PDF download.

9

Attitudinal Survey Responses* Basic Comparison Graphs The Basic Comparison Graphs illustrate how the responses of your students to the questionnaire section of the INSIGHT Assessment differ when compared to all INSIGHT students in the same cohort. For each cohort five graphs are produced that are entitled: • Attitudes To School

Assessment Feedback

• Career Choice • Cultural Capital • Job Aim • Likelihood Of Staying In Education These graphs provide both raw and comparative data. A Comparison graph looking at cultural capital may look something like this:

On each comparison graph: •   The questionnaire items are shown on the left of the bar chart. • The number immediately following each questionnaire item is the ‘overall average’ percentage of the whole INSIGHT sample that responded to the items as indicated.

10

•   The number on the right hand side of the graph corresponds to the percentage of your students thats responded to the items as indicated.

* Not included in INSIGHT Core

• The grey shaded background on the graph makes the ‘overall average’ percentages easy to visualise. • The bar graph allows you to visually compare your school’s data with the ‘background’ INSIGHT data. Red bars indicate that your data is significantly different, at the 95% confidence level, to that of the ‘background’ INSIGHT data—otherwise the bars are blue. • The symbol on the bar gives an indication of the uncertainty in the data by showing you whether your data is significantly above or below that of the ‘background’ INSIGHT data.

Assessment Feedback

In short, a school’s data is shown in the foreground as a bar graph which is plotted against a background of the average data for the whole INSIGHT sample. In this way these graphs present simple comparative data—whilst maintaining student confidentiality.

11

Predictions & Chances Graph

Predictions and Chances Graph Software This section includes examples of the type of predictions and chances graph software that you will receive from the INSIGHT Tests. Please note that for predictions to be made from INSIGHT scores, a pupil must have made a valid attempt at all four INSIGHT sections: Reading, Maths, Science and Developed Ability*.

Predictions & Chances Graphs Software The INSIGHT Predictions and Chances Graph Software includes the following: • Personalised Predicted Grades for each pupil in each subject. • These worksheets show, for each pupil, predictions of GCSE grades. Predictions can be displayed as point scores or as grades/levels, for example as 4.3, or as letters, e.g. D/E.

Why is this useful? Predicted grades can be used to aid the target setting process and can be analysed at the pupil, form, subject and cohort level. In addition to the chances graphs (described later), they can be used to monitor a pupil’s progress against the average progress made by pupils of similar ability and to highlight where intervention may be needed.

Personalised Pupil level chances graphs What these graphs show:

12

These example graphs show each individual pupil’s probability of achieving various grades/levels in different subjects at GCSE. * Not included in INSIGHT Core

Pupil level Chances Graph showing GCSE Maths

Why is this useful? Chances Graphs are an excellent way of working with pupils to show what range of grades/levels they are likely to achieve in external examinations. They can be used to: • Warn more able pupils against complacency • Encourage less able pupils to work hard and stay focused

Predictions & Chances Graph

Summary Chances Graph show all GCSE subjects for one pupil. Here all the pupil level chances graphs are summarised on A4 sheets (a set for each pupil) covering the likely subjects at GCSE. You can select which subjects to show depending on what is offered in your school or which subjects the pupil has opted to take.

13

• Show that a predicted grade/level is not a foregone conclusion. Pupils of a given ability ultimately achieve a range of grades/levels.

Predictions & Chances Graph

• Help parents understand the realistic likely achievements of their child.

14

• Illustrate different chances of success in different subjects. • Aid option choices.

Subject-level Chances Graphs These graphs summarise likely overall performance in each subject.

Why is this useful? These graphs can aid subject and departmental target setting.

Value-added Software What is value-added? Value-added measures the amount of progress made by pupils over a certain period of time, compared against pupils of similar abilities in other schools.

Why is value-added useful?

What value-added measures does INSIGHT offer? INSIGHT provides a comprehensive suite of value-added measures, providing data is available. • Progress in lower school, from the MidYIS Test, as applicable to INSIGHT • Progress in the upper school from INSIGHT to the majority of GCSE subjects. The examples shown here are applicable to both the above. Please note: Pupils must have made a valid attempt at all the INSIGHT assessments for us to provide value-added feedback.

Summary Worksheet What this worksheet shows This worksheet provides Value-added information for each pupil in each subject. The example shown here is for MidYIS Year 7 to INSIGHT.

For GCSE value-added analysis, we use the following scale to translate a pupil’s GCSE achieved grade into numerical point score.

Value-added software

If one school increases the achievement level of its pupils more than other schools, then its pupils gain an additional advantage. This relative advantage is what is called Value-added.

15

GCSE Grade Equivalent Point Score

A*

A

B

C

D

E

F

G

U

8

7

6

5

4

3

2

1

0

Value-added software

Below is an example of INSIGHT to GCSE value-added. Although only three subjects are shown here due to space limitations, for GCSE we expect to offer analysis for the majority of GCSE, Short Course or Vocational subjects, where sufficient data allow.

Why is this useful? As the information is displayed in spreadsheet format it can be used to identify individual pupil performance, cohort or class level averages and can assist the user in further analysis. Over and under-performing subjects can be easily identified.

Subject Reports for each subject in INSIGHT or at GCSE: Scatter Graphs and Pupil Level Residuals (PLR) Tables. For each subject, this report allows you to display a scatter graph and a Pupil Level Residuals (PLR) Table.

What these reports show Each Scatter Graph shows the regression line for that subject with your pupils’ results superimposed. A regression line describes the relationship between either:

16

• The MidYIS scores and INSIGHT Assessment scores, or • The INSIGHT Assessment scores and GCSE grades.

Example of Scatter Graph and Pupil Level Residuals (PLRs) Table from INSIGHT to GCSE Business Studies

• The overall INSIGHT Test Score. • The expected performance predicted using the INSIGHT Test score. • The pupil’s actual GCSE result. • The raw residual or value-added, i.e. the difference between the achieved result and the expected result. • The standardised residual. This is the raw residual re-scaled to show its position in the whole sample, allowing fair comparison to be made between subjects and over time.

Value-added software

And the table shows:

Why is this information useful? The scatter graph provides a visual overview of your pupils’ Value-added results. • Pupils plotted above the line have performed better in that particular subject than expected, given their INSIGHT Test score. They have made more progress relative to other pupils of similar ability therefore achieving positive Value-added.

17

• Pupils who fall on the line have performed exactly as we would have expected given their INSIGHT Test score, these pupils achieved zero Valueadded. • Those pupils under the line have negative Value-added and have made less progress of those of similar ability.

Value-added software

Value-added Summary Graph

18

This graph shows the average raw residual for each GCSE subject analysed for your school. It shows the average raw residual (difference between predicted and actual performance) for each subject analysed. It shows how well pupils in each subject performed in comparison with similar pupils in the same subject in other schools. A similar graph is available for MidYIS to INSIGHT Reading, Maths and Science Assessments.

Why is this useful? A positive score means that, on average, pupils in your school did better in a subject than pupils of a similar ability in the rest of the INSIGHT sample. By the same token, a negative score means that, on average, your pupils did not do as well in that particular subject when compared to pupils of similar ability who took the subject.

Average Standardised Residual Graph Standardised residuals are raw residuals (or value-added) rescaled so that all measurements are presented on the same, fair scale. They allow comparisons to be made fairly between subjects and from year to year. Examples of Average Standardised Residuals Graphs: a)  Value-added: MidYIS to INSIGHT Average Standardised Residuals by Subject MidYIS Year 7 2006/2007 to INSIGHT Assessments 2009

1.5

Average Standardised Residual

1 0.6 0.5

0 -0.1 -0.5 -0.7 -1

-1.5 Overall Science

Overall Reading

Overall Mathematics

b)  Value-added: INSIGHT to GCSE

Value-added software

What are Standardised Residuals?

19

Why is this graph useful?

Value-added software

This is perhaps the most powerful graph in value-added terms. It allows you to put subjects’ performance into context and question whether any trends (good or bad), are occurring in your pupils’ achievements.

20

• Subjects performing as expected lie completely within the confidence bounds. Subjects may be positive one year and negative the next and are displayed in white. These changes in results are not significant and can be attributed to the chance variation in the different groups of pupils taught each year. Here achievement is roughly in line with predictions. • Subjects performing in excess of expectations are positive, displayed in blue and lie outside the inner confidence bound. Here the performance of pupils in a subject is largely beyond statistical error and pupils are achieving, on average, beyond their predicted grades. • Subjects performing below expectations are negative, displayed in yellow and lie beyond the inner confidence bound. Here the performance in a subject is largely beyond statistical error and pupils are achieving, on average, below their predictions.

Confidence Bounds The two shades of grey behind the bars represent the confidence limits of the data. These confidence bounds allow you to see whether a positive or negative standardised residual is significant or whether it can be attributed to chance variation.

Standards Referencing As a way of helping teachers assess their pupils’ performances INSIGHT provides guidance in the form of three Standards Referencing documents which show the relationship between INSIGHT standardised scores in Reading, Maths and Science and the corresponding Key Stage levels.

These indicators provide teachers and administrators with the detailed guidance needed to assess pupils’ performance on the scales they are used to. By giving both objective measures and the levels that pupils can be expected to achieve they aid the checking of performance against expectations and teacher assessed levels. An example from the Reading Standards Referencing Document:

An example from the Maths Standards Referencing Document:

An example from the Science Standards Referencing Document:

Standards Referencing

By using a pupil’s INSIGHT standardised scores for each of the three assessments you can look up what level has been achieved, and see broad descriptions and examples of what the pupil can be expected to know, understand and do in a variety of topics. In the case of Reading, example passages are shown at each level that a pupil would be able to read with a fair degree of understanding. In Maths and Science, the example items show what a pupil would have an even or better chance of answering correctly.

21

22

notes

notes

23

For more information visit: www.ceminsight.org Email: [email protected] Tel: +44 (0) 191 334 4200/4210 www.cemcentre.org