IEA Secretariat Herengracht 487 1017 BT Amsterdam, The Netherlands Tel: +31 20 625 3625, Fax: +31 20 420 7136 Email: [email protected]

MINUTES OF THE 55TH GENERAL ASSEMBLY MEETING OF THE INTERNATIONAL ASSOCIATION FOR THE EVALUATION OF EDUCATIONAL ACHIEVEMENT DOCUMENT No. IEA-HQ/GA/14



13–16 OCTOBER 2014 Vienna AUSTRIA 

HOSTED BY AUSTRIAN FEDERAL MINISTRY OF EDUCATION AND WOMEN’S AFFAIRS (BMBF)

Minutes of the 55th IEA General Assembly Meeting 2 ______________________________________________________________________________________________________________

LIST OF PARTICIPANTS (GA-55/B) GA Representatives 1. Australia 2. Austria 3. Belgium (Flemish) 4. Belgium (French) 5. Botswana 6. Canada 7. Chile 8. Croatia 9. Cyprus 10. Czech Republic 11. Denmark 12. England 13. Estonia 14. Finland 15. France 16. Georgia 17. Germany 18. Hong Kong SAR 19. Hungary 20. Iceland 21. Indonesia 22. Iran 23. Italy 24. Japan 25. Jordan 26. Korea 27. Kuwait 28. Latvia 29. Lithuania 30. Luxembourg 31. Malaysia 32. Mexico 33. Morocco 34. Netherlands 35. New Zealand 36. Norway 37. Philippines 38. Poland 39. Portugal 40. Qatar 41. Russian Federation 42. Singapore 43. Slovak Republic 44. Slovenia 45. South Africa 46. Spain 47. Sweden 48. Thailand 49. United Arab Emirates 50. United States

i.a. Sue Thomson Mark Német, SC Isabelle Erauw Dominique Lafontaine Brian Mokopakgosi Pierre Brochu i.a. Gabriela Cares i.a. Michelle Braš Roth Elena Papanastasiou i.a. Josef Basl Hanne Løngreen Lorna Bertrand Anu Toots Jouni Välijärvi Thierry Rocher i.a. Natia Mzhavanadze i.a. Rainer Lehmann Frederick Leung Sándor Brassói, SC Sigurgrímur Skúlason Furqon Ebrahim Talae Roberto Ricci Fumi Ginshima Abdalla Yousef Awad Al-Ababneh Sung Hoon Kim Ridha Al-Khayat Andris Kangro Rita Dukynaitė Amina Kafai i.a. Zainal Aalam Hassan Gilberto Guevara Niebla Mohammed Sassi Paul van Oijen Megan Chamberlain Annette Qvam Josette Biyo i.a. Michał Sitek Hélder Diniz de Sousa i.e. Saada Al-Obaidli Galina Kovaleva Chew Leng Poon, SC Romana Kanovská Eva Klemenčič i.a. Vijay Reddy i.a. Francisco Javier García Crespo Camilla Thinsz Fjellström Piniti Ratananukul Ali Mehad Alsuwaidi, SC Dana Kelly, SC

IEA Officers 51. Roel Burgers, Secretariat

______________________________________________________________________________ 13–16 October 2014

Minutes of the 55th IEA General Assembly Meeting 3 ______________________________________________________________________________________________________________

52. Dirk Hastedt, Executive Director, TEG Chair 53. Anne-Berit Kavli, Chair, SC 54. Paulína Koršňáková, Secretariat 55. Heiko Sibberns, DPC, TEG 56. Hans Veenhuys, IEA Legal Advisor 57. Alana Yu, Secretariat 58. Catherine Zwaneveld, Secretariat IEA Publications and Editorial Committee (PEC) 59. Tom Loveless 60. David Robitaille, PEC Chair, Honorary Member IEA Technical Executive Group (TEG) 61. Henry Braun, Boston College, United States 62. Jostein Ryssevik, Ideas2evidence, Norway 63. Norman Verhelst, Eurometrics, Netherlands International Coordinators 64. Ralph Carstens, ICCS 65. Julian Fraillon, ICCS, ICILS 66. Steffen Knoll, ECES 67. Michael Martin, PIRLS, TIMSS, TIMSS Advanced, TEG (ex officio) 68. Ina Mullis, PIRLS, TIMSS, TIMSS Advanced, TEG (ex officio) 69. Wolfram Schulz, ICCS, ICILS Honorary Members 70. Seamus Hegarty 71. Kimmo Leimu 72. Jack Schwille Observers 73. Fatma Al Marri (KHDA, UAE) 74. Shaikha Ali Al Zaabi (ADEC, UAE) 75. Elsebeth Aller (Ministry of Education, Denmark) 76. Ayesha Almerri (Ministry of Education, UAE) 77. Khawla Almualla (Ministry of Education, UAE) 78. Ameena Abdulla Alobaidli (Supreme Education Council, Qatar) 79. Emin Amrullayev (Ministry of Education, Azerbaijan) 80. Newman Burdett (NFER, England) 81. Ahmed Chaibi (Ministry of National Education, Morocco) 82. Kyong Hee Chon (KICE, Korea) 83. Gabriela Fröhlich (BMBF, Austria) 84. Lilia T. Habacon (DOST, Philippines) 85. Myung Ae Lee (KICE, Korea) 86. Jacques Mapel (CONFEMEN–PASEC, Senegal) 87. Nizam (Center for Education Assessment, Indonesia) 88. Claudia Paller (BMBF, Austria) 89. David Rutkowski (Indiana University, United States) 90. José María Sánchez-Echave (INEE, Spain) 91. Claudia Schreiner (BIFIE, Austria) 92. Prapa Tantasuparuk (Office of the Education Council, Thailand) 93. Christina Wallner-Paschon (BIFIE, Austria) 94. Carole Willis (NFER, England)

______________________________________________________________________________ 13–16 October 2014

Minutes of the 55th IEA General Assembly Meeting 4 ______________________________________________________________________________________________________________

95. Ruangrat Wongpramote (Office of the Education Council, Thailand) Speakers 96. Sigrid Blömeke (University of Oslo, Norway) 97. Svein Lie (University of Oslo, Norway) 98. Helge Lund (Norwegian Directorate for Education and Training, Norway) 99. Albert Motivans (UNESCO Institute For Statistics) 100.Ingrid Munck (University of Gothenburg, Sweden) 101.Kajsa Yang Hansen (University of Gothenburg, Sweden) APOLOGIES (GA-55/I/03) GA Representatives Khalid Al-Harqan (Qatar) Peter Archer (Ireland) Jusuf Arifi (Macedonia) Osman Çelίk (Turkey) Dimitris Chassapis (Greece) Luiz Cláudio Costa (Brazil) Žaneta Džumhur (Bosnia and Herzegovina) Michał Federowicz (Poland) Hagit Glickman (Israel) Karine Harutyunyan (Armenia) Faridah Abu Hassan (Malaysia) Carlos Henríquez Calderón (Chile) Eckhard Klieme (Germany), PEC Neda Kristanova (Bulgaria) Almagul Zhabataena Kultumanova (Kazakhstan) Patricia Martínez (Colombia) Geoffrey Masters (Australia) Maia Miminoshvili (Georgia) Mohammed O. Matar Mustafa (Palestine) Rufus Poliah (South Africa) Ismael Sanz Labrador (Spain) Jonathan Wright (Scotland) Ching-Shan Wu (Chinese Taipei) Tomáš Zatloukal (Czech Republic) Honorary Members Albert Beaton Christiane Brusselmans-Dehairs Robert Garden Jur Hartenberg John Keeves Barbara Malak-Minkiewicz Tjeerd Plomp Alejandro Tiana Judith Torney-Purta Hans Wagemaker Committee Members John Ainley, PEC Jan-Eric Gustafsson, TEG

______________________________________________________________________________ 13–16 October 2014

Minutes of the 55th IEA General Assembly Meeting 5 ______________________________________________________________________________________________________________

Larry Hedges, TEG Marc Joncas, TEG Observers Huda Al-Awadi (Ministry of Education, Bahrain) Jameela Al Muhairi (KHDA, UAE) Rabaa Alsumaiti (KHDA, UAE) Marguerite Clarke (The World Bank) Dequan Li (CSE, China) Pierre Mairesse (European Commission) Toziba Masalila (SACMEQ) Emiliana Vegas (IDB)

______________________________________________________________________________ 13–16 October 2014

Minutes of the 55th IEA General Assembly Meeting 6 ______________________________________________________________________________________________________________

MONDAY, 13 OCTOBER 2014 09:00–10:15 Session I Welcome Addresses Ms Anne-Berit Kavli, Chair of IEA, opened the 55th IEA General Assembly meeting and introduced Ms Heidrun Strohmeyer, Director of the Section Information Technology, Statistics, Gender Issues at the Austrian Federal Ministry of Education and Women’s Affairs (BMBF). Ms Strohmeyer welcomed all participants on behalf of the BMBF. She characterized IEA as a pioneer in the provision of high quality, comparable data on educational achievement and its contexts, thus facilitating public and policy dialogue on how to raise education quality and innovation in Austria and abroad. Opening Statements Ms Kavli reported briefly on the Standing Committee’s February and October 2014 meetings, noting the good progress that was seen in IEA’s studies over the year and high anticipation over the upcoming release of the ICILS 2013 international results. Mr Dirk Hastedt’s new position as IEA Executive Director commenced, as planned, on 1 April 2014. Ms Kavli also signaled that future changes in Dutch legislation about the management of foundations necessitate a careful review of the IEA statutes, a process that has been initiated under the guidance of IEA Legal Advisor Mr Hans Veenhuys and will continue in 2015. Mr Hastedt welcomed participants and thanked the meeting hosts. In addition to the smooth running of projects, he highlighted a number of accomplishments since the last General Assembly meeting, including a rich training portfolio, the second volume of the IERI journal Large-scale Assessments in Education1, and preparation of the 6th IEA International Research Conference (22–26 June 2015, Cape Town, South Africa), which will include a new strand linking secondary analysis to policy developments. He also pointed to new initiatives, particularly in relation to computer-based assessment (CBA) and the development of a communication strategy. He concluded with a call for attention to important financial challenges facing the organization, which will continue to be closely monitored. Approval/Amendment of Agenda (GA-55/I/01) Ms Kavli referred to the agenda that was previously distributed to participants. Any changes will be reflected in the minutes of the meeting. That the agenda be approved: Moved: Jouni Välijärvi (Finland) Seconded: Elena Papanastasiou (Cyprus) Carried: unanimously The agenda was adopted. Approval/Amendment of Minutes of 54th GA in Lisbon, Portugal (GA-55/I/02) No amendments were made to the minutes of the 54th IEA General Assembly meeting. That the minutes be approved: Moved: Chew Leng Poon (Singapore) Seconded: Ali Mehad Alsuwaidi (United Arab Emirates) 1

http://www.largescaleassessmentsineducation.com/

______________________________________________________________________________ 13–16 October 2014

Minutes of the 55th IEA General Assembly Meeting 7 ______________________________________________________________________________________________________________

Carried: unanimously The minutes from Lisbon were adopted. Apologies (GA-55/I/03) Ms Kavli referred to the list of invitees who sent apologies (see pp. 4–5 of the minutes). Approval of New Standing Committee Members (GA-55/I/04) Ms Kavli welcomed new Standing Committee member Dr Dana Kelly (United States). Mr Sándor Brassói (Hungary) and Dr Margarita Peña, former representative for Colombia, concluded their terms as members of the Standing Committee, and Ms Kavli thanked them for their work. The Standing Committee nominated Dr Elena Papanastasiou (Cyprus) and Dr Thierry Rocher (France) as new candidates. That Dr Elena Papanastasiou, General Assembly representative for Cyprus, be approved as a member of the Standing Committee: Moved: Sándor Brassói (Hungary) Seconded: Frederick Leung (Hong Kong SAR) Carried: unanimously Dr Papanastasiou was approved as a member of the Standing Committee. That Dr Thierry Rocher, General Assembly representative for France, be approved as a member of the Standing Committee: Moved: Pierre Brochu (Canada) Seconded: Lorna Bertrand (England) Carried: unanimously Dr Rocher was approved as a member of the Standing Committee. Honorary Membership for Dr Svein Lie (GA-55/I/05) On behalf of the Standing Committee, Ms Kavli presented the nomination of Dr Svein Lie for honorary membership of IEA, in recognition of his significant, longstanding contributions to the organization. That Dr Svein Lie be approved as an honorary member of IEA: Moved: Galina Kovaleva (Russian Federation) Seconded: Dominique Lafontaine (French Community of Belgium) Carried: unanimously Dr Lie was approved as an honorary member of IEA. Announcements (GA-55/I/06) Mr Hastedt announced a call for proposals for a thematic report using PIRLS data: “The Home, Parental Involvement, and Literacy Achievement at Grade 4.” He noted that in the future, IEA aims to publish regular thematic/in-depth reports, and invited suggestions for topics of interest.

______________________________________________________________________________ 13–16 October 2014

Minutes of the 55th IEA General Assembly Meeting 8 ______________________________________________________________________________________________________________

IEA Awards 2014 (GA-55/I/07) Dr David Robitaille, Chair of the IEA Awards Committee, announced the winners of the 2014 IEA awards competition, recognizing outstanding empirical research that makes use of IEA study data. He commented on the high quality of submissions. The 2014 Bruce H. Choppin Award at the doctoral level was given to Dr Rolf Strietholt for his dissertation, “ ‘Using the World as an Educational Laboratory’ Revisited: Methodological Foundations for Utilizing Recent and Older International Large-scale Studies for Educational Effectiveness Research.” The 2014 Bruce H. Choppin Award at the master’s level was given to Mr Francisco Cerón Acevedo for his thesis, “Cross National Differentiation in School Systems and Achievement: Compositional Effects and Contextual Factors in Case of Chile.” The 2014 Richard M. Wolf Award was given to Dr Kajsa Yang Hansen and Dr Ingrid Munck for their article, “Exploring the Measurement Profiles of Socioeconomic Background Indicators and their Differences in Reading Achievement: A Two-level Latent Class Analysis,” published in IERI Monograph Series: Issues and Methodologies in Large-Scale Assessments, Volume 5. PEC Annual Report (GA-55/III/13) Dr David Robitaille, Chair of the IEA Publications and Editorial Committee (PEC), thanked retiring committee members Mr Robert Garden, Dr Svein Lie, and Dr Anu Toots, and welcomed incoming chairperson Dr Tom Loveless and new members Dr John Ainley and Dr Eckhard Klieme. He announced that PEC is actively seeking additional members. Over the last 12 months, the committee conducted reviews of three policy briefs, the ICILS 2013 international report, and a revised edition of the IEA Style Guide. Dr Robitaille noted that the policy brief series, while highly appreciated by PEC members for making IEA findings accessible to new audiences, raised concerns about the kinds of policy implications that can be drawn. Efforts have been taken to address these concerns in full, and publication of the series will resume shortly. Ms Kavli thanked Dr Robitaille for his years of dedicated service to PEC and the Awards Committee. Mr Hastedt recognized PEC members for “making the impossible possible” in upholding high standards of quality and working under strict timelines to provide feedback for improvement of IEA publications. 10:45–12:30 Session II ICILS 2013 Progress Report (GA-55/I/08) The IEA International Computer and Information Literacy Study (ICILS) 2013 progress report was given by Research Director Mr Julian Fraillon and Assessment Coordinator Dr Wolfram Schulz, both of the Australian Council for Educational Research (ACER). Mr Fraillon summarized recent milestones of ICILS 2013, a study in 21 education systems of the ways in which young people are developing computer and information literacy (CIL) to support their capacity to participate in the digital age. The assessment framework was published in 2013.2 He highlighted the collaborative development of the survey instruments, involving a project advisory committee and extensive country consultation and technical review. He next gave a demonstration of the student assessment module, After-school Exercise. The module includes a series of small tasks, followed by a large task requiring students to create an advertising poster for an afterschool exercise program. Mr Fraillon displayed examples of actual responses to the open2 Fraillon, J., Schulz, W., & Ainley, J. (2013). International Computer and Information Literacy Study: Assessment framework. Amsterdam: IEA.

______________________________________________________________________________ 13–16 October 2014

Minutes of the 55th IEA General Assembly Meeting 9 ______________________________________________________________________________________________________________

ended task, showing a broad range of student work. He noted that one of the major contributions of ICILS is an empirically derived scale and description of CIL learning. The international report will be released on 20 November 2014, and the international database training is scheduled for March 2015 in conjunction with the 1st ICILS 2018 national research coordinator (NRC) meeting. Preliminary considerations for a second cycle of ICILS include incorporating a broader range of devices and software contexts, though Mr Fraillon noted that this will depend on the level of country participation and interest. A confidential preview of the ICILS 2013 international results followed, presented by Dr Schulz and Mr Fraillon. Mr Hastedt reminded participants that the launch of ICILS 2018 requires timely confirmations of country participation. There was a brief discussion about some innovative and challenging aspects of ICILS 2013. Some difficulties were noted with the timeline and scoring of the large tasks. A number of delegates considered the study data relevant for reform efforts (e.g., in relation to the organization of the learning process and ICT resourcing/use), and expressed interest in the 2018 cycle. Mr Kimmo Leimu (Honorary Member) gave importance to the broader concept of opportunity to learn, which was captured in ICILS 2013. Ms Gabriela Cares (Chile) praised the study team’s support of the computer-based test administration in her country. Dr Jouni Välijärvi (Finland) and Mr Sándor Brassói (Hungary) inquired about the nature/types of ICTrelated subjects in countries, and their relationship to achievement outcomes. Mr Fraillon and Dr Schulz explained that the national context survey includes questions on countries’ approaches to CIL education in the curriculum (i.e., as a separate subject, integrated into different subjects, or as a cross-curricular responsibility) and the types of ICT-related subjects offered at different stages of school education. They cautioned that it is not possible to associate achievement with the described provision of subjects in countries, given the fairly diverse range of subject characteristics and limited number of participating countries. Dr Ebrahim Talae (Iran) asked about criteria for the scoring of large tasks. Mr Fraillon answered that the constructed responses/large tasks were scored by trained scorers using criteria covering both the use of information and the degree to which the information product met the specified communicative purpose. Ms Lorna Bertrand (England) considered the potential for additional analyses linking TIMSS and ICILS results. She also spoke of increasing policy interest in measuring students’ understanding of computers and programming ability. 14:00–16:00 Session III ICILS 2013 Progress Report (cont’d) Mr Fraillon mentioned cross-comparisons across studies as potentially of interest at the national level. He clarified that the CIL construct includes an aspect on students’ basic understanding of computer use but, like most similar constructs developed in recent years, does not emphasize advanced computer-based technical skills (e.g., programming). He noted that the framework for 2018 will be adapted as necessary to keep the study contemporary, while still observing the requirement to measure and report trends. Ms Kavli and Mr Hastedt commended the ICILS study team, registered positive feedback on a possible second cycle, and confirmed that country representatives would be approached following the General Assembly meeting for more formal expressions of interest and commitment. ICCS 2016 Progress Report (GA-55/I/09) The IEA International Civic and Citizenship Education Study (ICCS) 2016 progress report was given by International Coordinators Dr Wolfram Schulz, ACER, and Mr Ralph

______________________________________________________________________________ 13–16 October 2014

Minutes of the 55th IEA General Assembly Meeting 10 ______________________________________________________________________________________________________________

Carstens, IEA DPC. Mr Carstens highlighted aspects of the ICCS study design and progress since last year. He noted high interest in the regional modules as an enrichment of ICCS 2016, but in light of operational constraints (due to late confirmations of participation) and shifting priorities of interested NRCs, it was decided that the European and Latin American regional modules would focus on affective-behavioral dimensions only, and the Asian student questionnaire would be offered (without changes) as a national option. He reported that ICCS has been discussed prominently in many international initiatives related to Global Citizenship Education and Education for Sustainable Development. The study’s next steps include completion of the field test (October–November 2014), review of the field test results, and refinement of the main survey instruments—a challenging endeavor, which requires maintaining strong links to ICCS 2009 and incorporating new focus areas. The study team will also conduct a careful review of the cross-national validity and comparability of measures as part of the field test analysis. The study directors announced the publication of the second edition of the ICCS 2009 international database user guide, which contains a further set of released items.3 Dr Paul van Oijen (The Netherlands) reflected on interesting findings from a national study of civic and citizenship education in which Grade 8 students scored lower than expected, in contrast to students in other grade levels. Dr Gilberto Guevara Niebla (Mexico) expressed interest in collecting data on contexts of violence. Dr Schulz noted some coverage of this theme in ICCS 2009, particularly the Latin American module, which included questions about students’ experiences of and attitudes toward violence, and about attitudes toward the peaceful coexistence of diverse groups. Mr Carstens added that ICCS exemplifies the mechanism that exists in all IEA studies for countries’ interests to inform and guide the development of the framework. ETLS Progress Report Mr Hastedt provided a brief update on the status of the IEA English Teaching and Learning Study (ETLS). Recruitment was highly impacted by countries awaiting determination of the European Commission’s plans in the field of languages, and development of ETLS was thus put on hold this year. Currently only a small number of European countries have made formal expressions of interest. Mr Hastedt and Ms Kavli reminded delegates that without sufficient country participation, the study cannot move forward. Given the promising reception of the ETLS proposal at previous General Assembly meetings, they expressed the hope that more countries will affirm their interest and officially join the study. IEA Communication Strategy Mr Helge Lund, Norwegian Directorate for Education and Training, gave a presentation on the IEA communication profile. He noted a striking dichotomy of IEA’s image: IEA is an established and respected institution among researchers, but seems to be a “well-kept secret” among some other audiences (e.g., practitioners and the public). He characterized IEA’s unique strength as a research organization, which takes an objective, independent approach to presenting study findings—without entering into policy decisions directly. Possible areas of weakness, branding, and resourcing needs (particularly in terms of internal personnel) were also considered important points of attention. He stressed that communication should be an integrated part of operations planning, involving the cooperation of management staff and possibly also the organization’s membership and wider network. He concluded with a message about the strong potential of IEA, which is ideally positioned in the current debate over comparative assessment to take a more pronounced position in the field. 3 Brese, F., Jung, M., Mirazchiyski, P., Schulz, W., & Zuehlke, O. (2014). ICCS 2009 user guide for the international database (2nd ed.). Amsterdam: IEA.

______________________________________________________________________________ 13–16 October 2014

Minutes of the 55th IEA General Assembly Meeting 11 ______________________________________________________________________________________________________________

Mr Hastedt reminded participants of some of the communication initiatives already underway and the various target audiences under consideration (researchers, policymakers, the media, practitioners, the general public, and funders/donor organizations). One such initiative is to create a communication taskforce, drawing on country experiences and expertise to provide input for a new communication strategy for IEA. Mr Hastedt invited nominations of communication experts who can support and donate their time to the taskforce (the first meeting will take place in early 2015). He also welcomed suggestions for further initiatives, but cautioned that IEA has limited resources to invest in this effort. Dr Elena Papanastasiou (Cyprus) suggested a simple change: prefixing “IEA” to all study names in promotional materials (e.g., IEA-TIMSS, IEA-PIRLS), so as to reinforce the understanding that these studies are conducted under the IEA umbrella. Other suggestions from participants included developing more frequent, diversified reports at the international or regional level, improving the targeting of communication materials to stakeholders, and increasing IEA’s capacity building activities to support the use of data within countries. Dr Ridha Al-Khayat (Kuwait) considered it a primary challenge to meet policymakers’ needs for clear, straightforward messages. Dr Eva Klemenčič (Slovenia) shared her experience of the importance of fostering cooperation between researchers and the ministry toward common goals. Mr Lund and Ms Kavli thanked participants for their insightful comments. 16:15–17:30 Session IV Discussion Groups: IEA Communication Strategy (Session I) Participants gathered into pre-assigned groups to discuss their views on an IEA communication strategy, and the potential aims and products of the IEA communication taskforce. TUESDAY, 14 OCTOBER 2014 09:00–10:30 Session V TIMSS 2015 Progress Report (GA-55/II/11) The IEA Trends in International Mathematics and Science Study (TIMSS) 2015 progress report was given by Co-Directors Dr Ina Mullis and Dr Michael Martin, Boston College. Dr Martin presented the report. Dr Martin announced the completion of the field test administration of TIMSS 2015 in 53 participating countries, TIMSS Advanced 2015 in 10 countries, and TIMSS Numeracy 2015 in 7 countries. TIMSS Numeracy, a new initiative for the 2015 cycle, will offer better measurement for countries where many fourth grade students are still developing fundamental skills, and will report achievement results on the TIMSS fourth grade mathematics scale. The TIMSS Advanced 2015 assessment frameworks were published in March 2014.4 After extensive review of the field test results in meetings of the expert committees (July 2014) and NRCs (August 2014), the questionnaires and achievement items for all TIMSS assessments were finalized. The main survey data collection will take place in October–December 2014 and March–June 2015 for southern and northern hemisphere countries, respectively. Dr Thierry Rocher (France) asked about the TIMSS Advanced field test outcomes and implications for item development. Dr Mullis and Dr Galina Kovaleva (Russian Federation), member of the science expert group, responded that some of the TIMSS 4 Mullis, I.V.S., & Martin, M.O. (Eds.). (2014). TIMSS Advanced 2015 assessment frameworks. Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Boston College.

______________________________________________________________________________ 13–16 October 2014

Minutes of the 55th IEA General Assembly Meeting 12 ______________________________________________________________________________________________________________

Advanced field test items were considered too challenging for the student population. In close consultation with the item review committee, the items were revised where needed, while still assuring the same coverage of the framework. PIRLS 2016 Progress Report (GA-55/II/12) The IEA Progress in International Reading Literacy Study (PIRLS) 2016 progress report was given by Co-Directors Dr Ina Mullis and Dr Michael Martin, Boston College. Dr Mullis described how PIRLS 2016 was expanded for better measurement and policy relevance through the introduction of the new ePIRLS assessment of online informational reading and development of PIRLS Literacy (from prePIRLS), which, as a companion to TIMSS Numeracy, will offer better measurement and reporting (on the PIRLS scale) for countries where many fourth grade students are still developing fundamental reading skills. In November 2013, the PIRLS 2016 assessment framework was published.5 Dr Mullis noted the considerable work over the past year in developing engaging, culturally appropriate passages/items and updating the context questionnaires. The PIRLS 2016 and PIRLS Literacy 2016 instruments for the field test were finalized and sent to national centers for translation, and the ePIRLS field test materials and computer-based delivery system are in the process of being finalized. The field test will take place in March–April 2015. Dr Martin and Dr Mullis next demonstrated the beta version of the ePIRLS software, showing examples of different items/tasks and specific features of the software as a student would progress through the assessment. They thanked all partners involved at the international and national level for their successful collaboration, and reported positive feedback from countries on this innovative assessment, which will be further refined after the field test. eTIMSS Initiative Dr Ina Mullis and Dr Michael Martin, Boston College, described the initiative to include a computerized option in TIMSS 2019. They stated that a dual system is envisaged, which supports both electronic and paper-and-pencil modes so as not to preclude any countries from participating in TIMSS. eTIMSS is planned to be administered on tablets with a stylus and (optionally) a keyboard for higher grades. The potential benefits of the eTIMSS initiative include increasing student engagement and operational efficiency over time, and the option to assess complex problem-solving and inquiry tasks not possible on paper. Dr Mullis noted the important financial resources and technical capabilities necessary for this initiative. One major challenge is to maintain comparability between delivery modes, which will be addressed through a careful step-by-step plan beginning in 2014–2015 with the conversion of TIMSS items to e-format and development of the eTIMSS system and complex tasks. Work in subsequent years would include small scale studies to assess comparability, a pilot study, further item development work, and a full field test in 2018. The main data collection would incorporate a bridge study with countries assessing in both paper and electronic modes. Delegates from a number of countries (French Community of Belgium, France, Iran, United States) discussed the possibilities and challenges of CBA, recognizing both the value of trend measurement and the need to stay relevant to current learning/school contexts. Dr Frederick Leung (Hong Kong SAR) considered whether a mode effect study should be conducted earlier in the project schedule. Dr Michał Sitek (Poland) mentioned concerns over the cost of CBA, and the need to be flexible to adapt to ever-changing technology. Dr Martin and Dr Mullis acknowledged the challenges involved, affirming that the eTIMSS initiative will be closely informed by IEA’s developing expertise in other studies (e.g., ICILS, ePIRLS) and will build on the robust linking methodology that has 5 Mullis, I.V.S., & Martin, M.O. (Eds.). (2013). PIRLS 2016 assessment framework. Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Boston College.

______________________________________________________________________________ 13–16 October 2014

Minutes of the 55th IEA General Assembly Meeting 13 ______________________________________________________________________________________________________________

been developed to bridge assessments. Mr Hastedt confirmed the United States’ commitment to provide initial funding to support the eTIMSS development. He noted the significant development effort but also the potential for synergies across various IEA studies, and assured that resources would be used carefully and effectively. Further funding is needed to finance this ambitious but worthwhile initiative, which will provide benefit to the whole organization in the advance towards CBA. 11:00–12:30 Session VI Experiences from Countries: Use of IEA Study Results Ms Kavli opened the floor to the panel presentation on the use of IEA study results in countries. Dr Anu Toots (Estonia) illustrated how IEA study findings can support constitutional change by describing her country’s experience with Vote@16, an initiative to reduce the voting age. She began by discussing the legal contexts for Vote@16 in Estonia (where voting age is defined by the constitution), and traced the development of public opinion and policy support. She stated that the initiative is now high on the national agenda; after a series of parliamentary hearings, the first local elections with 16-year-old voters are anticipated in 2017. In describing how IEA study results were used, she stressed the importance of being honest about the limitations of the research evidence. Data from ICCS, CIVED, and a national assessment contributed to a 2013 impact assessment study requested by the Ministry of Justice. ICCS results on the knowledge, perceptions, and behaviors of lower-secondary students also helped to address key public concerns about the maturity of young adults in this age group. Data were further used for comparisons between Estonia and Austria (where 16-year-olds are allowed to vote), and between 14year-olds and adults on aspects considered relevant to voting behavior. Dr Toots concluded by identifying the continued need for data on election-related issues and voting practices/initiatives, as well as opportunities for further analyses (e.g., comparisons of Vote@16 countries, impact studies, and educational governance studies) if ministries and other stakeholders work together. Dr Vijay Reddy (South Africa) described how TIMSS data are used in South Africa to support education outcomes, provide insights into quality, and extend in-country analyses. She explained that participation in TIMSS 1995 and TIMSS 1999 represented an important first step in the assessment of educational achievement at the national level. Low scores sparked political and public debate, and provided a catalyst for investments in mathematics and science programs. After the TIMSS 2003 results came out, researchers focused on disaggregating the national average scores in order to tell a fuller story on the performance of different groups, while TIMSS 2011 data was seen as an opportunity to report on trends in achievement. With TIMSS 2015 data, researchers intend to extend the trend line, continue reporting on group-level performance, and use more sophisticated modeling analyses. Dr Reddy presented some examples of withincountry analyses of TIMSS data. She showed, for instance, how TIMSS results could inform the setting of realistic improvement targets (at both the lower and top ends of the performance scale) and support the notion that different policies and strategies are needed for schools of different type (e.g., public, no fee, fee-based, independent) and socioeconomic contexts. She emphasized the importance of secondary analysis to gain maximum benefit from participation in large-scale assessments. Ms Anne-Berit Kavli, IEA Chair, presented key findings from Northern Lights on TIMSS and PIRLS 2011, a report analyzing Nordic results in TIMSS 2011 and PIRLS 2011.6 By conceiving of Nordic countries as a unique “laboratory” for educational research because of the countries’ similar culture and background but different approaches to education, the report aims to shed light on similarities and differences between the systems for further education policy development and joint initiatives in the region. Ms Kavli gave an 6

http://www.udir.no/Upload/Forskning/2014/Nlights%20TIMSS%20and%20PIRLS.pdf

______________________________________________________________________________ 13–16 October 2014

Minutes of the 55th IEA General Assembly Meeting 14 ______________________________________________________________________________________________________________

overview of Nordic participation in IEA studies, noting how they form part of a broader national knowledge base. She next displayed trend results for Norway across several international assessments (showing a similar pattern in the development of achievement over time) and discussed some Nordic characteristics. She presented findings from the report’s collection of articles, exploring questions about the relationship between school performance and policy variations; the teaching of reading literacy, and its relation to curricula; how to improve mathematics teaching; the relationship between learning outcomes and teachers’ attitudes, beliefs, and practices; and the characteristics of top performers. The report highlighted the importance of an individual approach and support from teachers and parents for learning, as well as schools’ emphasis on academic success. Ms Kavli thanked the other presenters for their insightful examples of the use of IEA study results for informing policy. 14:00–15:30 Session VII ECES Progress Report (GA-55/I/10) The IEA Early Childhood Education Study (ECES) progress report was given by International Coordinators Ms Juliane Hencke and Dr Steffen Knoll, both of the IEA DPC. Dr Knoll presented the report. Dr Knoll reported on the design and recent activities of ECES. He explained that the study is organized in two phases, with Phase 1 (policy questionnaire) focusing on the national contexts for early childhood education (ECE). Due to the low number of confirmed country participants, the 2nd NRC meeting (originally scheduled for February 2014) was postponed until September 2014 and the development of Phase 2 (assessment module and context questionnaires) was put on hold. He summarized progress in the development of the study framework and the Phase 1 policy questionnaire, which was piloted in five countries (November–December 2013), revised based on the pilot results and NRCs’ feedback, and finalized for administration in October–November 2014. The policy report will provide information on the wider policy contexts for ECE (with a focus on the year before ISCED 1), variations in ECE policy and systems, and key similarities and differences across a range of indicators (related to public policy, delivery models and setting types, access and participation, quality, and expectations for outcomes). Dr Knoll noted that while the development of an ECE policy typology was initially envisaged, a case study approach (including country profiles) might be more appropriate for the smaller number of participating countries; this could also generate fruitful hypotheses for further research/interest in Phase 2. Dr Dana Kelly (United States) noted the positive comments she received on the policy questionnaire for its potential to capture her country’s complex system. Dr Michał Sitek (Poland) commented that it can be important for stakeholders to receive plans on the whole project (e.g., schedule and costs) before making participation decisions. Mr Hastedt explained that although a schedule and budget for both phases of ECES had been prepared, the timeline depends on countries’ timely commitment, and the low enrollment to date has made Phase 2 uncertain. He noted IEA’s continuing efforts to promote the study, and expressed the hope that more countries would join and enable Phase 2 to move forward. Post-2015 Global Education Agenda Mr Albert Motivans, UNESCO Institute for Statistics, gave a presentation on the development of the post-2015 global education agenda. He first discussed the context for the millennium development goals and post-2015 frameworks. In assessing global education progress to 2015, there is an evident need for continued improvement, not only in access to education but also in education quality and equity. Different processes

______________________________________________________________________________ 13–16 October 2014

Minutes of the 55th IEA General Assembly Meeting 15 ______________________________________________________________________________________________________________

and extensive deliberation among global and regional stakeholders have formed part of a complex effort to agree on goals for a new development framework. Mr Motivans noted that quality education is central to the proposed goals; the Education for All Steering Committee’s proposed targets also include an explicit focus on learning outcomes. Measurement challenges include a lack of consensus on certain concepts, definitions, and measures; insufficient data to regularly monitor some aspects of education; and the need for a longer term approach to the collection of education data, requiring country commitment and leadership. He identified and described several initiatives (consultation on post-2015 indicators, proposal for a global metric, Learning Metrics Task Force/IPAL, and UIS catalogue of learning assessments) where IEA members can contribute to post2015 debates on measurement and learning. Dr Jack Schwille (Honorary Member) pointed to higher education and private education as important areas of attention. Dr Vijay Reddy (South Africa) considered the risk that governments—particularly in developing countries—could be pulled in different directions by outside agencies, and that single indicators at the global level may not capture the nuances of what is happening locally. Mr Motivans said that the aims of the post-2015 global education agenda are not normative, but rather a way of supporting countries, mobilizing actors, and broadening communities of educational stakeholders. Specific goals and targets will need to be set at the national level, feeding into policy processes and monitoring. Ms Kavli and Mr Hastedt thanked Mr Motivans for his presentation of activities that are important for IEA work. Mr Hastedt spoke about the IEA studies’ contribution to the post-2015 millennium development goals. Although the UN-defined millennium goals for 2015 can be seen to have had positive impacts on the emphasis on education overall (as shown by the increase in the percentage of children in school), he explained that in some countries, this may have come at the expense of quality, with many end-of-primary students not reaching expected levels in reading, writing, and numeracy. IEA experts were invited to participate in three expert groups on reading and numeracy indicators and Global Citizenship Education to discuss new post-2015 goals, targets, indices, and benchmarks. He reported that a high level of agreement was reached on the importance of measuring learning outcomes, the need for high quality, comparable data (with IEA studies as a point of reference), and the need for universal goals with country-specific targets. One idea is to generate a global learning metric by linking regional assessments (e.g., SACMEQ, PASEC, TERCE) to TIMSS and PIRLS. The frameworks and high quality trend data from TIMSS, PIRLS, and ICCS could provide an important resource for measuring education progress to 2030. 16:00–17:30 Session VIII Discussion Groups: Post-2015 Millennium Development Goals (Session II) Participants gathered into pre-assigned groups to discuss their views on the post-2015 millennium goals for education and the ways in which IEA studies can contribute. WEDNESDAY, 15 OCTOBER 2014 09:00–10:30 Session IX Feedback from Discussion Groups Group 1 Chair: Pierre Brochu (Canada), Rapporteur: Dana Kelly (United States) Participants: Isabelle Erauw (Flemish Community of Belgium), Fumi Ginshima (Japan), Romana Kanovská (Slovak Republic), Steffen Knoll (ECES), Rainer Lehmann (Germany), Nizam (Center for Education Assessment, Indonesia), Vijay Reddy (South Africa), Sigurgrímur Skúlason (Iceland), Ebrahim Talae (Iran).

______________________________________________________________________________ 13–16 October 2014

Minutes of the 55th IEA General Assembly Meeting 16 ______________________________________________________________________________________________________________

Dr Kelly reported on two themes emerging from the group’s discussion of target groups for an IEA communication strategy. First, group members considered more frequent dissemination of news and information important; reports should go beyond the international report and be accessible and engaging to various audiences (e.g., policymakers, parents, practitioners), keeping in mind the need for technical rigor. Second, the group suggested broadening IEA’s provision of training and support to countries in the use of data. Dr Kelly described a wide-ranging conversation about how IEA studies can contribute to the post-2015 millennium development goals. IEA’s potential to contribute was seen in terms of quality (expertise in assessment, development of instruments for appropriate measurement in diverse countries), capacity building, and “leading the way” in new assessment areas/foci (e.g., vocational education, improving measurement of process quality). Group 2 Chair: Anu Toots (Estonia), Rapporteur: Mark Német (Austria) Participants: Saada Al-Obaidli (Qatar), Michelle Braš Roth (Croatia), Megan Chamberlain (New Zealand), Zainal Aalam Hassan (Malaysia), Eva Klemenčič (Slovenia), Brian Mokopakgosi (Botswana), Thierry Rocher (France). Dr Német reported that group members mostly attributed high importance to IEA’s communication strategy, but recognized some lack in effectiveness. A high quality communication strategy should cope with diversity (different institutional settings, regions, cultural traditions) and sharpen the profile of IEA (e.g., balance its research orientation with policy interests, capitalize on the studies’ curriculum focus and gradebased sampling). Dr Német noted differences in awareness of the post-2015 millennium goals among representatives of the group. Members discussed the technical difficulty of aligning TIMSS/PIRLS to regional studies, the cultural appropriateness of studies in different countries, and the possible impact of (normative) targets on frameworks. Possible actions for IEA include promoting cooperation within regions/clusters of countries, and between experienced countries and ‘newcomers’ to international assessment. Group 3 Chair: Jouni Välijärvi (Finland), Rapporteur: Chew Leng Poon (Singapore) Participants: Josette Biyo (Philippines), Rita Dukynaitė (Lithuania), Lilia T. Habacon (DOST, Philippines), Amina Kafai (Luxembourg), Sung Hoon Kim (Korea), Myung Ae Lee (KICE, Korea), Roberto Ricci (Italy), David Robitaille (PEC), Paul van Oijen (The Netherlands), Norman Verhelst (TEG). Dr Poon stated that group members underscored the role of national centers in an IEA communication strategy, as they can adapt messages for their local target groups and networks. Current activities by national centers include reporting to schools and policymakers, and linking IEA and national study data. A communication expert able to bridge the needs of policymakers, researchers, and practitioners could support this work. Members also suggested making better use of social media and the encyclopedias. Dr Poon reported several ways in which IEA was seen as a potential contributor to the post-2015 agenda. For example, IEA could share expertise in methodology and benchmarking, participate in technical working groups, advocate for the needs of countries to collect data that is relevant to their specific national contexts, and run workshops to strengthen countries’ analysis capabilities. Some concerns were expressed about the quality of a global learning metric. Group 4 Chair: Frederick Leung (Hong Kong SAR), Rapporteur: Sándor Brassói (Hungary) Participants: Abdalla Yousef Awad Al-Ababneh (Jordan), Lorna Bertrand (England), Kyong

______________________________________________________________________________ 13–16 October 2014

Minutes of the 55th IEA General Assembly Meeting 17 ______________________________________________________________________________________________________________

Hee Chon (KICE, Korea), Furqon (Indonesia), Gilberto Guevara Niebla (Mexico), Dirk Hastedt (IEA), Seamus Hegarty (Honorary Member), Andris Kangro (Latvia), Natia Mzhavanadze (Georgia), Annette Qvam (Norway), Jack Schwille (Honorary Member), Michał Sitek (Poland), Sue Thomson (Australia). Mr Brassói listed a number of open questions discussed by the group, for instance, in relation to communication priorities, target groups and products, the possibilities for data use at the school level, and the focus of evaluation (different subjects/grades vs. system level). The different needs of stakeholders and the potential study benefits were stressed. Suggestions included increasing the focus on the school/classroom level, supporting early career researchers, and improving online/media dissemination of findings. Mr Brassói next summarized a rich discussion providing evidence of various withincountry activities related to the post-2015 millennium goals. Education quality, equity, and sustainability were all considered important focal points on the agenda. Looking forward, members recommended that IEA ensure good representation in international discussions of post-2015 goals, become a resource center (especially for developing countries), and use its network to promote IEA studies and share expertise. Group 5 Chair: Elena Papanastasiou (Cyprus), Rapporteur: Ali Mehad Alsuwaidi (United Arab Emirates) Participants: Josef Basl (Czech Republic), Gabriela Cares (Chile), Francisco Javier García Crespo (Spain), Galina Kovaleva (Russian Federation), Dominique Lafontaine (French Community of Belgium), Piniti Ratananukul (Thailand), Camilla Thinsz Fjellström (Sweden). Mr Mehad Alsuwaidi reported that group members recognized the value of a long-term communication strategy. They identified principals and teachers as key stakeholders, and noted the need for targeted tools and techniques, various dissemination activities (e.g., at conferences), and increased communication to member countries. It was considered a shared responsibility of members to support IEA and produce communication products in their local languages. Mr Mehad Alsuwaidi noted that awareness of the post-2015 millennium goals was lower among some countries represented by the group. IEA’s major contribution to the development of these goals was seen to lie in providing technical expertise and data, particularly in relation to numeracy, literacy, and civic and citizenship education outcomes. IEA studies could help countries identify the main challenges related to poor performance in these areas. Ms Kavli thanked group members for their suggestions and constructive feedback. Mr Motivans underlined two key points from the discussion of the post-2015 millennium goals: 1. Robust, valid measures of education outcomes are an important area in which IEA can contribute, and 2. Country representatives have a responsibility to voice their interests and needs in the debate. Mr Hastedt registered a strong message of support for IEA’s engagement in further discussions in this area, as well as a number of concrete ideas for how IEA can help countries build expertise in educational monitoring and evaluation. 11:00–12:30 Session X The TEDS-M experience Dr Sigrid Blömeke, University of Oslo, discussed experiences and lessons from TEDS-M. She reminded participants of the study’s aims and summarized some major challenges of conducting a cross-national study of higher education, from missing links between research groups and between research models, to concerns about response rates, comparability, and lack of consensus on the nature of mathematics pedagogical content

______________________________________________________________________________ 13–16 October 2014

Minutes of the 55th IEA General Assembly Meeting 18 ______________________________________________________________________________________________________________

knowledge and general pedagogical knowledge. However, she emphasized that such challenges could also be seen as the study’s biggest successes: TEDS-M advanced the field of higher education research (conceptually, methodologically, and operationally) and provided proof that an assessment of the quality/content of teaching could be carried out at the international level. Analysis of TEDS-M data showed that both opportunity to learn (OTL) and selectivity are important predictors of teacher education outcomes, and that OTL in mathematics, mathematics pedagogy, and practical experiences are relevant. TEDS-M triggered a number of reports and follow-up studies (including longitudinal studies and video-based performance assessments). Dr Blömeke concluded that although many controversies made the TEDS-M beginnings difficult, researchers now have a more sophisticated view on how teacher education and school/teacher effectiveness are linked. Ms Kavli thanked Dr Blömeke for her informative presentation. Dr Furqon (Indonesia) inquired about cross-cultural differences, and Dr Blömeke responded that follow-up studies indicate some interesting differences in relationships (e.g., between OTL and teacher education outcomes) across countries. Mr Pierre Brochu (Canada) commented on the high value of TEDS-M, and asked about options for using the study instruments in other research. Mr Hastedt confirmed that permission requests to use IEA reports, data, and instruments are all considered on an individual basis depending on the intended use (i.e., commercial vs. non-profit/educational).7 Similarities and Differences between Groups of Countries Concerning Relative Weaknesses and Strengths Dr Svein Lie, University of Oslo, gave a presentation on the use of TIMSS data for an indepth study of relative strengths and weaknesses of groups of countries. He described how secondary information can be gained from test items by looking for meaningful patterns in the item residuals (i.e., what remains when the score aspect has been taken out). By calculating the p-value residuals—how much better or worse a particular country scored than expected from the international item difficulty and the country’s overall score—it is possible to identify patterns of countries (showing similar strengths and weaknesses) and patterns of items (that tend to discriminate similarly between countries). Using TIMSS 1995 mathematics data, Dr Lie presented the correlations of p-value residuals between Norway and other participating countries; the correlations were highest between Norway and Sweden, Iceland, and Denmark, making a Nordic cluster visible. Other clusters (e.g., East Asia, English-speaking, Northern Europe) also emerged from a similar method of analysis with TIMSS 1995 science data. He displayed results of the clustering process in dendrograms for TIMSS 2003, emphasizing that the data ‘drive’ the labeling of clusters and there are many mechanisms that can make countries different or similar (language, geography, political and cultural history, etc.). Such groupings of countries can trigger follow-up studies to identify what constitutes the particularity of each group and encourage regional (or group-wise) reports so that countries can learn from each other’s approaches. Ms Kavli spoke of Dr Lie’s long involvement in IEA studies as a TIMSS NRC, advisory committee member, and member of PEC; his outstanding academic career and contributions to the field; and his work at the University of Oslo EKVA (Unit for Quantitative Analysis in Education). On behalf of IEA and the Norwegian Directorate for Education and Training, she presented the award of IEA honorary membership to Dr Lie. Dr Lie expressed appreciation to his colleagues at the ministry and IEA for the honor. He reminisced about TIMSS and the lessons learned from the “heroes” of that study and colleagues in Nordic countries and abroad.

7

http://www.iea.nl/permissions.html

______________________________________________________________________________ 13–16 October 2014

Minutes of the 55th IEA General Assembly Meeting 19 ______________________________________________________________________________________________________________

14:00–15:30 Session XI Wolf Award Presentation: Exploring the Measurement of Socioeconomic Status Dr Kajsa Yang Hansen and Dr Ingrid Munck, University of Gothenburg, discussed the development of research into the influence of socioeconomic status (SES) on achievement. Dr Munck reflected on progress made since the early studies of SES using Six Subject Survey data, noting that while software possibilities have changed, the research questions formulated at the time are still relevant today. She described some shifts in the conception and measurement of SES with examples from earlier research, leading up to the first latent variable SES application (which received the 1984 IEA Choppin Award8). Dr Yang Hansen noted how this approach could be extended to measure SES as a multidimensional, multilevel construct. She presented one application of a two-level latent class analysis (TLCA), drawing on Swedish PIRLS 2006 data. The TLCA approach can take into account the measurement error in the response patterns of SES indicators, hierarchical IEA data structure (students nested in classes in schools), and variation of SES indicators across different schools. The analysis identified three latent SES classes of individuals: the economically and culturally affluent group, the culturally disadvantaged group, and the culturally well-off group. It showed that reading achievement in Sweden differed significantly across the SES classes, and that almost half of the between-school achievement differences could be explained by SES composition at the school level. Speaking the test language at home was significantly related to reading achievement only for the immigrant-concentrated, culturally disadvantaged group, indicating that a well-integrated societal and school environment may compensate for students’ disadvantaged family background. Ms Kavli thanked Dr Munck and Dr Yang Hansen for their interesting analysis, and presented them with the 2014 IEA Wolf Award.9 IEA DPC Annual Report (GA-55/III/14) Mr Heiko Sibberns, Director of the IEA Data Processing and Research Center (DPC), referred to the written report detailing the center’s activities over the last 12 months. He highlighted developments in three broad areas of work. Activities in the area of international studies included country support, consultation, study-related and methodological workshops, and secondary analysis. In the area of national studies, Mr Sibberns noted a demand for services in a large variety of studies, a tendency towards longitudinal studies, and new areas of interest (special education needs, vocational education and training). The National Educational Panel Study (NEPS), which in previous years reflected a very high level of activity for the IEA DPC (about 15–20 sub-studies per year in 2009–2013 and 10 sub-studies in 2014) will decline to only two sub-studies per year in 2015–2018; this will be reflected in the IEA budget, with full effects seen in 2016. Within the cross-sectional units, efforts have focused on the preparation of hardware for computer-based testing, CBA software development, implementation of an online study management system, and alignment of budgeting procedures. He concluded by noting the closer cross-center cooperation within IEA. Ms Kavli thanked Mr Sibberns and IEA DPC colleagues for their excellent work. TEG Annual Report (GA-55/III/15) Mr Dirk Hastedt, Chair of the IEA Technical Executive Group (TEG), announced new TEG member Mr Jostein Ryssevik. He summarized the 10 October 2014 meeting of TEG, which focused on technical issues in relation to TIMSS (linking TIMSS and TIMSS Numeracy, 8

http://www.iea.nl/choppin_award.html The article is available online at: http://www.ierinstitute.org/fileadmin/Documents/IERI_Monograph/IERI_Monograph_Volume_05_Chapter_3.pd f 9

______________________________________________________________________________ 13–16 October 2014

Minutes of the 55th IEA General Assembly Meeting 20 ______________________________________________________________________________________________________________

measuring trends in the context questionnaire scales), ICCS (cultural invariance of the attitudinal scales), and ECES (feasibility of a typology approach, handling data from federal systems). He reported that TEG members appreciated the step-by-step approach of the eTIMSS initiative but stressed the need to carefully monitor comparability. Members also discussed their involvement in the review process for the call for proposals for thematic reports. Financial Report (GA-55/III/16–18) The IEA financial report was given by Mr Dirk Hastedt, Executive Director, and Mr Roel Burgers, Financial Manager. Mr Hastedt informed the General Assembly about changes in IEA membership fees, as reviewed and approved by the Standing Committee. He explained that the current fees were established in 1996 and have not been adjusted for inflation since then. Membership fees are used to maintain the crucial infrastructure behind the organization’s activities, including the provision of study information, publications, data, and software; support from IEA experts; and key meetings (e.g., General Assembly, Standing Committee, TEG) and conferences. Membership in IEA provides the opportunity to govern and influence the development of IEA research, become a part of IEA’s worldwide network, and benefit from professional development and capacity building. Mr Hastedt showed the increase in costs since 1996 due to the rising number and complexity of studies, as well as investments to facilitate study development and dissemination. Under the new scheme, membership fees will increase 10% per year over the next 5 years, thereafter increasing according to the current inflation rate, with the exception of developing countries. He stressed that the intention is not to cover all management and administration costs, but rather to correct for the situation that these costs have increased over nearly two decades while the value of membership fees has decreased given the rate of inflation. Mr Burgers described the change to the study participation fee scheme. Currently the international study fees are equally spread over all years of a project. He explained that IEA’s cash flow position has weakened under this flat billing schedule for several reasons: higher international coordination costs incurred in the first 2.5 project years, the long period of uncertainty in country participation (resulting in delayed invoicing), and late fee payments. Under the new scheme, planned to begin with ICILS 2018, a higher percentage of the total international participation fee would be invoiced during the first 3 project years than in the remaining 2 years (when national costs are typically higher). This would bring the payment schedule more in line with IEA’s cost pattern and help offset the cash flow challenges arising from late payments. Mr Burgers next referred General Assembly members to IEA’s “Annual Report for the Year 2013” and provided an overview of IEA’s financial position, 2014 forecast, and 2015 budget, as approved by the Standing Committee. He also distributed an updated list of outstanding fees. Mr Burgers presented IEA’s balance sheet as of 31 December 2013. Fixed assets (mainly computer hardware and software) invoiced at $3.7 million were 70% depreciated. Intangible fixed assets of $735,000 relate to capitalized development costs for ECES (which was postponed until early 2014 because of a lack of participation at that time). Receivables remained at a similar level as in 2012, but unpaid membership and participation fees, which accounted for 50% of receivables in 2013, increased by $545,000. As a result of the increased debtor level, cash assets decreased by $2.3 million. Mr Burgers noted that the launch of several studies in parallel in 2013—along with the delayed collection of fees—has caused temporary cash flow constraints. With average monthly expenses of about $2 million, IEA’s cash position covers only 1.8 months. Project funds, earmarked for four international projects, increased to $3 million. Current liabilities decreased to $3 million and were covered by cash assets. Net assets (a

______________________________________________________________________________ 13–16 October 2014

Minutes of the 55th IEA General Assembly Meeting 21 ______________________________________________________________________________________________________________

combination of the continuity reserve and project reserve) increased by 1.5% to $6.4 million in 2013, due to a modest year-end result of $95,000. With respect to IEA’s profit and loss statement for the year ending 31 December 2013, Mr Burgers stated that revenue increased by $2.1 million, mainly due to the new cycles of TIMSS and PIRLS. Operating expenditure increased by $2.3 million, following the revenue pattern. Revenue levels for 2014 and 2015 are projected at about $22 million, with cost levels expected to follow closely. That the 2013 Annual Report be approved: Moved: Elena Papanastasiou (Cyprus) Seconded: Hélder Diniz de Sousa (Portugal) Carried: unanimously The Annual Report was approved. Mr Hastedt strongly appealed to all member and participating countries to pay fees on time. More formalized participation agreements with countries will be arranged, and a discount of 5% will be offered to incentivize countries that are able to pay the full international study participation fee in the first year. Ms Kavli underscored Mr Hastedt’s message and confirmed the Standing Committee’s support for taking necessary measures to secure funds. Announcement of 2015 General Assembly Dr Gilberto Guevara Niebla (Mexico) said that he was pleased to host the 56th IEA General Assembly meeting in Mexico City on 5–8 October 2015. He looked forward to welcoming all participants next year, and to the productive and interesting discussions that will take place. Closing Remarks Ms Kavli thanked the General Assembly meeting participants for their feedback and support. She remarked that realizing the ambitions of IEA requires the shared effort of all members of the IEA community. She extended thanks to the IEA Secretariat and IEA DPC staff, Mr Hastedt, and Dr Mark Német (Austria) and Austrian colleagues for hosting the meeting. The meeting was adjourned at 15:30.

______________________________________________________________________________ 13–16 October 2014