Literacy Assessment Using Mobile Technology
assessment in developing countries. Wagner (2003) identifies three major areas of concern for assessments: size/scope, efficiency, and cost. He aptly names his method for reforming international assessments “smaller, quicker, cheaper,” and defines each term as follows:
Sarah Muffly The Earth Institute – Columbia University
[email protected]
Overview Current educational monitoring in Haiti is insufficient and doesn’t give policy makers information to make spatially relevant and targeted interventions. The lack of available results on quality of literacy and numeracy, key indicators of primary education, mean that local schools don’t have basis to improve and don’t allow regional and national ministries to optimize support. Globally, India and the east African countries of Tanzania, Kenya, and Uganda have successful models for school report cards that are greatly advanced with the use by tools for rapid tabulation and review of the results. This same model could be applied in Haiti at a lower-‐cost by utilizing the cell phone network and new mobile monitoring technologies. This could be carried out by existing employed government school inspectors to ensure a more accurate monitoring, provide incentives for local schools and communities, and support government and funder’s policy actions. Such a model also aligns with current scholarship on effective practices for literacy
Smaller: Assessment methods do not need to be major entrepreneurial enterprises, but rather just robust enough to answer key policy questions at the national and local levels […] Quicker: Literacy assessments need to be completed in ‘real time’ so that results can affect policy and spending in the ‘lifetime’ of current ministerial appointments. […] Cheaper: LDCs [less developed countries] cannot afford either the fiscal or human resources costs of deep involvement in highly technical assessment exercises. (Wagner, 2003, p. 294).
The literacy assessment project described in this paper follows a method that adheres to these guidelines in that it is done at the local level, disseminates results rapidly, and is low-‐cost. Most notably, it makes use of mobile technology to record and disseminate results that is reliable and easy to use.
Global examples of monitoring literacy and numeracy Assessment Case studies: India, East Africa Literacy assessment endeavors in India and East Africa provide applicable examples for assessment in Haiti. In India, the Annual Status of Education Report (ASER) conducts household-‐based reading tests for children. Their method serves to complement existing measures of monitoring progress in schools. The Indian government’s District Information System for Education (DISE) releases report cards
containing general information on schools, teachers, and students, such as the state of school infrastructure, funding, and the figures on student enrollment and grade repetition (Banerji and Wadhwa, 2010). These data are disaggregated by state, district, block, and cluster, but “independent, national data is scarce, especially on children’s achievement” (ibid). ASER’s method is independent from DISE, and also collects data on out-‐of-‐school children. One of the goals of ASER is to survey households in each rural district in India (Banerji and Wadhwa, 2010). In 2012, Results 16,166 villages were show 5 surveyed, in which levels of 596,846 children in reading 331,881 households were ability: no assessed (ASER, 2012). reading, Below are the results of letters, the 2012 literacy test in words, Hindi. The table provides paragraph, an example of how test story results can be organized and shared. Table 1: 2012 ASER literacy test results by grade (Std.)
plurality of children assessed, there are still nearly 20% of children who can only recognize letters, which may lead some educators or administrators to devote more effort toward getting children to read words. Another educational organization following this method of assessment is Uwezo, which operates in Tanzania, Kenya, and Uganda, with the goal of increasing children’s literacy and numeracy. Uwezo conducts the same kind of household-‐based assessments that ASER does in India, using the same test format (the content is translated and edited or rewritten for context). In addition to administering the tests, Uwezo writes annual reports on the findings of the assessments, one for each of the three countries surveyed, and one comprehensive East Africa report. The 2012 East Africa Report notes that in 2011, 350,000 children from more than 150,000 households were assessed (Uwezo, 2012). The report states that across East Africa, 32% of children in grade 3 passed—meaning they could read at “story level”—the Kiswahili literacy test, and only 16% passed the English literacy test.
Use of Mobile Phones in Data Collection
Source: ASER 2012 The ASER method allows relevant stakeholders and actors in education to easily understand how children are reading in broad terms. For example, while “story level” competency claims the greatest
The Earth Institute’s unique approach to literacy assessment incorporates cutting-‐ edge mobile technology into the process of recording and sharing results. Consequently, communication and action surrounding test results happens more quickly and has the potential to serve the very EI uses children who are assessed. mobile phones to Using mobile phones for collect data data collection in developing countries is
not a new concept, and has been done in many places by various organizations. DeRenzi et al (2007) make the point that using mobile phones enables field workers to “stay in disconnected environments for extended periods of time” (DeRenzi et al, 2007, p. 1), as workers can record data on the phones and then back up or transmit them when in a connected environment. Simply put, mobile phones—especially smartphones, which are essentially small computers—are a convenient, reliable tool to collect and store data when in the field: they eliminate the need to carry around sheaves of paper, pens, and other materials that are easily lost or damaged. Many documented instances of using mobile phones for data collection involve research in the health sector (DeRenzi et al, 2007, Tomlinson et al, 2009), but there is not a great deal of equivalent research in the education sector. There is, however, a growing interest in using mobile technology for learning, called m-‐learning (Traxler & Kukulska-‐Hulme, 2005, Barker et al, 2005). Using mobile phones to conduct literacy assessments, as described in this paper, is an innovative way of combining the known effectiveness of mobile data collection with the concept of m-‐learning. This is because assessments are more than data points; they serve to show educators and administrators how and what children are learning.
Education Monitoring in Haiti: What is being done, what can be improved Current Educational Monitoring in Haiti The MENFP’s 2010-‐2015 Operational Plan for the education system outlines its method for monitoring and evaluation in schools.
Large-‐scale educational assessments in Haiti take the form of summative exams to progress from Fundamental Current schooling (primary and junior assessment secondary) to secondary school, methods and again to graduate from are time-‐ secondary school. Both types of consuming exams are given at the end of the school year, which means that any information useful for school improvement can only be implemented during the following school year (at the earliest) and often does not benefit the actual students who took the exams. The Operational Plan also calls for reinforcing current means of monitoring and evaluation, as well as devising new methods for it. The goals include—among several others— making available academic statistics for the whole country on an annual basis, having all subsectors of education covered in the MENFP’s database, and creating a documentation center for the Ministry (MENFP, 2010, p. 25). The MENFP will organize several different agencies to put these M&E activities into effect, including Departmental Directorates, School District Offices, and Zone Inspection Offices. In other words, some M&E activities will be managed at relatively local levels.
The Earth Institute Method: Pilot Project Since current and planned activities to monitor educational quality in Haiti still place heavy emphasis on end-‐of-‐school exams and other annual forms of evaluation, a desirable shift would be for future assessment endeavors to align with Wagner’s “smaller, quicker, cheaper” model while also collecting much-‐needed information about other
elements of education, such as student and teacher attendance, sex ratios, and age of students. The Earth Institute’s literacy assessment pilot project aimed to address these issues, especially that of “quicker” assessments. In the Port-‐à-‐Piment Watershed project, in early summer, 2012, members of the Education Team piloted a new literacy assessment method in the watershed. In total, 50 children were assessed, 40 of whom were enrolled in traditional schools (either public or private) and 10 of whom were enrolled in a new, special program attached to a public school that caters to children who have not previously attended school. The tests used follow the ASER format. What made the testing in Haiti unique, however, was the way in which the results were recorded and Mobile disseminated. The Earth technology Institute uses formhub, used for an online platform— faster data created by the Modi collection Research Group at and Columbia University—for processing authoring surveys that can then be uploaded to Android-‐powered mobile phones and used to collect data. The surveys for the Haiti project, for example, recorded information about children’s age, sex, enrollment status, and their performance on the literacy test. Once the phones have access to wireless Internet, the surveys can be sent electronically back to formhub, where the data can be organized and disseminated. This type of technology allows for automated generation of report cards, which means that data from assessments can be used nearly instantly, thereby benefiting not just school administrators and policymakers, but potentially the students themselves who take the tests.
Implementation model There are two basic models for carrying out this type of assessment. One option is household-‐based, in which testers assess children in their homes, with family members present to observe the process. The other typical model is school-‐based, in which testing happens as part of regular school activities. The Earth Institute pilot project, because it worked with a relatively small sample size, was able to merge elements of these models. The tests were administered in schools to students who had been randomly selected ahead of time by school directors. The school directors notified these students’ families ahead of time, so that a parent or other family member could attend and observe the assessment. Each school director chose 5 students in the 3rd year of the 1st cycle of Fundamental education (primary school). Additionally, the Education Team met with Parent-‐Teacher associations to introduce and explain the process. Testing The team went to schools at specific times arranged by the school directors and by the Education Team. Many Family children were members accompanied by their observed mothers, but others were testing accompanied by fathers, grandparents, or in some cases, by siblings. Children were assessed one at a time, in order to make them less nervous and distracted. Testing spaces included the director’s office, an empty classroom, the yard behind the school, or in one case, a closed-‐off corner of a one-‐room schoolhouse. For each child and chaperone, a member of the Education Team would
introduce us as a group, explain what the test was and how it worked, and chatted with the child to make him or her feel at ease. Then one of them would ask identifying questions: name of the child, name of the child’s mother (or the name of the chaperone present), and the child’s age. Once this information was recorded on the phone, the tests were given. The tests and method of administering them were adapted from ASER’s practices. Students were asked to read aloud from a short test consisting of a short story (about ten sentences), two short paragraphs (four sentences each), a series of words (three to four letters each), and a series of letters. The tests were administered individually. First, one of the CRS animators held the test in front of the child as he or she read the short paragraphs aloud. The test was folded in half so that only the paragraphs are visible. If it was clear that the child could read the paragraphs, the tester flipped the paper so that only the story is visible; the child was then asked to read the story. If the child could successfully read the story and answer basic comprehension questions about it, he or she was marked at “story level.” If not, he or she is marked at “paragraph level.” If the child could not read the paragraphs— or read them with clear difficulty and did not truly understand them—he or she was asked to read the series of words. The tester repeated the action of folding the test in half so that at first only the words are visible. If the child could read the words, he or she is marked at “word level.” If not, the tester flipped the test to the other side and asked the child to read the series of letters. Results The results of the assessment covered a wide range of reading aptitude. In some schools,
the majority—if not all—of the children could read at some level, but these children were also often overage for their grade level. Furthermore, at other schools, Range of the majority of children chosen results; could not read at all—or could majority read only letters—regardless of could age. There were, however, some read story schools where the majority of the children were able to read the paragraphs or the story. Of the fifty children assessed, 14% were marked at “no reading,” but 56% were able to read the story.
Implications What the pilot project shows first and foremost is that this type of literacy assessment is fast, reliable, and easy to use. The technology in question, while sophisticated, is accessible to the multiple users. Therefore, anyone from teachers, to school directors, to regional educational inspectors could be trained to use it and to conduct assessments. It is also a low-‐cost and rapid tool. The phones needed to conduct assessments need only be purchased once, and the surveys can be created on any computer. Moreover, because tests can be administered multiple times during a school year, the Report Cards can be used to monitor students’ progress throughout the year; their information Frequent thus benefits teachers testing = and students, as well more as officials. The opportunities frequent testing could for be a means to measure improvement the effects of certain policy or curriculum changes in schools. The regular assessments can also serve as a method of monitoring student attendance and related details (such
as age and sex of children in school) and teacher presence. Consequently, the assessment process can evaluate educational quality in terms of learning as well as other relevant indicators.
Scaling up The Earth Institute is already scaling up these types of assessments in other locations. In partnership with EI is Uwezo, assessments conducting have been conducted assessments in the Millennium in other MVP Villages of Mbola, in Tanzania, Ruhiira, in Uganda, and Dertu and Sauri, in Kenya. Additionally, a quasi-‐experimental study using the mobile technology method is currently underway in the Millennium Village of Bonsaaso, in Ghana. Furthermore, these assessments are administered to five students per month as part of regular M&E surveys in schools. As for Haiti, given the ease and low cost of acquisition and use of this technology, this type of project has great potential to be scaled up to the whole country. Such an endeavor could be a solution to the lack of consistent monitoring of schools in Haiti. For example, the country already has a system of regional and zonal inspectors, responsible for monitoring public and private schools. These regional school inspectors could be easily trained in the survey method in order to administer tests in all the schools in their region. Report Cards would then be generated from the results, which could be transmitted to other Ministry of Education officials and back to the school administrators. If multiple inspectors conducted tests in their regions, the Report Cards could serve as a means of comparing
performance across regions and schools in the country. Such a scale-‐up would require a certain amount of advance planning, including developing tests, frequent communication between MOE officials and school directors (and possibly teachers), buying and programming mobile phones, and training the relevant actors in the process. Once the preparation is completed, however, the method is self-‐sustaining, especially because some of the costs are one-‐time expenses. That is, after the initial preparation, assessments can be conducted multiple times. Overall, this kind of endeavor aligns with the MENFP’s plan for gathering academic statistics and using local government agencies to carry out M&E tasks.
Conclusion The type of literacy assessment method described here is useful in two main ways: first, it is a fast, easy, and low-‐cost Assessments way to measure measure children’s reading learning as well skills. Second, it can as other M&E serve as a tool for indicators monitoring school quality, not only as far as learning is concerned, but in terms of other important factors such as enrollment, attendance, teacher presence, age and sex of children. For Haiti, this endeavor has the potential to complement and promote existing M&E activities in schools. Additionally, it can improve educational quality for the children who are assessed by attending to their reading needs in real time.
Appendix A) Screen shots of the phone-‐based survey
B) Photos from pilot
C) Sample Report Card
Bibliography ASER. (2012). Enrollment and learning report card. Retrieved from http://www.asercentre.org/education/India/status/p/143.html Banerji, R. & Wadhwa, W. (2010). Annual status of education report 2005-‐2010: Key features of ASER. Retrieved from www.asercentre.org Barker, A., Krull, G., & Mallinson, B. (2005, October). A proposed theoretical model for m-‐ learning adoption in developing countries. In Proceedings of mLearn (Vol. 2005, p. 4th). DeRenzi, B., Anokwa, Y., Parikh, T., & Borriello, G. (2007, August). Reliable data collection in highly disconnected environments using mobile phones. In Proceedings of the 2007 workshop on Networked systems for developing regions (p. 4). ACM. Ministère de l’Education Nationale et de la Formation Professionelle (Ministry of Education and Professional Training). (2010). Vers la refondation du système éducatif haïtien: Plan opérationnel 2010-‐2015 des recommendations de la commission présidentielle éducation et formation (Towards the overhaul of the Haitian educational system: Operational Plan 2010-‐2015 from recommendations from the presidential education and training commission). Port-‐au-‐Prince, Haiti: Ministère de l’Education Nationale et de la Formation Professionnelle. Traxler, J., & Kukulska-‐Hulme, A. (2005). Mobile learning in developing countries. Uwezo. (2012). Are our children learning? Literacy and numeracy across East Africa. Retrieved from http://www.uwezo.net/publications/reports/ Wagner, D. (2003). Smaller, quicker, cheaper: alternative strategies for literacy assessment in the UN Literacy Decade. International Journal of Educational Research, 39, pp. 293– 309.
