Proceedings of the 8th WSEAS International Conference on EDUCATION and EDUCATIONAL TECHNOLOGY

Smart Question (sQ): Tool for Generating Multiple-Choice Test Questions H. FATIH UGURDAG1, ERHAN ARGALI1, OGUZ E. EKER1 ALI BASARAN2, SEZER GÖREN1, HÜSEYIN ÖZCAN1 1 Computer Engineering Department 2 Electrical and Electronics Engineering Department Bahcesehir University Ciragan Cad. 34353, Besiktas, Istanbul TURKEY [email protected] http://ugurdag.com Abstract: - When putting together collections of practice questions for multiple-choice standardized tests in math/science, it is important that we test a student’s ability on a particular type of question over and over again. Thus, it is best to present a question with different numbers and/or shuffled answers. As a result of that, thousands of combinations can be produced from a particular question. Smart question is a concept where a static question, with fixed numbers and answer choices, is converted to a dynamic question with parameters. The parameters have a list of possible values, and the answer choices are unordered. A smart question is turned into specific static question instances through question generation software. Smart question concept simplifies generation of numerous similar questions from a question template, and it also saves tremendous space in a database. We have developed a software tool called sQ, which facilitates question design with its sQd interface, allows creation of parameter specification lists with sQp, and generates questions through sQg. Key-Words: - Test questions, Question authoring, Automatic generation, Storage optimization. sitting close to each other to prevent cheating. In addition to shuffling the order of questions, slightly modifying every question increases the reliability of the test. Besides shuffling the answer choices, we can turn the numbers given in a question into parameters and have a list of possible parameter combinations or a range or formula for each parameter. Consider a question that asks the hypotenuse length of a right triangle. There are at least the well-known combinations 3-4-5, 5-12-13, and multiples of them. On top of parameterizing supplied data and the right answer, we can also parameterize wrong answers. One can easily come up with more than a million versions of the same math/science question. On multiple-choice tests, there are usually 5 answer choices, meaning at least 5! (=120) versions come from just shuffling the answer options. Also, extra options for wrong answers can be included to pick from. For instance, if the number of possible wrong answers for a question is 10 in the database, that means the number of question versions has another factor of C(10,4)=210. Number of possibilities drastically increases as we analyze further. If there are, for example, 50 versions as long as the parameters are concerned, they all add up to 1,260,000 (=120*210*50) versions for that question.

1 Introduction Today millions of practice questions are authored for the purpose of preparing students for multiplechoice standardized tests in math/science. The goal of a practice test is to measure how well a student is able to solve certain question templates. To make sure a student is able to consistently answer a question type, test centers/books need to expose the student to the same question several times over various practice tests. However, when we repeat a question, presenting an identical copy of it may not be wise. The student may answer the question correctly with the help of his/her photographic memory. That is, he/she remembers the correct answer or simply position of the answer choice. Therefore, to be sure a student is able to answer a particular type of question, we need to at least shuffle the multiple answer choices, or better yet, we should present a new version of the question to the student with different numbers and different answer choices. Creating different versions of a question is necessary also when handing out several versions of a test to a class of students. In this case, it is important that we ask the same questions to every student for fairness reasons. However, we should give different versions of the same test to students

ISSN: 1790-5109

173

ISBN: 978-960-474-128-1

Proceedings of the 8th WSEAS International Conference on EDUCATION and EDUCATIONAL TECHNOLOGY

Authoring and storing so many versions of a question is troublesome. We cannot manually create them. Also, they take up too much space once they are created. To address this problem, we came up with a concept called Smart Question. Smart question is a way of storing all static versions of a question in one dynamic question. When accompanied by a software tool, this concept makes it possible to create a myriad number of versions of the same question on the fly, which is otherwise impossible. It also saves incredible amount of disk space, which, in turn, reduces hardware costs and database access times when a practice test is put together. Without smart questions, we have to manually create many versions of a question with fixed answer choices and parameters, and then tag them to show that they are actually the same question in disguise. A smart question contains all that information in a very compact and easy-to-modify way. We have developed a software tool called sQ to realize automatic generation of questions. The rest of this paper includes a literature review and then the details of sQ tool and the specific type of smart questions it creates. In the last section, we outline the future work necessary to make this concept and software a part of a complete system that administers practice tests.

The above works have to do with test generation. Our work is on generating individual questions. Very few commercial software currently do that. One of them is TestMakPro2, which is listed under Image-ination.com in [1]. TestMakPro2 can author questions with 3 parameters at the maximum and is designed mainly for text-based algebraic questions. In academia, the problem of question generation has been mostly addressed in specific domains so far. Stanescu et al. [5] generate questions from course material. Cristea et al. [6] generate questions in the area of electrical circuit analysis. Hoshino et al. [7] generate grammar and vocabulary questions. Aldabe et al. [8] have a similar work in which they generate questions for teaching the Basque language. Hsiao et al. [9] generate questions in the domain of object-oriented programming. Although the type of question generation in the work of Holohan et al. [10] is ontology based and more general purpose in its essence, it has been show-cased only in generation of SQL related computer science questions. The work of Papasalouros et al. [11] is also ontology based and is shown in the context of generating multiple-choice language reading questions from an essay, where answer choices are sentences derived from the ontology. This work also pays special attention on generation of meaningful wrong answers, which they call distractors.

2 Literature Review

3 Smart Question

There is a plenty of prior work on generating tests from a databank of static questions. Software for that is common place and commercially (and even freely) available. You may see a list of such software in [1, 2]. Academic community is currently focusing on adaptive methods of picking question sequences [3]. An adaptive method can be run before a student takes a test, and the test can be composed based on the student’s profile [4]. Another form of adaptive selection of questions is used when a question is picked only after the student answers the previous question. This is currently used even in some electronically administered standardized tests. It is also used in online tutoring systems. The purpose here is to assess the knowledge of a student in a particular subject area. If a student cannot answer a question of a certain difficulty level, there is no reason one should ask him/her a harder question. With this in mind, systems with adaptive question sequences use a decision making process before posing each question.

Ideally, a smart question would be metadata, which includes all kinds of data such as text, formulas, images, video, animation, etc. However, the scope of our work in this paper is limited to practice questions for math/science standardized tests in the form of an image file. In Turkey, there is a huge industry around standardized tests with multiple choice questions, most of which contain math/science questions. Just the portion of this industry that serves the takers of the national university entrance exam (1.5 million students annually and mostly high school seniors) produces a yearly revenue of at least 1.5 billion US dollars. When other standardized tests are also included, test centers and publishers in Turkey generate a combined revenue of 5 billion US dollars in a year. Some of the test centers and publishers of this industry easily trace their roots back to 40 years ago. They have created an archive of questions over time. When computers and computerized publishing came into mainstream use, they transformed their question archives into electronic databases by scanning

ISSN: 1790-5109

174

ISBN: 978-960-474-128-1

Proceedings of the 8th WSEAS International Conference on EDUCATION and EDUCATIONAL TECHNOLOGY

questions that were previously on paper. Hence, such question archives currently consist of image files (one per question). That is why our software tool sQ starts by reading in an image file (i.e., a regular static question). A smart question consists of an unaltered image file (i.e., the original static question), a parameter definition file (.sQd file), and a parameter specification file to generate instances of the smart question (.sQp file). sQ authors a smart question through its 3 interfaces (or tabs), namely, sQd (sQ design), sQp (sQ parameterize), and sQg (sQ generate). sQ starts with the image of a static question. If we are to design a smart question from scratch, we are still expected to design a static question (a particular instance of the smart question) with ordinary software tools and save it as an image file in bitmap or JPEG format. sQ is like a drawing tool; it lets the question designer mark areas in the image as parameters (job of sQd) and save them in an sQd file. sQp part of sQ reads in the sQd file and lets the user create a specific instance of the question by entering parameter values. Parameter values immediately show up in the image displayed as they are entered; however, the image file is not modified. A question instance is saved as an entry in the sQp file. sQg reads in an sQp file, turns numbers or symbols into subimages, updates the image and saves the generated question as a new image file (i.e., a new static question). Creating new versions through shuffling of the multiple choices is also sQg’s responsibility.

cover the text underneath (i.e., old values). Arrange Answer lines up all choices vertically. When we start creating a new parameter/answer, the new one is shown in red, whereas the previously created ones show in blue. We can select a blue box by pointing on inside it and click. When a box is selected, we can resize it by pushing the plus and minus signs under where it says Resize. In the dropdown box where it says Y-Max, we have 3 other choices. Y-Max means the y-coordinate of the bottom of the box to be resized. This is called Bottom in the parameter definition (parDef) lines in the bottom subwindow. Y-Min is Top; X-Min is Left; X-Max is Right. The tuple called Color in a parDef line is the background color of the parameter. This is used when sQg generates the pixels of a parameter value in the default foreground color of black. However, background color may vary and has to be saved in the sQd file for each parDef. sQd picks up the background color from the first corner we started drawing the parameter box. The last value (Type) in a parDef line indicates a regular parameter when it says parameter and indicates an answer choice when it says answer.

3.1 Smart Question Design (sQd) sQd tab (Fig. 1) is the interface where the user defines the location (i.e., box) of each parameter. After defining a parameter, a new line is created in the subwindow in the bottom. The contents of that subwindow are what are saved in the sQd file. We start by loading an image file through a selection in the File menu. The two buttons, under where it says Toolbox, let us start defining a new parameter. Left button (with a pencil) creates a new regular parameter, whereas the one with a plus sign creates a new answer choice. Once one of those two buttons is clicked, we can draw a rectangular box. That box is where sQg embeds a new subimage to create a new version of the question. If you click on Match Size, sizes of all choice boxes are made equal (height and width) and match the size of the first choice created. This is critical because when the answer choices are shuffled, the boxes have to be interchangeable (i.e., swappable). When we swap choice B and A, they have to fully

ISSN: 1790-5109

Fig. 1 sQd interface The trash can deletes all parameters in the current session and sQd file that is being manipulated. The next button (dash) adds the current parameter just defined. The next button is the refresh button, in case the display gets out of sync. The dropdown box to the right shows the name of the selected (red) box. It is a4 because an answer is selected (not parameter). Parameter names start with letter p. (Note that parameter/answer indices start from 0.)

3.2 Smart Question Parameterize (sQp) As the parameters are tagged (i.e., defined) in the sQd interface, they are saved in an sQd file. This way, the parameters can be located every time the

175

ISBN: 978-960-474-128-1

Proceedings of the 8th WSEAS International Conference on EDUCATION and EDUCATIONAL TECHNOLOGY

sQd file is loaded. Basically, the sQd file plus the image file represents the parameterized question (i.e., smart question), whereas sQp and sQg function when the user needs to create specific instances of that template. Hence, sQd is definitely authoring tool. sQp can be regarded as either an authoring or generation tool while sQg is definitely a generation tool. In order to select an answer or parameter; all the user has to do is click a pixel nearby. Once a parameter is selected, a new value can be entered in the big textbox near the top. When the plus button in Fig. 2 is pushed, a new parameter value is added to the current parameter specification line (parSpecLine), which appears right on top of the bottom subwindow, and then the image of the question on the screen is updated. A parSpecLine is a list of parameter specifications (parSpecs). Each parSpec is a pair of parameter name and value. When a parameter is not specified in a parSpecLine, it is not overwritten by a new value (hence a subimage) when the question is regenerated. In Fig. 2, the current parSpecLine has 6 parSpecs. a1 is the name of the parameter of the first parameter specified in this parSpecLine and 10\p (10π) is its value. p6 is the last parameter of the current parSpecLine (highlighted in blue). parSpecs are separated with a forward slash. When the check button is pushed, the current parSpecLine is added to the list of parSpecLines displayed in the bottom subwindow, and the sQp file is updated.

Toolbox and use a GUI that enters the code for LaTex symbols.

Fig. 3 LaTex toolbox

3.3 Smart Question Generate (sQg) sQg is the final stage of sQ, which generates subimages for modified parameters, embeds them in the parameter boxes to create the complete image of a new question instance. sQg shuffles the answers. It can read in an sQp file as well as an sQd file, then displays the sQp entries (parSpecLines) in its lower subwindow. sQg creates a new image for the selected parSpecLine and displays it on the screen.

Fig. 4 sQg interface When the icon with a pencil is clicked, sQg generates images based on the checkboxes in the bottom and shows the image of the last question instance. The floppy icon saves the images. sQg turns each parSpec into a subimage. For that it has to parse a parameter value in a parSpec, which may have LaTex keywords, and create a subimage with the specified numbers/symbols and appropriate background color. It then overlays all created subimages on the original image of the question and creates a single image.

Fig. 2 sQp interface The red stop button resets the current parSpecLine, whereas the trash button trashed the whole sQp file and clears the bottom subwindow. The big textbox in the top left corner is where we can enter the spec for the selected parameter. This textbox supports LaTex syntax. If the user is not familiar with LaTex language, he/she can click on

ISSN: 1790-5109

176

ISBN: 978-960-474-128-1

Proceedings of the 8th WSEAS International Conference on EDUCATION and EDUCATIONAL TECHNOLOGY

[2] http://www.filetransit.com/topfile.php?name= Free_Download_Question_Paper_Generation_ Software [3] A.T. Fong, H.H. Siew, P.L. Yee, L.C. Sun, IOAS: An Intelligent Online Assessment System, Journal of WSEAS Transactions on Computers, Vol. 6, No. 3, 2007, pp. 552-559. [4] P.D. Cristea and R. Tuduce, Test Authoring for Intelligent E-Learning Environments, Inter. Workshop on Authoring of Adaptive and Adaptable Educational Hypermedia (part of WBE), 2004. [5] L. Stanescu, C.S. Spahiu, A. Ion, and A. Spahiu, Question Generation for Learning Evaluation, Inter. Multiconf. on Computer Science and Information Technology, 2008, pp. 509-513. [6] P.D. Cristea and R. Tuduce, Automatic Generation of Exercises for Self-Testing in Adaptive E-Learning Environments: Exercises on AC Circuits, Inter. Workshop on Authoring of Adaptive and Adaptable Educational Hypermedia (part of WBE), 2005. [7] A. Hoshino, L. Hunan, H. Nakagawa, A Framework for Automatic Generation of Grammar and Vocabulary Questions, Inter. Conf. on Using Technologies for Language Learning (WorldCALL), 2008. [8] I. Aldabe, M.L. de Lacalle, M. Maritxalar, E. Martinez, and L. Uria, ArikIturri: an Automatic Question Generator Based on Corpora and NLP Techniques, (originally appeared in Inter. Conf. on Intelligent Tutoring Systems – ITS) Lecture Notes in Computer Science, Vol. 4053, pp. 584594, 2006. [9] I. Hsiao, P. Brusilovsky, and S. Sosnovsky, Web-based Parameterized Questions for ObjectOriented Programming, World Conf. on ELearning in Corporate, Government, Healthcare, and Higher Education (ELEARN), 2008. [10]E. Holohan, M. Melia, D. McMullen, and C. Pahl, The Generation of E-Learning Exercise Problems from Subject Ontologies, Inter. Conf. on Advanced Learning Technologies (ICALT), 2008. [11]I. Hsiao, P. Brusilovsky, and S. Sosnovsky, Web-based Parameterized Questions for ObjectOriented Programming, World Conf. on ELearning in Corporate, Government, Healthcare, and Higher Education (ELEARN), 2008.

4 Future Work and Conclusion Besides many small and not so small improvements and additional features, one has to think about how smart questions could become a part of bigger solutions. Let’s first consider a test generation environment. sQ could be used just to solve the generation problem but not the storage space problem. In that case sQ would quickly generate question instances, and they would be added to the question bank. To solve the space problem, one has to write a layer between the test generation software and sQp files. We could modify the sQp files such that all questions are kept in a big sQp file, and every parSpecLine points to an image file as well as an sQd file. This way, parSpecLines become the questions that the test generation software (or other learning software) actually queries. sQ could be extended to author and generate text based questions, and later on could be even extended to questions with metadata. If sQ generated questions in the standard SCORM format, then its generated questions would be available for import into major software packages, including WebCT and Blackboard. Being able to run sQg in batch mode would greatly speed up question generation process. We need to create a unique and meaningful file name for generated questions, especially to see if an answer shuffling produced one of the previously generated answer combinations. This way, there would not be duplicate question instances. We should also look into generating subimages for parameter boxes with transparent background. Currently, it is a hassle to start drawing the box from a background pixel. Also, the background may sometimes not be a single plain color. In conclusion, we have demonstrated the concept of smart question and a prototype software tool that lets people author smart questions and generate static question instances from a smart question. Smart question is unique from similar works in terms of its question authoring paradigm and target domain. It is geared more towards math/science questions, although it could be applicable to other domains. In terms of its authoring process, it starts with a static question in the form of an image and offers a graphical tool to annotate the image and saves the annotation data into sQd and sQp files. References: [1] http://www.dmoz.org/Computers/Software/ Educational/Teachers_Help/Test_Authoring

ISSN: 1790-5109

177

ISBN: 978-960-474-128-1