Expert Appraisal of 2011 Superior Multimedia Award Finalists
The purpose of this appraisal is to provide a determination of the 2011 Superior Multimedia Award winner among the two selected finalists. The award finalists are ZeGenie’s interactive, online high school mathematics course, and Conspiracy Code: American History, a 3D edutainment product developed by Florida Virtual School. To choose which product is superior, the products will be evaluated using a rubric based on Baumgartner and Payr’s (1997) Qualitative Weight and Sum (QWS) method (pp. 44-50). The rubric covers five Usability categories and six Pedagogical categories. The original rubric was developed by the DETC 620 Group 2 jury, which is comprised of Shawn Grant, Mary Hollister, Susan Johnston, Cheryl Jones-Strong, Jennifer Keer, Jill Kriss, Charles Lawson, and Teresa Marcheskie, and then further modified by the jury Chairperson, Charles Lawson. The completed rubrics for both ZeGenie and Conspiracy Code can be found following the Reference section.
Appraisal Method
Online and computer based multimedia instructional programs have become so plentiful that many have taken to cataloguing them. One such catalog is the Merlot collection of learning materials found at www.merlot.org. As of this writing, the catalog boasts almost 28,000 unique learning materials. Understanding the importance of systematically evaluating the growing array of interactive multimedia in a more accurate way, Baumgartner and Payr set out to analyze a variety of methods. The method they advanced, QWS, lacked the flaws of the traditional numerical weight and sum (NWS) method of evaluation, since it allowed for a qualitative basis, instead of a quantitative basis which could not be justified (1997, p.47). Using Scriven’s (1991) method of weighting with symbols instead of numbers, the QWS eliminates numerical scales in both the weighting of categories as well as the rating of criteria (p. 294). For this reason, QWS was selected as the evaluation method for this appraisal.
Reeves and Harmon (1994) stipulate that both usability and pedagogical criteria should be selected to evaluate interactive multimedia (IMM) instruction. “Pedagogical dimensions are concerned with those aspects of the design and implementation of IMM that directly affect learning. User interface dimensions are concerned with those aspects of IMM that ensures the learner can actually engage in a meaningful interaction” (p.474). The jury for Group #2 set out to analyze the most essential usability and pedagogical criteria, and to determine how the evaluator would be able to discern whether the criteria was met. To do this, the group created two sub-groups, one that focused primarily on usability, the other pedagogy. The group then placed each criterion into a broader category. Finally, each category was assigned a weight, using the symbols of * (very important), # (important), and + (less important). The E (essential) weight was not needed, since these two products were finalists that had already been established to have met all essential criteria.
Assessment of Usability
The first finalist evaluated, ZeGenie, is a comprehensive set of short math lessons that cover the major learning objectives of a typical high school algebra, geometry, trigonometry, statistics, and probability. The student would drill down through a menu to locate the specific lesson he or she wishes to view. For example, the student could choose Trigonometry > Radians > Length of Arc, at which point the Virtual Teacher window would appear. The student would view an animated, narrated lesson which would focus on that single topic. Within the lesson, there may or may not be one or more interactive exercises, such as typing text into a field to solve a problem. The student can rewind, fast forward, and pause the animation, or make it full screen, but unfortunately there is no text equivalent of the majority of the narration, so the hearing impaired would struggle with this application. While the interface is simple and intuitive, it does lack clear directions for the interactive exercises.
Conspiracy Code was the second finalist evaluated. It is a game-like immersive environment in which the student takes on the role of a pair of high school aged characters. The product has little initial information, but directions are given on a gradual, as needed basis. It will take students some time to learn the intricacies of the game play, but students used to video games will find this exploration rewarding. There does not appear to be a “game save” option, which seems to require a student to take the entire lesson at one sitting. The interface appearance is very polished, which easy to perform mouse driven controls. While this application may not run on some older computers, it will operate in a compatibility mode to allow most Windows computers to operate the software well. There is no Mac compatibility though. This application would be impossible for a visually impaired person to use.
Assessment of Pedagogy
The ZeGenie content is written at a high school reading level, which is appropriate for its audience. The content is well chunked, allowing for a student to reflect often on what they have seen and heard. Each subject if well scaffolded; if the student chooses to follow the order presented, he or she will encounter prerequisite information prior to being presented new information. The student is given a great deal of flexibility to choose his or her own path through the software. While the assessments are limited, there are sufficient interactions with the student to keep him or her interested. The content does focus on higher level math thinking skills. It would have been useful to have made some sort of access to other students in the course, to help create a more connectivist and/or constructivist learning environment. There is a nice feature where students can send comments to an instructor who would respond to their email account on file.
Conspiracy Code, like ZeGenie, is written at an appropriate, high school, reading level. Since it does not use a traditional format, one must consider the pedagogical criteria in a different light. For example, chunking is not as apparent, but since information is released gradually to the user, this can be considered a form of chunking – in other words, there is a clear and guiding structure to the content that provides a manageable cognitive load. On the other hand, the degree of interaction was extremely apparent. Interaction is furthermore both meaningful and plentiful. One area that could use improvement is the use of exercises that could be considered authentic for the purpose of either reflection or assessment – the learning material sometimes is overwhelmed by the game play.
Results
The evaluation of the finalists produced the following results:
ZeGenie: *: 24 #: 16 +: 5 0: 2
Conspiracy Code: *: 25 #: 15 +: 5 0: 2
By the slimmest of margins, Conspiracy Code bested ZeGenie, making it the winner of the 2011 Superior Multimedia Award. The most surprising aspect of the results was not the winner, though, but the fact that the winner, which used very costly and impressive technology, had barely won. Many who had simply seen both finalists had assumed that Conspiracy Code would easily best ZeGenie. But the QWS evaluation showed that both products had significant strengths in certain areas, such as Navigation and Content, but were somewhat lacking in the same areas of Objectives, Higher Order of Thinking, and Accessibility. This close result shows that when the core criteria of successful multimedia learning materials are thoughtfully considered, this will outweigh the charismatic look and feel that a program such as Conspiracy Code has, but a more traditional multimedia product such as ZeGenie does not.
Conclusion
ZeGenie’s High School Mathematics course and Conspiracy Code’s American History course proved to be worthy finalists, with each receiving high marks in most areas. While Conspiracy Code’s game-like 3D interface helped propel it to victory, its 2011 Superior Multimedia Award was obtained by incorporating key usability and pedagogical standards. Bells and whistles can only take a multimedia learning product so far; to meet its learning objectives, a product requires sound instructional and technological design, which both of the finalists had in abundance.
References
Baumgartner, P., & Payr, S. (1997). Methods and practice of software evaluation: The case of the European Academic Software Award (EASA). In Proceedings of ED-MEDIA 97 - World Conference on Educational Multimedia and Hypermedia (pp. 44-50). Charlottesville: AACE.
Reeves, T. C., & Harmon, S. W. (1994). Systematic evaluation procedures for interactive multimedia for education and training. In S. Reisman (Ed.), Multimedia computing: Preparing for the 21st century (pp. 472-505). Hershey, PA: Idea Group Publishing.
Scriven, M. (1991). Evaluation Thesaurus. 4th ed. Newbury Park: SAGE.
Appraisal Method
Online and computer based multimedia instructional programs have become so plentiful that many have taken to cataloguing them. One such catalog is the Merlot collection of learning materials found at www.merlot.org. As of this writing, the catalog boasts almost 28,000 unique learning materials. Understanding the importance of systematically evaluating the growing array of interactive multimedia in a more accurate way, Baumgartner and Payr set out to analyze a variety of methods. The method they advanced, QWS, lacked the flaws of the traditional numerical weight and sum (NWS) method of evaluation, since it allowed for a qualitative basis, instead of a quantitative basis which could not be justified (1997, p.47). Using Scriven’s (1991) method of weighting with symbols instead of numbers, the QWS eliminates numerical scales in both the weighting of categories as well as the rating of criteria (p. 294). For this reason, QWS was selected as the evaluation method for this appraisal.
Reeves and Harmon (1994) stipulate that both usability and pedagogical criteria should be selected to evaluate interactive multimedia (IMM) instruction. “Pedagogical dimensions are concerned with those aspects of the design and implementation of IMM that directly affect learning. User interface dimensions are concerned with those aspects of IMM that ensures the learner can actually engage in a meaningful interaction” (p.474). The jury for Group #2 set out to analyze the most essential usability and pedagogical criteria, and to determine how the evaluator would be able to discern whether the criteria was met. To do this, the group created two sub-groups, one that focused primarily on usability, the other pedagogy. The group then placed each criterion into a broader category. Finally, each category was assigned a weight, using the symbols of * (very important), # (important), and + (less important). The E (essential) weight was not needed, since these two products were finalists that had already been established to have met all essential criteria.
Assessment of Usability
The first finalist evaluated, ZeGenie, is a comprehensive set of short math lessons that cover the major learning objectives of a typical high school algebra, geometry, trigonometry, statistics, and probability. The student would drill down through a menu to locate the specific lesson he or she wishes to view. For example, the student could choose Trigonometry > Radians > Length of Arc, at which point the Virtual Teacher window would appear. The student would view an animated, narrated lesson which would focus on that single topic. Within the lesson, there may or may not be one or more interactive exercises, such as typing text into a field to solve a problem. The student can rewind, fast forward, and pause the animation, or make it full screen, but unfortunately there is no text equivalent of the majority of the narration, so the hearing impaired would struggle with this application. While the interface is simple and intuitive, it does lack clear directions for the interactive exercises.
Conspiracy Code was the second finalist evaluated. It is a game-like immersive environment in which the student takes on the role of a pair of high school aged characters. The product has little initial information, but directions are given on a gradual, as needed basis. It will take students some time to learn the intricacies of the game play, but students used to video games will find this exploration rewarding. There does not appear to be a “game save” option, which seems to require a student to take the entire lesson at one sitting. The interface appearance is very polished, which easy to perform mouse driven controls. While this application may not run on some older computers, it will operate in a compatibility mode to allow most Windows computers to operate the software well. There is no Mac compatibility though. This application would be impossible for a visually impaired person to use.
Assessment of Pedagogy
The ZeGenie content is written at a high school reading level, which is appropriate for its audience. The content is well chunked, allowing for a student to reflect often on what they have seen and heard. Each subject if well scaffolded; if the student chooses to follow the order presented, he or she will encounter prerequisite information prior to being presented new information. The student is given a great deal of flexibility to choose his or her own path through the software. While the assessments are limited, there are sufficient interactions with the student to keep him or her interested. The content does focus on higher level math thinking skills. It would have been useful to have made some sort of access to other students in the course, to help create a more connectivist and/or constructivist learning environment. There is a nice feature where students can send comments to an instructor who would respond to their email account on file.
Conspiracy Code, like ZeGenie, is written at an appropriate, high school, reading level. Since it does not use a traditional format, one must consider the pedagogical criteria in a different light. For example, chunking is not as apparent, but since information is released gradually to the user, this can be considered a form of chunking – in other words, there is a clear and guiding structure to the content that provides a manageable cognitive load. On the other hand, the degree of interaction was extremely apparent. Interaction is furthermore both meaningful and plentiful. One area that could use improvement is the use of exercises that could be considered authentic for the purpose of either reflection or assessment – the learning material sometimes is overwhelmed by the game play.
Results
The evaluation of the finalists produced the following results:
ZeGenie: *: 24 #: 16 +: 5 0: 2
Conspiracy Code: *: 25 #: 15 +: 5 0: 2
By the slimmest of margins, Conspiracy Code bested ZeGenie, making it the winner of the 2011 Superior Multimedia Award. The most surprising aspect of the results was not the winner, though, but the fact that the winner, which used very costly and impressive technology, had barely won. Many who had simply seen both finalists had assumed that Conspiracy Code would easily best ZeGenie. But the QWS evaluation showed that both products had significant strengths in certain areas, such as Navigation and Content, but were somewhat lacking in the same areas of Objectives, Higher Order of Thinking, and Accessibility. This close result shows that when the core criteria of successful multimedia learning materials are thoughtfully considered, this will outweigh the charismatic look and feel that a program such as Conspiracy Code has, but a more traditional multimedia product such as ZeGenie does not.
Conclusion
ZeGenie’s High School Mathematics course and Conspiracy Code’s American History course proved to be worthy finalists, with each receiving high marks in most areas. While Conspiracy Code’s game-like 3D interface helped propel it to victory, its 2011 Superior Multimedia Award was obtained by incorporating key usability and pedagogical standards. Bells and whistles can only take a multimedia learning product so far; to meet its learning objectives, a product requires sound instructional and technological design, which both of the finalists had in abundance.
References
Baumgartner, P., & Payr, S. (1997). Methods and practice of software evaluation: The case of the European Academic Software Award (EASA). In Proceedings of ED-MEDIA 97 - World Conference on Educational Multimedia and Hypermedia (pp. 44-50). Charlottesville: AACE.
Reeves, T. C., & Harmon, S. W. (1994). Systematic evaluation procedures for interactive multimedia for education and training. In S. Reisman (Ed.), Multimedia computing: Preparing for the 21st century (pp. 472-505). Hershey, PA: Idea Group Publishing.
Scriven, M. (1991). Evaluation Thesaurus. 4th ed. Newbury Park: SAGE.