Here is my preliminary thinking on the topic:
Evaluating learning outcomes in a 3D Virtual Immersive Environment (VIE) should not be done in the traditional fashion of a paper and pencil multiple choice assessment. Using a multiple-choice assessment for 3D learning is like using a yard stick to measure the amount of steam produced by boiling water. You cannot possibly obtain an accurate measurement. See my discussion with Sherry Engel on this topic in her blog post Learning in 3D: Measuring the Impact.
Learning in a 3D VIE is meant to be experiential. While it is possible to craft 3D environments to teach facts and declarative knowledge, the real power of these virtual immersive environments is to allow learners to experience concepts, solve problems and work on skills like negotiation, sales acumen and team work (so called “Soft skills”).
Assessment of knowledge gained in terms of concepts, problem-solving and soft skills cannot be accurately measured with true/false or multiple choice questions. They need to be measure using different techniques such as rubrics and performance-based checklists.
Rubric
One way to construct a rubric for assessing in-world activity is to create a hierarchy of desired skills. The lowest level (and fewest points) is the minimal acceptable behavior or activity from a learner. For example, if you are teaching sales skills, it might be the opening used by the learner to begin a sale. The learner might say “Hi, my name is Juan” that might be the minimum and the learner received 5 points.
The best opening might be “Hello, my name is Juan Hertz and I am from ABC company and I’m here to talk to you today about maximizing your company’s revenue by minimizing idle inventory, do you have a few moment?” and the learner would receive 25 points.
When observing the learner role-playing in an authentic situation, the instructor assigns a skill level to the behavior and the skill level is associated with a score. This can even be done when asking learners to create content in-world. For example, you might teach a lesson on sustainability in home construction and then ask the learner to build a “green” home in a virtual world. Then the students could be assessed by the instructor walking around the green home and checking observing the sustainability element the learners included such as solar-panels, tankless hot water heater. If they included the minimum, they would receive a low score. If the integrated the elements together and carefully thought through the design, they’d receive a higher score.
The key is to design the rubric as a way of moving a learner from the state of minimal knowledge, to the state of mastery. The skill involved in creating the rubric is to understand how learners move from basic skills to the level of mastery.
Performance-based Checklist
A checklist can be used when observing the learner for specific behaviors that can be seen or not seen during an activity. Such as, “Did the officer ask to see the driver’s license?” or “Did the student check to ensure the safety guard was locked before attempting to operate the equipment?” These performances can be taught in 3D virtual spaces and then assessed either in the space as the learner runs through a simulation or, more appropriately, have the assessment occur in the physical world to note the transfer of skills and knowledge from the virtual to physical world.
Performance-based checklists can be assigned points to each item (weighted if necessary) and calculated based on how many of the items on the checklist were properly completed by the learner, they can be timed as well to see if time requirements are met.
Assessing learners in a 3D VIE should use techniques similar to assessing the learners in a physical location. You want to assess transfer of learning, ability to perform the right tasks and the ability to critically problem-solve. Two methods that help are rubrics and performance checklists but there are more.
How are you assessing learning in 3D virtual worlds, I’d love to hear some ideas?
__
Recommended Games and Gadgets
Recommended Books
Content Guide
Posts
4 comments:
Hi Shelly,
Check out TIE - http://k12.emantras.com/product-tie.html TIE measures learning in a virtual world through many interactive exercises. Let me know if you are interested and I can give you a demo o how we use labs, quizzes, pick and drop and various gaming interactive exercises to measure learning.
Regards,
Ajay (Mobl21)
While I agree with your post I want to add a voice to those who are using VIE for things other then the soft skills described in your post.
While I often dream of teaching architecture or marketing or a real estate course, etc. that leverages the power of VIE in the ways you describe in your post and your book I don't. I teach accounting and use Second Life effectively (I think) in using that platform to do two things: 1) Provide a way to engage students in what is often considered a not so engaging course, and 2) Providing a 3-D tangible interactive model of what is otherwise an abstract concept (the accounting equation).
To that end, I don't think assessing learning should be any different using a VIE then any other learning tool, but it IS important to assess if the VIE is accomplishing what is expected. To that end I have a paper that is forthcoming in the journal Issues of Accounting Education that shows a relationship between student engagement while using Second Life and exam performance. I also collect data about students perceptions of the the VIE and blogged about that here: http://www.mydebitcredit.com/2010/01/08/second-life-what-do-the-students-think/
Ajay--thanks for your comment, I will check it out, seems interesting. I would certainly like a tour at some point, thanks for the offer and I'm looking forward to it.
RobinsHermano--You make a really good point, creating a model to visualize data that is not normally visualized is a great teaching advantage of 3D worlds and determining the associated learning would be using established evaluation tools, didn't think of that before. Thanks, that has given me another avenue to consider. Very helpful.
Ajay - I did see the virtual world platform that you referenced TIE at the VSS conference. Pearson had developed several courses that they were showcasing there. I thought it was a very interesting platform especially for K12. It does not answer all of Karl's questions but definitely something that makes learning exponentially more fun and engaging for kids.
Post a Comment