Adaptive Assessment

I attended a thought-provoking session facilitated by @chrischampion yesterday(11/12/11) at EdCamp Harrisburg. The session focused on “doing different things” versus just “doing things differently.” Chris pushed the audience to really think creatively about the possibilities for technology’s impact on learning and to go beyond the immediate efficiencies realized by using technology. One topic that Chris explored was adaptive testing. I tweeted:

#edcamphbg instead of online quiz with immediate feedback, try adaptive testing @chrischampion

The following tweet thread ensued:

lauriev88
@apetroski "Adaptive testing" ? Can you explain? #edcamphbg

apetroski
@lauriev88 short version of adaptive testing for twitter = student's answer to question determines their next question . . .

lauriev88
@apetroski Hmmm...would love to hear more. #edcamphbg

So lauriev88, this blog post is in response to your request. I’m not an adaptive testing expert by any means, but it is a topic I’ve investigated in a number of classes I teach (LTMS 510 and LTMS 608 in in the HU graduate program) and in a couple of projects we’ve pursued through the Center for Advanced Entertainment and Learning Technologies at Harrisburg University.

The common assessment approach is to provide the same test to each student. Adaptive assessments dynamically adjust for each student based on their performance as they answer questions in the assessment. The question sequence and difficulty is based on their performance on previous questions.

Adaptive assessment goes beyond randomized assessment. In a randomized assessment, students might receive the questions in a different order than another student or receive a random set of questions from a question pool. Another randomized approach is to present the answers in a randomized order for each student. In all of these cases the assessment is still not adaptive. The assessment doesn’t change based on the student’s performance. In an adaptive assessment, a student who answers question #1 incorrectly will receive a different question #2 than the student who answers question #2 correctly. In the session, Chris also gave an example of a student being able to complete an exam in a short series of questions by answering the most difficult questions correctly while another student might experience a longer series of questioning or even be directed to remediation before completing the assessment, based on their performance.

Here are links to just a few of the resources to provide a more in-depth exploration of adaptive testing.

- Computer-adaptive testing: an innovative tool for teachers, students
- C.A.T Central
- JATT Special Issue on Adaptive Testing: Welcome and Overview
- SMARTER Balanced Assessment Consortium
- Computer-Adaptive Testing Poses Challenges
- Adaptive and Interactive Assessments with E-Learning

As Chris emphasized in the session, adaptive testing is most often experienced in certification exams. A number of the educational publishers (e.g. Pearson, Edison, etc.) also employ versions of adaptive testing as an assessment option. In these cases there is a large investment of time and dollars to create and validate the assessments. But, there are ways that you can begin to use the basic constructs of adaptive assessment in the classroom without a large dollar investment (time will still need to be invested).

- Moodle Lesson (Lesson in Moodle: An Illustrated Guide, Adaptive mode in Moodle Quiz is not truly adaptive assessment)
- “Go to Page Based on Answer” in Google Forms
- Assessment Center
- FastWeb
- WebExaminer

I hope that’s more lauriev88. Thanks for asking. It was good to go back and reference some information about adaptive testing that I hadn’t looked at in a while. And, it’s got me started on preparation for LTMS 510: Learning Technologies and Solutions for the spring semester. And, thanks again to @chrischampion for the session that got this discussion, and blog post, started.

Could you imagine Architects or Accountants having this discussion?

Do you need an instructional design degree? This is an argument that comes up about every 3-4 months in the instructional design field. This time it's on Tom Kuhlmann's Rapid eLearning Blog. Tom does a good job of indicating how you can do it without a degree and the benefits of getting a degree.

I'm not doubting the skills of the many that have commented about how they have succeeded without a degree, but also have some thoughts about why a degree is necessary.

For those that have been working in the field for a while, the need for a degree is most likely not as important, especially if you're someone who's not "afraid" of technology or experimenting with new ideas. But, if you're looking to move positions in the company or move to a new company, that degree may be handy.

If you are a career changer or careet starter, YOU NEED A DEGREE! You will not get the breadth of knowledge that you will receive in a graduate degree program from books, tutorials and social networks. You may be able to develop a specific expertise, but what happens when the next project isn't exactly like the project before it?

Degree or not, you need to keep learning in this field. So, even if you don't feel the degree is required, it would benefit you to get it . . . or at least take some classes. Formal, credit classes can be another tool in your personal learning. If you're not participating in some sort of professional development on at least a yearly basis, you are not going to advance your practice in the field.

Finally, whether you agree with any of my previous statements or not, I suggest we think of our profession as a whole when having this debate.

The instructional design field will not advance nor will the ID position indicate any expertise, specific skillset or explicit value to those outside the discipline until those doing the work are properly and similarly credentialed.

Could you imagine Architects or Accountants having this discussion?

Yes, there is "Trainer." But, that's outdated. Otherwise, an instructional designer in one organization is most likely not doing the same tasks, nor requires the same skillset as an instructional design in another organization in the current environment.

Maybe a degree isn't the ultimate indicator, but at least a recognized national certification is needed so that others know what we do and that "you" have undergone some type of preparation to do it.

Unfortunately, this national standard does not currently exist.

Probable-to-Possible: Mapping and Flowcharting

This article originally appeared in the January central PA ASTD chapter newsletter . . .

With each turn of December on the calendar comes renewed optimism, new ideas and plans for a new approach. Resolutions abound, they often fall of the to-do list in favor of the more mundane, and urgent requirements. So, take some time this year to become familiar with a few technologies that will help you generate ideas, manage them and create processes for implementation.

Mind mapping
A mind map is a diagram of terms, tasks or resources arranged around a central keyword or idea. Mind maps are often used to visualize or structure ideas as an aide to everything from taking notes to solving problems.

Probable
Mindmeister (collaborative, browser-based and free)
http://www.mindmeister.com

Mind 42 (collaborative, browser-based and free)
http://mind42.com/

Middle Ground
Visual Thesaurus (creates word maps that blossom with meanings and branch to related words )
http://www.visualthesaurus.com

Spinscape (auto-discover capability that allows you to pull in all the information you want from the web) http://www.spinscape.com

Possible
Flash Brainer (generate up to 530% more ideas than traditional brainstorming)
http://www.flashbrainer.com/

Axon Idea Processor (provides a variety of thinking tools)
http://bit.ly/UBKLl

Application for learning: Needs analysis, theming, creating scenarios and simulations, assessment
Other resources: Mind mapping blog, http://mindmappingsoftwareblog.com/
Related concepts: Crowdsourcing, http://bit.ly/dwRniE; Idea Management, http://rww.to/8XTiyS

Concept Mapping
A concept map is a diagram showing relationships among concepts. Unlike mind maps, concept maps are not centered on a singular topic. A concept map can be a free form, system view of a set of concepts and can include multiple hubs and clusters.

Probable
IHMC CMap Tools (free concept mapping)
http://bit.ly/FiW4

SMART Ideas (SMART board –related product)
http://bit.ly/g22cqm

Middle Ground
Inspiration (multi-functional tool)
http://www.inspiration.com/

Mind Manager (maps with an intuitive visual framework)
http://bit.ly/9850m0

Possible
Design Web (automatic construction
of interactive conceptual maps)
http://bit.ly/gB7rt3

Application for learning: Needs analysis, curriculum design, creating scenarios and simulations, assessment
Other resources: The Theory Underlying Concept Maps and How to Construct Them, http://bit.ly/aDY5Fb; List of concept mapping and mind mapping software, http://bit.ly/9VGVFl
Related concepts: Cognitive tutor, http://bit.ly/eJDOGv

Flowcharting
We’re all familiar with the concept of flowcharting and almost everyone has probably done it at some point in their academic or professional careers. However, until recently there were very few computer-based tools available for flowcharting, beyond Visio in Microsoft Office. If you haven’t used Visio, you may have attempted flowcharting in Microsoft Word or PowerPoint (= pounding a nail in with a slipper).

Probable
Gliffy (online diagram software – free version)
http://www.gliffy.com/

Flowchart.com (online multi-user, real-time collaboration flow charts service)
http://flowchart.com/

Middle Ground
Microsoft Visio (advanced diagramming with flowchart linking)
http://bit.ly/aZcZ4F

Possible
FlowBreeze (create flowcharts from Excel)
http://bit.ly/3Cus4O

Application for learning: Instructional analysis, process analysis, storyboarding, job aides
Other resources: Standard flowchart symbols, http://bit.ly/caXc7w
Related concepts: Storyboarding, http://bit.ly/f4vZiV

So, enter the New Year with the same vim and vigor for new ideas and new approaches, but consider using some of the tools listed in this article to keep that momentum going. Capture and brainstorm new ideas with mind mapping, map new approaches and strategies with concept maps and create new and more efficient processes with flowcharting.

Also consider how these tools can impact your learning analysis, design and development directly. I’ve listed some applications for learning with each list of tools. What do you think? Do you already use the tools? If so, what are the results? Can you think of any other ways in which these tools can impact your learning design and development activities? Let’s continue the conversation on Twitter (@apetroski), LinkedIn (http://linkd.in/eFBs2j) or on the Learning Evolution Blog (http://bit.ly/8YKc3E).

The Evolution of the Learning Game

When I was designing and devleoping my first games for learning in 2000, the framework of most serious games was fairly well established . . . "An interactive activity to reinforce rote leearning; built in Flash." Fast forward a decade-plus adn the options and opportunities in games for learning have exploded. The strategies for creating and using serious games has advanced. No longer confined to rote learning at the end of a tutorial or even to the desktop, serious games come in all shapes and sizes. While the desktop is still a primary delivery method, today's experiences can be much more immersive. And, there are some new and evolving ways to design, deliver and experience games for learning.

Read the remainder of this post on the LEEF blog . . .