Data, once a word reserved for researchers and computer scientists, has become a buzzword in the field of instructional design. Some instructional designers might have some vague notion that it’s important to collect learner data, but they might not know which data is important or why.
The importance of data in any educational intervention cannot be overemphasized. The data from a course lets us know whether the course is effective and whether any aspects of the course should be adjusted to better meet the needs of the learners. For example, if an unusually high percentage of learners get a post-test question wrong, it’s possible that the question was written unclearly or is incorrect. In that case, we would use that information to improve the test. Alternatively, if we see that most learners answer questions on a concept correctly on a pre-test, then maybe the curriculum should focus more on the concepts that they got wrong.
We cannot make these improvements if we do not have the data necessary to inform us that a change might be needed. For example, page-turning SCORMs do not report data such as individual user responses; they only report whether users completed the SCORM, which is not enough information to inform specific instructional design decisions. How do you check whether learners frequently get a question wrong in a SCORM? Answer: you don’t. All you know is that they completed it.
Assessments are just one form of data collection: we can, for example, track learners’ decision paths in branching scenarios and create models of those paths. We can see whether some options are more popular than others and review the content to understand why. Maybe one of the options is unrealistic, or maybe the problem presented is too simple. We use that information to revise the branching scenario accordingly.
Instructional design is iterative. We use subject matter experts to inform our curricula at the beginning of the instructional design process and make necessary adjustments to ensure that the content is accurate. Similarly, we need to use data from our learners after they’ve completed our curricula to find instructional design problems we don’t even know exist. When we use course tools in our curricula that don’t capture this essential information about our learners, we miss a vital step in our design process: revising. We might be able to make some guesses, but without data, that’s all they’ll be.
Our learners deserve intelligent, informed curriculum designs. As instructional designers, we need to leverage educational tools that afford data collection.