The Test Blueprint: Aligning your teaching, learning objectives and tests.

| 15 Comments | 0 TrackBacks
In both our campus trips, we discussed objectives.Mostly from the context of informing your students. Objectives have a much bigger and more important job though. It helps you create assessments that directly reflect what you hope your students learn. If the assessment indicates that the learning is not reflective of the objective (students don't do well), what does that mean. Well, most of us want to assume that there is a lack of effort on the students' part, and indeed that could be true. But sometimes it's more than that. Sometimes we simply are not testing what we're teaching, or teaching to our objectives. That is we should be "teaching to our tests, and testing to our objectives". Each point of this triangle informs the other:assessmenttriangle.JPG
Follow along with the Voice Thread (which should be accessible now), and see how a blue print works. Then try it for a few of the questions on your exams. Let us know what you think.

No TrackBacks

TrackBack URL: https://blogs.psu.edu/mt4/mt-tb.cgi/360509

15 Comments

How can you imagine using a test blueprint in a course that you teach? If you can’t imagine it, why not?

I do not think I can imagine using the blueprint. It looks like a lot more work. I also include items on my review sheets that I want them to learn but may not be tested on this exam. As my exams are cumulative in nature, I can only cover so much material on a 50 minute exam. So for the next exam, I may have them contrast a new theory with an older one, which may not have appeared on the earlier exam.

Additionally, I am concerned that if I change my mind about questions, this table of specification locks me into the exam. If I say I have one question on attributions, I will need to have one question on it.

(I left this comment on the VoiceThread, but I'll leave it here as well.)

I do actually like the idea of a test blueprint. I think I frequently do something similar (particularly for my final exams) where I track what I think of as important course "content" and how many points I've devoted to each topic. I can see how matching points to Learning Objectives rather than content areas would be beneficial to my test design.

My one concern about providing a blueprint to students is that my tests (in physics) are typically comprised of about 10 MC items, followed by two free-response pages: one long problem, and one set of conceptual questions. I suspect (though I haven't tried to map it out yet) that most of my MC items test Lower-order outcomes, and that the free-response pages are more likely to test Higher-order outcomes. So by giving students a Table of Specifications in advance, they may be able to figure out what the "topic" of the free-response problem will be. But I don't want my students to know in advance what the topic of the free-response problem is going to be - one of my Learning Objectives is that students should be able to identify the relevant physics principles when presented with a problem. On the other hand, as I said, I haven't acutally tried to map my test items to the Outcome levels yet, so maybe I'm mistaken about whether or not students could pick out the free-response problem based on the Table of Specifications...

Also, as Kristine said, on many of my tests I cannot test *everything* that I think is important (students complain that my exams are too long as it is!) I am a bit concerned that in giving my students a Test Blueprint, I'd be making it "OK" for them not to study certain important ideas.

How can you imagine using a test blueprint in a course that you teach? If you can’t imagine it, why not?

I use MC questions in a number of my quizzes, midterm exams and final exams. They're usually in combination with problems and/or essay questions. The number of questions depends upon whether it is a quiz versus a final exam, the course I'm teaching and the course content.

I can see how using a test blueprint would be helpful particularly to verify that the current MC questions I'm using align with the test objectives and cover the content and outcomes I'm trying to assess.

The course I'm focusing on for this Course in College Teaching is IB303, International Business. This is a course that can appear somewhat “alien” to many students as it deals with the international aspects of business versus domestic business. When I survey my students in the first class, I usually find out that most of them have not been outside the US/Canada. This course covers many concepts that students are not yet familiar with, but will encounter in the work environment.
Although I focus on trying to be sure that the MC questions I use align with the objectives, course content and outcomes, it will be interesting to see how closely I have actually accomplished this given I indicated above that this can be a little tricky given the amount of new concepts for students taking this course.
I will start with one of the quizzes (there usually three in this course) using the test blueprint approach to assess the MC questions. After listening to the voice thread, I think it might be especially helpful for this course, given the large volume of new information, to provide students with a test blueprint. I have not done that in the past but will consider when I teach this course this fall.
Bernie

How can you imagine using a test blueprint in a course that you teach? If you can’t imagine it, why not?

I like the idea of the test blue print but because of the large volume content of each course am not sure how well this will work. Students sometimes complain that my exams are very lengthy and I do try to combine MC with short answer and essay questions. I categorize my MC questions as easy, medium and difficult so I am using somewhat of a blueprint but I do not give it to the students. My concern with the blueprint is that students will study for the blueprint and not master of the material. On a positive note if I used the test blue print and aligned the objectives with content and outcomes I may be able to cut down on the length of some exams while making sure the material is assessed. I am going to try the test blue print with one of the courses in the fall.

Our accrediting body requires assessment of the program. I have had to develop an assessment plan and this test blue print may fit in the plan. I just hope I have enough time to sit down and put the effort required to complete the blueprint.

I'm thinking that the purpose of the test blue print is not clear.

Faculty who adopt this strategy usually find that they streamline their assessments, get rid of the "dead wood", and really target their learning objectives clearly. It doesn't mean that you're trying to include everything. It's pretty simple really. Pick an exam, maybe one that students think is difficult, then work through a test blueprint. If you're asking questions that are not aligned with your objectives on the exam (e.g., weather instruments vs storms), perhaps you need to rethink those questions. Maybe drop them from this exam and move them to another. It's all about reflecting about what you're teaching and ultimately assessing the learning. Is it more work...it is additional work, yes. But the folks who take the time to do it find it has a big payoff. If your students sometimes complain that your tests are "too hard", this is one strategy to figure out if they may have a point. That "too hard" flag sometimes means, not that they didn't study, but they didn't get to "practice" in class or homework, and now they are being asked to do something totally new in a pressure cooker. And heaven help them if they are reflective thinkers. It's a recipe for disaster.

Kristine, to your point, it doesn't "lock" you in to anything. It gives you a strategy to better define the content of your exams, and makes it easier for you to evaluate your students. At least that's the intent.

One of my personal goals has been to identify those topics which I consider essential for my students to master in order for me to "pass" them through CHEM 110, in the midst of the hundreds of topics we cover. I am working on this list. A test blueprint may be a way for me to be sure I am using the test as a way to be sure these items are mastered and then the other topics become those that separate out the "A" students from the "C" students.

Hi, All. Cindy invited me to join your conversation. She and I did some workshops on test blueprints (and other assessment topics) for faculty at Hazleton and Fayette this past academic year. Folks at those events raised some of the same issues as those of you commenting here. We worked with them over time, and for those who tried blueprints, they loved what they learned from doing it.
A few thoughts…

1. We don’t have to test everything that we teach—At best, our tests can only be a sample of all that we ‘cover’ in class. But given that, it must be a very intentional sampling. That’s where the blueprint comes in. It allows us to map out our intentions. But caution: The assessment must provide evidence of the extent to which students are meeting our objectives. Again, the blueprint allows us to be certain we are doing so.

2. Beth raises an important issue: Students may see topics on a blueprint that they know were covered, but not see a corresponding item or items indicated. Then they’ll be inclined not to review it. There are a couple strategies for getting around this. One is to collapse topics. For example, if you’ve taught the topics “Dogs” and “Cats” but only intend to test Dog knowledge, then you don’t want to leave Cats empty on the blueprint. So call the topic: Dogs and Cats. Or Pets. Either way, students will have to study both to be sure they’re prepared. A second strategy is to organize the blueprint by Objectives. The issue is far less likely to arise if you organize the blueprint this way.

3. Finally, you don’t have to offer it to students. If you’re uncomfortable with that, don’t do it. But we strongly recommend that you at least create the blueprint for YOU. Doing so will going a long way toward improving the validity of students’ scores. You want their score to reflect their knowledge of something. A blueprint is the best way to make sure you’re actually testing that knowledge. All major exams (standardized tests, certification exams, etc.) are constructed using test blueprints. We don’t think about it, because we typically see the tests only when we’re taking them. Here’s a link to an example I found just with a quick Google search. It appears on a website designed for students preparing to take the Nephrology Nursing Certification Exam: http://www.nncc-exam.org/cgi-bin/WebObjects/NNCCMain.woa/wa/viewSection?s_id=1073744584&ss_id=1&tName=cdnTestTips

Good luck with your blueprints!

I was glad to view Crystal's comments. I think the idea for a blueprint is a sound one for me as an instructor. I can see its benefits in making sure I am indeed testing on what I want my students to learn from the course, i.e., as a tool for making sure I am testing on objectives for the course.

However, I can't imagine giving it to my students at the education level that they are at. I think it would generate more questions than answers for them. I can see its value for students at a higher level and even at masters level courses. I'm just not sure sharing that with my students would help them. I often pause during class discussion to comment, "You need to understand this," or "This will definitely be on the exam." I've found that helps students who pay attention in class. But no matter what you do, it always seems not everyone gets the message.

Honestly I don't see myself having the time to make the blue print, but I think having the idea of it in my head will help me be more careful about selecting questions. I do make sure there are about the same number of questions from each chapter, and I do provide a study sheet of materials that may be covered on the exams.

I think the real purpose of the exams for me is to make sure students are doing the textbook reading, and so I get the questions from the textbook. I have recently decided to have many more smaller quizzes so that the students feel the need to keep up with the reading instead of trying to do it all right before two big exams. I think this plan is more in line with my purpose to make sure they are keeping up with the reading.

I realize that my goal is not a learning objective, but I think my other assessments match up with my learning objectives more.

I like the idea of the blueprint as an instruction tool, but I agree with John's assessment that it does not feel appropriate to give it to the students at the very early stages of college coursework (like those at Mont Alto). However, as I work through my testing/evaluation strategies, I think that creating a blueprint would be immensely helpful (time consuming, but helpful).

(I left this comment on VoiceThread as well. I welcome any feedback you might have...)

I have always operated as though one class meeting in college is equivalent to a week's worth of class meetings in high school. In light of the amount of information we cover each week, and the fact that I usually give only 3 (or fewer) exams per semester, each exam covers a vast range of learning objectives. That said, I don't see the harm in writing out every learning objective (even if there are 25 or more). This will only make my exams more focused, and help me weed out any questions that stray from the vast range of objectives. On a separate note: I find it difficult to craft MC questions that have higher-order outcomes (analyze, evaluate, create). Do others have this problem? This is something I hope we can discuss at some point.

How can you imagine using a test blueprint in a course that you teach? If you can’t imagine it, why not?

I believe that using a test blueprint is be very helpful to the students but the problem with that is the students mainly focus on what is going to be covered
on the test. Sometimes they seem to wait for the test blueprint to start their review.
That’s why I believe that it is always good to put in the test blueprint two or three questions on material that was covered during the previous exam so they will know that they need to understand those topics.

I second David's desire to talk about crafting higher-order MC questions! I have a few that fit the bill -- but only very few. Most of my questions top out at "apply".

This is the first time I have ever seen a test blueprint. I think it's an interesting idea - especially if you are having issues writing an assessment that covers that topics you think are important and over a range of higher and lower order thinking. I'm not sure I see myself using one, in part, like others mentioned, it seems like quite a bit of additional work. The voicethread was useful for me to at least get thinking about taking an inventory of my exam items, even if it's not as formal as in a test blueprint.
Also, if I were to ever teach a class that was less familiar to me than the ones I currently teach, I could see it being an especially useful tool.
I don't think I would give my students a copy of the blueprint. I have found that providing students study guides has actually led to worse exam performance because it is used as a crutch for parsing out the material and identifying important topics - a skill I like students to develop during the semester.

Leave a comment

Subscribe to receive notifications of follow up comments via email.
We are processing your request. If you don't see any confirmation within 30 seconds, please reload your page.

Search This Blog

Full Text  Tag

Recent Entries

To Multiple Choice or Not .....
We've all written questions that have violated one or two of these"rules". Even if it was on a low-stakes quiz.…
The Test Blueprint: Aligning your teaching, learning objectives and tests.
In both our campus trips, we discussed objectives.Mostly from the context of informing your students. Objectives have a much bigger…
The Fribbled Breg of Multiple Choice
Many years ago I gave a collegue from a completely different discipline a multiple choice exam I'd written for my…