TEST/SCALE CONSTRUCTION MODEL
MEASUREMENT STAGE NAME OF STEP ACTION
PREMEAS Objectives What is it you want to
instruct on? Attitudes
to change?
PREMEAS Instruction Provide actual instruction
MEAS Table of Specs 2 dimensional table of
topics/cog levels
MEAS Item compilation Construct/gather/ items
appropriate to TOS
MEAS Administration Give the darn thing!
MEAS Analysis I Make test scores and
show group summary
POSTMEAS Evaluation Item analysis and
other info ... make
final judgement as to
"worth" of this test
--------------------
Start the whole process over again ... have fun!
The pre measurement phase includes bothy 1) defining objectives, and 2) allowing for instruction/activities/experience. As for 1, it is assumed that a typical course of instruction will necessarily need to have some up front objectives ... both for what the instructor is planning on doing and (more important I guess) for what the students are expected to accomplish. I mentioned in class today that back in the late 60s and early 70s .. there was a wave of interest in "behavioral objectives" ... (3 parts: task, setting, behavior criterion) ... and this swept the nation. But ... and it took about 10 years to realize this ... that if one spent all ones time (and you could) on developing behavioral objectives ... one did NOT have any leftover time to provide instruction. So ... there was a gradual diminution of the stress on these ... but some overall listing of objectives is still a good idea. As for 2, after the objectives have been defined ... listed ... there needs to be time for students to have opportunities to accomplish what was laid out in front of them as THEIR tasks. So ... instruction occurrs ... activities for practive are given ... ie, one experiences things to help achieve the pre defined objectives. The measurement stage is where you begin to get down to the brass tacks of planning and developing and giving an ACTUAL test. Here, there are 4 subheadings: 3) table of specifications, 4) gather/construct items, 5) test administration, and 6) analysis I. Again, each is mentioned briefly below.
For 3, it is helpful to develop what is called a table of specifications or TOS. This is nothing more than a 2 dimensional table ... where you have topics on the top (item rules, item analysis, etc.) and several levels of thinking/processing on the side (facts, skills, application, etc.). Then, after making the TOS ... one tries to assign percentage weights to each cell in the table accoriding to how much time/emphasis was spent in that area. NOTE: THESE PERCENTAGES SHOULD not REFLECT WHAT YOU wanted TO DO ... BUT RATHER WHAT YOU did! With the TOS, one can then get a feel for how many items you will have to find or construct to fit into each cell. Then, for 4 for example, if you were making a 50 item MC test and, one of the cells in the able was valued at 10%, then you would need approximately 5 items to adequately cover that area ... to have a balanced representation on your test, given the number of items you will be using. Of course, here is where the concept of an ITEM POOL IS VERY USEFUL .... where you collect over time, items that have been tried out and fit various categories so that come test building time ... you are SELECTING items ... rather than worrying about will you have time to CONSTRUCT items. 5 is test administration and that is rather self explanatory. But 6 ... Analysis I ... means the basic scoring you use to assign scores to students ... which is actually the end of the MEASUREMENT phase (since measurement is the assignment OF the numbers ... and now you have done that). You also might do some descriptive stats to show what happened on the test.
The last stage is the POST MEASUREMENT which in a nutshell, is the final analysis part and coming to grips with how WELL the students accomplished the objectives or to make final determinations as to how well your TEST worked. Here we might do some item analysis ... and make judgements about keeping some of the items in your item pool ... but possibly revising and/or discarding some that don't work too well.
If you want to look at the macro file ... follow this link ... Minitab Macros ... down the list of macros to #37 ... called paired comparisons.
Friday is the deadline for students to submit their topics for the classroom/cognitive tests they will be working on ... keep posted!
TABLE OF SPECIFICATIONS
| TOPICS
Cog Lev |Item Rules (20) Item Anal (20) Rel (30) Val (30)
-------------------------------------------------------------
Appl 20 | 4 4 6 6
|-------------------------------------------------
Skills 30 | 6 6 9 9
|-------------------------------------------------
Facts 50 | 10 10 15 15
------------------------------------------------------------
Look at this very simple table of specifications. Assume that you are planning a test that covers units on Item Rules, Item Analysis, Reliability, and Validity ... and you have SPENT approximately and EMPHASIZED approximately at the percentages listed: 20,20, 30, and 30. Also look at the side heading where I have indicated several different cognitive processing levels that you have focused on (note: Bloom's taxonomy and others can be used there) and have ESTIMATED that you have spent about 50% on facts, 30% on Skills, and the remaining 20% on Application. If you then crossmultiply a topic value times a processing value ... you will get the approximate emphasis IN THAT CELL ... so that applications for item rules is about 20% of 20% or about 4% overall. This then gives you are rather easy way to figure out about how many ITEMS you will need for the test ... and this helps you to make a BALANCED test in terms of topic coverage and cognitive processing coverage.
Now ... it this were a 100 item MC test ... the values above would also be the number of items you would need ... but say you wanted a 50 item MC test .... then we would simply need HALF of the value above ... the # of FACTS items would be (for example) accordingly ... 5, 5, 7/8 , and 7/8.
We then had some discussion about the following issues: creating an ITEM BANK to allow you to SELECT items for the test ... rather than having to constuct them each time, and how the number of cells you need to cover says something about sampling that is possible and, perhaps the types of items that you will be using. For example, what if you wanted to use essay items ....? In the first place, essay items are poor choices for assessing whether students know the facts ... so, does this mean you would NOT test in those cells on this test? If not ... content coverage suffers. Even if you decide on MC items ... you can see that some cells will have rather few items ... some only 2 perhaps ... and that does leave alot of room for sampling ... and perhaps MISSING the mark.
I also tried to say that far too many times ... if one were to compare what they actually DID in the class with your ESTIMATES of how much time you spent on various things .. you would be amazed at the discrepancies that can occur. Thus ... the table of specifications can assist in making a balanced test ... one that fairly covers the material in about the same proportions that were emphasized in the course. If that is not done ... how is the student to possibly know how to study for the test?