|About Us | Press Room | Demos & Downloads | Contact Us | Site Map|
Feedback in Computer-Assisted Learning
Roy B. Clariana
Clariana, R.B. (2000). Feedback in
NETg University of
Limerick Lecture Series.
What is feedback? In our physical environment, feedback is the return of information that is the result of an activity or process. This is often a cause-effect association, and we come to understand processes and activities both by passive observation and also by active manipulation. In our social environment, feedback is an evaluative response to our actions. Our neural systems have been tuned to allow us to note associations, both physical and social, to our survival benefit. In addition, our neural systems process language, which a fairly recent, artificial, and symbolic form.
Feedback is a natural part of human life. Our physical and social environments are full of feedback, and we are wired to use this feedback, often automatically. Also, research clearly shows that feedback is instructionally powerful. Thus, authentic instruction must bristle with overt and covert feedback information. However, after more than eight decades and hundreds of studies, there is not a generally accepted model of how feedback works in instruction (Kulhavy & Stock, 1989). There are a number of unanswered questions and perhaps even more unquestioned answers.
This chapter begins with a brief overview of the comparative effects of different forms of instructional feedback, especially those that are available in computer-assisted learning (CAL). Next, several important factors that relate to feedback effects are presented. Finally, a tentative computational model of feedback is presented based on associative learning and connectionist approaches. A visual metaphor termed high-dimensional semantic space is provided to describe the effects of feedback.
In CAL, learner response is generally limited to recognition (clicking on) and recall (typing in). Fairly soon, CAL will be capable of properly responding to extended learner input, such as a 200-word essay response utilizing automated approaches such as Latent Semantic Analysis (Foltz, Kintsch, & Landauer, 1998). However, currently, CAL feedback research must focus on recall or recognition learning outcomes; and CAL feedback is fundamentally limited to something that can happen on the screen (and sometimes audio). This would include displays of words, numbers, highlights, motion, animations, arrows, icons, color changes, and other cues.
Traditionally investigated types of feedback available in CAL include the following: Knowledge of response (KR) that states "right" or "wrong" or otherwise tells learners whether their response is correct or incorrect; Knowledge of correct response (KCR) that states or indicates the correct response; and Elaborative feedback that includes several more complex forms of feedback that explains, directs, or monitors (Smith, 1988). Elaborative feedback includes the forms listed below:
All of these are specifically and intentionally directed at individual learner responses except for monitoring feedback, which provides information over a group of learner responses, such as over a section, a lesson, or an entire unit of CAL instruction.
Advisement is worth a brief diversion. As mentioned, monitoring feedback or advisement (Tennyson, 1980) is meta-level feedback, such as ongoing on-screen lesson scores, recommendations for instructional branching, and end-of-unit and end-of-lesson scores. For this reason, advisement may primarily affect continuing motivation rather than item-specific error correction.
How does advisement function? Current advisement research views advisement forms within a continuum of effectiveness that relates to the information quality and quantity provided by the advisement. Advisement relates to instructional decision-making by the learner, and much CAL advisement research includes learner- versus program-control conditions (Coorough, 1991; Hannafin, 1984; Johansen & Tennyson, 1983; Santiago & Okey,1992; Tennyson & Buttrey, 1980). Learners usually appreciate this sort of information although they don’t always use it appropriately.
Advisement effectiveness results are mixed. Milheim and Azbell (1988) reported that advisement, in general, improves posttest scores, increases the number of students that reach mastery, increases time on task, and increases instructional efficiency. Clariana (1992, 1993a) reported that advisement in CAL positively effects persistence, lesson production, attendance, and achievement. Thus, advisement should be provided to learners in CAL.
Different forms of feedback like those mentioned above can be provided immediately after a learner's response; or feedback information can be delayed for some amount of time such as seconds, minutes, or days after the learner’s response. Feedback timing is important for at least two reasons. First, because our neural systems are temporally based, immediate feedback is likely to be processed differently than delayed feedback (Shanks, Pearson, & Dickerson, 1989). Second, previous research has shown performance differences for immediate and delayed feedback.
Two different models may explain the differences between immediate feedback (IF) and delayed feedback (DF), the Interference Perseveration Hypothesis (IPH; Kulhavy & Anderson, 1972) and a dual-trace information processing explanation (Glover, 1989; Kulik & Kulik, 1988). The IPH depends on initial lesson errors interfering or not interfering with learning the correct response. IPH holds that with IF, the error memory trace activation interferes with encoding the correct response memory trace. The delay with DF allows the initial question and error response association to subside before the association between the question and the correct response is formed or strengthened. In addition, with the IPH, interference occurs very late during encoding and likely requires overnight consolidation.
Alternately, the dual-trace explanation holds that two processing opportunities occur with DF and only one with IF. Thus, DF groups will remember both their initial error response as well as the question–correct response associations on the retention test. For the IF group, the question–error response association is weakened and the question–correct response association is strengthened. Therefore, with the dual-trace explanation, differences between DF and IF are due to retrieval response competition between the dual memory traces under DF compared to no retrieval competition for the single memory trace under IF. The trace with the greatest activation level would be selected as the correct answer. In a study of IF and DF, Clariana, Wagner, and Roher-Murphy (2000) reported that DF groups remembered their initial errors significantly better than the IF group 24-hours later. This supports the dual-trace explanation. Also, no difference was observed between the IF and DF groups in 24-retention of correct response scores. Remembering initial lesson errors did not relatively improve the DF groups’ retention test score; in fact, errors perseverated under DF.
Many feedback timing studies and meta-analyses of studies provide guidelines for when to use DF or IF (Bangert-Drowns, Kulik, Kulik, & Morgan, 1991; Kulik & Kulik, 1988). For example, in situations that use questions only with feedback as instruction, such as in a CAL drill, DF is superior to IF with an effect size of 0.36. However, in studies that use questions with feedback and include additional instructional materials, such as in a CAL tutorial, IF is superior to DF with an effect size of 0.28 (Kulik & Kulik, 1988). These are only general guidelines. There is usually little difference in learning for immediate versus delayed feedback, though Clariana et al. (2000) recommended that DF be used when it is advantageous to remember both initial error responses and correct responses, such as in CAL simulations. For example, Lewis and Anderson (1985) reported that learners receiving DF in an adventure game were better at detecting errors relative to the IF group. However, when it is critical to inhibit, suppress, or forget errors, such as in CAL typing skills programs and when acquiring speeded or automatic responses involving critical, difficult, or dangerous content, IF should be used.
Comparative Feedback Effectiveness
Which type of item-specific feedback is best? Some studies have shown KCR superior to KR, while KR was superior to no feedback (Gilman, 1969; Kulhavy, 1977; Travers, Van Wagenen, Haygood, & McCormick, 1964; Waldrop, Justin, & Adams, 1986); however, the results have been mixed (Schimmel, 1986). A meta-analysis by Bangert-Drowns, Kulik, Kulik, and Morgan (1991) compared KR, KCR, AUC, and elaborative feedback to no feedback. Compared to no feedback, KR feedback was least effective (e.s. = -0.08), followed next by KCR (e.s. = 0.22). AUC, (e.s. = 0.53) and elaborative feedback (e.s. = 0.53) were equally most effective. The performance results may be summarized as:
KR < no feedback < KCR < AUC = Elaborative feedback
In general, more feedback information means more learning.
A review of 30 studies by Clariana (1993b) that compared no feedback, KR, KCR, and DF to multiple-try feedback (MTF, mostly AUC) reported a similar superiority for MTF over no feedback, e.s. = 0.56. However, unlike Bangert-Drowns et al., he reported no difference between KR, KCR, DF, and MTF. These results may be summarized as:
no feedback < KR = KCR = DF = MTF(AUC)
This finding is consistent with a meta-analysis by Schimmel (1983) involving 15 studies that compared KR and KCR to no feedback. KR and KCR were each more effective than no feedback (e.s. = 0.47). Clariana’s and Schimmel’s findings can only be summarized as any feedback is better than no feedback, a rather disappointing conclusion after having come so far.
Factors Related to Feedback Effectiveness
Then, besides DF, are all forms of feedback about equivalent? At least differences in DF and IF can be readily explained (see above) with regard to the number of memory traces established or altered by feedback. Several other factors are emerging that may explain when certain kinds of feedback are more effective. These include learner prior knowledge, the intended kind and level of learning outcome, and the separate contribution of question effects in relation to overall feedback effects. Note that all of these directly relate to lesson item difficulty. These factors will later provide a bridge to describe a model of how feedback works.
A Possible Trait-Treatment Interaction with Multiple-Try Feedback
Clariana (1993b) reported a trait-treatment interaction (TTI) relating MTF and prior knowledge. Specifically, for low-prior-knowledge students, KCR was more effective than MTF, e.s. = 0.11 (see Figure 1). For high-prior-knowledge learners, MTF was better, e.s. = 0.39 (Clariana, 1999a).
Figure 1. Interaction of prior-knowledge and type of feedback (Clariana, 1999a).
Since MTF and KCR feedback information is usually identical when the learner’s response is correct, this TTI likely relates to affective and cognitive differences on errors. An affective explanation of reduced effects for MTF is straightforward. If a learner makes many lesson errors, then frustration is likely and MTF would have a negative effect. This negative effect for MTF would increase as the number of errors increases. Thus, MTF would likely obtain a relatively negative effect with difficult or new lesson content and with low-prior knowledge students.
For example, Dick and Latta (1970) reported that low-ability students became frustrated during a repeat-until-correct lesson using constructed response study tasks relative to high-ability students. Morrison, Ross, Gopalakrishnan, and Casey (1995) reported that AUC students’ selection of review screens deteriorated in relation to the number of lesson errors committed. Further, they reported that the AUC treatment groups’ lesson scores were considerably lower than the lesson scores for KCR and other groups. Similarly, Clariana (1999a) reported increased frustration by students in an MTF treatment relative to students in a KCR treatment. During the lesson, students in the MTF treatments pressed the Enter key without trying to respond at nearly three times the rate of the KCR group, thus obtaining lower lesson scores than the KCR group.
Alternately, some learners such as high-ability and/or high-prior knowledge students may prefer MTF, which should positively impact learning. For example, Clariana and Lee (2001) reported that 71% of learners (graduate education majors) preferred MTF to KCR with multiple-choice lesson tasks. Regarding MTF, the students wrote comments like "I learn more when I get it on my own" and "Trying a second time makes me think more about the question and what it is really asking." This positive effect for MTF should be more likely for easier, familiar, personal, and interesting lesson content and for high-prior-knowledge students (Nishikawa, 1985), and with recognition study tasks relative to recall study tasks. Thus, a TTI for MTF can be readily accounted for by increased or decreased engagement in the lesson due to motivational issues related to learner characteristics and the lesson content.
From a cognitive amount-of-information viewpoint, MTF essentially provides the same information as KCR. However, MTF requires information from within the learner while KCR information is presented from outside of the learner and as true. When an error occurs, MTF information acts to alter the existing mental representation of the item and response, perhaps directly, allowing the item and correct response to fit within the learner’s broader preexisting understanding of this content. Alternately, KCR information acts to reject the preexisting mental structure and provides the item and correct response as a new instance of this content. Thus, MTF and KCR, though outwardly similar, have very different effects on episodic memory.
A number of studies have shown that MTF is only equal to or is relatively less effective than single-try forms of feedback, such as KR, KCR, and DF, when posttest questions are identical (as repeated memory instances) to lesson questions (Anderson, Kulhavy, & Andre, 1971; Bardwell, 1981; Clariana & Smith, 1989; Clariana, Wagner, & Roher-Murphy, 2000; Hodes, 1984; More, 1969; Morrison et al. 1995). However, when posttest questions are different than the lesson questions (not as repeated instances, but meaningful), such as trouble-shooting new problems, applying principles, and classifying new concept examples, then MTF is often relatively more effective than single-try forms of feedback (Angell, 1949; Cantor & Brown, 1956; Clariana, Ross, & Morrison, 1991; Dempsey & Driscoll, 1989; Lee, Smith, & Savenye, 1991; Nielson, 1990; Pressey, 1950, Richards,1989).
For example, Clariana et al. (1991) examined the effects of several forms of feedback on identical and higher-level learning outcomes. High school students (N=103) in a summer remedial program completed a difficult computer-based lesson on science principles with text portions, verbatim and inference adjunct lesson questions, and one of three forms of feedback including DF, KCR, or MTF. Posttest questions were identical, paraphrased, and transformed following Anderson (1972), Bormuth (1970), and Bormuth, Manning, Carr, and Pearson (1970) in order to measure higher-order learning outcomes. All three feedback groups performed significantly better than the no-feedback control on identical retention test items (repeated instances, rote learning), but there was no difference between the feedback and control groups for higher-order posttest questions. However, note that the MTF scores were relatively lower than DF and KCR scores for identical posttest items, but MTF scores were significantly better than DF scores for paraphrased and transformed-paraphrased questions. This lends support to the interpretation that MTF is better than single-try forms of feedback (at least DF) with higher-order learning outcomes (see Figure 2).
Figure 2. MTF (triangles, dashed line), DF (diamonds), and KCR (boxes) immediate posttest scores from Clariana et al. (1991).
This finding is hardly surprising. According to Bransford (1979) and Jonassen (1985), instructional activities that require the learner to actively generate a response from their own internal understanding should be more powerful than activities that only present information, where meaning comes from outside of or external to the learner. Clariana et al. (1991) states,
Allowing unassisted multiple response tries has considerable intuitive appeal. Answering until correct may engage learners in additional active processing following errors and also ensures that the last response is a correct one, a principle espoused over half a century ago in the contiguity theory of Edwin Guthrie. (p. 6)
Therefore, high prior-knowledge learners, by definition, would have meaningful or more developed mental representations or schemas relative to low prior-knowledge learners; and MTF can take advantage of a rich schema while KCR cannot. Thus, MTF tends to act on an individual’s internal and personal meaning of the lesson content, and DF, KCR, and other forms of feedback that present the correct response to the learner tend to establish strong specific instances of the content. If so, MTF should logically have a greater effect on higher-order, meaning-related learning outcomes, and DF and KCR would lock down facts as memory instances. This explanation can account for both hypotheses, the TTI for MTF and prior-knowledge, and MTF effects on higher-order learning outcomes.
Next, we provide a description of the effects of adjunct questions without feedback and of text. This will allow us to later separate the effects of text, questions, and feedback.
Adjunct Questions Without Feedback
A lot is known about adding questions without feedback to instructional text (Hamaker, 1986; Lindner & Rickards, 1985). For example, either providing questions before text, inserted within text, or massed after the text is effective. However, questions inserted in the text are usually most effective and probably influence the processing of both concurrently read text material as well as to-be-read text material (Andre, 1979; Duchastel, 1983; Hamaker, 1986). When comparing the response formats, short answer constructed response lesson questions have a stronger effect than multiple-choice lesson questions on both constructed response and multiple-choice posttests questions (Anderson & Biddle, 1975; Hamaker, 1986). Additionally, it is hardly surprising that posttest performance is higher when study time is adequate and the text is continually available. These are all ecologically valid suggestions that should be considered during the design of CAL.
Perhaps most significantly, higher-order lesson questions usually obtain larger effects than factual-level lesson questions for both lower-level learning outcomes, such as facts and verbal information, and also for certain higher-level learning outcomes, such as applying knowledge of concepts and principles in order to recognize new examples, and to solve problems involving the same concepts or principles. However, the relatively positive effect of higher-order lesson questions usually does not generalize to new/other concepts or principles (Andre, 1979; Hamaker, 1986; Lindner & Rickards, 1985). Also, as with MTF, a TTI has been suggested for adjunct questions and prior knowledge (Hudgins, Dorman, & Harris, 1979; Lindner & Rickards, 1985). Poor readers do better with frequently inserted questions, which may serve to place instances, mileposts, or anchors to populate and structure their mental representation with the new content. In contrast, good readers do better with post text questions, which may serve to refine or tune their already existing mental representations of the text. In summary, lesson questions without feedback, and especially higher-order lesson questions, are instructionally effective and should be used in CAL.
The Contribution of Questions and of Feedback to Higher-Order Learning
Unfortunately, the feedback literature is not as extensive or clear as the adjunct question literature on the effects of different forms of feedback on higher-order learning outcomes (Mory, 1996; Smith & Ragan, 1993). Much of the feedback literature has dealt with lower-level verbal learning outcomes (Merrill, 1985; Mory, 1996; Schimmel, 1983), and the approaches often involved multiple-choice lesson questions that were repeated in some form on the posttest. However, even though positive feedback effects are well established for lower-level learning outcomes, it is unclear that feedback can effect higher-order learning outcomes (Mory, 1996).
Morrison et al. (1995) compared several forms of feedback (DF, KCR, and MTF) and also included text-only and questions-only control groups. This allows us to determine the separate effects of questions and questions plus feedback for lower and higher-order learning outcomes. Undergraduate education majors (N=248) completed a computer-based lesson on writing instructional objectives. Incentives were manipulated by counting posttest achievement as a part of the student’s course grade in one condition, but in the other condition, participation without regard to posttest achievement was part of their course grade. It was expected and also observed that students in the grades condition would try harder to learn the lesson material relative to those in the no-grades condition. The posttest question-level design followed Clariana et al. (1991) using identical, transformed, and paraphrased questions. The results for the feedback and the question treatments are shown in Figure 3 in terms of effect sizes relative to the text-only control group.
For identical-level posttest questions (rote), both questions (e.s. = .52) and questions with feedback (e.s. = .75) had a strong effect. The question portion accounted for about 70% of the question plus feedback identical-level posttest scores. For transposed and paraphrased posttest questions, questions accounted for about 77% of the question plus feedback posttest scores. For combined transposed plus paraphrased posttest questions, the feedback actually subtracted from the effects of the question portion.
In a study by Clariana (2001) using higher-order lesson questions, questions accounted for an even greater proportion of the questions plus feedback effects (see Figure 4). High school students (N=78) were randomly assigned to one of five lesson treatments: text plus questions with DF, text plus questions with KCR, text plus questions with MTF, text plus questions with no feedback, and a text-only control group. A retention test given five days after instruction assessed both verbatim memory of principles from the lesson and also higher-level understanding of principles from the lesson (again paraphrased and transformed).
Figure 4. Effect size comparisons from Clariana (2001) showing feedback treatments (solid lines) and questions treatment (dashed lines) relative to the text-only control (where effect size equals 0).
Relative to the control group, feedback was beneficial for lower-level learning but was detrimental for transformed test items (see Figure 4). For identical-level posttest questions (rote), both questions (e.s. = .48) and questions with feedback (e.s. = .56) had a strong effect. The question portion accounted for about 85% of the question plus feedback identical-level posttest scores. For paraphrased posttest questions, questions accounted for about 50% of the question plus feedback posttest scores. For transposed and the combined transposed plus paraphrased posttest questions, the feedback again substantially subtracted from the effects of the question portion. Transposing posttest questions had a strong negative effect on all treatments.
The major difference between the Morrison et al. (1995) results and the Clariana (2001) results occurred with transposed (T & TP) posttest items (compare Figures 3 and 4). Why are the results for transposed posttest questions so different? The Morrison study mainly involved concept identification and definition while the Clariana study involved science principles, described as the interrelation between several concepts. Transposed stem and response for these two learning outcomes obtain very different forms of posttest questions. For example, from Morrison, "An objective is a statement of what the learner will be able to do…" transposed becomes "A statement of what the learner will be able to do is an objective." In this case, transposing concept identification and definition type questions produced nearly equivalent questions.
However, transposing principles does not produce equivalent questions, and in some cases, transposing a principle will produce a non-truth. For example, from Clariana, "increasing heat in a closed cylinder increases pressure" transposed becomes "increasing the pressure in a closed cylinder increases the heat." Though both are true, the first is truer than the second because heat will escape the closed system though pressure cannot. Similarly, Kintsch (2000) has shown that another form of principle, namely a metaphor such as "My lawyer is a shark," also is not reversible (cannot be transposed). Therefore, principles often should not be transposed and logically, our neural systems should not automatically allow these broader associations.
In addition, our internal neural networks act to reduce this type of over-generalization by balancing inferences produced by a principle with real-world experiences of related instances. Instances, especially if part of personal experience, can win out or dominate processing relative to generalization from learned principles. Concepts are closer to the experienced world than are principles, which are composed of concepts. Instances and examples that are experienced are likely to dominate the understanding of a principle. This is often referred to in the science education literature as na´ve misconceptions, such as the sun rises rather than the earth rotates under the sun. In the heat-and-pressure case above, common experience suggests that a soft drink can, which has increased internal pressure, usually feels cold, not warm. Thus, it is easier to believe that soft drink cans are cold than to believe that pressure increases heat in the can. The instance wins out, and the principle does not transpose.
Transposing and paraphrasing posttest questions provide particularly important measures of associative learning. These two studies, and our argument so far, strongly indicate that feedback mainly involves a highly specific question and response association. A recent study by Clariana, Wagner, and Rorher-Murphy (2000) has applied an associative learning, connectionist approach to describe feedback effects. Their approach and results will be described in more detail since it is central to our explanation of feedback effects.
A Connectionist Description of Feedback Timing Effects
Recently, Clariana (1999b) has suggested that an associative learning, connectionist approach can be used to predict the quantitative effects of feedback on posttest memory activation levels of errors and of correct responses (Haberlandt, 1997; Elman, 1993; McLeod, Plunkett, & Rolls, 1998; Plunkett & Marchman, 1993; 1996; Seidenberg & McClelland, 1989). He hypothesized a feed-forward network using the delta rule to describe feedback effects with multiple-choice lesson questions (see Figure 5). A feed-forward network consists of input nodes connected to multiple output nodes (Shanks, 1995). The input node represents the multiple-choice question. The output node would represent possible responses. The alternative with the greatest activation level represents the response the neural network will give to that input pattern.
Figure 5. A simple connectionist network with several possible output activation levels (aout) for a given input pattern (ain).
Feedback in this model acts to strengthen question–response associations that are correct and to inhibit associations that are incorrect. The amount of increase or inhibition was computed using the delta rule. The delta rule (Widrow & Hoff, 1960) describes the change in association weight, termed Dw, between the input unit and output unit at each learning trial, as:
Dwio = a ain (to - aout)
The term a is the learning rate parameter (a fudge factor constant that dampens the impact), ain is the activation level of input units, to is the desired response (the t refers to "teacher", in this case to comes from item feedback), and aout is the activation level of the output units (Shanks, 1995). In instructional terms, learning is an increase in the association, which is an increase in Dwio between the stimulus (ain) and the correct response (aout), with a relative decrease in association, which is a decrease in Dwio for incorrect responses.
To apply the delta rule, Clariana (1999b) assumed that lesson average item difficulty values are reasonable estimates of the association weights of the correct responses (aout). Item difficulty (p) is defined as the proportion (pg) of individuals who answer an item (g) correctly (item difficulty notation convention from Crocker & Algina, 1986). For example an item difficulty of .20 indicates that 20% of the learners responded correctly to that item. Item difficulty values can range from 0.00 to 1.00 with low values indicating difficult items and high values indicating easy items.
In the delta rule equation, feedback impacts learning in the term (to - aout). Customarily, the values for to and aout are constrained between 0 and 1 (McLeod, Plunkett, & Rolls, 1998). The value for to equals 1 if the activation level of the input unit matches the desired response (i.e., with correct responses), and to equals zero if the activation level of the input unit does not match the desired response (i.e., with incorrect responses). Therefore, with correct responses, the association weight increases since (1 - aout) is positive while with incorrect responses, the association weight decreases since (0 - aout) is negative. In other words, when feedback is provided as part of the responding instance, correct responses are strengthened and incorrect responses weakened.
Thus, given lesson item difficulties as initial aout, to equal to 0 or 1, and a ain = 0.4 (per Clariana, 1999b), the delta rule can be used to predict posttest item difficulties, aout after immediate and delayed feedback (see Figure 6).
Figure 6. Predicted retention test values generated by the delta rule across a range of lesson item difficulty values (from Clariana, 1999b). Predicted retention test memory of Initial Lesson Responses (ILR) are shown as dashed lines and Correct Responses (CR) are shown as solid lines for delayed feedback (DF) and immediate feedback (IF).
Clariana, Wagner, and Rorher-Murphy (2000) tested this computational model. High school students (n = 52) completed a computer-based lesson with text portions, verbatim and inference-level recognition study tasks, and either DF, STF (KCR), or MTF. The retention test given 24-hours after the lesson consisted of items identical to the lesson. On the retention test, students were asked to remember their initial responses from the day before and also the correct responses. There was a significant difference for type of feedback, with retention test memory of initial lesson responses greater under DF than under STF (see the left panel of Figure 7). These results closely match the predicted inhibition (or not) of initial lesson response errors for immediate feedback relative to DF. This shows that the computational model was able to predict inhibition of lesson errors fairly well; however the model over-estimated increased activation of correct response values (see the right panel of Figure 7), suggesting that different mechanisms are at work with question–correct response associations.
These findings show that both the direction and magnitude of feedback effects, at least for lesson errors, can be computationally determined. This suggests that automatic processes may be at work. Also, though lesson errors are important since they reflect a learner’s mental representation of a question–response association, previous models have not systematically accounted for lesson errors. For these reasons, we advocate that feedback researchers consider an associative learning, connectionist approach.
Explaining the effects of text, questions, and feedback
Below is a visual model of mental representations. This visual model is used to describe the effects of text, questions, and feedback on learners’ mental representations of CAL lesson material.
Visualizing Mental Representations
Mental representations can be visualized in several ways. Collins and Quillian (1972) proposed semantic networks that use nodes to represent concepts and lines between nodes to show the relationships between concepts. For example, some relationship data from McClelland (1981) that describes two fictitious gangs, the Sharks and Jets, can be displayed as a semantic network (see left panel of Figure 8). For descriptive purposes, added information about Gangs and about Sharks and Jets not in the original data set was added to fill out the semantic network. The Shark’s gang member named Dave is a divorced drug pusher. Semantic networks have been used to describe many aspects of information, such as the hierarchical structure of the information.
Figure 8. A semantic network (left) and a MDS (right) of associations for two fictitious gangs (from McClelland, 1981).
Graphical displays of psychological space have also been called high-dimensional semantic space (HDSS; Foltz, Kintsch, & Landauer, 1998) and cognitive maps (Diekhoff & Wigginton, 1982). Several recent computational approaches show similarity in mental representations as distances in psychological space rather than as nodes and links (Diekhoff & Wigginton, 1982; McLeod, Plunkett, & Rolls, 1998). These approaches provide another way of displaying and of thinking about an individual’s mental representation of information.
A scaling procedure, such as multi-dimensional scaling (MDS), can display relationship data visually in fewer dimensions. For example, to use MDS to display HDSS, first relationship data is described in a weight matrix (see Table 1). A 1 in the matrix indicates an association between the column and row instance, while a 0 indicates no association. Each row (or column) in the table is a high-dimensional vector. Next MDS can be applied to that matrix. MDS was conducted with Table 1 data using SPSS 9.0 including the standard default values for MDS except for selecting create Euclidean distances and selecting display group plots (see right panel of Figure 8).
Table 1. Weight matrix of some of McClelland’s (1981) data (with 8 examples of gang members).
First, note that Art is nearer Clyde than Alan (see right panel of Figure 8) showing that the Art and Clyde (with four overlaps) have more similar weight vectors (see Tables 1 & 2) than Art and Alan (with only two overlaps). Simple correlations of vector elements from Table 2 for Art, Clyde, and Alan provide the same information: Art vs. Clyde, r = .54; Art vs. Alan, r = .08; and Clyde vs. Alan, r = .08.
Table 2. Vector elements for Alan, Art, and Clyde.
Next, Art is located near single, bookie, and Jets, as well as Junior High School, and 40-ish. Next, note that in this MDS representation (see Figure 8), Dave is nearest divorced, pusher, and Sharks. Category data from the original data set, such as age 20-ish and High School graduate, are shown as instances along with gang-member instances. Data information that all Sharks share, such as hanging on the north-side and wearing blue, stack precisely on top of Sharks in the MDS, since Sharks, blue, and north-side all have identical weight vectors. In this sparse MDS with only 8 gang members, instances of gang members and characteristics of the instances are all mixed together. Note how the MDS changes when more instances of gang members are included in the matrix (in this case 27 instances, see resulting MDS in Figure 9).
Figure 9. MDS of McClelland (1981) data including 27 gang members. Top and bottom panels are identical, except the bottom panel has been stretched to display details.
The gang member instances grouped together in a cluster or cloud because of the relative similarity of their weight vectors while the vectors that characterize the instances, such as pusher, bookie, and burglar, move away on the dimensional scales (see upper panel of Figure 9). The concept of gang member is represented in this HDSS as a cloud or cluster of instances of gang members, even though there is no weight vector called gang member in the data set to cause this grouping. The cloud of instances is highly structured. Similar gang members will be nearer each other because their vectors are similar. For example, why are Doug and Nick near each other (see center of the bottom panel of Figure 9)? At first glance, they seem dissimilar since Doug is a Jet pusher and Nick is a Shark bookie. However, since Doug and Nick are both 30-ish, high school graduates, and single, they actually have a lot in common.
HDSS as Structural Knowledge
The association between instances in HDSS looks like structural knowledge, which is the knowledge of the inter-relationship of knowledge elements. It has been suggested that developing structural knowledge of a content area is necessary in order to flexibly use that knowledge-base (Jonassen & Wang, 1992). This MDS approach visually depicts structural knowledge (Diekhoff & Wigginton, 1982) and demonstrates that hierarchical structural knowledge can emerge in HDSS when enough cases or examples (vectors) are included in the weight matrix. This visual approach would further suggest that if you directly teach structural knowledge relationships before multiple instances are established, then the structural knowledge associations would simply be another vector like gang member or pusher or Alan. Thus, teaching structural knowledge relationships too soon would result in relatively weaker and even possibly meaningless one-vector effects on HDSS rather than regional (cloud) multi-vector effects that would be quite meaningful to the learner.
If structural knowledge emerges in actual mental representations in a similar way to this MDS model, then CAL should present many cases or examples per coherent knowledge base (in this example, the cluster did not emerge until at least 22 gang members were included in the MDS matrix). Probably human neural networks are more capable of deriving structural knowledge than this MDS approach, suggesting that fewer cases would be required for the emergence of structural knowledge. Nevertheless, learners can only interact with existing instances in order to generate structural knowledge (Jonassen & Cole, 1993). Given the likely importance of structural knowledge, it is reasonable in CAL to include many cases and examples and then require the learner to interact with the cases.
Note that HDSS is a model of association weights, not of activation weights. This is an important distinction. The entire weight matrix is probably never activated at the same time, only parts are activated. An input pattern, such as a question, interacts with the entire weight matrix to produce a network of activation that is a subset of HDSS, and the HDSS response to the question would be one instance or several instances within that subset. For example, if the learner were asked to list all Jets who are burglars, first Jets and burglar would activate (see Figure 9). Note that Jets and burglar, besides being instances in HDSS, are also an element of every other vector in the matrix. Any other instance in the matrix that has a 1 in the Jets and/or burglar element of its vector also activates to some degree. This process has been referred to as spreading activation and will include nearest neighbors in HDSS, such as George, Lance, John, Jim, and possibly Alan (also a correct response) and Ken (an incorrect response). This is a likely subset of HDSS that is activated by the question.
Next, this subset is reduced to one or just a few instances. Kintsch (1994) has described a process of how the activation level of only one or a few instances in a weight matrix increases as the question vector cycles upon the weight matrix while activations of all of the other instances decrease (relatively). A small number of instances with the greatest activation levels when the pattern stabilizes, such as George, John, and Lance, are the HDSS response to the question.
In addition, a vector with a one-element association is established between the question pattern "list all Jets who are burglars" and the response pattern "George, John, and Lance." If the same question is asked later, the response will be considerably quicker because it involves vectors with fewer elements, which will cycle faster. This process could be called rote responding. It is also referred to as the familiarity-based process of recognition, which stabilizes in about 200ms (Rotello & Heit, 1999), compared to the slower (800ms to 2000ms) recall-like or recollection process that involves many more vectors representing more qualitative information and requiring more cycles for the pattern to stabilize (Dobbins, Kroll, Yonelinas, & Liu, 2000). If the system doesn’t immediate identify an input pattern, then it resorts to the slower recollection process. It is not surprising that our neural system would act this way, since there is obviously a powerful survival benefit in shortening the response time to danger.
This vector-based approach is easy to apply and understand and is highly explanatory. However, it oversimplifies how a question is handled by an individual’s mental representation. First, it reduces weight matrix associations between instances to either a 1, 0, or –1; and there is possibly a gradation in associations. Second, the model assumes a relatively na´ve HDSS since it tends to disregard preexisting association data. Regarding gradation in association, many neural-systems rely on sigmoid activation functions which drive activation towards 1, 0, or –1 (McLeod, Plunkett, & Rolls, 1998); therefore, this simplification may not be a problem. Regarding the naivetÚ of the HDSS representation, a relatively uncluttered area of HDSS can be established for new content by simply adding vectors of unique instances, such as context characteristics. For example, the more unique context vectors that are included, the further the entire content representation is driven away from preexisting content instances and thus into clear HDSS space (new dimensional space). This easily accounts for context differences in recall memory, such as a list learned underwater is not as well remembered on land and vice versa (Godden & Baddeley, 1975, 1980). The underwater context variables, which are many, compartmentalize the list words together within a generally less accessible area of HDSS when recalling the list on land. Reestablishing context variables, such as wearing a mask and breathing through a mouthpiece while on land, should tend to bring up the underwater list.
In addition, instances and associations (the elements in an instance) in HDSS may have idiosyncratic activations. For example, if Ken was just previously mentioned, the residual activation from that thought would add to the weak activation that results from the "Jets who are burglars" question. Thus, Ken would become active enough to emerge as a correct answer, though Ken is not a Jet. Also, specific instances in HDSS may have higher baseline activation than other instances, probably due to individual familiarity with those instances. Idiosyncratic activations would account for idiosyncratic responses to text, questions, and feedback.
Using HDSS to Describe the Effects of Text, Questions, and Feedback
The effects of altering a vector element on HDSS can be used to explain the effects of text, questions, and feedback. First, reading a text loads HDSS with additional content by changing 0s to 1s in the weight matrix for associated information, such as "Doug is a high-school graduate." Some instances and associations may already be filled in due to prior-knowledge and experiences. Note that unless associations are specifically stated or implied by the text, there are biologically conservative reasons why the remaining elements in the vector will be 0. In this case, the effect of replacing a 0 with 1 is to make the Doug vector and the high school graduate vector more alike, therefore closer together in HDSS.
Like word instances, propositions and sentences will also be placed as single instances in HDSS (Kintsch, 1994). Foltz, Kintsch, and Landauer (1998) reported that text coherence can be improved by making sure that consecutive sentences have semantic overlap, either by repeating words or repeating synonyms of words in successive sentences. This type of semantic overlap should allow for consecutive sentences to obtain more similar vectors and thus be nearer each other in HDSS, which intuitively seems like a good idea. Low prior-knowledge readers benefit the most from highly coherent text (Britton & Gulgoz, 1991; McNamara, Kintsch, Butler, & Kintsch, 1996). Unfortunately, this type of writing style can be wordy and boring, especially for high-prior knowledge readers.
Questions add to the effects of text in at least two ways. Questions can just repeat text information or can cause the reader to make inferences from the text. When a question repeats text information, it should simply strengthen the association between the instances, changing a 0 vector element to 1, similar to the text case above. In the same way, text summaries should have the same effect as questions that repeat the text.
Question can also drive inferences. For example, a text may say in one place that Doug and Nick had a big argument after their graduation and, later, that they were 18 at the time. The inference is that they graduated from high school. A question could ask, "Did Nick graduate from high school?" This question would act to strengthen and complete the weight matrix by associating several separate instance vectors, thus establishing a more accurate representation in HDSS.
Does requiring the learner to make the inference on their on result in a stronger memory trace association? In other words, do questions result in more fully associated and thus better and more accurate HDSS? As described earlier in the chapter, previous research has shown that questions, and especially higher-order questions, have a stronger effect than lower-order questions and than text only. Questions are fundamentally different than text in that questions probe and activate the learner’s existing mental representation. Therefore, questions tune the HDSS as long as the learner is able to answer the question correctly.
How does feedback complement the effects of questions? Immediate feedback has been shown to strongly inhibit error associations (Clariana et al., 2000). What does this mean in terms of HDSS? Suppose that the learner was asked whether Doug was a Shark or Jet. Note that Doug is near the boundary for Shark (see center of the bottom panel of Figure 9). If Shark were primed at that point due to preceding activity, the learner could reasonably though incorrectly respond that Doug is a Shark. Without feedback, that incorrect association between Doug and Shark would be strengthened. The Doug vector would point more towards Shark territory, and the learner would more likely repeat this error association in the future.
However, if immediate verification feedback (stating yes, no, right, wrong, and so on) is provided, it would inhibit the direct association between Doug and Shark. Since that association is 0 in the weight matrix table, then it would become negative. When the MDS is recomputed with a -1 association for Doug and Shark, Doug moves away from Shark territory and deeper into Jet territory just beside Clyde. This new MDS suggests that immediate verification feedback can establish an orthogonal relationship between Doug and Shark by altering only one element of the Doug vector.
The effect of immediate verification feedback on HDSS can be weak or strong. If there are many examples related to Doug in HDSS, such as should occur with high-prior knowledge learners who have read and understood the text, then these instances will tend to hold Doug’s position as is; and feedback has a relatively smaller effect. But if the HDSS is sparse, such as with low-prior knowledge students who did not read or else did not capture the text, then there are fewer instances holding Doug in place; and Doug will be moved quite a distance in HDSS. Therefore, immediate verification feedback only slightly alters or tunes associations of a rich mental representation but can substantially alter a sparse mental representation.
Multiple-try feedback usually provides verification and with errors, states "try again." Like repeated questioning, providing MTF to high-prior-knowledge learners further enriches an already rich psychological landscape by slightly adjusting the positions of existing content, especially by establishing orthogonal relationships for error associations. Knowledge of correct response feedback, after providing verification, then provides the correct response. KCR establishes a familiarity-based fast association between the question pattern and the response pattern. For posttest questions that are identical to the lesson questions, this is good. However, for more meaningful and new posttest questions, familiarity-based associations bias an already rich landscape with strong distractions. The associations compete with the meaningful existing structure.
On the other hand, low-prior-knowledge learners have relatively flat or unpopulated mental representations for that content. Therefore, additional processing tries required by MTF would involve an HDSS that cannot activate a correct or meaningful response. This would be both useless and frustrating since MTF would tend to scramble up the few correct associations that exist as the learner tries and rejects existing instances, disassociating each instance in turn from other existing instances. No wonder MTF obtains negative effects with low-prior knowledge learners. Therefore, immediate KCR feedback is best for low-prior-knowledge students. KCR provides question–correct response associations to populate schema with familiar instances, and those instances would tend to dominate mental representations of the content.
In CAL, we must do a better job of designing instructional text. Text coherence related to semantic overlap between sentences has been shown to be critical for learning from text. Since the limited screen area in CAL relative to print reduces re-reading of previous sentences (Clariana & Smith, 1988), text coherence must be greater in CAL than with paper-based text.
Questions are effective and account for a substantial portion of the effects of questions with feedback. Questions apparently establish more accurate mental representations of text content. Thus, providing questions in CAL, and especially higher-order questions, is a good idea as long as the learner is able to correctly answer the question. When the learner answers a question incorrectly, then feedback should be provided.
Regarding feedback, here are several tentative conclusions:
If the conclusions of this paper are confirmed, then the effects of feedback are fairly direct and can be predicted. However, substantial research remains to be done to validate and confirm the views expressed above.
Anderson, R. C. (1972). How to construct achievement tests to assess comprehension. Review of Educational Research, 42(2), 145-170.
Anderson, R. C., Kulhavy, R. W., & Andre, T. (1971). Feedback procedures in programmed instruction. Journal of Educational Psychology, 62, 148-156.
Anderson, R. C., & Biddle, W. B. (1975). On asking people questions about what they are reading. In G. Bower, ed., The psychology of learning and motivation, 89-132. New York, NY: Academic Press.
Andre, T. (1979). Does answering higher-level questions while reading facilitate productive learning? Review of Educational Research, 49 (2), 280-318.
Angell, G. W. (1949). The effect of immediate knowledge of quiz results on final examination scores in freshman chemistry. Journal of Educational Research, 42(5), 391-394.
Bangert-Drowns, R. L., Kulik, C. C., Kulik, J. A., & Morgan, M. (1991). The instructional effect of feedback in test-like events. Review of Educational Research, 61 (2), 213-238.
Bardwell, R. (1981). Feedback: How does it function? Journal of Experimental Education, 50, 4-9.
Bormuth, J. R. (1970). On the theory of achievement test items. The University of Chicago Press.
Bormuth, J. R., Manning, J., Carr, J., & Pearson, D. (1970). Children's comprehension of between- and within-sentence syntactic structure. Journal of Educational Psychology, 61, 349-357.
Bransford, J. D. (1979). Human cognition: Learning, understanding, and remembering. Belmont, CA: Wadsworth Publishing.
Britton, B. K., & Guloz, S. (1991). Using Kintsch’s computational model to improve instructional text: Effects of repairing inference calls on recall and cognitive structures. Journal of Educational Psychology, 83, 329-345.
Cantor, J. H., & Brown, J. S. (1956). An evaluation of the trainer-tester and punchboard tutor as electronics trouble-shooting training aids. Technical Report 1257-2-1, Naval Training Device Center, Orlando, Florida. (ERIC Document Reproduction Service No. ED 129 336)
Clariana, R, B. (2001). Determining the separate contributions of higher-order questions and of feedback. An unpublished manuscript.
Clariana, R, B. (1999a). CBT design: A feedback achievement treatment interaction. Twenty-first Annual Proceedings of Selected Research Presentations of the Association for Educational Communications and Technology, 22, 87-92.
Clariana, R. B. (1999b). Differential memory effects for immediate and delayed feedback: A delta rule explanation of feedback timing effects. A Presentation at the Annual Convention of the Association for Educational Communications and Technology, February, 1999 in Houston, TX. (ERIC Document Reproduction Center ED 430 550).
Clariana, R. B. (1993a). The motivational effect of advisement on attendance and achievement in computer-based instruction. Journal of Computer-Based Instruction, 20 (2), 47-51.
Clariana, R. B. (1993b). A review of multiple-try feedback in traditional and computer-based instruction. Journal of Computer-Based Instruction, 20 (3), 67-74.
Clariana, R. B. (1992). The effects of public reports of progress on rate of lesson completion in computer-based instruction. Journal of Computing in Childhood Education, 3 (2), 127-136.
Clariana, R. B., & Lee, D. (2001). Recognition and recall study tasks with feedback. Educational Technology Research and Development, under review.
Clariana, R. B., Ross, S. L., & Morrison, G. R. (1991). The effects of different feedback strategies using computer-assisted multiple-choice questions as instruction. Educational Technology Research and Development, 39 (2), 5-17.
Clariana, R. B., & Smith, L. J. (1989). The effects of AUC and KCR feedback on learners of different ability. A paper presented at the Annual meeting of the Mid-South Educational Research Association, Little Rock, Arkansas November. (ERIC Document Reproduction Service No. ED 313 387)
Clariana, R. B., & Smith, L. J. (1988). Learning style shifts in computer-assisted instruction. A paper presented at the Annual Meeting of the International Association for Computers in Education (IACE) in New Orleans, LA., April, 1988. (ERIC Document Reproduction Service: ED 295 796)
Clariana, R. B., Wagner, D., & Roher-Murphy, L. (2000). A connectionist description of feedback timing. Educational Technology Research and Development, under revision.
Collins, A., & Quillian, M. (1972). Experiments on semantic memory and language comprehension. In L. Gregg, ed., Cognition in learning and memory. New York: Wiley.
Coorough, R. P. (1991). The effects of program control, learner control, and learner control with advisement lesson control strategies on anxiety and learning from computer-assisted instruction. In Proceedings of the 13th Annual Meeting of the Association for Educational Communications and Technology, Orlando, FL.
Crocker, L. M. & Algina, J. (1986). Introduction to classical and modern test theory. New York: Holt, Rinehart and Winston.
Dempsey, J. V., & Driscoll, M. P. (1989). The effects of four methods of immediate corrective feedback on retention, discrimination error, and feedback study time in computer-based instruction. A paper presented at the Annual Meeting of the American Educational Research Association, San Francisco, CA. March.
Dick, W., & Latta, R. (1970). Comparative effects of ability and presentation mode in computer-assisted instruction and programmed instruction. Audio-Visual Communication Review, 18(3), 34-45.
Diekhoff, G. M., & Wigginton, P. (1982). Using multidimensional scaling-produced cognitive maps to facilitate the communication of structural knowledge. A paper presented at the Annual Meeting of the Southwestern Psychological Association. (ERIC Document Reproduction Service No. ED 218 245)
Dobbins, I. G., Kroll, N. E. A., Yonelinas, A. P., & Liu, Q. (2000). Distinctiveness in recognition and free recall: The role of recollection in the rejection of the familiar. Journal of Memory and Language, 38, 381-400.
Duchastel, P. C. (1983). Improving adjunct question research: Processes and ecological validity. Human Learning, 2, 1-5.
Elman, J. L. (1993). Learning and development in neural networks: The importance of starting small. Cognition, 48, 71-99.
Foltz, P. W., Kintsch, W., & Landauer, T. K. (1998). The measurement of textual coherence with latent semantic analysis. Discourse Processes, 25, 285-307.
Gilman, D. A. (1969). Comparison of several feedback methods for correcting errors by computer-assisted instruction. Journal of Educational Psychology, 60 (6), 503-508.
Glover, J. A. (1989). The "testing" phenomenon: Not gone but nearly forgotten. Journal of Educational Psychology, 81 (3), 392-399.
Godden, D. R., & Baddeley, A. D. (1975). Context-dependent memory in two natural environments: On land and underwater. British Journal of Psychology, 66, 325-331.
Godden, D. R., & Baddeley, A. D. (1980). When does context influence recognition memory. British Journal of Psychology, 71, 99-104.
Haberlandt, K. (1997). Cognitive Psychology 2nd edition. Boston, MA: Allyn and Bacon.
Hamaker, C. (1986). The effects of adjunct questions on prose learning. Review of Educational Research, 56 (2), 212-242.
Hannafin, M. J. (1984). Factors affecting the locus of instructional control in the design of computer-assisted instruction. Journal of Instructional Development, 7(3), 6-10.
Hodes, C. L. (1984). Relative effectiveness of corrective and non-corrective computer feedback on the cognitive learning of science. Unpublished Masters Thesis, the Pennsylvania State University. (ERIC Document Reporduction No. ED 311 883)
Hudgins, B.B., Dorman, J., & Harris, M. (1979). Some effects of adjunct questions on intermediate grade children with differing reading comprehension abilities. Journal of Educational Research, 72, 259-265.
Johansen, K. J., & Tennyson, R. D. (1983). Effect of adaptive advisement on perception in learner-controlled, computer-based instruction using a rule-learning task. Educational Communications and Technology Journal, 31(4), 226-236.
Jonassen, D. H. (1985). Generative learning versus mathemagenic control of text processing. In D. H. Jonassen, ed. The Technology of text : Principles for structuring, designing, and displaying text, 9-45. Englewood Cliffs, NJ: Educational Technology.
Jonassen, D. H., & Cole, P. (1993). Learner-generated vs. instructor-provided analysis of semantic relationships. In Proceedings of Selected Research and Development Presentations at the 15th Annual Convention of the Association for Educational Communications and Technology. (ERIC Document Reproduction Service No. ED 362 170)
Jonassen, D. H., & Wang, S. (1992). Acquiring structural knowledge from semantically structured hypertext. In Proceedings of Selected Research and Development Presentations at the 14th Annual Convention of the Association for Educational Communications and Technology. (ERIC Document Reproduction Service No. ED 348 000)
Kintsch, W. (2000). Metaphor comprehension: A computational theory. Psychonomic Bulletin and Review, in press.
Kintsch, W. (1994). Discourse processing. In d’Ydewalle, Eelen, & Bertelson, eds. International perspectives on psychological science: Volume 2: The state of the art, 135-155. Hove, UK: Lawrence Erlbaum Associates Ltd.
Kulhavy, R. W. (1977). Feedback in written instruction. Review of Educational Research, 47 (1), 211-232.
Kulhavy, R. W., & Anderson, R. C. (1972). Delay-retention effect with multiple-choice tests. Journal of Educational Psychology, 63 (5), 505-512.
Kulhavy, R. W., & Stock, W. A. (1989). Feedback in written instruction: The place of response certitude. Educational Psychology Review, 1, 279-308.
Kulik, J. A., & Kulik, C. C. (1988). Timing of feedback and verbal learning. Review of Educational Research, 58 (1), 79-97.
Lee, D., Smith, P. L., & Savenye, W. (1991). The effects of feedback and second try in computer-assisted instruction for a rule-learning task. Thirteenth Annual Proceedings of Selected Research presentations of the Association for Educational Communications and Technology, 13, 411-432.
Lindner, R. W., & Rickards, J. P. (1985). Questions inserted in text: Issues and implications. In D. H. Jonassen, ed. The Technology of text : Principles for structuring, designing, and displaying text, 131-157. Englewood Cliffs, NJ: Educational Technology.
Lewis, M. W., & Anderson, J. R. (1985). Discrimination of operator schemata in problem solving: Learning from examples. Cognitive Psychology, 17 (1), 26-65.
McClelland, J. L. (1981). Retrieving general and specific information from stored knowledge of specifics. In Proceedings of the Third Annual Meeting of the Cognitive Science Society, 170-172.
McLeod, P., Plunkett, K., & Rolls, E. T. (1998). Introduction to connectionist modeling of cognitive processes. Oxford, UK: Oxford University Press.
McNamara, D., Kintsch, E., Butler, S. N., & Kintsch, W. (1996). Are good texts always better? Interactions of text coherence, background knowledge, and levels of understanding in learning from text. Cognition and Instruction, 14, 1-43.
Merrill, J. (1985). Levels of questioning and forms of feedback: Instructional factors in courseware design. A presentation at the Annual Meeting of the American Educational Research Association in Chicago, Illinois. (ERIC Document Reproduction Service No. ED 266 766)
Merrill, J. (1987). Levels of questioning and forms of feedback: Instructional factors in course ware design. Journal of Computer-Based Instruction, 14 (1), 18-22.
Milheim, W. D., & Azbell, J. W. (1988). How past research on learner control can aid in the design of interactive video materials. Tenth Annual Proceedings of Selected Research Papers at the 1988 Annual Convention of the Association for Educational Communications and Technology, 10, 459-472. (ERIC Document Reproduction Service No. ED 295 652).
More, A. J. (1969). Delay of feedback and the acquisition and retention of verbal materials in the classroom. Journal of Educational Psychology, 60(5), 339-342.
Morrison, G. M., Ross, S. M., Gopalakrishnan, M. & Casey, J. (1995). The effects of feedback and incentives on achievement in computer-based instruction. Contemporary Educational Psychology, 20, 32-50.
Mory, E. H. (1996). Feedback research. In D. H. Jonassen, ed. Handbook of research for educational communications and technology, 919-956. New York: Simon & Schuster Macmillan.
Nielsen, M. C. (1990). The impact of informational feedback and a second attempt at practice questions on concept learning in computer-aided instruction. Doctoral dissertation, University of Texas at Austin.
Nishikawa, S. S. (1985). Feedback: Implications for further research and study. A paper presented at the Annual Meeting of the Association for Educational Communications and Technology, Anaheim, CA. (ERIC Document Reproduction Service No. ED 256 328)
Plunkett K., & Marchman, V. A. (1993). From rote learning to system building: Acquiring verb morphology in children and connectionist nets. Cognition, 48, 21-69.
Plunkett K., & Marchman, V. A. (1996). Learning from a connectionist model of the acquisition of the English past tense. Cognition, 61, 299-308.
Pressey, S. L. (1926). A simple apparatus which gives tests and scores - and teaches. School and Society, 23, 373-376.
Pressey, S. L. (1950). Development and appraisal of devices providing immediate automatic scoring of objective tests and concomitant self-instruction. The Journal of Psychology, 29, 417-447.
Richards, D. R. (1989). A comparison of three computer-generated feedback strategies. Eleventh Annual Proceedings of Selected Research Presentations of the Association for Educational Communications and Technology, 11, 358-367.
Rotello, C. M., & Heit, E. (1999). Two-process models of recognition memory: Evidence for recall-to-reject. Journal of Memory and Language, 40, 432-453.
Santiago, R. S., & Okey, J. R. (1992). The effects of advisement and locus of control on achievement in learner-controlled instruction. Journal of Computer-Based Instruction, 19(2), 47-53.
Schimmel, B. J. (1983). A meta-analysis of feedback to learners in computerized and programmed instruction. Paper presented at the Annual Meeting of the American Educational Research Association, Montreal, Canada. (ERIC Document Reproduction No. ED 233 708)
Schimmel, B. J. (1986). Feedback use by low-ability students in computer-based education. (Doctoral dissertation, Columbia University Teachers College). Dissertation Abstracts International, 47 (11), 4068.
Shanks, D. R. (1995). The psychology of associative learning. Cambridge, UK: Cambridge University Press.
Shanks, D. R., Pearson, S. M., & Dickinson, A. (1989).Temporal contiguity and the judgment of causality. Quarterly Journal of Experimental Psychology, 41B, 139-159.
Seidenberg, M., & McClelland, J. (1989). A distributed model of word recognition and naming. Psychological Review, 96, 523-568.
Smith, P. L. (1988). Toward a taxonomy of feedback: Content and scheduling. A paper presented at the Annual meeting of the Association for Educational Communications and Technology, New Orleans, LA.
Smith, P. L., & Ragan, T. J. (1993). Designing instructional feedback for different learning outcomes. In J. V. Dempsey & G. C. Sales, eds., Interactive instruction and feedback, 75-103. Englewood Cliffs, NJ: Educational technology.
Spock, P. A. (1987). Feedback and confidence of response for a rule-learning task using computer-assisted instruction. (Doctoral dissertation, University of Texas, Austin). Dissertation Abstracts International, 48 (5), 1109.
Tennyson, R. D. (1980). Advisement and management strategies as design variables in computer-assisted instruction. A paper presented at the Annual Meeting of the American Educational Research Association, Boston, MA. (ERIC Document Reproduction Service No. 195 246).
Tennyson, R. D., & Buttrey, T. (1980). Advisement and management strategies as design variables in computer-assisted instruction. Educational Communications and Technology Journal, 28(3), 169-176.
Travers, R. M. W., Van Wagenen, R. K., Haygood, D. H., & McCormick, M. (1964). Learning as a consequence of the learner's task involvement under different conditions of feedback. Journal of Educational Psychology, 65 (3), 167-173.
Waldrop, P. B., Justin, J. E., & Adams, T. M. (1986). A comparison of three types of feedback in a computer-assisted instruction task. Educational Technology, 26, 43-45.
Widrow, B., & Hoff, M. E. (1960). Adaptive switching circuits. 1960 IRE WESCON Convention Record (Pt. 4), 96-104.