DBR_11

After summarizing the primary benefits of Design-Based Research (e.g., iterative design, articulation of proto-theories), Koschmann, et al. argue that researchers are still left with the considerably immense challenge of how to actually do it: "How do we go about systematically and rigorously studying [instructional] practice?" (p. 134). One promising option they identify is Garfinkel's Ethnomethodology because of its specific focus on meaning-making practices and "'local rationality.'"

They allocate a good chunk of their chapter-length discussion to Garfinkel's 5 policy statements on ethnomethodology: Indifference; Contingently Achieved Accomplishment; Relevance; Accountability; and Indexicality. In the aggregate, they argue that these Ethnomethodological principles can help uncover "meaning-making practices." More specifically, these principles emphasize the researcher's responsibility to locate what people within these scenes of study are "actually accomplishing" and "making relevant."

Koschmann, T., Stahl, G., & Zemel, A. (2007). The video analyst’s manifesto. In R. Goldman, R. Pea, B. Barron, & S. J. Denny (Eds.), Video Research in the Learning Sciences (pp. 133-143). Mahwah, N.J.: Lawrence Erlbaum Associates.

After summarizing the primary benefits of Design-Based Research (e.g., iterative design, articulation of proto-theories), Koschmann, et al. argue that researchers are still left with the considerably immense challenge of how to actually do it: "How do we go about systematically and rigorously studying [instructional] practice?" (p. 134). One promising option they identify is Garfinkel's Ethnomethodology because of its specific focus on meaning-making practices and "'local rationality.'"

They allocate a good chunk of their chapter-length discussion to Garfinkel's 5 policy statements on ethnomethodology: Indifference; Contingently Achieved Accomplishment; Relevance; Accountability; and Indexicality. In the aggregate, they argue that these Ethnomethodological principles can help uncover "meaning-making practices." More specifically, these principles emphasize the researcher's responsibility to locate what people within these scenes of study are "actually accomplishing" and "making relevant."

Koschmann, T., Stahl, G., & Zemel, A. (2007). The video analyst’s manifesto. In R. Goldman, R. Pea, B. Barron, & S. J. Denny (Eds.), Video Research in the Learning Sciences (pp. 133-143). Mahwah, N.J.: Lawrence Erlbaum Associates.

Ash focuses on implementing design-based research methods in informal learning contexts. Drawing from her background as a biologist, her particular scope of interest is science or discovery museums. She presents a three-level approach that allows the researcher to gain insights at both micro and macro levels. These three levels comprise a (1) Flow chart that presents an overview of a single event, (2) a Significant Event [SE] that includes one significant meaning-making event, and (3) a Microgenetic level where a detailed analysis pairs an utterance to a function. Her emphasis on discontinuity refers to the natural flow of conversational discourse in which there are pauses and interruptions, but viewed in the aggregate, these discontinuous segments can be cohered and used to infer certain meanings. Ash provides a valuable overarching rationale for her study when she explains that while significant financial sums are invested in various educational after-school and museum programs, little is known about the nature of learning that occurs there.

Ash, D. (2007). Using video data to capture discontinuous science meaning making in Nonschool settings. In R. Goldman, R. Pea, B. Barron, & S. J. Denny (Eds.), Video Research in the Learning Sciences (pp. 207-226). Mahwah, N.J.: Lawrence Erlbaum Associates.

Phillips addresses the challenge of obtaining funding for DBR research. He argues that two problems contribute to this difficulty. The first is an oversimplification of scientific research in that it sets experimental methods (e.g., randomized trials) as the singular, "gold" standard and as a result, there is inadequate recognition given to the vital creative and investigative processes that establish the scope and purpose of the research (i.e., it's raison d'être). Accordingly, Phillips presents three suggestions. First, DBR researchers should specify their anticipated results by describing them in one of three ways (e.g., enhance understanding of design process, design an artifact). Second, researchers should also demonstrate a clear correspondence between claim(s) that are made and their supporting warrants. Third, funding panels should consider adopting something akin to a "creative arts" model where a researcher's proposal can be evaluated as part of a larger portfolio of work, and therefore an indicator of its potential. Phillips analogizes it to the MacArthur's Foundation's Genius Grant where although the evaluation process is fuzzy, there is a "surprising degree of inter-rater reliability" among its judges.

Phillips, D. C. (2006). Assessing the quality of design research proposals. In J. J. H. van den Akker, K. Gravemeijer, S. McKenney, & N. Nieveen (Eds.), Educational Design Research (pp. 93-99). London ; New York: Routledge.

Kelly appears to essentially assert that Design Experiments cannot yet be considered a full-fledged, research methodology because it tends to define itself more in terms of "descriptors" (e.g., iterative, interventionist) than those characteristics commonly associated with a mature methodology. He prefers the term Teaching Experiments because it is more "centrally concerned with curricular subject matter" (p. 117). For DBR to evolve into a mature methodology, it must demonstrate or reflect characteristics such as an Argumentative Grammar, Contributions to Problems of Meaningfulness, and the Balancing of Contingent with Necessary Claims. Argumentative Grammar, for example, provides the foundation for making claims based on solid warrants.

Kelly, A. (2004). Design Research in Education: Yes, but is it Methodological? Journal of the Learning Sciences, 13(1), 115-128.

Dede argues that one problem DBR faces is that it leans too heavily on methodology (over-methodologized) and not enough on theoretical grounding (i.e., under-conceptualized); consequently, DBR's most pressing need is epistemological. Such a challenge can be seen as part of a broader evolution in which the field moves from a "Pre-paradigmatic" status to a "Paradigmatic" one (e.g., Kuhn). In developing theoretical foundations, more attention needs to be given to falsifiable propositions. Acknowledging the formidable difficulties, Dede nevertheless suggests that the DBR community needs to consensually arrive at a set of standards that essentially determines what offers a substantive contribution to the field, or more formally, what induces a new "ontological insight"?

Dede, C. (2004, January). If Design-Based Research is the Answer, What is the Question? A Commentary on Collins, Joseph, and Bielaczyc; diSessa and Cobb; and Fishman, Marx, Blumenthal, Krajcik, and Soloway in the JLS Special Issue on Design-Based Research. Journal of the Learning Sciences, pp. 105-114.

I came across an intriguing piece in the NYTimes that reminded me of some of our recent discussions in class centering around the questions related to measurement and assessment (Measurement and its Discontents). Essentially, the author (Robert Crease, a philosophy prof at SUNY-Stony Brook) uses the recent 24th International conference on Weights & Measures to discuss two forms of measurement: ontic and ontological. As I understand it, Ontic measurement pertains to the quantitative measures we're well acquainted with (e.g., meters, kilograms). Ontological has to do with more of an experientially informed basis of evaluation. Crease essentially argues that we don't do enough of the ontological, because we are so fixated on the measuring itself that we tend to forget why we are measuring in the first place. In laying out this case, he references education as one area that has suffered from too much ontic measurement.

One way is to ask ourselves what is missing from our measurements. Are the tests administered by schools making students smarter and more educated, or just making us think we know how to evaluate education?

Lobato asserts that current approaches to studying transfer are limited because they are primarily based on what the experimenter considers to be “models of normative or expert performance” rather than the unique processes used by the subject (learner). She also finds Campione, et al.’s (1995) alternate model of “flexible understanding” limiting because it doesn’t adequately distinguish between learning and transfer. In studying a high school algebra class, she proposes action-oriented transfer, which focuses on how a student constructs “relations of similarity,” or “how the 'actors' see situations as similar.” Instead of hypotheses, Lobato argues that profiles (Collins, 1999) are better suited to investigating action-oriented transfer. Observations should be guided on “focusing phenomena” that signal the “regularities in the ways in which teachers, students, artifacts, and curriculum act together to direct students’ attention toward certain mathematical properties over others.” Benefits of action-oriented transfer include identifying what is relevant for students and how they perceive affordances. Lobato provides a table of eight differences between the theoretical assumptions made by traditional transfer and action-oriented. Overall, Lobato helps researchers see how transfer is more complicated than a simple, top-down communication between teacher (expert) and student.

Lobato, J. (2003). How Design Experiments Can Inform a Rethinking of Transfer and Vice-Versa. Educational Researcher, 32(1), 17 -20.

Borko, et al. discuss Design-Based Research (DBR) in the context of a broader review of four different methodological genres. They select a study from each genre and evaluate it according to three criteria:quality, recency, and topical focus.

They describe central features of DBR and assert that there are commonalities between DBR and "practitioner research" (e.g., spiraling cycles and iterative cycles). As a representative example of DBR, they refer to a three-year design experiment, the Construction of Elementary Mathematics (CEM) project, which centered on the study of an experimental teacher preparation program. The authors explain that the design experiment generated both practical products such as (a) pedagogical practices for supporting pre-service teachers' quantitative reasoning skills and (b) theoretical contributions, notably an extension of "existing taxonomies of mathematical justification.” Borko, et al. conclude this chapter by urging a pluralistic view of research methodologies as essential for teasing out the complexity inherent in 21st century learning environments and present several recommendations for making it feasible (e.g., build capacity by coordinating researchers with various types of expertise). For inspiration, they note two promising examples of such a vision (Teacher Pathways, Teacher Quality Partnership). Overall, this helps see where DBR is positioned within the broader educational research landscape, and more importantly, how it is configured less as an adversary to traditional research and more as a contributor to a collective effort to understanding complex learning environments that no one method is capable of fully explaining.

Borko, H., Whitcomb, J. A., & Byrnes, K. (2008). Genres of research in teacher education. In M. Cochran-Smith, S. Feiman-Nemser, & D. J. McIntyre (Eds.), Handbook of Research on Teacher Education: Enduring Questions in Changing Contexts. Feiman-Nemser, Sharon. (pp. 1017-1049). New York: Routledge.

diSessa and Cobb argue that Design Studies needs a new theoretical basis because existing theoretical types (e.g., Grand Theory, Orienting Frameworks, Frameworks for Action, and Domain-Specific Instructional Theories) are inadequate. Accordingly, they propose Ontological Innovation, which they define as "the invention of new scientific categories, specifically categories that do useful work in generating, selecting among, and assessing design alternatives" (p. 78).

They present two case studies as illustrations: (1) Meta-Representational Competence, and (2) Sociomathematical Norms. The first, MRC, occurs within the context of teaching physics to elementary school students and find that students already possessed considerable expertise in inventing, evaluating, and refining a spectrum of representational forms even though the content matter was beyond their grade level. The second case, Sociomathematical Norms, also is conducted within an elementary school context and focuses on math instruction. Here they examine the “renegotiation of classroom social norms” which leads to an analysis of mathematical difference - i.e., the discussion of different approaches to solving math problems. Among other conclusions, this suggests that mathematical difference involves a joint negotiation between teacher and students. Overall, most of the articles we have read have fallen in two camps: the need (rationale) for Design-Based Research as a viable alternative to traditional research methods, and the many intricate details and challenges associated with carrying it out - i.e., methodology. diSessa and Cobb’s article presents a new area of need: a theoretical foundation.

diSessa, A. A., & Cobb, P. (2004). Ontological Innovation and the Role of Theory in Design Experiments. Journal of the Learning Sciences, 13(1), 77-103.