Considering different research methodologies as complementary is certainly more appealing rather than seeing them as adversary. Each research methodology serves particular functions that will not be effectively met using different research methodologies. For the purpose of justifying causal conclusion, randomized experiments are “the most efficient and reliable method” (Cook and Sinha, 2006, p. 563). This method, however, seems inappropriate for many other educational inquiries. Two examples, i.e. theories of change and design experiments, are delineated to describe the complexity of educational inquiries.
Cook and Sinha (2006) identify three limitations of a theory of change when it is used to study the causal relationship among variables, namely its lack of explicitness, its tendency of linearity, and its lack of specific timeline. Such claim is valid, since program theory serves a different function. Part of my assignment to study at LUC [from Sanata Dharma University] is to develop a leadership program for school principals in Indonesia. In the group of 12 students, we developed the training plan and objectives, learning materials, and evaluation plan. As program developers, we are interested in the success of our program, so we developed a number of instruments to obtain five-level program evaluation, i.e. participants’ reactions, participants’ learning, organization support and change, participants’ use of new knowledge and skills, and student learning outcomes (Guskey, 2000). In a sense, by employing a program theory to evaluate our program, we will gather much information that allows us to improve our service. Program theory will not necessarily tell us the causal relationship among variables (e.g. structural support from participants’ schools and their efforts to bring significant changes in their schools). However, it does provide an early description or portrayal of the conditions of each school and participants, and it serves our main objective, i.e. to inform us what to do to improve our service.
Another case in point, the inquiries done in design experiments could not simply be conducted under randomized experimental procedures. Two major issues emerge. First, the epistemological beliefs between the randomized experiments and design experiments are different. Geared to the concept of “learning ecology” (Cobb, et al., 2003, cited in Schoenfeld, 2006, p. 200), design experiment researchers consider learning context as a systematic whole – thus any related issues that constitute real learning (comprising personal characteristics of students, peer interactions, teacher skills and commitment, curriculum, external forces such as high-stakes testing and other regulations) cannot be necessarily broken down. Such a viewpoint is starkly against randomized experiments that take particular variables into isolated conditions (like cells put into a petri dish for analysis). Second, randomized experiments inherit traditional research methods and positivist epistemology with its nomothetic goal. Design experiments were developed in the context of theory change (Brown, 1992).Ann L. Brown (1943-1999) is widely known for her contribution to develop renewed constructivist perspective in learning. The book How People Learn: Brain, Mind, Experience, and School (Bransford, Brown and Cocking, 1999), is considered to capture the essence of the three-decade research in learning science. Brown’s work is certainly not based on a pre-existing structure as randomized experiments are. Research on memory, information-processing theory, nature of expertise, and metacognition in the last three decades has built a substantive body of knowledge. This theory change has made design experiment advocates find the rigid tradition of randomized experimental research hard to implement. When real learning process is considered as systemic (i.e. not an isolated action done by a particular individual on a specific area, but also as social enterprise done with other people), design experiments work best. There will be too many confounding variables that make randomized experiments unfeasible to accomplish.
References
Bransford, John D., Brown, Ann L., & Cocking, Rodney R. (1999). How People Learn: Brain, Mind, Experiences, and School. Washington, D.C.: National Academy Press.
Brown, Ann L. (1992). Design Experiments: Theoretical and Methodological Challenges in Creating Complex Interventions in Classroom Settings, The Journal of the Learning Sciences, 2(2), 141-178.
Cook, Thomas D, & Sinha, Vandna (2006). Randomized Experiments in Educational Research. In Judith L. Green, Gregory Camili, & Patricia B. Elmore (Eds.). Handbook of Complementary Methods in Education Research, (pp. 551-565). Mahwah, NJ: American Educational Research Association.
Guskey, Thomas R. (2000). Evaluating Professional Development. Thousand Oaks, CA: Corwin Press, Inc.
Schoenfeld, Alan H. (2006). Design Experiments. In Judith L. Green, Gregory Camili, & Patricia B. Elmore (Eds.). Handbook of Complementary Methods in Education Research, (pp. 193-205). Mahwah, NJ: American Educational Research Association.
