Evidence on “What Works”: An Argument for Extended-Term Mixed-Method (ETMM) Evaluation Designs

@article{Chatterji2004EvidenceO,
  title={Evidence on “What Works”: An Argument for Extended-Term Mixed-Method (ETMM) Evaluation Designs},
  author={Madhabi Chatterji},
  journal={Educational Researcher},
  year={2004},
  volume={33},
  pages={13 - 3}
}
Federal policy tools for gathering evidence on “What Works” in education, such as the What Works Clearinghouse’s (WWC) standards, emphasize randomized field trials as the preferred method for generating scientific evidence on the effectiveness of educational programs. This article argues instead for extended-term mixed-method (ETMM) designs. Emphasizing the need to consider temporal factors in gaining thorough understandings of programs as they take hold in organizational or community settings… 
View on SAGE
219 Citations
Highly Influential Citations
15
Background Citations
88
Methods Citations
46
Results Citations
3

219 Citations

Gathering evidence on an after-school supplemental instruction program: Design challenges and early findings in light of NCLB.

The No Child Left Behind (NCLB) Act of 2001 requires that public schools adopt research-supported programs and practices, with a strong recommendation for randomized controlled trials (RCTs) as the…

Grades of Evidence

This article argues with a literature review that a simplistic distinction between strong and weak evidence hinged on the use of randomized controlled trials (RCTs), the federal “gold standard” for…

An Analysis of the Fidelity Implementation Policies of the What Works Clearinghouse

Simulations show that the What Works Clearinghouse policy minimizes the positive impact of highly effective programs and the negative impact ofhighly ineffective programs.

Causal inferences on the effectiveness of complex social programs: Navigating assumptions, sources of complexity and evaluation design challenges.

Lessons Learned from the Use of Randomized and Quasi-Experimental Field Designs for the Evaluation of Educational Programs.

Conclusion: Agency in the face of complexity and the future of assumption-aware evaluation practice.

A model for fidelity of implementation in a study of a science curriculum unit: Evaluation based on program theory

This work was based on the work conducted by SCALE-uP: A collaboration between George Washington University and Montgomery County Public Schools (MD); Sharon Lynch, Joel Kuipers, Curtis Pyke, and…

The development of the concept of “evidence” in the U.S. educational policies

It has been approximately twenty years since the word “evidence” entered in educational research thanks to Evidence Based Education and a debate grew immediately cause to the complexity in defining…

“The Research is Continuing, Whether We Are There or Not”1. Research on Continuing Education Between Scientific Traditions and Evidence Expectations

Evidence-based governance – international and supranational stakeholders of educational policy such as the European Union (EU) or the Organisation for Economic Co-operation and Development (OECD)…

Building Confidence in Qualitative Research

The quality of qualitative research has been subject to considerable criticism recently, partly driven by the development of an international movement for “evidence-based policy and practice.” In the…
...

References

SHOWING 1-10 OF 58 REFERENCES

Models and Methods for Examining Standards-Based Reforms and Accountability Initiatives: Have the Tools of Inquiry Answered Pressing Questions on Improving Schools?

This article synthesizes research on standards-based reforms and accountability, with specific attention to purposes, models, and methods of inquiry. Starting with the premise that the reforms were…

Comment: Culture, Rigor, and Science in Educational Research

In this article the authors argue that both the Feuer, Towne, and Shavelson article and the larger National Research Council (NRC) report on which it is based must be understood in the context of…

Contestation and Change in National Policy on “Scientifically Based” Education Research

In this article, we examine the definitions of “scientifically based research” in education that have appeared in recent national legislation and policy. These definitions, now written into law in…

Randomized Experiments in Educational Policy Research: A Critical Examination of the Reasons the Educational Evaluation Community has Offered for not Doing Them

This article notes the paucity with which reform efforts in education have been evaluated experimentally, despite well-nigh universal acknowledgement that experiments provide the best justification…

Scientific Culture and Educational Research

The No Child Left Behind Act of 2001 requires federal grantees to use their funds on evidence-based strategies. The law includes definitions of research quality, which are also featured prominently…

Advances in mixed-method evaluation : the challenges and benefits of integrating diverse paradigms

Evaluation theory and practice today are characteristically pluralistic, embracing diverse perspectives, methods, data, and values within and across studies that aim to generate more insightful and…

Beyond Use: Understanding Evaluation’s Influence on Attitudes and Actions

Although use is a core construct in the field of evaluation, neither the change processes through which evaluation affects attitudes, beliefs, and actions, nor the interim outcomes that lie between…

Transcending the Qualitative-Quantitative Debate: The Analytic and Systemic Approaches to Educational Research

There is a distinction to be made that transcends the one between the quantitative and qualitative research paradigms. It is a distinction between that kind of research that suits best the study of…

Comment: “Science” Rejects Postmodernism

The National Research Council report Scientific Research in Educationclaims to present an inclusive view of science as it responds to federal government attempts to legislate educational research.…

Comment: Be Careful What You Wish For—You May Get It: Educational Research in the Spotlight

Two matters of deep concern in the Feuer, Towne, and Shavelson article are the focus of this commentary. First, the authors agree that legislative mandates reifying particular methodological…
...