Rigor - Free Research

**************************************************************
Educational CyberPlayGround http://www.edu-cyberpg.com/
**************************************************************
Net Happenings Mailing List

Net Happenings Service
<http://www.edu-cyberpg.com/Community/Subguidelines.html>

Subscribe | Unsubscribe | Change Email Preferences -
<http://www.edu-cyberpg.com/Community/NetHappenings.html>
**************************************************************
National Children's Folksong Repository www.edu-cyberpg.com
Integrate Literacy, Music, and Technology into the classroom.
**************************************************************

Rigor-Free Research?
By Joanne Jacobs Published  09/27/2004

http://www2.techcentralstation.com/1051/printer.jsp?CID=1051-092704D

Forget the anecdotes and assumptions. Under the No Child Left Behind Act, 
federal education dollars are supposed to fund only programs proven 
effective by "scientifically based research." That's spotlighting a 
problem: A lot of what passes for education research isn't reliable or 
rigorous, and many education professors aren't keen on the scientific method.

Education has a "dirty little secret," writes Jeffrey Mervis in the June 
11, 2004 Science Magazine:

"No program has yet met that rigorous standard, because none has been 
scientifically evaluated and shown to be effective. (A related secret is 
that there's no consensus on the type of evaluation studies that are needed.)"

Bush's Education Department wants controlled studies, like the tests that 
determine whether a new drug is safe and effective. Is Panacea Z more 
likely to cure ignorance than Brand X? It would be nice to know before 
investing millions of dollars. And yet the research often provides no guidance.

In May, the National Research Council tried to determine the effectiveness 
of middle-school math curricula developed by the National Science 
Foundation and by commercial publishers. After identifying 698 studies of 
19 curricula, NRC concluded it was impossible to decide which programs work 
and which don't. While about 20 percent of studies met NRC's minimal 
standards, no one program was backed by sufficient research to prove its 
effectiveness.

Most of the NRC's 212-page report discussed how to evaluate education 
programs in a scientifically valid way, Mervis writes. "The problem is 
complicated by the many factors that influence student achievement: 
students' previous knowledge, their teachers' quality of training, the 
level of resources available, the degree of parental and community support, 
and so on."

In July, the Education Department unveiled the new, improved What Works 
Clearinghouse which reports on which educational programs, products, 
practices and policies are backed by research, and evaluates the strengths 
or weaknesses of the studies.

For example, the clearinghouse looked at 300 studies on peer-assisted 
learning (students tutoring each other) and found 15 that met evidence 
standards; 176 studies didn't pass the screen and 109 are still being 
reviewed.

Of 70 studies on two middle school math curricula, only one met evidence 
standards fully, another met the standards with some reservations and 20 
are still being reviewed.
Controlled studies are harder to do with children than with lab rats, 
especially if everyone assumes that the experimental group is getting 
something special that should be available to everyone.

Evaluating some questions requires following students for many years, but 
students move around so much it's hard to keep track of them. Some studies 
-- for example on the effectiveness of bilingual vs. English immersion 
classes -- get muddled because teachers aren't following the model they're 
supposed to be using. Controlling for home factors also is challenging. If 
the study is small, a few atypical students or teachers can throw off the 
results, critics say. If it's large, it costs a fortune.

Some educators say there's no point in doing controlled studies: The 
evidence will be ignored by policy makers. Or they complain that schools 
will focus on measurable outcomes -- test scores -- and ignore what's hard 
to measure.

Yet without scientific rigor, education researchers can't answer any of the 
interesting questions.

"Education is often degraded by the use of pseudoscience or weak science or 
anecdote in lieu of better methods," writes Grover J. (Russ) Whitehurst, 
research director for the U.S. Education Department, in a Chronicle of 
Higher Education story.

Academics in the American Education Research Association typically do 
"qualitative" and "ethnographic" research, notes the Chronicle. Like 
anthropologists, they describe what goes on in a classroom or at a school, 
but don't provide any data that makes it possible to figure out whether one 
approach works better than another.

"Fewer than 10 percent of AERA members are knowledgeable about randomized 
trials," Robert F. Boruch, a Penn education and statistics professor, tells 
the Chronicle. "And even fewer have actually worked on a randomized trial."

As a result, education professors have been frozen out of major new 
studies, often in favor of private research firms like Mathematica, which 
is evaluating software that claims to boost reading and math skills, and 
MDRC, which has specialized in job training and welfare reform. Labor 
economists, statisticians and psychologists have the skills to do 
controlled studies. For the most part, the education professors do not.

Of course, some are converting to rigorous research. It's where the money 
is. And plenty of academics really do want to know what works.

But there's a deep well of hostility to cold, hard, number-heavy science, 
poisoned further by liberal elites' loathing of the Bush administration. 
Though the move to controlled studies started in the Clinton 
administration, it didn't take off till Bush pushed through No Child Left 
Behind, which greatly increased federal education funding and insisted that 
all federally funded programs be research-based. If Bush's guys want 
scientific rigor, it must have something to do with Halliburton, right?

Joanne Jacobs blogs on education at JoanneJacobs.com. She's writing a book 
on a start-up charter high school. She is a TCS contributor.


<>~~~~~<>~~~~~<>~~~~~<>~~~~~<>~~~~~<>~~~~~<>~~~~~<>~~~~~<>~~~~~<>
EDUCATIONAL CYBERPLAYGROUND 
http://www.edu-cyberpg.com

Net Happenings, K12 Newsletters, Network Newsletters
http://www.edu-cyberpg.com/Community/index.html

FREE EDUCATION VENDOR DIRECTORY LISTING
http://www.edu-cyberpg.com/Directory/default.asp

HOT LIST OF SCHOOLS ONLINE
http://www.edu-cyberpg.com/Schools/default.asp

Educational CyberPlayGround Services
http://www.edu-cyberpg.com/PS/Home_Products.html
<>~~~~~<>~~~~~<>~~~~~<>~~~~~<>~~~~~<>~~~~~<>~~~~~<>~~~~~<>~~~~~<>

Other related posts: