[net-gold] Re: How Reliable Are the Social Sciences?
- From: "David P. Dillard" <jwne@xxxxxxxxxx>
- To: Other Net-Gold Lists -- Educator Gold <Educator-Gold@xxxxxxxxxxxxxxx>, Educator Gold <Educator-Gold@xxxxxxxxxxxxxxxx>, net-gold@xxxxxxxxxxxxx, NetGold <netgold@xxxxxxxxxxxxxxx>, Net-Gold <net-gold@xxxxxxxxxxxxxxxx>, K-12ADMINLIFE <K12ADMIN@xxxxxxxxxxxxxxxxxxx>, K12AdminLIFE <K12AdminLIFE@xxxxxxxxxxxxxxx>, NetGold <netgold@xxxxxxxxxxxxxxxx>, Net-Platinum <net-platinum@xxxxxxxxxxxxxxx>, Net-Gold <NetGold_general@xxxxxxxxxxxxxxxxx>, Temple Gold Discussion Group <TEMPLE-GOLD@xxxxxxxxxxxxxxxxxxx>, Temple University Net-Gold Archive <net-gold@xxxxxxxxxxxxxxxxxxx>, Health Lists -- Health Diet Fitness Recreation Sports Tourism <healthrecsport@xxxxxxxxxxxxxxxx>, Health Diet Fitness Recreation Sports <healthrecsport@xxxxxxxxxxxxxxx>, HEALTH-RECREATION-SPORTS-TOURISM@xxxxxxxxxxxxxxxxxxx
- Date: Sun, 20 May 2012 23:29:56 -0400 (EDT)
.
.
Date: Sun, 20 May 2012 20:08:07 -0700
From: Richard Hake <rrhake@xxxxxxxxxxxxx>
Reply-To: Net-Gold@xxxxxxxxxxxxxxx
To: AERA-L@xxxxxxxxxxxxxxxxx
Cc: Net-Gold@xxxxxxxxxxxxxxx
Subject: [Net-Gold] Re: How Reliable Are the Social Sciences?
.
.
If you reply to this long (26 kB) post please don't hit the reply
button unless you prune the copy of this post that may appear in your
reply down to a few relevant lines, otherwise the entire already
archived post may be needlessly resent to subscribers.
.
****************************************************
.
ABSTRACT: Rick Froman of the TIPS discussion list has pointed to a
New York Times Opinion Piece "How Reliable Are the Social Sciences?"
by Gary Gutting at <http://nyti.ms/K0xVQL>. Gutting wrote that
Obama, in his State of the Union address <http://wapo.st/JnuBCO>
cited "The Long-Term Impacts of Teachers: Teacher Value-Added and
Student Outcomes in Adulthood" (Chetty et al., 2011 at
<http://bit.ly/KkanoU>) to support his emphasis on evaluating
teachers by their students' test scores. That study purportedly shows
that students with teachers who raise their standardized test scores
are "more likely to attend college, earn higher salaries, live in
better neighborhoods, and save more for retirement."
.
After comparing the reliability of social-science research
unfavorably with that of physical-science research, Getting wrote [my
CAPS): "IS THERE ANY WORK ON THE EFFECTIVENESS OF TEACHING THAT IS
SOLIDLY ENOUGH ESTABLISHED TO SUPPORT MAJOR POLICY DECISIONS?" THE
CASE FOR A NEGATIVE ANSWER lies in the [superior] predictive power of
the core natural sciences compared with even the most highly
developed social sciences."
.
Most education experts would probably agree with Getting's negative
answer. Even economist Eric Hanushek, as reported by Lowery
<http://nyti.ms/KnRvDh>, states: "Very few people suggest that you
should use value-added scores alone to make personnel decisions."
.
But then Getting goes on to write (slightly edited): "While the
physical sciences produce many detailed and precise predictions, the
social sciences do not. The reason is that such predictions almost
always require randomized controlled trials (RCT's) which are seldom
possible when people are involved. . . . . . Jim Manzi. . .
.[[according to Wikipedia <http://bit.ly/KqMf1M>, a senior fellow at
the conservative Manhattan Institute <http://bit.ly/JvwKG1>]]. . . .
in his recent book "Uncontrolled" <http://amzn.to/JFalMD> offers a
careful and informed survey of the problems of research in the social
sciences and concludes that non-RCT social science is not capable of
making useful, reliable, and nonobvious predictions for the effects
of most proposed policy interventions." BUT:
.
(1) Randomized controlled trails may be the "gold standard" for
medical research, but they are not such for the social science of
educational research - see e.g., "Seventeen Statements by
Gold-Standard Skeptics #2" (Hake, 2010) at <http://bit.ly/oRGnBp>.
.
(2) Unknown to most of academia, and probably to Getting and Manzi,
ever since the pioneering work of Halloun & Hestenes (1985a) at
<http://bit.ly/fDdJHm>, physicists have been engaged in the social
science of Physics Education Research that IS "capable of making
useful, reliable, and nonobvious predictions," e.g., that
"interactive engagement" courses can achieve average normalized
pre-to-posttest gains which are about two-standard deviations above
*comparison* courses subjected to "traditional" passive-student
lecture courses. This work employs pre/post testing with Concept
Inventories <http://en.wikipedia.org/wiki/Concept_inventory> - see
e.g., (a) "The Impact of Concept Inventories on Physics Education
and It's Relevance For Engineering Education" (Hake, 2011) at
<http://bit.ly/nmPY8F>, and (b) "Why Not Try a Scientific Approach to
Science Education?" (Wieman, 2007) at <http://bit.ly/anTMfF>.
.
****************************************************
.
Psychologist Rick Froman of the TIPS discussion list pointed to a
"New York Times" Opinion Piece "How Reliable Are the Social
Sciences?" by Gary Gutting (2012), a professor of philosophy at the
University of Notre Dame <http://bit.ly/LhiZey>.
.
Gutting wrote [bracketed by lines "GGGG. . . . . ."; my inserts at ".
. . . .[[insert]]. . . ."; replacing concealed links with overt
URL's; slightly edited]:
.
GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG
.
Public policy debates often involve appeals to results of work in
social sciences like economics and sociology. For example, in his
State of the Union address . . . .[[see the transcript published in
the Washington Post (2012), and Valerie Strauss' (2012) report "Obama
on education in State of the Union address."]]. . . . this year,
President Obama cited a recent high-profile study. . . . .[["The
Long-Term Impacts of Teachers: Teacher Value-Added and Student
Outcomes in Adulthood" (Chetty et al., (2011)]]. . . to support his
emphasis on evaluating teachers by their students' test scores. The
study purportedly shows that students with teachers who raise their
standardized test scores are "more likely to attend college, earn
higher salaries, live in better neighborhoods, and save more for
retirement."
.
How much authority should we give to such work in our policy
decisions? The question is important because media reports often
seem to assume that any result presented as "scientific" has a claim
to our serious attention. But this is hardly a reasonable view.
There is considerable distance between, say, the confidence we should
place in astronomers' calculations of eclipses and a small marketing
study suggesting that consumers prefer laundry soap in blue boxes. .
.
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
.
Scientists working in a discipline generally have a good sense of
where a given piece of works stands in their discipline. But often,
as I have pointed out . . . . .[["Trying to Live Forever" (Gutting,
2011)]]. . . . for the case of biomedical research, popular reports
often do not make clear the limited value of a journalistically
exciting result. Good headlines can make for bad reporting.
.
Second, and even more important, there is our overall assessment of
work in a given science in comparison with other sciences. The core
natural sciences (e.g., physics, chemistry, biology) are so well
established that we readily accept their best-supported conclusions
as definitive. . . . . Even the best-developed social sciences like
economics have nothing like this status.
.
Consider, for example, the report President Obama referred to. By
all accounts it is a significant contribution to its field. As
reported in The Times . . . .[["Big Study Links Good Teachers to
Lasting Gain" (Lowrey, 2012)]]. . . . . , the study, by two
economists from Harvard and one from Columbia, "examined a larger
number of students over a longer period of time with more in-depth
data than many earlier studies, allowing for a deeper look at how
much the quality of individual teachers matters over the long term."
As such, "It is likely to influence the roiling national debates
about the importance of quality teachers and how best to measure that
quality."
.
But how reliable is even the best work on the effects of teaching? .
. . . . . IS THERE ANY WORK ON THE EFFECTIVENESS OF TEACHING THAT IS
SOLIDLY ENOUGH ESTABLISHED TO SUPPORT MAJOR POLICY DECISIONS? THE
CASE FOR A NEGATIVE ANSWER . . . .[[my CAPS]]. . . . . .lies in the
predictive power of the core natural sciences compared with even the
most highly developed social sciences. Social sciences may be
surrounded by the "paraphernalia" of the natural sciences, such as
technical terminology, mathematical equations, empirical data, and
even carefully designed experiments. But when it comes to generating
reliable scientific knowledge, there is nothing more important than
frequent and detailed predictions of future events. We may have a
theory that explains all the known data, but that may be just the
result of our having fitted the theory to that data. The strongest
support for a theory comes from its ability to correctly predict data
that it was not designed to explain.
.
While the physical sciences produce many detailed and precise
predictions, the social sciences do not. The reason is that such
predictions almost always require randomized controlled trails
(RCT's) which are seldom possible when people are involved. For one
thing, we are too complex: our behavior depends on an enormous number
of tightly interconnected variables that are extraordinarily
difficult to distinguish and study separately. Also, moral
considerations forbid manipulating humans the way we do inanimate
objects. As a result, most social science research falls far short
of the natural sciences' standard of controlled experiments. . . . .
.
. . . . . .
.
Jim Manzi. . . . .[[according to Wikipedia <http://bit.ly/KqMf1M>, a
senior fellow at the conservative Manhattan Institute
<http://bit.ly/JvwKG1>]]. . . . in his recent book "Uncontrolled"
[Manzi (2012)], offers a careful and informed survey of the problems
of research in the social sciences and concludes that
"nonexperimental". . . .[[psychologists' code-word for "non-RCT"]]. .
. social science is not capable of making useful, reliable, and
nonobvious predictions for the effects of most proposed policy
interventions." BUT:
.
(1) Randomized controlled trails may be the "gold standard" for
medical research but they are not such for the social science of
educational research - see e.g., "Should Randomized Control Trials Be
the Gold Standard of Educational Research?" (Hake (2005a,b,c) and
"Seventeen Statements by Gold-Standard Skeptics #2" (Hake (2010)].
.
(2) Unknown to most of academia, and probably to Getting and Manzi,
ever since the pioneering work of Halloun & Hestenes (1985a at
<http://bit.ly/b1488v>, physicists have been engaged in the social
science of Physics Education Research that IS "capable of making
useful, reliable, and nonobvious predictions," e.g., that
"interactive engagement" courses can achieve average normalized
pre-to-posttest gains which are about two-standard deviations above
*comparison* courses subjected to "traditional" passive-student
lecture courses. This work employs pre/post testing with Concept
Inventories <http://en.wikipedia.org/wiki/Concept_inventory> - see
e.g., (a)"Promising Practices in Undergraduate Science, Technology,
Engineering, and Mathematics Education: Summary of Two Workshops"
(NRC, 2011a); (b) "Successful K-12 STEM Education: Identifying
Effective Approaches in Science, Technology, Engineering, and
Mathematics" (NRC, 2011b); (c) "The Impact of Concept Inventories on
Physics Education and It's Relevance For Engineering Education"
(Hake, 2011); and (d) "Why Not Try a Scientific Approach to Science
Education?"(Wieman, 2007) at <http://bit.ly/anTMfF>). .
.
Those who wish to dig deeper into social-science research might consider:
.
(1) 'Experimental and Quasi- Experimental Designs for Generalized
Causal Inference" [Shadish, Cook, & Campbell (2002)]; and
.
(2) "Expanded Social Scientist's Bestiary: A Guide to Fabled Threats
to, and Defenses of, Naturalistic Social Science" [Phillips (2000)].
.
.
.
Richard Hake, Emeritus Professor of Physics, Indiana University
Honorary Member, Curmudgeon Lodge of Deventer, The Netherlands
President, PEdants for Definitive Academic References
which Recognize the Invention of the Internet (PEDARRII)
<rrhake@xxxxxxxxxxxxx>
Richard Hake <rrhake@xxxxxxxxxxxxx>
Links to Articles: <http://bit.ly/a6M5y0>
Links to SDI Labs: <http://bit.ly/9nGd3M>
Blog: <http://bit.ly/9yGsXh>
Twitter <http://bit.ly/juvd52
.
.
.
"In some quarters, particularly medical ones, the randomized
experiment is considered the causal 'gold standard.' IT IS CLEARLY
NOT THAT IN EDUCATIONAL CONTEXTS, given the difficulties with
implementing and maintaining randomly created groups, with the
sometimes incomplete implementation of treatment particulars, with
the borrowing of some treatment particulars by control group units,
and with the limitations to external validity that often follow from
how the random assignment is achieved."
- Tom Cook & Monique Payne (2002, p. 174)
.
". . .the important distinction. . .[between, e.g., education and
physics]. . . is really not between the hard and the soft sciences.
Rather, it is between the hard and the easy sciences."
David Berliner (2002)
.
"Physics educators have led the way in developing and using objective
tests to compare student learning gains in different types of
courses, and chemists, biologists, and others are now developing
similar instruments. These tests provide convincing evidence that
students assimilate new knowledge more effectively in courses
including active, inquiry-based, and collaborative learning, assisted
by information technology, than in traditional courses."
Wood & Gentile (2003)
.
.
.
REFERENCES [All URL's shortened by <http://bit.ly/> and accessed on
20 May 2012.]
Berliner, D. 2002. "Educational research: The hardest science of
all," Educational Researcher 31(8): 18-20; online as a 49 kB pdf at
<http://bit.ly/GAitqc>.
.
Chetty, R., J.N. Friedman, & J.E. Rockoff. 2011. "The Long-Term
Impacts of Teachers: Teacher Value-Added and Student Outcomes in
Adulthood," revised January 2012, online as a 741 kB pdf at
<http://bit.ly/Jntweg>. NBER Working Paper No. 17699. revised January
2012, CAUTION: A footnote states: "NBER working papers are
circulated for discussion and comment purposes. They have not been
peer- reviewed or been subject to the review by the NBER Board of
Directors that accompanies official NBER publications." (NBER =
National Bureau of Economic Research). An executive summary is
online as a 94 kB pdf at <http://bit.ly/M4Fy96>. Graphs of the
primary results are online at <http://bit.ly/KkanoU> with the
conclusion "great teachers create great value, and that test-score
based value-added measures are one useful input into identifying such
teachers."
.
Cook, T.D. & M.R. Payne. 2002. "Objecting to the Objections to Using
Random Assignment in Educational Research" in Mosteller & Boruch
(2002).
.
Froman, R. 2012. "How reliable are the social sciences?" 0nline on
the OPEN! TIPS archives at <http://bit.ly/J9D9Za>. Post of 18 May
2012 07:05:10-0700 to TIPS.
.
Gutting, G. 2011. "Trying to Live Forever" New York Times, Opinion
Pages, August 9, 2011,; online at <http://nyti.ms/Ku5Dwv>.
.
Gutting, G. 2012. "How Reliable Are the Social Sciences?" New York
Times, Opinion. Pages, 17 May; online at <http://nyti.ms/K0xVQL>.
.
Hake, R.R. 2005a. "Should Randomized Control Trials Be the Gold
Standard of Educational Research?" online on the OPEN AERA-L archives
at <http://bit.ly/qrUfFz>. Post of 15 Apr 2005 20:28:30-0700 to
AERA-L and various other discussion lists.
.
Hake, R.R. 2005b. "Should Randomized Control Trials Be the Gold
Standard of Educational Research? "online on the OPEN AERA-L archives
at <http://bit.ly/nBDInF>. Post of 17 Apr 2005 20:30:46 0700to AERA-L
and various other discussion lists.
.
Hake, R.R. 2005c. "Should Randomized Control Trials Be the Gold
Standard of Educational Research? "online on the OPEN AERA-L archives
at <http://bit.ly/ppBsyC >. Post of 19 Apr 2005 09:48:12 -0700 to
AERA-L and various other discussion lists.
.
Hake, R.R. 2010. "Seventeen Statements by Gold-Standard Skeptics #2,"
online on the OPEN! AERA-L archives at <http://bit.ly/oRGnBp>. Post
of 27 Apr 2010 10:03:58-0700 to AERA-L and Net-Gold. The abstract and
link to the complete post were transmitted to various discussion
lists and are also on myblog "Hake'sEdStuff" at
<http://bit.ly/bSmKI9 > with a provision for comments.
.
Hake, R.R. 2011. "The Impact of Concept Inventories On Physics
Education and It's Relevance For Engineering Education," invited
talk, 8 August, second annual NSF-sponsored "National Meeting on STEM
Concept Inventories," Washington, D.C., online as a 8.7 MB pdf at
<http://bit.ly/nmPY8F>.
.
Halloun, I. & Hestenes, D. 1985a. "The initial knowledge state of
college physics,"Am. J. Phys. 53(11): 1043-1055; online as a 197 kB
pdf at <http://bit.ly/fDdJHm>. See also Halloun & Hestenes (1985b).
.
Halloun, I. & D. Hestenes. 1985b. "Common sense concepts about
motion." Am. J. Phys. 53(11) 1056-1065; nline as a 70 kB pdf at
<http://bit.ly/hVIFD9>.
.
Lowrey, A. 2012. "Big Study Links Good Teachers to Lasting Gain," New
York Times, Education, 6 January; online at <http://nyti.ms/KnRvDh>.
.
Manzi, J. 2012. "Uncontrolled: The Surprising Payoff of
Trial-and-Error for Business, Politics, and Society." Basic Books,
publisher's information at <http://bit.ly/JeNS82>. Amazon.com
information at <http://amzn.to/JFalMD>. Note the searchable "Look
Inside" feature.
.
Mosteller, F. & R. Boruch, eds. 2002. "Evidence Matters: Randomized
Trials in Education Research." Brookings Institution. Amazon.com
information at <http://amzn.to/n6T0Uo> . A searchable expurgated
Google Book Preview is online at <http://bit.ly/mTcPIE>.
.
NRC. 2011a. "Promising Practices in Undergraduate Science,
Technology, Engineering, and Mathematics Education: Summary of Two
Workshops," online at <http://bit.ly/nCMLk7>. Natalie Nielsen,
Rapporteur. Planning Committee on Evidence on Selected Innovations in
Undergraduate STEM Education: Susan Singer (Chair), Melvin George,
Kenneth Heller, David Mogk, & William B. Wood, Board on Science
Education, Division of Behavioral and Social Sciences and Education.
The National Academies Press; online at <http://bit.ly/nCMLk7>.
.
NRC. 2011b. "Successful K-12 STEM Education: Identifying Effective
Approaches in Science, Technology, Engineering, and Mathematics,"
online at <http://bit.ly/opEXhn>," Committee on Highly Successful
Science Programs for K-12 Science Education" of the Division of
Behavioral and Social Sciences and Education.
.
Phillips, D.C. 2000. "Expanded Social Scientist's Bestiary: A Guide
to Fabled Threats To, and Defenses Of, Naturalistic Social Science"
Rowman & Littlefield; publisher's information at
<http://bit.ly/Jb1cXz>. Amazon.com information at
<http://amzn.to/cstR0B>. Paul E. Meehl
<http://en.wikipedia.org/wiki/Paul_E._Meehl> wrote: "Should be
required reading for all Ph.D. candidates in social science. It is a
mind clearing analysis of the highest order, prophylactic and
curative of the numerous methodological and substantive ills that
afflict us. It is especially needed today when the
'positivist-bashers' are using the Vienna Circle's mistakes and
Kuhn's exaggerations for obscurantist purposes.
.
Shadish, W.R., T.D. Cook, & D.T. Campbell. 2002. "Experimental and
Quasi- Experimental Designs for Generalized Causal Inference."
Houghton Mifflin. Amazon.com information at <http://amzn.to/JWWMVi>.
A goldmine of references on social science research.
.
Strauss, V. 2012. "Obama on education in State of the Union address"
New York Times, "The Answer Sheet," 24 January; online at
<http://wapo.st/KBgR6p>.
.
Wieman, C. 2007. "Why Not Try a Scientific Approach to Science
Education?" Change Magazine, September/October; online as a 804 kB
pdf at <http://bit.ly/anTMfF>.
.
Wood, W.B., & J.M. Gentile. 2003. "Teaching in a research context,"
Science 302: 1510; 28 November; online to subscribers at
<http://bit.ly/9izfFz>. A summary is online to all at
<http://bit.ly/9qGR6m>.
.
Washington Post. 2012. "State of the Union 2012: Obama speech
transcript," Washington Post, 24 January, online at
<http://wapo.st/JnuBCO>. Obama states: "At a time when other
countries are doubling down on education, tight budgets have forced
states to lay off thousands of teachers. WE KNOW A GOOD TEACHER CAN
INCREASE THE LIFETIME INCOME OF A CLASSROOM BY OVER $250,000. . . . .
[[my CAPS]]. . . . The foregoing was probably derived from Chetty et
al. (2011). Their executive summary states: "Replacing a teacher
whose true VA is in the bottom 5% with one of average quality would
generate cumulative earnings gains of $52,000 per student or more
than $1.4 million for the average classroom; discounting at a 5%
interest rate to age 12 YIELDS A PRESENT VALUE GAIN OF MORE THAN
$250,000 PER CLASSROOM. . . . . . [[my CAPS]]. . . .
.
.
Other related posts:
- » [net-gold] Re: How Reliable Are the Social Sciences? - David P. Dillard
