[net-gold] Multiple Choice Exam Questions #2
- From: "David P. Dillard" <jwne@xxxxxxxxxx>
- To: Temple University Net-Gold Archive <net-gold@xxxxxxxxxxxxxxxxxxx>, Temple Gold Discussion Group <TEMPLE-GOLD@xxxxxxxxxxxxxxxxxxx>, Net-Gold <net-gold@xxxxxxxxxxxxxxxx>, Educator Gold <Educator-Gold@xxxxxxxxxxxxxxx>, Educator Gold <Educator-Gold@xxxxxxxxxxxxxxxx>, K12AdminLIFE <K12AdminLIFE@xxxxxxxxxxxxxxx>, Net-Platinum <net-platinum@xxxxxxxxxxxxxxx>, NetGold <netgold@xxxxxxxxxxxxxxx>, "Net-Gold @ Nabble" <ml-node+3172864-337556105@xxxxxxxxxxxxx>, K-12ADMINLIFE <K12ADMIN@xxxxxxxxxxxxxxxxxxx>, net-gold@xxxxxxxxxxxxx
- Date: Wed, 7 Apr 2010 05:25:34 -0400 (EDT)
.
Date: Tue, 6 Apr 2010 17:57:46 -0700
From: Richard Hake <rrhake@xxxxxxxxxxxxx>
Reply-To: Net-Gold@xxxxxxxxxxxxxxx
To: AERA-L@xxxxxxxxxxxxxxxxx
Cc: Net-Gold@xxxxxxxxxxxxxxx
Subject: [Net-Gold] Re: Multiple Choice Exam Questions #2
This post contains a few additions and corrections to the previous
post of today "Re: Multiple Choice Exam Questions."
If you reply to this long (25 kB) post please don't hit the reply
button unless you prune the copy of this post that may appear in your
reply down to a few relevant lines, otherwise the entire already
archived post may be needlessly resent to subscribers.
***********************************************
ABSTRACT: Karol Dean of the POD list asked: "Is there any research or
folklore to support the 1 question/minute formula [for
multiple-choice questions] that I've heard?"
To which Ken Bain replied: "multiple-choice questions that simply
require the regurgitation of isolated information, or worse yet, the
ability to recognize correct answers . . . tend to foster surface or
strategic rather than deep approaches to learning. . . . .THIS
DOESN'T MEAN THAT YOU CANNOT DEVELOP MULTIPLE-CHOICE EXAMINATIONS
THAT CAN FOSTER DEEP APPROACHES. Look, for example, at the way Eric
Mazur develops what are basically multiple-choice questions for his
Peer Learning approach. But that approach is embedded in an
environment designed to promote deep considerations. . . . . . To
understand and appreciate Mazur's approach, you must understand both
the way he develops the questions and how he uses them. Once you
understand that (and both the need to promote deep approaches to
learning and the research on what fosters deep approaches), I think
you will quickly see that requiring students to answer multiple
choice questions in less than a minute each (130 questions in 90
minutes) will foster the most shallow of approaches to learning, and
cannot possibly foster deep approaches."
In this post I:
(a) quote psychometricians Mark Wilson and Meryl Bertenthal in
support of Bain's claim that "multiple-choice examinations that can
foster deep approaches," and
(b) elaborate on the physics education environment in which Mazur
came to desert the traditional passive student lecture for an
"Interactive Engagement" method.
***********************************************
Karol Dean (2010) in a POD post of 5 April wrote:
". . . . .I talked with the faculty member about a general rule I've
heard that 1 question per minute is about the right amount for
standard test bank exam questions. Is there any research or folklore
to support the 1 question/minute formula that I've heard? Are there
better ways to estimate the amount of time it would take a student to
respond to multiple choice questions drawn from a test bank? If it
helps, the course being discussed is an Anatomy course for 1st year
students who are pursuing a nursing degree."
To which Ken Bain (2010) <http://www.montclair.edu/center/Bain.html>
replied [bracketed by lines "BBBBBB. . . . ."; my insert at ". . . .
.[[insert]]. . . . ."; my CAPS]:
BBBBBBBBBBBBBBBBBBBBBBBBBB
As others may have already pointed out, one important consideration
comes from the literature on surface, strategic, and deep approaches
to learning. Briefly, thirty years of research finds that
multiple-choice questions that simply require the regurgitation of
isolated information, or worse yet, the ability to recognize correct
answers in a multiple-choice exam tend to foster surface or strategic
rather than deep approaches to learning. Without a deep approach
(intention) students are highly unlikely to develop a deep
understanding.
THIS DOESN'T MEAN THAT YOU CANNOT DEVELOP MULTIPLE-CHOICE
EXAMINATIONS THAT CAN FOSTER DEEP APPROACHES. Look, for example, at
the way Eric Mazur develops what are basically multiple-choice
questions for his Peer Learning approach. But that approach is
embedded in an environment designed to promote deep considerations.
You can see Eric discuss his ideas here . . . . . .[[Bain gives
<http://www.bestteachersinstitute.org/id106.html> but closer to the
source is the actual UTube video at
<http://www.youtube.com/watch?v=WwslBPj8GgI> (Mazur, 2010)]]. . . . .
To understand and appreciate Mazur's approach, you must understand
both the way he develops the questions and how he uses them. Once
you understand that (and both the need to promote deep approaches to
learning and the research on what fosters deep approaches), I think
you will quickly see that requiring students to answer multiple
choice questions in less than a minute each (130 questions in 90
minutes) will foster the most shallow of approaches to learning, and
cannot possibly foster deep approaches. I don't think you could
possibly ask the kinds of questions that Eric has developed in such a
130 question, 90 minute exam, and you obviously do not create the
kind of learning environment that promotes deep learning. . . . .
[[my own practice was to allow students a full 50 minutes to complete
the 30 multiple-choice question Force Concept Inventory (Hestenes et
al. (1992)]]. . . .
BBBBBBBBBBBBBBBBBBBBBBBBBB
TWO POINTS:
1111111111111111111111111111111111111111111111
1. Regarding Bain's provocative claim: "THIS DOESN'T MEAN THAT YOU
CANNOT DEVELOP MULTIPLE-CHOICE EXAMINATIONS THAT CAN FOSTER DEEP
APPROACHES," psychometricians Mark Wilson and Meryl Bertenthal
(2005, p. 94) wrote:
"Performance assessment is an approach that offers great potential
for assessing complex thinking and learning abilities, but multiple
choice items also have their strengths. For example, although many
people recognize that multiple-choice items are an efficient and
effective way of determining how well students have acquired basic
content knowledge, many do not recognize that they can also be used
to measure complex cognitive processes. For example, THE "FORCE
CONCEPT INVENTORY" . . . [Hestenes, Wells, & Swackhamer, 1992] . . .
IS AN ASSESSMENT THAT USES MULTIPLE-CHOICE ITEMS TO TAP INTO
HIGHER-LEVEL COGNITIVE PROCESSES." [My CAPS.]
22222222222222222222222222222222222222222222222
2. Regarding the "environment" alluded to by Bain in his statement:
"[Mazur's" approach is embedded in an environment designed to promote
deep considerations," consider Mazur's (1997, p. 4) account of the
physics education environment in which he came to desert the
traditional passive student lecture for an "Interactive Engagement"
method.
[Here "Interactive Engagement" courses are operationally defined in
Hake (1998a) as "those designed at least in part to promote
conceptual understanding through the active engagement of students in
heads-on (always) and hands-on (usually). . . .[[but not always -
witness Mazur's "Peer Instruction"]]. . . . activities that yield
immediate feedback through discussion with peers and/or instructors."]
Mazur wrote: "When reading this. . . . .[Halloun & Hestenes
(1985a,b]. . . my first reaction was 'Not my students. . .!'
Intrigued, I decided to test my own students' conceptual
understanding, as well as that of physics majors at Harvard. . . . .
the results of the test came as a shock: The students faired hardly
better on the Halloun and Hestenes test [1985a] than on their midterm
exam. Yet the Halloun and Hestenes test is SIMPLE, whereas the
material covered by the examination (rotational dynamics, moments of
inertia) if of far greater difficulty, or so I thought."
In Table 1, p. 972, "Force Concept Inventory and Mechanics Diagnostic
Test results" of Crouch & Mazur (2001) note:
a. The abrupt increase in the average *normalized* gain <g> [i.e.,
the actual average gain (<%post> - <%pre>) divided by the *maximum*
possible average gain (100% - <%pre>)] where the angle brackets
indicate class averages] from 0.25 in 1990 to 0.49 in 1991 when Mazur
replaced his passive-student lectures (THAT NETTED VERY POSITIVE
STUDENT EVALUATIONS - many administrators erroneously regard student
evaluations as valid measures of students' learning!) -with the
"interactive engagement" of peer instruction.
b. The gradual increase in the average normalized gain <g> from 0.49
in 1991 to 0.74 in 1997 as various improvements (Crouch & Mazur,
2001) were made in the implementation of peer instruction.
The Harvard results are consistent with those from hundreds of other
introductory physics courses employing either traditional or
interactive engagement methods [for reviews see Hake (1998 a,b;
2002a, 2007)]. And yet lessons from the physics education reform
effort [e.g., deHaan (2005); Hake (2002a,b, 2005b, 2007, 2008, 2010);
Heron & Meltzer (2005); Michael (2006); Stokstad (2001); Wood &
Gentile (2003)]; continue to be generally ignored throughout academia
[McCray et al. (2003), Millar & Osborne (2009), Osborne (2007),
Shelly et al. (2009), Zemsky (2003); and the NRC (1997, 1999, 2003).
Richard Hake, Emeritus Professor of Physics, Indiana University
24245 Hatteras Street, Woodland Hills, CA 91367
Honorary Member, Curmudgeon Lodge of Deventer, The Netherlands
President, PEdants for Definitive Academic References (PEDAR)
<rrhake@xxxxxxxxxxxxx>
<http://www.physics.indiana.edu/~hake/>
<http://www.physics.indiana.edu/~sdi/>
<http://HakesEdStuff.blogspot.com/>
<http://iub.academia.edu/RichardHake>
**************************************
NAYSAYERS:
"In science education, there is almost nothing of proven efficacy."
Grover Whitehurst, director, Institute of Education Sciences, USDE,
as quoted by Sharon Begley (2004)
"One striking feature of science education is that no standard or
commonly agreed outcome measures exist for any major topic."
Millar & Osborne (2009)
"50 years of research, curriculum development, and implementation
have not presented consistent and compelling patterns of outcomes."
Jonathan Osborne (2007)
**********************
YEASAYERS:
"Physicists are out in front in measuring how well students learn the
basics, as science educators incorporate hands-on activities in hopes
of making the introductory course a beginning rather than a finale."
Stokstad (2001)
"Physics educators have led the way in developing and using objective
tests to compare student learning gains in different types of
courses, and chemists, biologists, and others are now developing
similar instruments. These tests provide convincing evidence that
students assimilate new knowledge more effectively in courses
including active, inquiry-based, and collaborative learning, assisted
by information technology, than in traditional courses."
Wood & Gentile (2003)
"There is substantial evidence that scientific teaching in the
sciences, i.e., teaching that employs instructional strategies that
encourage undergraduates to become actively engaged in their own
learning, can produce levels of understanding, retention and transfer
of knowledge that are greater than those resulting from traditional
lecture/lab classes. But widespread acceptance by university faculty
of new pedagogies and curricular materials still lies in the future."
Robert DeHaan (2005)
"One of the most striking findings came from comparison of the
learning outcomes (as measured by the FCI and a related inventory on
mechanics) from 14 traditional courses (2,084 students) and 48
courses using "interactive-engagement" (active learning) techniques
(4,458 students). The results on the FCI assessment showed that
students in the interactive engagement courses outperformed students
in the traditional courses by 2 SDs. Similarly, students in the
interactive-engagement courses outperformed students in the
traditional courses on the. . . . . .[Mechanics Baseline Test
(Hestenes & Wells, 1992]. . . , a measure of problem-solving ability.
This certainly looks like evidence that active learning works!
Research in physics education is having a profound effect on the
development of instructional materials."
Joel Michael (2006)
**************************************
REFERENCES [Tiny URL's courtesy <http://tinyurl.com/create.php>. All
URL's accessed on 6 April 2010.]
Bain, K. 2010. "Re: Multiple Choice Exam Questions," POD post of 5
Apr 2010 22:08:23-0400; online on the OPEN! POD archives at
<http://tinyurl.com/ylqfolb>.
Begley, S. 2004. "To Improve Education, We Need Clinical Trials To
Show What Works," Wall Street Journal, 17 December, page B1. Copied
into the Appendix of Hake (2005a) in accord with the "fair use"
provision of U.S. copyright law as provided for in section 107 of the
US Copyright Law, Title 17 - see e.g.,
<http://www.law.cornell.edu/uscode/17/107.shtml>.
Crouch, C.H. & E. Mazur. 2001. "Peer Instruction: Ten years of
experience and results," Am. J. Phys. 69: 970-977; online at
<http://tinyurl.com/sbys4>.
Dean, K. 2010. "Multiple Choice Exam Questions," POD post of 5 Apr
2010 13:19:17-0700; online on the OPEN! POD archives at
<http://tinyurl.com/yzusuyb>.
DeHaan, R.L. 2005. "The Impending Revolution in Undergraduate Science
Education," Journal of Science Education and Technology 14(2):
253-269; abstract online at <http://tinyurl.com/ymwwe3>.
Hake, R.R. 1998a. "Interactive-engagement vs traditional methods: A
six-thousand-student survey of mechanics test data for introductory
physics courses," Am. J. Phys. 66: 64-74; online at
<http://www.physics.indiana.edu/~sdi/ajpv3i.pdf> (84 kB).
Hake, R.R. 1998b. "Interactive-engagement methods in introductory
mechanics courses," online at
<http://www.physics.indiana.edu/~sdi/IEM-2b.pdf> (108 kB). A crucial
companion paper to Hake (1998a).
Hake, R.R. 2002a. "Lessons from the physics education reform effort,"
Ecology and Society 5(2): 28; online at
<http://www.ecologyandsociety.org/vol5/iss2/art28/>. For an update on
six of the lessons on "interactive engagement" see Hake (2007).
Hake, R.R. 2002b. "Assessment of physics teaching methods.
Proceedings of the United Nations Educational, Scientific, and
Cultural Organization's ASPEN (ASian Physics Education Network)
workshop on active learning in physics, University of Peradeniya, Sri
Lanka; online at
<http://www.physics.indiana.edu/~hake/Hake-SriLanka-Assessb.pdf> (86
kB).
Hake, R.R. 2005a. Re: "To Improve Education, We Need Clinical Trials
To Show What Works," AERA-L post of 10 Jan 2005 16:01:05 -0800;
online at <http://tinyurl.com/yesqk7n>.
Hake, R. R. 2005b. "The Physics Education Reform Effort: A Possible
Model for Higher Education?" online at
<http://www.physics.indiana.edu/~hake/NTLF42.pdf> (100 kB). This is a
slightly edited version of an article that was (a) published in the
National Teaching and Learning Forum 15(1), December, online to
subscribers at
<http://www.ntlf.com/FTPSite/issues/v15n1/physics.htm>, and (b)
disseminated by the Tomorrow's Professor list
<http://ctl.stanford.edu/Tomprof/postings.html> as Msg. 698 on 14 Feb
2006.
Hake, R.R. 2006. "Possible Palliatives for the Paralyzing Pre/Post
Paranoia that Plagues Some PEP's" [PEP's = Psychologists, Education
Specialists, and Psychometricians], Journal of MultiDisciplinary
Evaluation, Number 6, November, online at
<http://survey.ate.wmich.edu/jmde/index.php/jmde_1/article/view/41/50>.
This even despite the admirable anti-alliteration advice at
psychologist Donald Zimmerman's site
<http://mypage.direct.ca/z/zimmerma/> to "Always assiduously and
attentively avoid awful, awkward, atrocious, appalling, artificial,
affected alliteration."
Hake, R.R. 2007. "Six Lessons From the Physics Education Reform
Effort," Latin American Journal of Physics
<http://journal.lapen.org.mx/sep07/HAKE%20Final.pdf> (124 kB).
Hake, R.R. 2008. "Design-Based Research in Physics Education
Research: A Review" in Kelly et al. (2008). A pre-publication version
of Hake's chapter is online at
<http://www.physics.indiana.edu/~hake/DBR-Physics3.pdf> (1.1 MB).
Hake, R.R. 2010. "Should We Measure Change? Yes! online at
<http://www.physics.indiana.edu/~hake/MeasChangeS.pdf> (2.5 MB) and
as ref. 43 at
<http://www.physics.indiana.edu/~hake>. To appear as a chapter in
"Evaluation of Teaching and Student Learning in Higher Education"
[Hake (in preparation)]. For a severely truncated version see Hake
(2006).
Halloun, I. & D. Hestenes. 1985a. "The initial knowledge state of
college physics students." Am. J. Phys. 53:1043-1055; online at
<http://modeling.asu.edu/R&E/Research.html>. Contains the "Mechanics
Diagnostic" test, precursor to the widely used "Force Concept
Inventory [Hestenes et al. (1992). This landmark work is NOT
referenced in McCray et al. (2003); NRC (1997, 1999, 2003); or the
NRC volumes Pelligrino et al. (2001) and Shavelson & Towne (2002).
Halloun, I. & D. Hestenes. 1985b. "Common sense concepts about
motion." Am. J. Phys. 53:1056-1065; online at
<http://modeling.asu.edu/R&E/Research.html>.
Heron, P.R.L. & D.E. Melzer. 2005. "The future of physics education
research: Intellectual challenges and practical concerns." Am. J.
Phys. 73(5): 390-394; online at
<http://www.physicseducation.net/docs/Heron-Meltzer.pdf> (57 kB).
Hestenes, D., M. Wells, & G. Swackhamer. 1992. "Force Concept
Inventory," The Physics
Teacher 30(3): 141-158; online at
<http://modeling.asu.edu/R&E/FCI.PDF> (100kB) [but without the test
itself.] The 1995 revision by Halloun, Hake, Mosca, & Hestenes is
online (password protected) at the same URL, and is currently
available in 16 languages: Chinese, Czech, English, Finnish, French,
German, Greek, Italian, Malaysian, Persian, Portuguese, Russian,
Spanish, Slovak, Swedish, & Turkish. My own habit was to allow
students a full 50 minutes to answer the 30 multiple-choice questions
on this test. For recommendations on administration and grading see
Hake (2002b).
Hestenes, D., and M. Wells. 1992. "A mechanics baseline test,"
Physics Teacher 30: 159-166; online at
<http://modeling.asu.edu/R&E/MechBaseline.pdf> (283 kB) [but without
the test itself.]
Kelly, A.E., R.A. Lesh, J.Y. Baek. 2008. "Handbook of Design Research
Methods in Education: Innovations in Science, Technology,
Engineering, and Mathematics Learning and Teaching." Routledge
Education; publisher's information at <http://tinyurl.com/4eazqs>.
Amazon.com information at <http://tinyurl.com/ygbotlh>.
Mazur, E. 1997. "Peer instruction: a user's manual." Prentice Hall;
information online at
<http://mazur-www.harvard.edu/education/pi_manual.php>.
Mazur, E. 2010. "Confessions of a Converted Lecturer" talk at the
University of Maryland on 11 November 2009. The abstract reads: "I
thought I was a good teacher until I discovered my students were just
memorizing information rather than learning to understand the
material. Who was to blame? The students? The material? I will
explain how I came to the agonizing conclusion that the culprit was
neither of these. It was my teaching that caused students to fail! I
will show how I have adjusted my approach to teaching and how it has
improved my students' performance significantly." That talk is now on
UTube at <http://www.youtube.com/watch?v=WwslBPj8GgI>; and the
abstract, slides, and references - sometimes obscured in the UTube
talk - are at <http://tinyurl.com/ybc53jw> as a 4 MB pdf.
As of 6 April 2010 17:23:00-0700 Eric's talk had been viewed by
16,855 UTube fans, up from 12,800 on 16 March 2010. In contrast,
serious articles in the education literature, often read only by the
author and a few cloistered academic specialists, usually create
tsunamis in educational practice equivalent to those produced by a
pebble dropped into the Pacific Ocean.
McCray, R.A., R.L. DeHaan, J.A. Schuck, eds. 2003. "Improving
Undergraduate Instruction in Science, Technology, Engineering, and
Mathematics: Report of a Workshop" Committee on Undergraduate STEM
Instruction," National Research Council, National Academy Press;
online at <http://www.nap.edu/catalog/10711.html>. Physicists
attending the workshop were Paula Herron, Priscilla Laws, John
Lehman, Ramon Lopez, Richard McCray, Lillian McDermott, Carl Wieman,
and Jack Wilson.
Michael, J. 2006. "Where's the evidence that active learning works?"
Advances in Physiology Education 30: 159-167, online at
<http://tinyurl.com/ykzp7lt>.
Millar, R. & J. Osborne. 2009. "Research and Practice: A Complex
Relationship?" Chapter 3, pages 41-62, of Shelley et al. (2009).
Surprisingly, the Google book preview of Shelley et al. (2009) at
<http://tinyurl.com/yddphh3> contains all of pages 41-62. To see this
use the ">" at the top of the first page to go to page vi and then
click on chapter 3.
NRC. 1997. "Science Teaching Reconsidered: A Handbook," National
Research Council, Committee on Undergraduate Science Education,
National Academy Press; online at
<http://www.nap.edu/catalog/5287.html>.
NRC. 1999. "Transforming Undergraduate Education in Science,
Mathematics,Engineering, and Technology," National Research Council,
Committee on Undergraduate Science Education, National Academy Press;
online at <http://www.nap.edu/catalog/6453.html>.
NRC. 2003. "Evaluating and Improving Undergraduate Teaching I Science
and Technology, Engineering, and Mathematics," ed. by M.A. Fox & N.
Hackerman, National Research Council, Committee on Undergraduate
Science Education, National Academy Press; online at
<http://www.nap.edu/catalog/10024.html>.
Osborne, J. 2007. "In praise of armchair science education,"
contained within E-NARST News 50(2) at
<http://www.narst.org/news/e-narstnews_July2007.pdf> (3.2 MB). The
talk itself is online at
<http://www.kcl.ac.uk/content/1/c6/01/29/36/joconference.pdf> (112
kB).
Pelligrino, J.W., N. Chudowsky, R. Glaser, eds. 2001. "Knowing What
Students Know: The Science and Design of Educational Assessment,"
National Academy Press; online at
<http://www.nap.edu/catalog/10019.html>.
Shavelson, R.J. & L. Towne, eds. 2002. "Scientific Research in
Education," National Academy Press, online at
<http://www.nap.edu/catalog/10236.html>.
Shelley, M.C., L.D. Yore, & B. Hand, eds. 2009. "Quality Research in
Literacy and Science Education: International Perspectives and Gold
Standards." Springer, publisher's information at
<http://www.springerlink.com/content/g2447682464446x2/>. Amazon.com
information at <http://tinyurl.com/yf7efra>, note the searchable
"Look Inside" feature. Barnes & Noble information at
<http://tinyurl.com/y8n9pe9>. An expurgated (teaser) version is
online as a Google "book preview" at <http://tinyurl.com/yddphh3>.
Stokstad, E. 2001. "Reintroducing the Intro Course," Science 293:
1608-1610, 31 August; online at <http://tinyurl.com/ybmcsol>.
Wood, W.B., & J.M. Gentile. 2003. "Teaching in a research context,"
Science 302: 1510; 28 November; online at
<http://bioquest.org/science_vol302_pg1510.pdf> (xx kB)
Wilson, M.R. & M.W. Bertenthal, eds. 2005. "Systems for State Science
Assessment." National Academy Press; the complete book and a search
engine are online at <http://www.nap.edu/catalog.php?record_id=11312>.
Zemsky, R. 2003. "On Encouraging Faculty to Pursue Instructional
Reform," in McCray et al. (2003).
.
Other related posts:
- » [net-gold] Multiple Choice Exam Questions #2 - David P. Dillard
