Nation’s Report Card Data Mining Competition 2019
Co-sponsored by NSF Northeast Big Data for Education Spoke
and Educational Testing Service
Technical Directors: Thanaporn "March" Patikorn and Neil Heffernan
Organizers: Ryan Baker, Beverly Woolf, Irvin Katz, Carol Forsyth,
Jaclyn Ocumpaugh
We are pleased to announce a worldwide competition to use data science
tools to analyze educational events in a large national test. The goal
of this competition is to engage leading researchers and promising
doctoral students in a Grand Challenge that pushes the field of
educational data mining forward, develops metrics for measuring
students’ test taking activities, and helps develop and test
evaluation methods for educational analysis. Competition participants
are invited to assess data produced by students early in a test to
predict students’ future activities later in the test. Thus,
competition participants will try to understand effective and
ineffective test-taking behaviors, and to determine how quickly these
behaviors can be detected. Details of the competition can be found
here https://sites.google.com/view/dataminingcompetition2019/home
This competition is organized by National Science Foundation Northeast
Big Data for Education Spoke (BDE) in conjunction with Educational
Testing Service. Please see below for answers to common questions:
What is the Dataset? This competition uses a 2017 data set provided by
Educational Testing Service, with permission from The Nation's Report
Card, also known as the National Assessment of Educational Progress
(NAEP). We will evaluate data from 2464 8th grade students who
received two blocks of math items. Block A contained 19 items and
Block B contained 15 items; students were given 30 minutes to complete
each block. Thus, competition participants will see data from the two
released blocks.
Also known as The Nation’s Report Card, NAEP is administered by the
U.S. Department of Education. The dataset for the competition includes
scores, responses to the test questions, a background questionnaire,
time to respond for each item, and events. Events include starting an
item, clicking a response (or typing something for constructed
response items), changing responses, and use of additional tools
(e.g., text-to-speech, drawing tool).
What will Competition Participants Predict? The nature of the exam
allows students to complete items at their own pace and even to skip
items. However, a hard, 30-minute cut-off time exists for completion
of each block. From initial inspection of the data, it appears that
some students simply go too fast (GTF) and do not actually read the
items. Alternately, some students may complete the test at a slower
pace but miss the opportunity to even see certain items. Competition
participants are only given data from Block A and they have to predict
whether the student will complete Block B efficiently. Thus,
competition participants will predict students’ behavior in Block B
and indicate whether students will (i) not work on all items (either
due to going too fast or too slow), or (ii) complete the exam as
intended. Competition participants will use Block A to create models
of students’ behavior and then predict their behavior in Block B,
specifically which of the two groups each student will fall in. A
secondary question considers whether participants’ predictions for
Block B can function effectively for different amounts of Block A data
(e.g., first 10 minutes of data, first 20 minutes of data, all data).
Dr Heffernan, one of the organizers says, "The NAEP's main task is to
measure achievement over time and determine whether the nation is
doing better. But if students do not take the test seriously and are
just "gaming it" with no effort, the data is less valuable. This
competition could potentially shed light on how to predict who is not
taking the test seriously. If it does, the NAEP might be able to use
this information to help teachers intervene early."
How will the winning solution be determined? A held-out leaderboard
data set will be used to evaluate the performance of solutions, which
can be submitted on an ongoing basis. A second held-out final test set
will be used to evaluate the performance of each participant’s last
submission, at the end of the competition. Solutions will be evaluated
using a combination of AUC ROC and Cohen’s Kappa.
How will winners be announced? We will announce winners once two
things happen: (i) competition participants will announce whether they
are comfortable listing their names on the final scoreboard and (ii)
participants will submit their code and a short description. A
scoreboard will show participants’ names and final scores. No e-mail
addresses will be made public. Participants may opt-out from the final
scoreboard. Each participant will be personally informed of their
final submission scores, regardless of whether they opt-out or not.
How will participants discuss the results? The BDE 2019 challenge
committee will organize a workshop at EDM 2020 for challenge
participants to discuss the results of BDE Grand Challenge 2019. We
anticipate that sharing information about what worked and what did not
will help the Big Data in Education research community to make even
more progress in the coming years. In addition, we will create a
special issue of the Journal of Educational Data Mining to publicize
the results of this competition; competitors will be invited to submit
articles on their solutions and findings.
What was the previous competition? In 2017, 74 researchers
participated in the BDE Grand Challenge competition. Participants were
provided with huge streams of student actions (47 megabytes of
compressed data) from students using an online mathematics environment
decades ago when they were students in middle school. Grand Challenge
participants predicted which students would go on to take jobs in
science, technology, engineering or mathematics (STEM). The purpose of
this prior competition was to help middle schools develop better tools
to understand and predict students’ interest in STEM careers. Three
winners of that 2017 competition won prizes, discussed their results
at EDM2018 and published results in international proceedings. See
https://sites.google.com/view/assistmentsdatamining
A special issue of the Journal of Educational Data Mining is forthcoming.
NSF is funding the Grand Challenge Big Data Competition 2019 and has
previously funded the creation and maintenance of the longitudinal
data from 2017 middle school data as well as tracking which careers
these students entered. We look forward to interesting findings and
conversations from the second Grand Challenge.
