analyst.andrewlengly.com/studies/ai-and-student-learning :: ai and student learning
# Do coursework grades show what students keep? 71.7% of 4,649 Open University students scored lower on the final exam than on coursework, and an open AI chatbot cut exam scores 17% in a field experiment. A data study by Andrew Leng Ly.
designed page ......... analyst.andrewlengly.com/studies/ai-and-student-learning
AI AND STUDENT LEARNING
analyst work .......... /machine/analyst data study ............ October 2026 question .............. Do coursework grades show what students remember, and what does an open AI chatbot do to the gap? headline finding ...... 71.7% of the 4,649 students scored lower on the final exam than on their coursework. Students who studied in the most weeks of the term scored 5.9 points higher on the exam than the least consistent quarter, even after accounting for coursework grade and total online activity. Both figures are stored Open University results, and the code behind them was not kept. my recommendation ..... The company should sell a hint-first tutor to schools, with weekly review sets built in. Schools should grade short weekly quizzes on old material and approve one guardrailed tutor. tools ................. JavaScript on Node for the derived rates · D3.js for the first dashboard · React and SVG for the charts on this page
THE QUESTION
the client ............ Two clients. One is a school or college deciding what to grade. The other is an edtech company that sells homework answers, with Chegg as the case. Chegg had 4.7 million subscribers in Q1 2024 and 3.2 million a year later. the decision they face The school has to choose what counts toward a grade and what to report beside GPA. The company has to choose what to sell now that 54% of US teens say they use an AI chatbot for schoolwork.
THE DATA, 8 TABLES
[01] oulad_summary.csv .............................................................. 6 rows /studies/ai-and-student-learning/data/oulad_summary.csv The size of the Open University sample and its totals. It covers 4,649 students in 2 STEM courses across 6 course runs, with the average gap between final exam and coursework and the share of students who scored below coursework on the exam. where it came from Computed from the Open University Learning Analytics Dataset (OULAD), which covers UK distance learning. why i chose it ... OULAD holds coursework scores and a cumulative final exam for the same students, along with their click logs. That lets a grade sit beside what the student kept. what it cannot tell us Be rerun. The raw OULAD files and the code that produced these rows are not in this repo, so the figures stand as stored results. The data was also collected before ChatGPT existed. [02] oulad_quartiles.csv ............................................................ 8 rows /studies/ai-and-student-learning/data/oulad_quartiles.csv Students split into 4 equal groups by weekly consistency and again by deadline cramming. Each row has the group’s coursework average, final exam average, exam pass rate and exam difference after controls. where it came from Computed from OULAD. Weekly consistency is the share of course weeks before the exam with any study activity. Deadline cramming is the share of a student’s clicks in the few days before each coursework deadline, divided by the share of course days those windows cover. why i chose it ... It puts a student’s study pattern beside both scores, the coursework average and the exam. what it cannot tell us Show cause. Students who study every week may differ in ways the data can’t see. The raw data and the code behind the group means and the adjusted differences are not in this repo. [03] oulad_gradeband.csv ........................................................... 20 rows /studies/ai-and-student-learning/data/oulad_gradeband.csv Final exam average for each consistency group inside 5 coursework grade bands, with the number of students in each cell. where it came from Computed from OULAD. why i chose it ... A reader can compare students with similar coursework grades without trusting the regression. what it cannot tell us Hold steady in the small cells. The smallest has 69 students. The raw data and the code that produced these rows are not in this repo. [04] oulad_model.csv ................................................................ 4 rows /studies/ai-and-student-learning/data/oulad_model.csv One regression of final exam score on weekly consistency, deadline cramming, coursework average and the log of total study clicks. It has course-run fixed effects and standard errors that do not assume equal variance across students. Each row holds one term with its estimate and p-value. where it came from Computed from OULAD. why i chose it ... It tests consistency and cramming in the same model, with coursework grade and total clicks held equal. what it cannot tell us Be refit or audited. The repo holds 4 coefficients with no code and no raw data behind them. P-values were stored to four decimal places, so 2 of them read 0.0. [05] wharton_rct.csv ................................................................ 2 rows /studies/ai-and-student-learning/data/wharton_rct.csv Practice and exam score changes against a control group for two AI arms, an open chatbot and a tutor with guardrails. The students were in high school math classes in Turkey (Bastani et al., PNAS 2025). where it came from Proceedings of the National Academy of Sciences. why i chose it ... It is a field experiment with a control group, and it tested students on an exam with the AI taken away. what it cannot tell us Speak for other subjects or school systems. The tutor arm’s exam change is stored as 0 because the paper reports it as about the same as control, so that cell is a reading of the paper’s wording. [06] pew_ai_use.csv ................................................................. 3 rows /studies/ai-and-student-learning/data/pew_ai_use.csv The share of US teens aged 13 to 17 who used ChatGPT for schoolwork in 2023 and 2024, and the share who use any AI chatbot for schoolwork in 2025. where it came from Pew Research Center survey releases. why i chose it ... It sizes how many students now do schoolwork with a chatbot at hand. what it cannot tell us Give a trend. Pew asked about ChatGPT in the first two surveys and about any AI chatbot in the third, so the last figure answers a broader question. [07] national_trends.csv ............................................................ 5 rows /studies/ai-and-student-learning/data/national_trends.csv Start and end values for US high school GPA and for ACT and NAEP grade 12 test scores. where it came from Two NAEP publications, the High School Transcript Study and the Nation’s Report Card, plus ACT’s grade inflation report. why i chose it ... It checks whether grades and test scores moved apart outside one university. what it cannot tell us Say anything about AI. The GPA series end in 2019 and 2021, before ChatGPT, and the 2024 NAEP seniors had a freshman year disrupted by the pandemic. Each series has two points, so the path between them is unknown. [08] edtech_signals.csv ............................................................. 4 rows /studies/ai-and-student-learning/data/edtech_signals.csv Chegg’s subscriber counts for Q1 2024 and Q1 2025 with the year-over-year change, and the share of students with Khanmigo access who use it regularly. where it came from Chegg’s quarterly results and a Khan Academy statement, both taken from news coverage. why i chose it ... It gives the company track two market readings, one for a paid answer service and one for a guardrailed tutor. what it cannot tell us Explain why subscribers left. The Khanmigo figure is Khan Academy’s own statement, and the table holds no definition of regular use.
THE PROCESS
[01] Sample
The Open University work used the 2 OULAD courses with a scored final exam, across 6
course runs. Students with almost no platform activity were removed, which left 4,649.
The cutoff for that filter was not recorded.
[02] Measures
Weekly consistency is the share of course weeks before the exam with any study
activity. Deadline cramming compares a student’s clicks in the few days before each
coursework deadline with the share of course days those windows cover. Each measure
splits the students into 4 equal groups.
[03] Model
One regression holds consistency, cramming, coursework average and the log of total
study clicks, with course-run fixed effects and standard errors that do not assume
equal variance. R² is 0.485. The raw files and the code for this step were not saved,
so derive.mjs reads the stored coefficients and cannot refit them.
[04] Pivot
oulad_gradeband.csv is long, with one row per grade band per consistency group. I
pivoted it to one row per band and added the gap between the most and least consistent
groups.
[05] Types
Every value in oulad_summary.csv is stored as a decimal, so the student count reads
4649.0. P-values are stored to four places, and 2 of them read 0.0. The code turns the
counts into whole numbers and prints those p-values as <0.001.
[06] Check
The stored tables cannot be rerun, so I checked them against each other. Group sizes
add up to 4,649 in both splits and in the grade band table. The student-weighted gap of
the groups lands within 0.03 points of the stored average, and each group’s exam
average rebuilt from its band rows lands within 0.05 points.
[07] Sources
The other 4 tables were compiled from published reports. They carry no source URL and
no retrieval date, and only national_trends.csv names a source on each row. Chegg
reported a 31% fall in subscribers. The two rounded counts give 31.9%, and the rounding
allows anything from 30.1% to 33.7%.
# One grade band before and after the pivot.
- Before, 4 rows of oulad_gradeband.csv
- After, one row per band
The 80 to 90 band holds 1,247 students. The spread is the most consistent group’s exam
average minus the least consistent group’s.
# The code for the key steps
- The pivot from long rows to one row per grade band.
derive.mjs ........ /studies/ai-and-student-learning/derive.mjs
- The check of the stored average gap against the group rows.
derive.mjs ........ /studies/ai-and-student-learning/derive.mjs
# Dead ends
cramming ............ Split on cramming alone, the heaviest crammers scored 2.6 points
lower after controls. With consistency in the same model, one
standard deviation more cramming goes with 0.51 fewer exam points,
and the p-value is 0.076. That is above 0.05, so no option here
rests on cramming.
the lowest grade band
In the “Under 60” coursework band the pattern is gone. The most
consistent group averaged 45.5 on the exam and the least consistent
group averaged 45.9. The gap favors the consistent group in 4 of 5
bands.
ai in the national series
The national table cannot carry a claim about AI. In the federal
transcript study GPA rose from 2.68 in 1990 to 3.11 in 2019, and
that series ends before ChatGPT. NAEP’s 2024 seniors also had a
pandemic freshman year.THE 3 FINDINGS THAT CHANGE THE DECISION
[01] 71.7% · Students whose final exam came in below their coursework average
71.7% of the 4,649 students scored lower on the final exam than on their coursework,
and the average gap was 9.7 points.
dataset .......... oulad_summary.csv
query ............ derive.mjs, summary()
# Every consistency group scored at least 8.6 points lower on the exam than on
coursework
Coursework average minus final exam average for 4 equal groups of students. Q1 studied
in the fewest weeks of the term and Q4 in the most. These are stored results from
OULAD.
Consistency group ........................... Points below coursework on the final exam
Q1 least ......................................................................... 11.3
Students: 1,165 · Coursework average: 68.6 · Final exam average: 57.2
Q2 ................................................................................ 9.6
Students: 1,160 · Coursework average: 72.6 · Final exam average: 63.0
Q3 ................................................................................ 9.4
Students: 1,160 · Coursework average: 77.2 · Final exam average: 67.7
Q4 most ........................................................................... 8.6
Students: 1,164 · Coursework average: 81.3 · Final exam average: 72.7
source ........... oulad_quartiles.csv, oulad_summary.csv, derive.mjs, groups()
[02] 5.9 points · Exam edge for the most consistent group over the least, after accounting
for grade and activity
After accounting for coursework grade and total online activity within each course, the
most consistent quarter of students scored 5.9 points higher on the final exam than the
least consistent quarter.
dataset .......... oulad_quartiles.csv
query ............ derive.mjs, groups()
# The most consistent group scored 5.9 points higher on the exam, after accounting for
grade and activity
Each bar compares a group with the least consistent group, which is the baseline and
sits at zero. Before accounting for grade and activity, the two end groups were 15.5
exam points apart. In the full model, each standard deviation of extra consistency goes
with 2.17 more exam points, and the result is very unlikely to be chance (p <0.001).
The code for these results was not kept.
Consistency group ........ Exam points above the least consistent group, after controls
Q1 least .......................................................................... 0.0
Students: 1,165 · Final exam average: 57.2 · Exam pass rate: 80.6%
Q2 ................................................................................ 2.9
Students: 1,160 · Final exam average: 63.0 · Exam pass rate: 87.3%
Q3 ................................................................................ 4.0
Students: 1,160 · Final exam average: 67.7 · Exam pass rate: 90.9%
Q4 most ........................................................................... 5.9
Students: 1,164 · Final exam average: 72.7 · Exam pass rate: 94.5%
source ........... oulad_quartiles.csv, oulad_model.csv, derive.mjs, groups()
[03] 17% lower · Exam score against the control group, students given an open chatbot
In a Wharton field experiment, students given an open chatbot scored 48% higher than
the control group on practice problems and 17% lower on the exam they took without it.
dataset .......... wharton_rct.csv
query ............ derive.mjs, rctArms()
# Students with an open chatbot scored 17% lower on the exam, and students with the
hint-first tutor matched the control group
Exam score below the control group, in percent, for high school math students in Turkey
(Bastani et al., PNAS 2025). The tutor gave hints and held back full answers. Its exam
change is stored as 0 because the paper reports it as about the same as control.
Arm ...................................................... Exam score below control (%)
GPT Base (open chatbot) ........................................................... 17%
Practice score against control: +48% · Exam score against control: -17%
GPT Tutor (guardrails) ............................................................. 0%
Practice score against control: +127% · Exam score against control: 0%
source ........... wharton_rct.csv, derive.mjs, rctArms()MY RECOMMENDATION
The company should sell a hint-first tutor to schools. Schools should grade weekly quizzes and approve one tutor. [A1] Guardrailed tutor as the default Company track. Hints first, with full answers only after the student shows work. evidence ......... Finding 3. Students given the hint-first tutor gained 127% on practice and scored about the same as the control group on the exam. main risk ........ Khanmigo’s low regular use suggests students drift back to open chatbots. 15% of students with access use it regularly. [A2] Sell to schools and colleges Company track. Instructors assign review sets and see exam readiness for each student. evidence ......... Finding 2. Consistency gives the company a measurable outcome to sell. Each standard deviation of extra consistency goes with 2.17 more exam points. main risk ........ Schools are slow to buy, and they’re a new kind of customer for the company. [B1] Grade weekly low-stakes quizzes School track. Short weekly quizzes that mix in old material and count toward the grade. evidence ......... Finding 2. After accounting for grade and activity, the exam edge grows with each step up in consistency and reaches 5.9 points for the top group. main risk ........ More grading, which auto-graded quizzes cut down. [B2] Approve one guardrailed AI tutor School track. Approve one tutor with guardrails and set rules for open chatbots on coursework. evidence ......... Finding 3. Students with an open chatbot gained 48% on practice and lost 17% on the exam. Students with the hint-first tutor gained 127% on practice and held level with the control group. main risk ........ Rules are hard to enforce at home. - The hint-first tutor is the one company option a trial has tested. In finding 3 it gained on practice and held level with control on the exam. - Selling through schools answers the tutor’s main risk. Students drift away from a tutor they picked themselves, and an instructor who assigns review sets keeps them coming back. - Weekly quizzes go straight at finding 2, where steady weekly study went with higher exam scores after controls. - One approved tutor with clear rules for coursework answers finding 3, where the open chatbot cost students 17% on the exam.
LIMITS AND NEXT STEPS
# What the data cannot support
- The four oulad tables are stored results. The raw Open University files and the code for
the group means and the regression are not in this repo, so this page cannot rerun them.
derive.mjs only checks that the stored tables agree with each other.
- The Open University data was collected before ChatGPT existed. It shows the gap between
coursework and exam was there before AI, and it holds nothing on AI use.
- The regression shows association. Students who study steadily may differ in ways the
data can’t see.
- A lower score on the exam than on coursework is no proof of forgetting. The two are
different assessments, and the tables hold no retest of the same material.
- Total study clicks stand in for study time. In the model more clicks go with a lower
exam score at the same coursework grade, 1.78 points per log unit, so I read clicks as
activity and make no claim about hours.
- The AI evidence is one experiment in high school math in Turkey. Pew’s 54% answers a
broader question than its 13% and 26%, so the Pew figures are separate readings.
# What I would test with more access
- With the raw OULAD files I would rerun the group means and the regression and save the
code here, so a reader can refit the model.
- With a school’s gradebook I would run a retention check on old material and compare the
retest scores with GPA.
- With an edtech company’s logs I would test a weekly review mode against a holdout group
and measure final exam scores.
- I would add a source URL and a retrieval date to every compiled row.TL;DR
studied ............... How well coursework grades predict final exams, and what an open AI chatbot does to that gap. found ................. 71.7% of the 4,649 students scored lower on the final exam than on their coursework. # by the numbers 71.7% ............... Exam below coursework average 5.9 points .......... Exam edge for the most consistent studiers 17% lower ........... Exam score with an open chatbot the catch ............. The Open University figures are stored results, and the code that produced them was not kept. next .................. With the raw OULAD files, rerun the group means and the regression and save the code here.
# end of /machine/ai-and-student-learning. andrew.leng.ly@gmail.com · leave machine mode