Skip to content

analyst.andrewlengly.com/studies/ai-and-student-learning :: ai and student learning

# Do coursework grades show what students keep? 71.7% of 4,649 Open University students scored lower on the final exam than on coursework, and an open AI chatbot cut exam scores 17% in a field experiment. A data study by Andrew Leng Ly.

designed page ......... analyst.andrewlengly.com/studies/ai-and-student-learning

AI AND STUDENT LEARNING

analyst work .......... /machine/analyst
data study ............ October 2026

question .............. Do coursework grades show what students remember, and what does an
                        open AI chatbot do to the gap?
headline finding ...... 71.7% of the 4,649 students scored lower on the final exam than on
                        their coursework. Students who studied in the most weeks of the term
                        scored 5.9 points higher on the exam than the least consistent
                        quarter, even after accounting for coursework grade and total online
                        activity. Both figures are stored Open University results, and the
                        code behind them was not kept.
my recommendation ..... The company should sell a hint-first tutor to schools, with weekly
                        review sets built in. Schools should grade short weekly quizzes on
                        old material and approve one guardrailed tutor.
tools ................. JavaScript on Node for the derived rates · D3.js for the first
                        dashboard · React and SVG for the charts on this page

THE QUESTION

the client ............ Two clients. One is a school or college deciding what to grade. The
                        other is an edtech company that sells homework answers, with Chegg
                        as the case. Chegg had 4.7 million subscribers in Q1 2024 and 3.2
                        million a year later.
the decision they face
                        The school has to choose what counts toward a grade and what to
                        report beside GPA. The company has to choose what to sell now that
                        54% of US teens say they use an AI chatbot for schoolwork.

THE DATA, 8 TABLES

[01] oulad_summary.csv .............................................................. 6 rows
     /studies/ai-and-student-learning/data/oulad_summary.csv
     The size of the Open University sample and its totals. It covers 4,649 students in 2
     STEM courses across 6 course runs, with the average gap between final exam and
     coursework and the share of students who scored below coursework on the exam.
     where it came from
                        Computed from the Open University Learning Analytics Dataset
                        (OULAD), which covers UK distance learning.
     why i chose it ... OULAD holds coursework scores and a cumulative final exam for the
                        same students, along with their click logs. That lets a grade sit
                        beside what the student kept.
     what it cannot tell us
                        Be rerun. The raw OULAD files and the code that produced these rows
                        are not in this repo, so the figures stand as stored results. The
                        data was also collected before ChatGPT existed.

[02] oulad_quartiles.csv ............................................................ 8 rows
     /studies/ai-and-student-learning/data/oulad_quartiles.csv
     Students split into 4 equal groups by weekly consistency and again by deadline
     cramming. Each row has the group’s coursework average, final exam average, exam pass
     rate and exam difference after controls.
     where it came from
                        Computed from OULAD. Weekly consistency is the share of course weeks
                        before the exam with any study activity. Deadline cramming is the
                        share of a student’s clicks in the few days before each coursework
                        deadline, divided by the share of course days those windows cover.
     why i chose it ... It puts a student’s study pattern beside both scores, the coursework
                        average and the exam.
     what it cannot tell us
                        Show cause. Students who study every week may differ in ways the
                        data can’t see. The raw data and the code behind the group means and
                        the adjusted differences are not in this repo.

[03] oulad_gradeband.csv ........................................................... 20 rows
     /studies/ai-and-student-learning/data/oulad_gradeband.csv
     Final exam average for each consistency group inside 5 coursework grade bands, with the
     number of students in each cell.
     where it came from
                        Computed from OULAD.
     why i chose it ... A reader can compare students with similar coursework grades without
                        trusting the regression.
     what it cannot tell us
                        Hold steady in the small cells. The smallest has 69 students. The
                        raw data and the code that produced these rows are not in this repo.

[04] oulad_model.csv ................................................................ 4 rows
     /studies/ai-and-student-learning/data/oulad_model.csv
     One regression of final exam score on weekly consistency, deadline cramming, coursework
     average and the log of total study clicks. It has course-run fixed effects and standard
     errors that do not assume equal variance across students. Each row holds one term with
     its estimate and p-value.
     where it came from
                        Computed from OULAD.
     why i chose it ... It tests consistency and cramming in the same model, with coursework
                        grade and total clicks held equal.
     what it cannot tell us
                        Be refit or audited. The repo holds 4 coefficients with no code and
                        no raw data behind them. P-values were stored to four decimal
                        places, so 2 of them read 0.0.

[05] wharton_rct.csv ................................................................ 2 rows
     /studies/ai-and-student-learning/data/wharton_rct.csv
     Practice and exam score changes against a control group for two AI arms, an open
     chatbot and a tutor with guardrails. The students were in high school math classes in
     Turkey (Bastani et al., PNAS 2025).
     where it came from
                        Proceedings of the National Academy of Sciences.
     why i chose it ... It is a field experiment with a control group, and it tested
                        students on an exam with the AI taken away.
     what it cannot tell us
                        Speak for other subjects or school systems. The tutor arm’s exam
                        change is stored as 0 because the paper reports it as about the same
                        as control, so that cell is a reading of the paper’s wording.

[06] pew_ai_use.csv ................................................................. 3 rows
     /studies/ai-and-student-learning/data/pew_ai_use.csv
     The share of US teens aged 13 to 17 who used ChatGPT for schoolwork in 2023 and 2024,
     and the share who use any AI chatbot for schoolwork in 2025.
     where it came from
                        Pew Research Center survey releases.
     why i chose it ... It sizes how many students now do schoolwork with a chatbot at hand.
     what it cannot tell us
                        Give a trend. Pew asked about ChatGPT in the first two surveys and
                        about any AI chatbot in the third, so the last figure answers a
                        broader question.

[07] national_trends.csv ............................................................ 5 rows
     /studies/ai-and-student-learning/data/national_trends.csv
     Start and end values for US high school GPA and for ACT and NAEP grade 12 test scores.
     where it came from
                        Two NAEP publications, the High School Transcript Study and the
                        Nation’s Report Card, plus ACT’s grade inflation report.
     why i chose it ... It checks whether grades and test scores moved apart outside one
                        university.
     what it cannot tell us
                        Say anything about AI. The GPA series end in 2019 and 2021, before
                        ChatGPT, and the 2024 NAEP seniors had a freshman year disrupted by
                        the pandemic. Each series has two points, so the path between them
                        is unknown.

[08] edtech_signals.csv ............................................................. 4 rows
     /studies/ai-and-student-learning/data/edtech_signals.csv
     Chegg’s subscriber counts for Q1 2024 and Q1 2025 with the year-over-year change, and
     the share of students with Khanmigo access who use it regularly.
     where it came from
                        Chegg’s quarterly results and a Khan Academy statement, both taken
                        from news coverage.
     why i chose it ... It gives the company track two market readings, one for a paid
                        answer service and one for a guardrailed tutor.
     what it cannot tell us
                        Explain why subscribers left. The Khanmigo figure is Khan Academy’s
                        own statement, and the table holds no definition of regular use.

THE PROCESS

[01] Sample
     The Open University work used the 2 OULAD courses with a scored final exam, across 6
     course runs. Students with almost no platform activity were removed, which left 4,649.
     The cutoff for that filter was not recorded.

[02] Measures
     Weekly consistency is the share of course weeks before the exam with any study
     activity. Deadline cramming compares a student’s clicks in the few days before each
     coursework deadline with the share of course days those windows cover. Each measure
     splits the students into 4 equal groups.

[03] Model
     One regression holds consistency, cramming, coursework average and the log of total
     study clicks, with course-run fixed effects and standard errors that do not assume
     equal variance. R² is 0.485. The raw files and the code for this step were not saved,
     so derive.mjs reads the stored coefficients and cannot refit them.

[04] Pivot
     oulad_gradeband.csv is long, with one row per grade band per consistency group. I
     pivoted it to one row per band and added the gap between the most and least consistent
     groups.

[05] Types
     Every value in oulad_summary.csv is stored as a decimal, so the student count reads
     4649.0. P-values are stored to four places, and 2 of them read 0.0. The code turns the
     counts into whole numbers and prints those p-values as <0.001.

[06] Check
     The stored tables cannot be rerun, so I checked them against each other. Group sizes
     add up to 4,649 in both splits and in the grade band table. The student-weighted gap of
     the groups lands within 0.03 points of the stored average, and each group’s exam
     average rebuilt from its band rows lands within 0.05 points.

[07] Sources
     The other 4 tables were compiled from published reports. They carry no source URL and
     no retrieval date, and only national_trends.csv names a source on each row. Chegg
     reported a 31% fall in subscribers. The two rounded counts give 31.9%, and the rounding
     allows anything from 30.1% to 33.7%.

  # One grade band before and after the pivot.
  - Before, 4 rows of oulad_gradeband.csv
  - After, one row per band
  The 80 to 90 band holds 1,247 students. The spread is the most consistent group’s exam
  average minus the least consistent group’s.

  # The code for the key steps
  - The pivot from long rows to one row per grade band.
    derive.mjs ........ /studies/ai-and-student-learning/derive.mjs
  - The check of the stored average gap against the group rows.
    derive.mjs ........ /studies/ai-and-student-learning/derive.mjs

  # Dead ends
  cramming ............ Split on cramming alone, the heaviest crammers scored 2.6 points
                        lower after controls. With consistency in the same model, one
                        standard deviation more cramming goes with 0.51 fewer exam points,
                        and the p-value is 0.076. That is above 0.05, so no option here
                        rests on cramming.
  the lowest grade band
                        In the “Under 60” coursework band the pattern is gone. The most
                        consistent group averaged 45.5 on the exam and the least consistent
                        group averaged 45.9. The gap favors the consistent group in 4 of 5
                        bands.
  ai in the national series
                        The national table cannot carry a claim about AI. In the federal
                        transcript study GPA rose from 2.68 in 1990 to 3.11 in 2019, and
                        that series ends before ChatGPT. NAEP’s 2024 seniors also had a
                        pandemic freshman year.

THE 3 FINDINGS THAT CHANGE THE DECISION

[01] 71.7% · Students whose final exam came in below their coursework average
     71.7% of the 4,649 students scored lower on the final exam than on their coursework,
     and the average gap was 9.7 points.
     dataset .......... oulad_summary.csv
     query ............ derive.mjs, summary()

     # Every consistency group scored at least 8.6 points lower on the exam than on
       coursework
     Coursework average minus final exam average for 4 equal groups of students. Q1 studied
     in the fewest weeks of the term and Q4 in the most. These are stored results from
     OULAD.
     Consistency group ........................... Points below coursework on the final exam
     Q1 least ......................................................................... 11.3
       Students: 1,165 · Coursework average: 68.6 · Final exam average: 57.2
     Q2 ................................................................................ 9.6
       Students: 1,160 · Coursework average: 72.6 · Final exam average: 63.0
     Q3 ................................................................................ 9.4
       Students: 1,160 · Coursework average: 77.2 · Final exam average: 67.7
     Q4 most ........................................................................... 8.6
       Students: 1,164 · Coursework average: 81.3 · Final exam average: 72.7
     source ........... oulad_quartiles.csv, oulad_summary.csv, derive.mjs, groups()

[02] 5.9 points · Exam edge for the most consistent group over the least, after accounting
     for grade and activity
     After accounting for coursework grade and total online activity within each course, the
     most consistent quarter of students scored 5.9 points higher on the final exam than the
     least consistent quarter.
     dataset .......... oulad_quartiles.csv
     query ............ derive.mjs, groups()

     # The most consistent group scored 5.9 points higher on the exam, after accounting for
       grade and activity
     Each bar compares a group with the least consistent group, which is the baseline and
     sits at zero. Before accounting for grade and activity, the two end groups were 15.5
     exam points apart. In the full model, each standard deviation of extra consistency goes
     with 2.17 more exam points, and the result is very unlikely to be chance (p <0.001).
     The code for these results was not kept.
     Consistency group ........ Exam points above the least consistent group, after controls
     Q1 least .......................................................................... 0.0
       Students: 1,165 · Final exam average: 57.2 · Exam pass rate: 80.6%
     Q2 ................................................................................ 2.9
       Students: 1,160 · Final exam average: 63.0 · Exam pass rate: 87.3%
     Q3 ................................................................................ 4.0
       Students: 1,160 · Final exam average: 67.7 · Exam pass rate: 90.9%
     Q4 most ........................................................................... 5.9
       Students: 1,164 · Final exam average: 72.7 · Exam pass rate: 94.5%
     source ........... oulad_quartiles.csv, oulad_model.csv, derive.mjs, groups()

[03] 17% lower · Exam score against the control group, students given an open chatbot
     In a Wharton field experiment, students given an open chatbot scored 48% higher than
     the control group on practice problems and 17% lower on the exam they took without it.
     dataset .......... wharton_rct.csv
     query ............ derive.mjs, rctArms()

     # Students with an open chatbot scored 17% lower on the exam, and students with the
       hint-first tutor matched the control group
     Exam score below the control group, in percent, for high school math students in Turkey
     (Bastani et al., PNAS 2025). The tutor gave hints and held back full answers. Its exam
     change is stored as 0 because the paper reports it as about the same as control.
     Arm ...................................................... Exam score below control (%)
     GPT Base (open chatbot) ........................................................... 17%
       Practice score against control: +48% · Exam score against control: -17%
     GPT Tutor (guardrails) ............................................................. 0%
       Practice score against control: +127% · Exam score against control: 0%
     source ........... wharton_rct.csv, derive.mjs, rctArms()

MY RECOMMENDATION

  The company should sell a hint-first tutor to schools. Schools should grade weekly quizzes
  and approve one tutor.

[A1] Guardrailed tutor as the default
     Company track. Hints first, with full answers only after the student shows work.
     evidence ......... Finding 3. Students given the hint-first tutor gained 127% on
                        practice and scored about the same as the control group on the exam.
     main risk ........ Khanmigo’s low regular use suggests students drift back to open
                        chatbots. 15% of students with access use it regularly.

[A2] Sell to schools and colleges
     Company track. Instructors assign review sets and see exam readiness for each student.
     evidence ......... Finding 2. Consistency gives the company a measurable outcome to
                        sell. Each standard deviation of extra consistency goes with 2.17
                        more exam points.
     main risk ........ Schools are slow to buy, and they’re a new kind of customer for the
                        company.

[B1] Grade weekly low-stakes quizzes
     School track. Short weekly quizzes that mix in old material and count toward the grade.
     evidence ......... Finding 2. After accounting for grade and activity, the exam edge
                        grows with each step up in consistency and reaches 5.9 points for
                        the top group.
     main risk ........ More grading, which auto-graded quizzes cut down.

[B2] Approve one guardrailed AI tutor
     School track. Approve one tutor with guardrails and set rules for open chatbots on
     coursework.
     evidence ......... Finding 3. Students with an open chatbot gained 48% on practice and
                        lost 17% on the exam. Students with the hint-first tutor gained 127%
                        on practice and held level with the control group.
     main risk ........ Rules are hard to enforce at home.

  - The hint-first tutor is the one company option a trial has tested. In finding 3 it
    gained on practice and held level with control on the exam.
  - Selling through schools answers the tutor’s main risk. Students drift away from a tutor
    they picked themselves, and an instructor who assigns review sets keeps them coming
    back.
  - Weekly quizzes go straight at finding 2, where steady weekly study went with higher exam
    scores after controls.
  - One approved tutor with clear rules for coursework answers finding 3, where the open
    chatbot cost students 17% on the exam.

LIMITS AND NEXT STEPS

  # What the data cannot support
  - The four oulad tables are stored results. The raw Open University files and the code for
    the group means and the regression are not in this repo, so this page cannot rerun them.
    derive.mjs only checks that the stored tables agree with each other.
  - The Open University data was collected before ChatGPT existed. It shows the gap between
    coursework and exam was there before AI, and it holds nothing on AI use.
  - The regression shows association. Students who study steadily may differ in ways the
    data can’t see.
  - A lower score on the exam than on coursework is no proof of forgetting. The two are
    different assessments, and the tables hold no retest of the same material.
  - Total study clicks stand in for study time. In the model more clicks go with a lower
    exam score at the same coursework grade, 1.78 points per log unit, so I read clicks as
    activity and make no claim about hours.
  - The AI evidence is one experiment in high school math in Turkey. Pew’s 54% answers a
    broader question than its 13% and 26%, so the Pew figures are separate readings.

  # What I would test with more access
  - With the raw OULAD files I would rerun the group means and the regression and save the
    code here, so a reader can refit the model.
  - With a school’s gradebook I would run a retention check on old material and compare the
    retest scores with GPA.
  - With an edtech company’s logs I would test a weekly review mode against a holdout group
    and measure final exam scores.
  - I would add a source URL and a retrieval date to every compiled row.

TL;DR

studied ............... How well coursework grades predict final exams, and what an open AI
                        chatbot does to that gap.
found ................. 71.7% of the 4,649 students scored lower on the final exam than on
                        their coursework.
  # by the numbers
  71.7% ............... Exam below coursework average
  5.9 points .......... Exam edge for the most consistent studiers
  17% lower ........... Exam score with an open chatbot
the catch ............. The Open University figures are stored results, and the code that
                        produced them was not kept.
next .................. With the raw OULAD files, rerun the group means and the regression
                        and save the code here.

# end of /machine/ai-and-student-learning. andrew.leng.ly@gmail.com · leave machine mode