San Diego State University

This paper examines whether interaction data from Marmoset, an auto-grading system for programming assignments, can predict students' midterm and final exam grades. Features include per-task passing rate, testcase outcomes (public/release/hidden), time between submission and deadline, and number of submissions. The authors built decision-tree classification models (grouping students as good/satisfactory/poor performers) and linear-regression models (predicting numeric grades) from these features. The regression approach performed much better than classification, with a mean difference of 0.92 on a maximum grade of 120; when regression predictions were converted into performance categories for direct comparison with the classifiers, results also improved. The authors separately tested a regression model using only the submission-time-interval feature. Sections: Introduction, Related Work, Student Performance and Marmoset, Modeling Techniques, Features, Experiments and Evaluation.

Predicting student performance using data from an Auto-grading system

Prior work on predicting student performance has targeted GPA, pass/fail rates, and exam grades using various data sources. Notably, one prior study used data from the PASTA auto-grading system to predict pass/fail rates in a final exam, but that dataset also included data from outside the auto-grading system. In contrast, the authors of this paper restrict their feature set exclusively to data from the Marmoset auto-grading system, without supplementing it with external student records.

Related Work (Predicting student performance using data from an Auto-grading system)

In the University of Vaterloo final and midterm exam contribute for 60%-90% of the final grade.  Marmoset is auto-grading system where students can submit coding assignments. Submitted code is tested on 3 different testcases: public, release and hidden. 

Student Performance and Marmoset (Predicting student performance using data from an 
Auto-grading system)

The paper 'Predicting student performance using data from an Auto-grading system' evaluates two modeling techniques: decision trees and linear regression. Decision trees are built from the training set by choosing, at each split, the feature that yields the maximum possible information gain. For the regression approach, the authors use multivariate linear regression because the model includes several independent variables (features).

Modeling Techniques (Predicting student performance using data from an Auto-grading system)

The student-performance models use four categories of data from the Marmoset auto-grading system: (1) the per-task passing rate for each student's best submission, (2) public, release, and hidden test-case outcomes for the best submission, (3) the interval between submission time and the task deadline, and (4) the number of submissions. The dataset contains 28 tasks: the first 16 were due before the midterm exam, and the remaining 12 were due before the final exam.

Marmoset Auto-grading Features for Predicting Exam Performance

In the study of Marmoset auto-grading data, decision-tree classifiers predicted students as good, satisfactory, or poor performers, while linear-regression models predicted numeric exam grades. Classification models trained separately on the four feature categories performed poorly and assigned most students to the good-performance category; confusing poor performers with good performers was especially problematic. Regression performed substantially better, with a reported mean difference of 0.92 on a 120-point grading scale. Converting the predicted grades into performance categories also produced better results than direct classification. A separate regression model using only the interval between submission and deadline indicated a strong correlation between that feature and exam performance.

Classification and Regression Results for Marmoset-Based Exam Prediction

Programming students benefit from submitting assignments multiple times, receiving rapid feedback, and refining their code, but manually reviewing every attempt would place a substantial burden on instructors and teaching assistants. Marmoset automates this feedback process. The study asks whether data collected by Marmoset can predict students' midterm and final exam grades.

Learn Before

Related