What Course Grades Don’t Capture About Step 1 Readiness

A new peer-reviewed study matched ScholarRx usage against CBSE scores and on-time Step 1 passage for 200 students in two consecutive cohorts, each taught under a different curriculum. Course grades and practice-question completion each predicted both outcomes. They weren’t related to each other, and that’s the part worth your attention.

Course grades are the best signal most schools have for Step 1 readiness. They’re also the signal your curriculum committee already watches most closely.

The harder question is what grades miss. A student can look fine in the gradebook and still be heading into the CBSE with less practice in the question format than their classmates. If nothing else in your data points that out, the first sign is often the score itself.

A study published in BMC Medical Education this summer asked whether data many schools already hold, practice-question completion, carries information that course grades don’t.

The study

Emma Sherman, Jack Stomberger and Kirsten Porter-Stransky, at the University of South Carolina School of Medicine Greenville, looked at two consecutive cohorts who completed the pre-clerkship biomedical sciences curriculum, one under a traditional discipline-based curriculum and one under a new integrated organ system curriculum, 200 students in total.

The school paid for access, so cost wasn’t deciding who used what. Students chose freely how much to use and which parts. The researchers then compared usage of Qmax practice questions, Rx Bricks, Flash Facts and Express Videos against each student’s highest CBSE equated percent correct score, and against whether they took and passed Step 1 before clerkships began.

They also included each student’s average biomedical science course grade in the models. Course performance is already known to track with exam outcomes, so accounting for it let the researchers ask a harder question: was question completion still associated with CBSE and Step 1 results once differences in course performance were taken into account?

The study was supported in part by a MERIC grant from ScholarRx, and the senior author previously received a ScholarRx Brick Builder Grant. ScholarRx provided the school’s usage data. As the paper discloses, ScholarRx had no role in study design, analysis, interpretation, manuscript preparation, or the decision to submit.

Two independent signals

Two variables came out significant, in both cohorts.

The first was the student’s average biomedical science course grade. Performance in the school’s own curriculum was the strongest signal in the data.

The second was the number of Qmax practice questions completed. Students who worked through more questions scored higher on the CBSE, and were more likely to take and pass Step 1 before clerkships began. For the 659 questions that made up the integrated cohort’s average, the models put the increase in odds of on-time passage at 274% in the traditional cohort and 622% in the integrated one. These are changes in the odds of on-time passage, not percentage-point changes in the probability of passing.

The part worth taking to your curriculum committee

Those two signals weren’t related to each other.

Question completion wasn’t simply a marker of the strong students. Strong course performance didn’t necessarily mean high question-bank engagement, and the reverse held too. Each one contributed something the other didn’t.

So a student can be doing perfectly well in your courses and still be well behind their classmates on practice volume. Your gradebook may not tell you that. In this study, course grades and question completion weren’t correlated in either cohort.

What this study doesn’t show

It’s correlational, not causal. It’s one school, and the two cohorts had different lengths of access, so they couldn’t be combined into a single model. The researchers knew how many questions students completed but not how many they answered correctly.

Students also chose for themselves how much of each resource to use, which means unmeasured traits, motivation, study habits, baseline preparation, could be driving both the usage and the outcomes. And the study couldn’t account for the personal reasons that lead some students to delay Step 1.

The authors are also explicit that similar results might appear with other good question banks, because the authors attribute the value of question practice to retrieval practice, which isn’t specific to any one product.

What the study does support is narrow and useful: across two different curricula at the same school, sustained practice-question work was associated with CBSE performance and with on-time Step 1 passage, independently of how students were doing in their courses.

Where to start

Put your usage data beside your grade data and look for students whose question engagement is low relative to their course performance. The study doesn’t set a threshold for when to step in, but it suggests those two reports are telling you different things.

Then pick one unit and assign a question set to it, rather than leaving students to find it themselves.

References

1. Sherman EC, Stomberger JI, Porter-Stransky KA. Association between ScholarRx Qmax question bank engagement and NBME CBSE and USMLE Step 1 performance. BMC Medical Education. 2026;26:1453. doi:10.1186/s12909-026-10013-3
2. Lidsky H, Kansupada J, Saran T, et al. Impact of commercial third-party resource integration on preclinical student resource use, preferences, and exam performance. Medical Science Educator. 2025;35(5):2441-2449. doi:10.1007/s40670-025-02456-y

The full paper is open access. The methods and limitations sections are worth your time if this is a decision you’re making for your school.

Faculty reporting in Qmax shows question completion at the individual and cohort level during the block, not only after it, and question sets can be assigned to a unit rather than left to students to find. Rx Bricks and Bricks Create exist so that foundational content lives inside your course rather than beside it. And the MERIC grant program funds independent research by medical educators, including research whose answers we can’t predict.

Start a conversation about your school

If you’re looking at your own usage and outcome data and want to talk through what it’s telling you, our Academic Solutions team works with faculty and curriculum leaders on exactly this: where the resources sit in a course, what gets assigned rather than offered, and what you can see during the block.

Contact Academic Solutions to set up a conversation, or ask for a walkthrough of what faculty reporting shows.