It turns out that the Clinical Informatics community is really smart! And that presents a challenge: how do you tune exam questions for very experienced test-takers?
From the perspective of a question bank author, making a useful Qbank is like walking a tightrope: the questions can't be too easy OR too hard. Easy questions become boring, while hard questions are dauntlessly demotivating.
To stay balanced on the tightrope, we first assess each question's accuracy, which is the percentage of students who get the question correct. On any given question, the rule of thumb is between 20-80% of students should get the question correct (this allows some questions to be easier, and some harder). CI Prep Pro updates questions that are too easy (>80% get it right) and too hard (<20% get it right) to fit in this Goldilocks zone.
Another metric is also total bank accuracy, which is the percentage of questions from the whole test bank that a student gets right. There is no magic percentage to aim for, but most commercial question banks aim for students to get 70% of questions correct (you get a majority of questions right to stay motivated but still have plenty of opportunities for learning). CI Prep Pro aims for a total bank accuracy of 65%, as the Clinical Informatics board exam is generally accepted to be on the difficult side.
But accuracy is not sufficient on its own to assess a question. A question could have a low accuracy because it's a badly worded trick question. Or, a question may have a high accuracy because all the distractors (wrong answers) are implausible.
How do we find these poor-quality questions? We use a concept called skill discrimination: how well a question differentiates between students who do know the material (top test-takers) versus those who don't know the material (bottom test-takers).
If the top test-takers get a question right and the bottom test-takers get a question wrong, the question has good skill discrimination. In other words, the question successfully "sorts" the high performers from the low performers.
And that's how we tune questions! The entire CI Prep Pro Qbank is monitored and updated to achieve just the right accuracy and high skill discrimination.