Michelle Meadows researchED Maths and Science

TL;DR

Experts struggle to predict exam difficulty accurately due to human judgment biases, impacting grade boundaries.

Transcript

being able to predict how hard students will find exam questions it's something of a holy grail for assessment researchers and experts are generally notoriously bad at it students find all sorts of things difficult that we don't anticipate and a thought being heir to describes some of their the biases in judgments about this task might be helpful a... Read More

Key Insights

• ❓ Human judgment biases make predicting exam difficulty challenging.
• 🎚️ Senior examiners aim for consistent exam difficulty levels.
• 🆘 Adjusting grade boundaries helps compensate for unexpected exam difficulty.
• ❓ Absolute judgments are weaker than relative judgments in assessing exam difficulty.
• 🎱 Pooled judgments from multiple examiners tend to be more accurate.
• 🌱 Accreditation panels scrutinize assessment plans to assess exam difficulty.
• 👨‍🔬 Research data collection can enhance predictions of exam difficulty levels.

Questions & Answers

Q: Why do experts struggle to predict exam difficulty accurately?

Experts, like all humans, are prone to judgment biases, making it difficult to anticipate how students will find exam questions. This leads to challenges in setting consistent difficulty levels.

Q: How do senior examiners mitigate the variability in exam difficulty?

Senior examiners aim to construct exams at a consistent difficulty level from year to year, but human judgment biases can still lead to unexpected variations. Grade boundaries are adjusted to level the playing field for students.

Q: What research supports the idea of absolute vs. relative judgments in assessing exam difficulty?

Studies suggest that absolute judgments are weaker than relative judgments in predicting exam difficulty. Pooled judgments, where multiple examiners assess, tend to be more accurate than individual judgments.

Q: How do accreditation panels evaluate exam difficulty when assessing specifications?

Accreditation panels scrutinize assessment plans and sample materials to gauge the likely difficulty of exams. Research data is collected to better estimate the appropriateness of the exam levels before students take them.

Summary & Key Takeaways

• Predicting exam difficulty accurately is challenging due to human judgment biases.

• Senior examiners construct exams aiming for consistent difficulty levels.

• Adjusting grade boundaries compensates for unexpected variance in exam difficulty.