Finding Great Participants for Your User Study: The Role of Difficulty and Success Rates in Optimizing Learning

Finding Great Participants for Your User Study: The Role of Difficulty and Success Rates in Optimizing Learning

Introduction:
When it comes to conducting a user study, one of the key challenges is finding the right participants who will provide valuable insights and feedback. In this article, we will explore two seemingly unrelated concepts - the process of finding great participants for a user study and the 85% rule for learning. By examining these ideas together, we can uncover valuable insights that can help us optimize the learning process and improve the quality of our user studies.

Step 1: Write a screener
To ensure that you find participants who align with your study objectives, it is crucial to develop a well-crafted screener. This screening questionnaire should include relevant questions that help you identify individuals who possess the necessary qualifications, characteristics, and demographics. By carefully designing your screener, you can filter out participants who may not be suitable for your study, saving time and resources.

The 85% Rule for Learning:
Learning, whether it is for humans or machines, appears to be optimized when we succeed approximately 85% of the time. This finding is consistent across various learning algorithms and educational theories. Barak Rosenshine's study on successful classrooms found that an 80% success rate was optimal. Similarly, stochastic gradient-descent based learning algorithms indicate that a training accuracy of around 85% leads to optimal results.

The Sweet Spot of Difficulty:
Lev Vygotsky's zone of proximal development theory suggests that tasks slightly beyond our current abilities, but manageable with assistance, maximize learning. This implies that learning is most effective when the difficulty level of the task is neither too easy nor too hard. When faced with a task that requires inference or problem-solving, individuals with higher background knowledge tend to perform better than those with less background knowledge. In such cases, encountering some difficulty can be beneficial as it prompts the retrieval of existing knowledge and the creation of a mental model.

Deliberate Practice and Learned Industriousness:
Anders Ericsson's model of deliberate practice asserts that when skills become automatic, we often plateau at levels below our true potential. This suggests that continually challenging ourselves with difficult tasks is necessary to reach higher levels of proficiency. Similarly, Robert Eisenberg's theory of learned industriousness highlights the motivational aspect of difficulty. Success on challenging problems can foster a sense of accomplishment and motivation, while failure can be demotivating.

Step 3: Schedule and Confirm Testers
Once you have identified potential participants through the screener, it is essential to schedule and confirm their participation. Clear and timely communication is key to ensuring that participants are committed and available for the study. Providing detailed instructions, addressing any concerns or questions, and confirming their participation beforehand will help streamline the process and minimize any last-minute cancellations or no-shows.

Conclusion:
Finding great participants for a user study involves a combination of careful screening, considering the optimal success rate, and incorporating appropriate difficulty levels. By aligning these concepts, we can enhance the learning experience and ultimately improve the quality of our user studies. To summarize, here are three actionable pieces of advice:

1. Craft a comprehensive screener that helps identify participants who align with your study objectives and demographics.
2. Strive for an 85% success rate in your learning or training process, as this seems to be the optimal point for effective learning.
3. Incorporate tasks that are challenging yet manageable, ensuring that participants can benefit from the struggle and retrieval of background knowledge.

By following these guidelines, you can increase the chances of finding great participants and enhance the overall success of your user study.