How to train knowledge workers like athletes with AI

TL;DR

Deliberate practice for knowledge work is possible with AI by building repeatable drills and clear rubrics. Start by defining what good looks like for artifacts, score them with a rubric, and use AI to provide consistent feedback. Over time, track patterns in performance to improve outcomes rather than relying on live performance alone.

Transcript

We need to move from a jobs format to a skills format for our roles and our career growth. And no one's ready to talk about it. That's what this video is all about. How do you think about your job differently and think about it in terms of skills you can train and improve, preferably with the help of AI? One of my inspirations for this post was a 2... Read More

Key Insights

• Knowledge work can be trained like athletics through deliberate practice and feedback loops.
• Rubrics transform fuzzy feedback into measurable scores that can be applied to multiple artifacts across disciplines.
• Five essential skills for the AI age are judgment, orchestration, coordination, taste, and updating.
• Artifacts, not adjectives, carry the evidence of skill in work output, and rubrics should be annotated on these artifacts.
• AI changes practice by providing consistent grading against clearly defined standards and prompts for edits.
• The first step in practice is to define what good looks like for each artifact and to agree on criteria with trusted colleagues.
• A structured reviewer process uses a red pen style critique to mark strengths and weaknesses before bringing in AI for scoring.
• Continuous logging of scores and patterns over time reveals personal growth and gaps in the skill set.
• More videos from this channel:
  • How Accurate Were AI Predictions for 2025 So Far?
  • How to build Claude skills quickly with meta skills
  • How did OpenAI tests let AI reach Hugging Face?
• More from this channel: AI News & Strategy Daily | Nate B Jones

Questions & Answers

Q: How can I start practicing like an athlete for knowledge work with AI

The first step is to define a concrete artifact that matters for your team, such as a decision document, and ask trusted colleagues what good looks like. Create a rubric with 1 to 5 scores for aspects like clarity, options, stakes, and risks. Mark up real examples to show how each dimension should be scored, then bring in AI to grade new documents using that rubric. This creates a repeatable practice loop and makes skill development observable and actionable.

Q: What are the five skills that matter most when AI handles execution

The five skills are judgment, orchestration, coordination, taste, and updating. Judgment covers framing decisions and evaluating options under uncertainty. Orchestration turns fuzzy goals into executable workflows for humans and AI. Coordination moves groups through ambiguity without adding chaos. Taste defines meaningful quality for products and writing, and updating governs how you adjust beliefs as evidence shifts. Together they form a practical skill set for AI powered knowledge work.

Q: How do artifacts help in practicing professional skills

Artifacts are the concrete outputs that demonstrate skill, such as decision documents, architecture memos, or project plans. They carry the evidence of judgment, orchestration, coordination, taste, and updating. By focusing on artifacts, you shift from abstract labels to measurable work products. Rubrics can then be applied to these artifacts to rate quality and guide targeted improvements.

Q: What role does AI play in the practice process

AI serves as a tool that can read text, apply a rubric consistently, and provide edits or suggested improvements. The key is to first build a human defined rubric and annotate examples. Then you use AI to score new work against that rubric and produce targeted edits. This keeps human judgment central while scaling feedback and repetition through AI.

Q: How can you ensure feedback is meaningful and not just fast

Start with trusted feedback from colleagues to establish what good looks like for each artifact. Use a red pen style critique to annotate strengths and weaknesses, then translate that into a rubric with explicit criteria and scores. Only after this human grounded rubric is in place should you bring in AI to grade new work, ensuring feedback remains grounded in concrete standards.

Q: Why is practice in knowledge work different from sports training

Knowledge work lacks clear binary feedback and highly variable outcomes, making immediate signals difficult. The video argues for narrow, repeatable practice scenarios with explicit metrics that can be graded. AI enables scalable, consistent feedback across artifacts, turning complex judgment into measurable practice, similar to drills in sports.

Q: What does the process look like to move from practice to real performance

Begin with defining good for key artifacts, then annotate real examples to create a rubric. Use AI to grade new outputs against that rubric, with explicit reasoning cited for each score. Track patterns over time to see improvements in the skill areas, turning practice into continuous performance growth rather than occasional, high stakes events.

Q: How can teams implement a skills first mindset at scale

Teams should identify core artifacts shared across roles and create rubrics that capture the five skills: judgment, orchestration, coordination, taste, and updating. Use AI to provide consistent feedback on these artifacts, while human mentors refine the criteria and scores. By documenting patterns and progress, teams can build a scalable system of deliberate practice that ties learning to tangible outputs and outcomes.

Summary & Key Takeaways

• A skills first approach reframes work by focusing on repeatable patterns you can practice with feedback rather than static job titles. It uses AI to scale custom feedback and to grade artifacts against defined rubrics, making learning measurable.

• The video identifies five core skills that matter in the AI age, and explains how to connect practice to real artifacts like decision docs and project plans to drive improvement.

• Practice loops move teams from vague performance to concrete, rubric driven evaluation, enabling continuous learning and visible progress through structured feedback and artifact based scoring.