How to run local AI models on Windows for hybrid intelligence

TL;DR

Local AI models run on the Windows PC through hybrid intelligence, combining local and cloud capabilities to keep the system responsive while enabling powerful on device computing. The demo covers MAI code, Flash, and other models with local execution, plus the new Windows ML runtime and the integration with GitHub Copilot for local tasks.

Transcript

models, runtime, and silicon, all working together to create a meaningful experience. We're working with the GitHub team to bring our vision for hybrid intelligence to life on Windows. Earlier this year, GitHub launched Hydro Fusion, which routes each task to the right model in the cloud. Today, we're extending intelligence to Windows, letting Hydr... Read More

Key Insights

• Local models run on the device to preserve responsiveness and leave room for other tasks.
• MAI code and Flash are optimized to run at reduced precision on hardware while keeping coding quality.
• Hybrid routing directs tasks to the appropriate model, cloud or local, based on complexity and resource needs.
• Windows ML provides a scalable runtime to deploy models across GPUs, NPUs, and CPUs.
• Llama CPP is coming to Windows ML to enable open source models on day one.
• Copilot gains local context, local actions, and local models to ground tasks on the PC.
• Users can automate workflows with automations and a visual canvas to manage sessions.
• Auto model selection can switch between local and cloud models depending on task complexity.

Questions & Answers

Q: How can I run AI models locally on Windows for better responsiveness?

To run AI models locally, you need a setup that supports local model execution alongside cloud models. The Windows ML runtime orchestrates deployment across GPUs, NPUs, and CPUs, enabling local models like MAI code and Flash to run on the device. This approach preserves responsiveness for everyday tasks while enabling more intensive processing when needed.

Q: What models are mentioned for local use on Windows?

The transcript mentions MAI code 1.x and Flash local as examples of coding focused models that run on the device. It also references 284 billion parameter models like Flash v4 and 256K context windows, illustrating the range of capabilities that can operate locally under the right optimizations and quantization techniques.

Q: What is Hydra Fusion and how does it relate to Windows?

Hydra Fusion is described as routing each task to the right model in the cloud and extending intelligence to Windows. It enables a hybrid approach where local models handle simpler tasks and cloud models handle complex ones, creating a seamless experience that keeps the PC responsive while leveraging powerful AI.

Q: How does Copilot integrate with local and cloud models on Windows?

Copilot gains local context, local actions, and local models, allowing it to ground tasks on the PC, perform actions like moving files or changing settings, and delegate to local models for privacy or cost reasons. This integration supports hybrid intelligence across home, code, and autopilot experiences.

Q: What benefits does the Windows ML runtime offer?

The Windows ML runtime provides a scalable way to deploy models across GPUs, NPUs, and CPUs, enabling efficient execution of both local and cloud models. This reduces latency, improves responsiveness, and broadens model compatibility for developers building on Windows.

Q: What does the term local context mean in Copilot on Windows?

Local context means Copilot can ground its understanding in all files on the user’s PC, enabling more relevant and precise responses and actions. This grounding allows Copilot to operate with awareness of local data while maintaining user permission and control over actions.

Q: How are quantization and context windows used for local models?

Quantization techniques reduce model size and memory footprint, enabling large models to run locally. A larger context window, such as 256K, allows the model to review more of the surrounding code or text, improving accuracy and usability for tasks like code cleanup or issue triage.

Q: What kind of tasks can be automated with local models and Copilot on Windows?

Users can automate tasks that involve local file management, window arrangements, or status changes through local actions. Automations can run on a schedule or trigger events, and the session canvas visualizes ongoing cloud and local tasks, helping users manage complex hybrid workflows without leaving the device.

Summary & Key Takeaways

• Hybrid intelligence on Windows enables local model execution alongside cloud models to power responsive AI workflows.

• Local models like MAI code and Flash are shown running on device with quantization and reduced memory usage.

• The Windows ML runtime and Llama CPP integration expand the choice of open source models for developers.