How to Run a Local AI Model with Ollama

TL;DR

Install Ollama, choose a model that fits your available memory, and run it from a terminal with the model name. Local models remove API fees and rate limits, keep data on the device, work offline, preserve access to a specific model version, and can provide an API server that lets other applications use downloaded models.

Transcript

Forget JGBT. Here is how to run your own LLM locally. So, a local AI model is one that runs entirely on your computer. Now, you might be thinking, "Okay, David, why would I even need local LLMs?" Well, there are many reasons. Number one, cost. No API fees, no subscriptions. Since it runs on your own computer, it's completely free. Number two, no ra... Read More

Key Insights

• Local AI models run entirely on the user's computer, so prompts and other data do not need to leave the device. They can also operate without an internet connection, making them usable during flights, in remote locations, or anywhere that cloud access is unavailable.
• Local model ownership preserves access to a specific downloaded version. Unlike a remotely managed model that may change, the locally stored version remains available, and an open-source model can be fine-tuned to support a particular workflow or use case.
• Open models suitable for consumer hardware are becoming more capable. The transcript argues that smaller models available for laptops and single-GPU computers are narrowing the benchmark gap with large cloud models, while the number of strong local options has increased substantially.
• An AI model is a collection of learned weights and biases, also called parameters. Training processes large amounts of tokenized data and adjusts these numerical values to reduce a loss function, leaving a model file that represents learned patterns, knowledge, and conclusions.
• Running an AI model requires the model file, a program capable of reading its weights, and an inference process that generates responses. Ollama combines these practical functions by downloading model files, loading their parameters into memory, and providing interfaces through which users or applications can access them.
• Ollama is an open-source tool for downloading and running local AI models. It stores model weights in an organized format, loads them into memory as an execution engine, supports terminal interaction, and starts an API server that allows other software to use installed models.
• Memory requirements depend on the computer architecture described in the transcript. Apple silicon systems use unified memory that is accessible to both the CPU and GPU, while a Windows computer with an Nvidia GPU primarily depends on the GPU's VRAM for loading and running models.
• Model selection should account for hardware capacity and the model's parameter count. Ollama's catalog includes models in multiple sizes, so users can begin with a model their existing Mac, Mac Studio, or Nvidia GPU can handle instead of building an expensive multi-GPU cluster.

Questions & Answers

Q: Why should you run an AI model locally?

Running an AI model locally removes API fees, subscriptions, and usage rate limits because inference happens on your own computer. It also keeps data on the device, works without an internet connection, and preserves access to the exact model version you downloaded. Open-source models can additionally be fine-tuned and run locally for a specialized use case.

Q: What is Ollama used for?

Ollama is an open-source tool that downloads and runs AI models on a computer. It organizes downloaded model files, reads their weights, loads billions of parameters into available memory, and performs inference to generate responses. It also offers a terminal interface and starts an API server so other applications can connect to locally installed models.

Q: How do you install and start Ollama?

Go to the Ollama website, select the download for your operating system, and open the installer. On macOS, move the application into the Applications folder and then open Ollama. After installation, Ollama automatically starts a local API server. The transcript recommends visiting the displayed localhost address and checking for the message confirming that Ollama is running.

Q: How do you download and run a model with Ollama?

Open a terminal, visit Ollama's model catalog to identify the model you want, and enter the Ollama run command followed by that model's name. Models may have multiple versions identified by parameter counts, such as the cited 20B and 120B options. The appropriate version depends on what your computer can handle and what you need it to do.

Q: What hardware is needed to run a local LLM?

A local model can be started on a computer with a single suitable GPU, including an Apple silicon Mac, Mac Studio, or a system with an Nvidia GPU. A large GPU cluster is not required. Because models come in different shapes and sizes, the practical approach is to select one that fits the memory and capabilities of the hardware already available.

Q: How does memory differ between Apple silicon and Nvidia systems?

Apple silicon computers with M-series chips use unified memory, which the transcript describes as accessible to both the CPU and GPU. On a Windows computer with an Nvidia GPU, the important resource for local model execution is the GPU's VRAM rather than ordinary system RAM. This architectural difference affects which model sizes each machine can load and run.

Q: What is an AI model made of?

An AI model is fundamentally a large collection of numerical weights and biases, collectively called parameters. During training, the system processes large amounts of tokenized data and adjusts those values while trying to reduce a loss function. The resulting parameters represent patterns, knowledge, and conclusions learned from the training data and are stored in a model file.

Q: How can other applications use a model running in Ollama?

Ollama automatically starts an API server when installed and opened. That server provides a connection point through which other tools and software can use models downloaded through Ollama. Without an API, the model would be limited to Ollama's own interface. With the server running, compatible applications, including the example of Cursor in the transcript, can access local inference.

Summary & Key Takeaways

• Local language models run entirely on a personal computer, offering privacy, offline operation, unlimited use, and freedom from API fees or subscriptions. Users retain a downloaded model version and can fine-tune open-source models for specialized needs. The range and capability of smaller open models have also expanded rapidly.

• Ollama performs three central roles: it downloads and organizes model files, acts as an engine that loads model parameters into memory, and provides an interface for running inference. It supports terminal use and automatically starts an API server, enabling compatible applications to connect to locally downloaded models on the computer.

• Beginners can install Ollama by downloading the appropriate installer, opening the application, and confirming that its local server is running. A model can then be obtained from Ollama's model catalog and launched through a terminal command. Model choice should reflect parameter count, computer specifications, available memory, and the intended use case.