How to run LLMs locally in 2027: Ollama, LM Studio and llama.cpp

To run an LLM locally, install a runtime such as Ollama, LM Studio or llama.cpp, download an open-weight model small enough for the computer's memory, and chat with it on the same machine. All three read GGUF model files, and a Q4_K_M file for Llama 3.1 8B is 4.58 GiB in the llama.cpp project's own table. Prompts stay on the machine unless a cloud feature is switched on.

Last updated: 11 November 2026.

Why run an LLM locally, and when is it the wrong choice?

Local models suit work where prompts must stay on the machine, where the connection is unreliable, or where the goal is to learn how inference works. Ollama's FAQ says it does not see prompts or data when a model runs locally, and LM Studio's documentation says nothing typed into the app leaves the device once a model is downloaded. The llama.app page, from the llama.cpp team, adds that its local API needs no API keys and brings no usage bills.

Local is the wrong choice when the model is too big for the machine. Ollama's download page warns that large models are slow on a computer without a strong GPU, and its library lists qwen3:235b at 142GB. Ollama's macOS documentation says model files can take tens to hundreds of GB of disk. For tasks that need no privacy, the hosted assistants listed in 10 best free AI tools in 2027 for work and study need no setup at all.

Which tool should you use: Ollama, LM Studio or llama.cpp?

Ollama, LM Studio and llama.cpp all run GGUF models locally and differ mainly in interface: Ollama centres on a command line and REST API, LM Studio on a desktop app and llama.cpp on the engine's own command line and server. The table compares the three as their own documentation describes them on 11 November 2026.

Ollama is an MIT-licensed tool that runs open models from a command line, a desktop app and a REST API. Its README lists llama.cpp as a supported backend and gives installers for macOS, Windows, Linux and Docker. Ollama's latest GitHub release on 11 November 2026 was v0.40.16, and the same app can now call Ollama's cloud models, which are not local.

LM Studio is a desktop app with a chat window, a model downloader backed by Hugging Face and a local server. Its terms grant a licence for personal or internal business purposes, and a blog post of 8 July 2025 says it became free at home and at work. On 16 July 2026 LM Studio added Bionic, an agent that can use local or cloud models, and the download page lists LM Studio version 0.4.28 beside it.

llama.cpp is an MIT-licensed C/C++ inference engine, and the project under both tools above. Its founding team, ggml.ai, joined Hugging Face on 20 February 2026, and the announcement says the projects remain open and community driven. The README now installs through llama.app and uses llama cli and llama serve, while the server README still shows a llama-server binary for source builds, so command names vary between guides.

All three read GGUF, so one downloaded file can be imported into another tool: Ollama through a Modelfile, LM Studio through lms import. Hugging Face's documentation lists llama.cpp, LM Studio and Ollama among the tools that use the format.

What is GGUF?

GGUF is a binary file format that stores a model's tensors and a standard set of metadata in one file, designed for inference with GGML-based programs such as llama.cpp. Hugging Face's documentation contrasts it with tensor-only formats such as safetensors, and the GGUF specification lists single-file deployment, extensibility and memory-mapped loading among its goals.

The specification puts a header at the start of every file: a magic number, a version, then a 64-bit tensor count and a 64-bit metadata count, little-endian by default. The script below writes a tiny synthetic file and reads the header back.

# pip install gguf numpy
import struct
import numpy as np
from gguf import GGUFWriter, GGUFReader

path = "tiny.gguf"
w = GGUFWriter(path, "llama")
w.add_name("tiny-demo")
w.add_block_count(2)
w.add_tensor("token_embd.weight", np.zeros((4, 8), dtype=np.float32))
w.write_header_to_file()
w.write_kv_data_to_file()
w.write_tensors_to_file()
w.close()

with open(path, "rb") as f:
    magic, version, n_tensors, n_kv = struct.unpack("<4sIQQ", f.read(24))
print("magic:", magic, "version:", version, "tensors:", n_tensors, "metadata entries:", n_kv)

r = GGUFReader(path)
for key in ("general.architecture", "general.name"):
    field = r.fields[key]
    print(key, "=", bytes(field.parts[field.data[0]]).decode())
print("tensor:", r.tensors[0].name, r.tensors[0].tensor_type.name)

Run with Python 3.12 and gguf 0.19.0, the output is below. The file is a 352-byte demo with no real weights, but the header is the same structure a full model carries.

magic: b'GGUF' version: 3 tensors: 1 metadata entries: 3
general.architecture = llama
general.name = tiny-demo
tensor: token_embd.weight F32

Ollama's import page says a GGUF file is imported with a one-line Modelfile and that Ollama does not quantize GGUF models during import. LM Studio's CLI imports one with lms import.

How much memory does a local model need?

A local model needs at least its file size in memory, plus room for the context and the operating system. The llama.cpp quantize README says memory and disk requirements are the same because models are currently loaded fully into memory, and Ollama's context page says a larger context length increases the memory needed.

The file size follows from the parameter count and the bits stored per weight. The table lists the llama.cpp README's measured figures for Llama 3.1 8B.

The formula is parameters times bits per weight, divided by 8 for bytes. The script recovers the parameter count from the F16 row and predicts the Q4_K_M row.

GIB = 2**30
f16_gib, f16_bits = 14.96, 16.0005   # llama.cpp README, Llama 3.1 8B
q4km_bits = 4.8944

params = f16_gib * GIB * 8 / f16_bits
print(f"Implied parameters: {params / 1e9:.2f} billion")
print(f"Predicted Q4_K_M size: {params * q4km_bits / 8 / GIB:.2f} GiB (README table: 4.58 GiB)")

Implied parameters: 8.03 billion
Predicted Q4_K_M size: 4.58 GiB (README table: 4.58 GiB)

The prediction matches the README to two decimals. Take a hypothetical laptop with 16 GB of memory: the 14.96 GiB F16 file would fill essentially all of it before the operating system is counted, while the 4.58 GiB Q4_K_M file leaves most of it free. The README warns that quantization may introduce some accuracy loss, and LM Studio's documentation advises choosing a 4-bit option or higher where the machine allows.

The vendors' own numbers point the same way. LM Studio recommends 16GB or more of RAM on a Mac and says 8GB Macs can still run smaller models with modest context sizes. The llama.app page recommends Gemma 4 for a Mac with 16 GB of memory or more, GPT-OSS from 24 GB and Qwen 3.8 from 32 GB. Ollama's library lists qwen3:4b at 2.5GB, qwen3:8b at 5.2GB, qwen3:14b at 9.3GB, qwen3:30b at 19GB and qwen3:32b at 20GB.

How do you run a first model?

To run a first local model, install Ollama, LM Studio or llama.cpp, download a model whose file fits in memory, run it, then point code at the local server. The commands below come from Ollama's README and library pages, the llama.cpp README, and the LM Studio and Hugging Face documentation. The steps are in order.

1. Check free memory and disk space, then pick a file size from the tables above.
2. Install one tool: the Ollama app or install script, the LM Studio installer, or llama.cpp through curl -LsSf https://llama.app/install.sh | sh, Homebrew or WinGet.
3. Download and run a model sized for the machine.
4. Check where it runs. In Ollama, ollama ps shows in its PROCESSOR column whether the model sits on the GPU, the CPU or both.
5. Point code at the local server instead of a hosted API.

# Ollama: a library tag
ollama run qwen3:8b

# llama.cpp: a GGUF repository on Hugging Face (quant defaults to Q4_K_M)
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

# LM Studio CLI: download a model and start the server
lms get qwen
lms server start

In the LM Studio app the steps are to download a model from the Discover tab, load it into memory and chat.

Ollama's default context is 4k tokens on machines with less than 24 GiB of VRAM, and its page says agents and coding tools should be set to at least 64000 tokens. The server addresses are in the table below. Each exposes OpenAI-style endpoints, and Ollama's documentation calls its support a subset of the OpenAI API, so an OpenAI client library mostly needs only its base URL changed. Ollama requires an API key value from the client but ignores it.

The same servers can feed other tools in this pack: What is the Model Context Protocol covers the standard that many agents use, and the best vector databases for RAG covers the stores that a local model can query for retrieval-augmented generation. Ollama's embeddings endpoint returns unit-length vectors, which suits that setup. For adding knowledge to a small model, prompt engineering vs RAG vs fine-tuning compares the options.

Is running a model locally private?

Running a model locally keeps prompts on the machine, but the privacy of the whole setup depends on three things. LM Studio's documentation says nothing entered when chatting leaves the device, though searching for models, downloading models, downloading runtimes and checking for app updates need a connection.

The first setting is cloud mode. Ollama's FAQ says cloud models are processed on Ollama's servers without storing or logging the content, and that setting OLLAMA_NO_CLOUD=1 turns the cloud features off. LM Studio says the cloud services behind its Bionic agent are Zero Data Retention. Neither makes a cloud model local.

The second is the network address. Ollama binds 127.0.0.1 port 11434 by default and OLLAMA_HOST changes that, and LM Studio's documentation says its server can listen on localhost or the local network. The third is the machine itself: LM Studio's documentation says conversations are stored on disk in JSON format, so chat history is a file.

For finance teams weighing hosted tools against local ones, the best AI tools for finance professionals sets out what each vendor says about data handling.

What are the limits of local models?

Local models are limited by speed, which depends on the hardware, by quality, which varies by model and quantization, and by licences, which differ. The Ollama download page says speed depends on the hardware. Quality is a matter of testing a candidate model on the actual task, since the quantization README itself notes some accuracy loss.

Licences are not uniform. Hugging Face lists Qwen3-8B, Gemma 4 and gpt-oss-20b as Apache-2.0, while Llama 3.1 carries its own community licence. That agreement requires a company with more than 700 million monthly active users on the release date to request a licence from Meta, asks for "Built with Llama" to be displayed when distributing, and access to the weights is gated. Read the model card before building a product on a model.

Tools also change. LM Studio added Bionic on 16 July 2026, and the llama.cpp README's quick start uses llama cli and llama serve while other pages in the same repository still name llama-server and llama-quantize, so check which command a guide means.

What mistakes should a first local setup avoid?

A first local LLM setup usually goes wrong in one of five ways.

• Choosing a model by parameter count alone and ignoring file size and context memory.
• Expecting Ollama to quantize a full-size GGUF during import, which its documentation says it does not do.
• Leaving the default 4k context on a task that needs a long document or an agent loop.
• Exposing the server on the network through OLLAMA_HOST with no proxy or other access control in front of it.
• Copying commands or ports from a guide that predates the current README.

Where does local LLM work fit in an AI and ML programme?

Certification in Artificial Intelligence and Machine Learning is an online programme from Vishlesan i-Hub, IIT Patna, delivered on Masai's online platform, that covers Python, machine learning, deep learning, generative AI and LLMs, agentic systems and deploying models to production. Its curriculum names LLM internals, Retrieval-Augmented Generation, fine-tuning with LoRA and REST APIs for models, the topics that a local setup lets a learner try on their own machine.

Frequently asked questions

Can you run an LLM on a normal laptop?

A normal laptop can run an LLM if the model file fits in memory with room to spare. LM Studio recommends 16GB or more of RAM on a Mac and says 8GB Macs can run smaller models with modest context sizes. A Q4_K_M file for Llama 3.1 8B is 4.58 GiB in the llama.cpp README, so it fits a 16 GB laptop comfortably.

What is GGUF?

GGUF is a binary file format that stores a model's tensors and a standard set of metadata in one file. It was developed by Georgi Gerganov, the creator of llama.cpp, for use with GGML-based programs. Hugging Face's documentation lists llama.cpp, LM Studio and Ollama among the tools that run GGUF files.

Is running a model locally private?

A model that runs locally keeps prompts on the machine, and LM Studio's documentation says nothing typed into the app leaves the device. Privacy changes if cloud models are switched on, because Ollama processes cloud prompts on its own servers, or if the local server is exposed to a network. Downloads and app updates still need the internet.

Do you need a GPU to run an LLM locally?

A GPU is not required to run an LLM locally, but it sets the speed. Ollama's download page says large models are slow on a computer without a strong GPU. llama.cpp supports hybrid CPU and GPU inference for models larger than the video memory, and it lists Apple silicon as a first-class target, so a recent Mac works without a separate GPU.

Is LM Studio free to use at work?

According to an LM Studio blog post of 8 July 2025, the app became free to use at home and at work from that day, and the earlier requirement for a separate commercial licence was removed. The app remains closed-source software under LM Studio's terms of use, which grant a licence for personal or internal business purposes.

What is the difference between Ollama and llama.cpp?

llama.cpp is the C/C++ inference engine, run from the command line with options for each backend. Ollama wraps a model library, a background server, a desktop app and a REST API around it, and its README lists llama.cpp as a supported backend. Both are MIT licensed and both read GGUF files.

Next step

Running a model on a personal machine is a quick way to see how size, quantization and context decide what a model can do. The topics behind it, from LLM internals to deploying models behind an API, are laid out in the Certification in Artificial Intelligence and Machine Learning curriculum.