Skip to content
>_devvkit
$devvkit learn --librarie llama.cpp-guide

llama.cpp Guide

[llm][cpp][inference][quantization]
AI / LLM Tools
Install
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
cmake -B build
cmake --build build --config Release
# Or one-liner:
# brew install llama.cpp

llama.cpp is the engine behind most local LLM tools. It supports 4-bit to 8-bit quantized GGUF models that run on CPU often faster than full-precision models on GPU. The project originated the GGUF format now used by Ollama, LM Studio, and GPT4All.

Key tools in the build: `main` (interactive chat), `server` (HTTP server with OpenAI-compatible API), `quantize` (convert models to lower precision), `perplexity` (benchmark), and `embedding` (text embeddings).

Quantization levels trade quality for speed: Q4_K_M is the sweet spot (4-bit, good quality). Q2_K is tiny but degraded. Q8_0 is near-lossless. Use `./quantize model.f16.gguf model.Q4_K_M.gguf Q4_K_M`. GUI: LM Studio (native app wrapping llama.cpp), Ollama (CLI wrapper), GPT4All.

Build & Setup

Build from source— Compile llama.cpp.
git clone https://github.com/ggerganov/llama.cpp && cd llama.cpp
cmake -B build -DGGML_CUDA=ON    # NVIDIA GPU
cmake -B build -DGGML_METAL=ON   # Apple Silicon
cmake -B build -DGGML_CUDA=OFF -DGGML_METAL=OFF  # CPU only
cmake --build build --config Release -j 8
Download models via script— Grab popular models.
# Hugging Face GGUF models:
# https://huggingface.co/TheBloke

# Download script:
python -c "
import huggingface_hub
huggingface_hub.hf_hub_download(
    repo_id='TheBloke/Llama-2-7B-GGUF',
    filename='llama-2-7b.Q4_K_M.gguf',
    local_dir='./models'
)
"

Inference

Interactive chat— Chat with a model.
./build/bin/main -m models/llama-2-7b.Q4_K_M.gguf \
  --color -n 512 -ngl 999 \
  -p "Building a website with Node.js:\n"
# -n: max tokens, -ngl: GPU layers (999 = all), -p: prompt

# Interactive mode:
./build/bin/main -m model.gguf --color -i -r "User:"
Embedding generation— Get text embeddings.
./build/bin/embedding -m model.gguf -p "Hello world"
# Outputs: list of floats (embedding vector)

# Batch embed from file:
./build/bin/embedding -m model.gguf -f sentences.txt

Server Mode

HTTP server— OpenAI-compatible API.
./build/bin/server -m models/llama-2-7b.Q4_K_M.gguf \
  --port 8080 -ngl 999 --host 0.0.0.0

# Test:
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-3.5-turbo","messages":[{"role":"user","content":"Hello"}],"stream":true}'

Quantization

Quantize model— Convert to smaller format.
# Convert to FP16 first, then quantize:
python convert.py model.gguf --outtype f16

# Quantize:
./build/bin/quantize model.f16.gguf model.Q4_K_M.gguf Q4_K_M
./build/bin/quantize model.f16.gguf model.Q8_0.gguf Q8_0
./build/bin/quantize model.f16.gguf model.Q2_K.gguf Q2_K

# All quantization types:
./build/bin/quantize --help

Benchmark

Perplexity benchmark— Measure model quality.
./build/bin/perplexity -m model.gguf -f wiki.test.raw
# Lower perplexity = better model

# Speed test:
./build/bin/main -m model.gguf -p "Hello" -n 256 -t 8 --no-display-prompt
# t: threads — depends on CPU cores
# Watch: tokens/second in output