experiment

llama.cpp easy-install for cloud instances

livetoolinginfralocal-first

A script that skips the 20-minute llama.cpp compile on 4-core Colab sessions.

I burned a lot of Colab sessions recompiling llama.cpp every time: 4 cores, 20+ minutes each. So I scripted the painful part away. Now a fresh cloud instance is usable in minutes.

binaries
per SM version
install time
seconds
stars
7

The problem

The official llama.cpp releases ship pre-built Windows CUDA binaries and nothing for Linux. Run llama.cpp across cloud instances and you recompile from source on every machine: 4 cores, 20+ minutes each, every single time. That is how I burned most of my early Colab sessions.

The fix

Build once per GPU SM version, store the binary on GitHub Releases, pull it anywhere in seconds. pull.sh detects the GPU, finds the matching binary, verifies its SHA256 checksum, and installs it to ~/.local/bin/llama. The archive bundles the CUDA runtime, so the target machine only needs a compatible driver.

Twenty minutes of compile time, every session, is a tax you only notice once you stop paying it.

The tooling

Six scripts cover the whole lifecycle: detect.sh for diagnostics, build.sh for compiling and uploading, pull.sh for installing, list.sh for browsing the store, verify.sh for checksums, and cleanup.sh for removing old versions. configs/gpu_map.json maps GPU model names to SM versions, and CI auto-builds every SM version on each new llama.cpp release.

Where it runs

T4, A100, L40S, RTX 4090, H100: if it has an SM number, it has a binary. The repo has picked up a small following, and the pattern generalizes to any project that ships Windows-only binaries.