Documentation

Deploy and control open LLMs on rented GPUs.

The essentials below get you from a RunPod key to a served model. The full reference (architecture, configuration, adding a provider) lives in the repo.

Install

The base install is the engine, CLI, and OpenAI proxy. The REST API and MCP server are optional extras, so the core stays lean.

pip install open-lease         # base: CLI + OpenAI proxy
pip install 'open-lease[all]'  # add the REST API and MCP server

Configure

Credentials come from environment variables, a local .env, or a TOML file, in that precedence. Secrets are never logged or written to state.

cp .env.example .env
# then set, at minimum:
#   RUNPOD_API_KEY   (required for the RunPod provider)
#   HF_TOKEN         (optional; needed for gated models)

gpu config              # effective config, secrets masked

Deploy and manage

A deployment is driven by the reconcile loop toward READY. First launch is download-bound; gpu status shows a percent or an elapsed-over-budget ETA. The catalog holds tuned, validated recipes, but the engine is model-neutral: --hf-repo runs any vLLM-servable model with no catalog entry (--gpu required).

gpu models                    # catalog of tuned, validated recipes
gpu availability qwen3-0.6b   # data centers with capacity right now
gpu deploy qwen3-0.6b --wait  # a catalog model
gpu deploy --hf-repo Qwen/Qwen3-14B --gpu A100-80GB   # or any HF model, no catalog entry
gpu status                    # id, state, endpoint, uptime, accrued $
gpu health dep-a1b2c3         # check-by-check health
gpu logs dep-a1b2c3           # provider/runtime logs
gpu restart dep-a1b2c3        # stop then redeploy the same profile
gpu stop dep-a1b2c3           # destroy the pod, keep the record
gpu delete dep-a1b2c3 --yes   # stop and remove the record

Talk to a model

Chat directly, or through the OpenAI-compatible proxy, which routes by the request model field (catalog id or HF repo) to a READY deployment and streams the response.

# one gesture: deploy, wait, drop into a chat REPL
gpu deploy qwen3-0.6b --wait --chat

# or the OpenAI proxy (routes by model name to a READY deployment)
gpu proxy                     # on :8080
curl localhost:8080/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"model":"qwen3-0.6b","messages":[{"role":"user","content":"hi"}]}'

Run it as a service

The daemon owns the reconcile, health, and orphan-sweep loops, so a non-blocking deploy progresses on its own. A non-blocking deploy with no daemon warns loudly rather than stalling.

gpu up                # daemon + proxy, backgrounded
gpu deploy qwen3-32b  # non-blocking; the daemon drives it to READY
gpu daemon --status
gpu down              # stop both

REST API

A thin FastAPI layer over the same core. Bodies are the domain models, auth is a single bearer token, and the OpenAI proxy is mounted at /v1/*.

gpu serve             # REST API on :8000, auto-docs at /docs

# routes mirror the core 1:1
POST   /deployments                 GET /deployments            GET /deployments/{id}
POST   /deployments/{id}/stop       POST /deployments/{id}/restart
DELETE /deployments/{id}            GET /deployments/{id}/logs|health|events
GET    /models  /providers  /availability  /costs  /volumes     POST /estimate
# the OpenAI proxy is mounted at /v1/*

MCP server

Agent-facing tools over the Orchestrator, so an agent can provision and run GPUs. Destructive tools require an explicit confirmation.

# an MCP stdio server of agent-facing tools over the same core
gpu-mcp
# tools: deploy_model, stop/restart/delete_deployment (delete needs confirm),
#        list_models, list_deployments, get_deployment, deployment_logs,
#        deployment_health, provider_status, gpu_availability, estimate_cost,
#        get_costs, chat_completion

Visual workbench

A visual layer over the same core: watch deployments climb to READY, deploy from a catalog or any HF repo, and chat with a model in the browser. gpu ui serves it locally alongside the API and proxy. The hosted workbench drives your own local server: start it with cross-origin allowed for that page (opt-in, off by default) and connect. Your server URL and token stay in your browser.

gpu ui   # local workbench: watch deployments come up, deploy, and chat. opens the browser

# or drive your local server from the hosted workbench (cross-origin, opt-in, off by default):
gpu serve --cors-origin https://workbench.openlease.canonicalresearch.dev

Full documentation and source on GitHub →