TensorFold

TensorFold is an open‑source inference engine that enables you to run large language models on the GPU hardware you already own. It provides native Metal support for Apple Silicon Macs and native CUDA support for NVIDIA GPUs, and it exposes an OpenAI‑compatible HTTP API so existing client libraries can talk to the locally hosted model.

The engine is targeted at developers, data scientists, and hobbyists who prefer on‑premise execution over cloud services. If you have a Mac with Apple silicon running macOS 13.3 or later, you can install the CLI with a single curl command (`curl -fsSL https://tensorfold.dev/install.sh | sh`). After installation you select a model checkpoint, such as Qwen, Nemotron, GLM‑5.3‑Flash, or other open models, and launch the server. The server accepts standard JSON payloads (`prompt`, `max_tokens`, etc.) and returns generated text in the same format used by OpenAI's completions endpoint, making integration straightforward for applications that already use that API.

Performance data is published per model and hardware configuration. For instance, Nemotron 3.5 on an RTX PRO 6000 (Blackwell) reaches 874 tokens per second, while GLM‑5.3‑Flash on two M5 Ultra instances delivers an aggregate 199 tokens per second across eight concurrent streams. These numbers come from community benchmarks that record the exact hardware, model version, and engine release used for each run, so you can compare your own machine against the reported figures. The engine also implements prompt reuse: when a new request shares part of a previous prompt, the previously computed tokens are reused, reducing latency to roughly 90–110 ms for Nemotron. Drafting is another optimization; a lightweight “drafter” proposes several tokens, the main model validates them, and only the matching prefix is committed, which can lower the amount of work per decode pass.

TensorFold's API is OpenAI‑compatible, meaning the typical input is a JSON object with a `model` identifier, a `prompt` string, and optional parameters like `temperature` or `top_p`. The output is a JSON response containing the generated `choices`, each with a `text` field and token usage statistics. This design suits projects that already rely on OpenAI‑style calls but need to run them locally for privacy, cost, or latency reasons.