uzu

Uzu is a Rust‑based inference engine created by Mirai Labs for iPhone, iPad and Mac devices that run on Apple Silicon. It runs large language models that have been quantized to 4‑bit or 8‑bit precision, which keeps the memory footprint small enough for mobile hardware while still delivering the language capabilities of larger models. The engine is marketed as a high‑performance component for on‑device AI that can respond in real time.

In practice the engine loads a quantized model file, such as a Qwen, LFM, Muse‑Glimmer or d3‑flash checkpoint, into memory and then accepts a single prompt as input. Because uzu is built around a batch size of 1, each request is processed independently, avoiding the latency that comes from waiting for a larger batch to fill. The runtime executes the model step by step, handling parsing, validation, tool calls, reasoning and rendering within a tight latency budget that aims to stay under 200 ms. The output is a text response that can be fed directly back to the user interface or to another part of an application, enabling continuous interaction without visible loading states.

The engine's design is tightly coupled to Apple Silicon's neural accelerator and memory hierarchy. Tensor multiplications and other core operations are optimized for the device's compute units, and the quantization scheme was co‑designed with the hardware to preserve interaction quality while reducing arithmetic intensity. This hardware‑aware approach lets uzu claim speeds around 1 000 tokens per second for “thinking” and 670 tokens per second for interactive rendering, numbers that the site uses to illustrate the difference between slow chat‑style latency and near‑instant AI‑native interfaces.

Developers who need private, offline AI on iOS, iPadOS or macOS will find uzu useful. Typical use cases include personal assistants that manage calendar events, retrieve files, or coordinate rides without sending data to a server. Because the stack includes a runtime layer, a model layer, a quantization layer and an application layer, teams can integrate the engine directly into automation scripts or native apps that already use Apple's device automation APIs. The tool is less suitable for developers who require large‑batch processing, cloud‑based scaling, or support for non‑Apple hardware, as the documentation focuses exclusively on Apple Silicon constraints.