gufo

Gufo is an inference engine built for AMD Strix Halo GPUs. It runs a curated set of open‑source models locally. The runtime exposes an OpenAI‑compatible API on port 8080 so any OpenAI SDK or front‑end like Open WebUI can talk to it.

The engine focuses on a few carefully selected models. For text and image generation it ships with the Qwen 3.8 27B model in various GGUF quantisations and a Qwen 3.8 Flash‑Next MoE model. For speech there are Qwen3‑ASR and Qwen3‑TTS, with the TTS model capable of a 12 Hz voice‑cloning pipeline that reaches up to 2.54× real‑time speed. Image and video generation modules are under development. Each model runs with a dedicated kernel and is benchmarked for both prefill (pp) and decode (tg) throughput. For example, Qwen 3.8 27B gives 656.33 tok/s in prefill and 123 tok/s summed over eight users in decode. The Flash‑Next MoE model delivers 1 702.93 tok/s in prefill and 184.74 tok/s summed over eight users.

Installing Gufo requires downloading the model weights from Hugging Face and spinning up the runtime container. A typical launch command pulls the latest image from ghcr.io and mounts a local models directory. The command includes flags for host, port, model path, and optional speculative decoding with a DFlash 2 model. It also exposes GPU devices with /dev/kfd and /dev/dri, and sets memory limits for the container.

Gufo is aimed at users who own Strix Halo hardware and want to use it fully. The design keeps the number of models small, each chosen for quality and performance. The community can contribute via GitHub, and there are already forks for Windows and for RDMA across two Halo GPUs. Benchmarks are published as part of the repository, and reproducible builds are encouraged. The documentation lists the supported models and their performance numbers, but it does not publish pricing or user limits beyond what the hardware can handle.

Key features

• OpenAI‑compatible API exposed on localhost:8080/v1
• Optimised kernels for Qwen 3.8 27B and Qwen 3.8 Flash‑Next
• Speech support with Qwen3‑ASR and Qwen3‑TTS
• Speculative decoding using DFlash 2 for higher throughput
• Reproducible builds with pre‑tested model weights
• Community‑driven forks for Windows and RDMA