CrowdGPT

CrowdGPT is an open‑source framework that lets you contribute GPU compute to train a language model without operating a datacenter. The project runs on a single Python client that you install on a personal computer, and the resulting model weights are published publicly on HuggingFace.

The system is aimed at users who have a consumer‑grade graphics card and want to help build an AI model that remains community‑owned. When the client starts, it contacts a server that assigns a 10 MB shard of tokenized text from the public CrowdGPT dataset along with the current global model weights. The client streams the shard, performs local training on the GPU, computes the weight delta, and then uploads that delta for validation. Updates are checked for NaN or Inf values, size conformity, and norm bounds before being aggregated into the global model. Because only the delta is sent, no personal files, browser history, or other local data leave your machine.

Installation is handled by a one‑line script that detects your hardware, installs the appropriate PyTorch wheel, and creates an isolated virtual environment. The installer works on Linux, macOS, and Windows and supports NVIDIA (CUDA), AMD (ROCm/DirectML), Apple Silicon (MPS) and Intel Arc (XPU). About 6 GB of free disk space is required, with PyTorch itself taking roughly 2.5 GB. A stable internet connection is needed to download model weights and dataset chunks; CPU‑only mode is possible but runs very slowly. Once installed, you can start the client, let it train for as long as you like, and stop at any time.

The model being trained has 1 billion parameters, a 151 K token vocabulary, 24 layers with grouped‑query attention, and uses BF16 precision. All code is a single human‑readable Python file, and the full training dataset is hosted on HuggingFace where anyone can inspect or contribute. Model weights are freely downloadable, and the project emphasizes privacy, ecological impact, and transparency.

Key features

• Open‑source Python client that runs on consumer GPUs
• Automatic hardware detection and dependency installation
• Training on 10 MB data shards streamed from a public dataset
• Secure delta submission with validation for NaN, size, and norm
• Publicly available model weights and dataset on HuggingFace
• Supports NVIDIA, AMD, Apple Silicon, and Intel Arc GPUs