Projects

LLM inference engine

2026 · C++ · Vulkan · AI

Runs large mixture-of-experts coding models on ordinary laptop GPUs.

A 30B parameter model does not fit on a 4 GB graphics card, but only a small part of it works on each token. The engine keeps that part on the GPU, keeps the rest of the experts in system RAM, and lets an integrated GPU help with the work.

It is built on Vulkan rather than CUDA, so it runs the same on NVIDIA, AMD and Intel hardware. The hot paths are hand-written: quantized matrix kernels, BF16 conversion and AVX2 SIMD on the CPU side.

It ships with an API server and a web chat that runs the code it writes in a sandbox and repairs it until the tests pass.