InferBox

Self-hosted GPU inference server compatible with the OpenAI API. Embeddings, rerankers, LLMs, STT, image generation. Micro-batching, speculative decoding, multi-GPU aware. In daily use on a 12 GB RTX 3060.

https://github.com/joseluissaorin/InferBox