Self-Hosting Qwen: Trivial to Brutal

David’s question: “It sounds like [you] can easily self-host Inkling Small, but it's hard to self-host Qwen. Is that right? How hard is it to stand up inference for a Qwen model?”

Eve: “Qwen's twenty-seven-billion is about as easy to self-host as it gets — twenty-seven gigs of weights at FP8, one GPU, done. The two-point-four-trillion flagship is the opposite extreme: over a terabyte of weights even quantized, so you're looking at a multi-GPU cluster, eight to sixteen cards minimum, and serious networking between them. That's not a weekend project, that's infrastructure. So Qwen isn't hard to host — it spans the whole range from trivial to brutal, and you pick your poison by model size.”