Recommendation for Self-hosted & open
Best Open-Weight LLM for Self-Hosting
Our top recommendation for Best Open-Weight LLM for Self-Hosting, based on the public evidence we track, is Qwen: Qwen3.8 27B.[1][2][3] Achieve 59 tok/sec generation on RTX 5090 for 8K prompts, scaling reasonably to 44 tok/sec at 128K context. Watch out: Dense architecture means every token processes full ~17GB weights, saturating memory bandwidth and limiting throughput compared to MoE alternatives. DeepSeek: DeepSeek V4 Flash 0423 is the next-ranked alternative. Fit on 16GB VRAM consumer cards (RTX 5080) for accessible self-hosting without enterprise hardware.
About this recommendation
- Updated
- Sep 25, 2026
- Evidence through
- Sep 25, 2026
- Sources
- 22
- Revision
- v79
Decision audit
Why this result
Inspect the inputs and the computed order behind the recommendation.
Models screened
20
live candidates
Evaluation feeds
5
task-weighted
Winner coverage
73%
intended feed weight
Largest provider share
4 of 8
deepseek
Sources evaluated
The task sets these weights before any model is scored.
| Evaluation feed | Weight | Winner result | Field measured |
|---|---|---|---|
| price weight | 33% | 91/100 | 20/20 |
| LiveBench Instruction Following | 22% | #2 | 11/20 |
| Route reliabilityunavailable | 22% | feed unavailable | 0/20 |
| LMArena Text | 17% | #14 | 17/20 |
| OpenRouter usage | 6% | 88/100 | 20/20 |
Provider concentration
Each exact model is scored separately; provider identity is not a ranking input.
- deepseek4 models
- Google1 model
- minimax1 model
- Qwen1 model
- Z.ai1 model
Decision table
Every published model is shown in computed order. Practitioner sources are distinct community threads, not the citations repeated in the prose below.
| Rank | Model | Relative score | Coverage | Practitioner evidence | Strongest measured reason |
|---|---|---|---|---|---|
| 01 | Qwen3.8 27BQwen | 77 | 73% | 28 threads · 23 families · 5 cautions | #2 LiveBench Instruction Following · #14 LMArena Text |
| 02 | DeepSeek V4 Flash 0423deepseek | 74 | 73% | 18 threads · 15 families · 5 cautions | #10 LiveBench Instruction Following · #15 LMArena Text |
| 03 | GLM 5.2Z.ai | 70 | 73% | 15 threads · 11 families · 3 cautions | #4 LMArena Text · #12 LiveBench Instruction Following |
| 04 | Gemma 4 26B A4B Google | 69 | 67% | 1 threads · 1 families · 0 cautions | #13 LMArena Text |
| 05 | MiniMax M3minimax | 68 | 73% | 6 threads · 5 families · 0 cautions | #11 LMArena Text · #13 LiveBench Instruction Following |
| 06 | DeepSeek V3.2deepseek | 67 | 67% | 4 threads · 4 families · 2 cautions | #18 LMArena Text |
| 07 | DeepSeek V4 Pro 0423deepseek | 67 | 73% | 14 threads · 12 families · 4 cautions | #7 LMArena Text · #11 LiveBench Instruction Following |
| 08 | DeepSeek V4.1 Flashdeepseek | 66 | 53% | 8 threads · 8 families · 1 cautions | #6 LiveBench Instruction Following |
Relative score combines normalized benchmark quality and signal coverage; independent practitioner evidence and freshness are bounded tie-breakers. It is an ordering score, not an absolute quality percentage. The writing model receives this order and cannot change it.
Qwen3.8 27B offers strong benchmark performance with Apache 2.0 licensing and runs on consumer hardware like RTX 3090 24GB with Q4_K_M quantization, though its dense architecture streams all ~17GB of weights per token.
Best when: Achieve 59 tok/sec generation on RTX 5090 for 8K prompts, scaling reasonably to 44 tok/sec at 128K context.
Tips
- Achieve 59 tok/sec generation on RTX 5090 for 8K prompts, scaling reasonably to 44 tok/sec at 128K context.
Watch out for
- Dense architecture means every token processes full ~17GB weights, saturating memory bandwidth and limiting throughput compared to MoE alternatives.
- Speculative decoding with `--gdn-replay` causes degenerate looping after ~900 characters, requiring careful flag management.
- SGLang scheduler busy-spins on idle models consuming one CPU core unless `--sleep-on-idle` is explicitly enabled.
DeepSeek V4 Flash runs on RTX 5080 16GB via community ports and serves reliably on DGX Spark clusters, though VRAM leaks accumulate 10MB per generation cycle with speculative decoding enabled.
Best when: Fit on 16GB VRAM consumer cards (RTX 5080) for accessible self-hosting without enterprise hardware.
Tips
- Fit on 16GB VRAM consumer cards (RTX 5080) for accessible self-hosting without enterprise hardware.
- Community ports verified on DGX Spark clusters with vLLM distributed serving for stable multi-node deployment.
Watch out for
- VRAM memory leak of ~10MB per PP+TG cycle with DSpark speculative decoding causes eventual OOM; disable speculative decoding for long-running services.
- SSE endpoint intermittently stops mid-generation without closing connection, leaving streams hanging indefinitely.
GLM 5.2 ranks #26 on LMArena with Elo 1472 and runs comfortably on dual RTX 3090s at Q8 quantization, making it a proven choice for quality-conscious self-hosters with moderate hardware budgets.
Best when: Run at Q8 quantization on two RTX 3090s for quality-preserving deployment without latest-generation hardware.
Tips
- Run at Q8 quantization on two RTX 3090s for quality-preserving deployment without latest-generation hardware.
- Strong human preference ranking (#26 of 146) indicates reliable output quality for interactive applications.
Watch out for
- vLLM 0.27.0 with `--reasoning-parser glm45` returns malformed reasoning fields requiring parser patches for correct tool-calling loops.
Gemma 4 26B-A4B was the incumbent local model before Qwen3.8-27B evaluation, with the comparison noting Qwen3.8's superior benchmarks but higher bandwidth demands from its dense architecture.
Best when: Established local deployment path with known resource requirements and tooling compatibility.
Tips
- Established local deployment path with known resource requirements and tooling compatibility.
MiniMax M3 completed 3/3 real-provider results with zero failures in evaluation, indicating reliable serving behavior, though evidence focuses on API usage rather than detailed self-hosting specifics.
Best when: Validated reliable serving with zero provider failures in benchmark testing for stable production deployment.
Tips
- Validated reliable serving with zero provider failures in benchmark testing for stable production deployment.
DeepSeek V3.2 shows OOM failures in CI testing and has fused attention implementations that bypass DCP handling, creating correctness issues with tensor parallelism greater than one.
Best when: MoE fine-tuning validated at scale in AutoModel for custom training workflows.
Tips
- MoE fine-tuning validated at scale in AutoModel for custom training workflows.
Watch out for
- Fused attention bypasses DCP handling, causing correctness problems when tensor parallelism exceeds one.
- OOM failures observed in automated testing on AMD hardware.
DeepSeek V4 Pro was considered too slow at launch for practical deployment, with API documentation inconsistencies and upstream provider blocking issues affecting reliability.
Best when: Consider only after reviewing the cited caution.
Watch out for
- Launch performance characterized as too slow for practical use cases.
- Upstream provider blocking causes AuthError for all model variants, breaking API compatibility.
DeepSeek V4.1 Flash brings native visual understanding in a 552B MoE with 8B/16B active parameters.
Best when: Smallest model in new architecture family with native multimodal capabilities for vision+text workloads.
Tips
- Smallest model in new architecture family with native multimodal capabilities for vision+text workloads.
Watch out for
- Crashes on RTX PRO 6000 Blackwell (SM120) with `ValueError` during KV-cache probing, blocking latest hardware deployment.
Frequently asked
- What is the top-ranked model for Best Open-Weight LLM for Self-Hosting?
- Qwen: Qwen3.8 27B ranks first in the current evidence-weighted comparison. Achieve 59 tok/sec generation on RTX 5090 for 8K prompts, scaling reasonably to 44 tok/sec at 128K context.[1]
- What should I watch out for with Qwen: Qwen3.8 27B?
- Dense architecture means every token processes full ~17GB weights, saturating memory bandwidth and limiting throughput compared to MoE alternatives.[2]
- What is an alternative to Qwen: Qwen3.8 27B?
- DeepSeek: DeepSeek V4 Flash 0423 is the next-ranked option. Fit on 16GB VRAM consumer cards (RTX 5080) for accessible self-hosting without enterprise hardware.[3]
Sources
- 1
“The numbers I was most interested in are tucked away in a chart towards the bottom - the speed comparison of the Mac Studios v.s. a RTX 5090: Qwen3.8 27B tokens sec generation speed Prompt size 8K 64K 128K 256K RTX 5090 PC 59 51 44 n a M5 Ultra 48 39 32 24 M3 Ultra 31 23.5 20 15 A whole bunch more comparison numbers in this section: https: www.macstories.net stories m5-ultra-mac-studio-revie...”
simonw · Hacker News · Sep 21, 2026 - 2
“## Context Qwen3.8-27B (released 2026-08-14, Apache 2.0, dense 27.78B, hybrid Gated DeltaNet + vision encoder) was evaluated as a replacement for the current local model (Gemma 4 26B-A4B Q8_0). The evaluation surfaced two pre-existing weaknesses in the local LLM layer that are worth fixing on their own merits, independent of whether Qwen ever becomes the default. **Why Qwen is not being made the default:** it is dense, so every token streams all ~17 GB (Q4_K_M) of weights through the UMA. The s…”
nbramia · GitHub · Aug 15, 2026 - 3
“Running DeepSeek V4 Flash on an RTX 5080 with 16GB VRAM Under Linux/WSL2 via DS4”
peppe200175 · Hacker News · Aug 12, 2026 - 4
“# `--gdn-replay` makes qwen35 generate degenerate looping text On `fb2cc35ab`, enabling `--gdn-replay` with MTP speculative decoding makes Qwen3.8-27B collapse into a repeated phrase until `max_tokens`. The flag is EXPERIMENTAL and defaults to off, so nothing is broken for anyone who leaves it alone — but it currently looks like a large speedup while actually destroying the output, which makes it easy to adopt by mistake. ## Symptom The first ~900 characters are coherent, then it locks into the…”
tmark00 · GitHub · Sep 3, 2026 - 5
“## Summary The pinned SGLang scheduler busy-spins on an idle model and consumes roughly one CPU core. Enabling SGLang's existing `--sleep-on-idle` path removes the zero-timeout polling loop while preserving the active request path. ## Environment - NVIDIA GB10 / DGX Spark - Qwen3.8 27B NVFP4 with DFlash2 - SGLang image: `lmsysorg/sglang@sha256:f6c809a2ebdeea97a3732e8bc139a32c18c9fb00a6e2fd770d257d9f72466ed3` - SGLang build: `0.0.0.dev0+qwen38.27b.g561c8f3` - 262144 context length - FP8 E4M3 KV…”
RyderFreeman4Logos · GitHub · Aug 20, 2026 - 6
“Thanks for this port — my two DGX Sparks have served DeepSeek-V4-Flash reliably on `0.1.1`. **Caveat up front:** both checkpoints I measured are community derivatives of the official `deepseek-ai` weights, not the official releases themselves. Per the publisher only 26 `attn.wo_b.weight` tensors differ and the `mtp.*` draft tensors are byte-identical to the originals — but I have not verified that, and I have not run the official checkpoints for comparison. My questions are about vLLM version a…”
philipy-devlog · GitHub · Sep 5, 2026 - 7
“# workaround `sparkrun run @eugr/deepseek-v4-flash-vision-exp -- --enforce-eager` ## Description DeepSeek V4 fails during CUDA graph capture while SparkRun starts vLLM. ## Error This occurs during: vLLM then reports: ## Environment * Hardware: 2x NVIDIA GB10 / DGX Spark * Launcher: SparkRun * Executor: vLLM distributed * Model: DeepSeek V4 Flash * Python: 3.12 * Worker: `Worker_TP0` ## Expected behaviour CUDA graph capture should complete and vLLM should start serving the model. ## Actual behav…”
mrpmorris · GitHub · Sep 15, 2026 - 8
“### Name and Version ### Description I'm observing a reproducible VRAM memory leak when running DeepSeek V4 Flash GGUF with DSpark speculative decoding enabled. After each PP+TG (prefill + generate) cycle, VRAM usage increases by approximately **10 MB**, and this accumulation continues until the process eventually runs out of memory (OOM). This issue does **not** occur with: - Step 3.7 + DSpark - Qwen3.8-27B + DSpark Both of the above maintain stable VRAM usage across multiple cycles. ### Envir…”
wxt-coder · GitHub · Aug 16, 2026 - 9
“## Summary The DeepSeek OpenAI-compatible `/chat/completions` SSE endpoint (both `api.deepseek.com` and the `opencode-go` proxy) **intermittently stops sending chunks mid-generation without closing the connection** — typically mid-`reasoning_content` (thinking) — with no terminal `finish_reason` chunk and no `[DONE]`. The stream simply goes silent and stays open indefinitely. Observed across multiple models: - `deepseek-v4-flash` - `deepseek-v4-flash-vision-exp` This does **not** happen with ot…”
iacore · GitHub · Aug 26, 2026 - 10
“On the other hand, it used about the same tokens as GLM 5.2 and got 1 point lower score. The fact that we have a GLM 5.2-class model that can run on two 3090's comfortably at Q8 is absolutely insane. It wasn't long ago that GLM 5.2 was considered amazing for open weight models.”
2001zhaozhao · Hacker News · Aug 17, 2026 - 11
“Ranks #26 of 146 on LMArena's overall text arena (Elo 1472), based on blind human preference votes.”
LMArena text arena · Benchmark · Sep 13, 2026 - 12
“Same failure on a different stack, so it is the reasoning-parser class and not one image: GLM-5.2 (QuantTrio Int4-Int8Mix) on vLLM 0.27.0 with `--reasoning-parser glm45`, TP=4 across four DGX Sparks. Quickstart `select()` returned `[0.5, 0.5, 0.5]`; the prefilled letter is in `message.reasoning`, `content` is `None`. #6 fixes it; #7 carries the same two changes plus a logprob fallback for servers that expose neither field. With either, the quickstart gives `[0.7245, 0.3858, 0.3896]` against the…”
joesinvestments · GitHub · Aug 18, 2026 - 13
“**Standing issue — intentionally left open. Revisit periodically; close only if the model is ruled out for good.** ## What this tracks Whether [Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) should replace Gemma 4 26B-A4B as the local model, and what would have to change upstream for that to make sense. ## Why it was not adopted on release (2026-08-15) The benchmarks are excellent — GPQA Diamond 89.2, LiveCodeBench v6 90.3, Terminal-Bench 2.1 73.0, SWE-bench Pro 61.7, Apache 2.0, 262K na…”
nbramia · GitHub · Aug 15, 2026 - 14
“Implementation is on draft PR #262: https://github.com/jeremylongshore/j-rig-skill-binary-eval/pull/262 The pinned MiniMax M3 run completed 3/3 real-provider results with ground_truth=true, zero provider failures, and no credential values committed. Evidence is in 000-docs/040-AA-AACR-minimax-m3-phase3-dogfood-2026-08-02.md and evidence/minimax-m3-spotcheck-20260802/summary.json. The branch is stacked on PR #260; the Phase 3 parent remains open.”
jeremylongshore · GitHub · Aug 2, 2026 - 15
“Summary MoE fine-tuning is well supported in AutoModel and validated at scale (GLM-5, MiniMax-M2.5, DeepSeek-V3.2, Step-3.5), but MoE knowledge distillation has no counterpart, there is no MoE example under examples/llm_kd/, and the KD recipe has diverged from train_ft.py in ways that break EP paths that already exist elsewhere in the codebase. This request is for a validated, end-to-end MoE KD example covering the common combinations users are trying today. Even after the individual bugs are f…”
Bijoyroy1 · GitHub · Jul 30, 2026 - 16
“# [Bug][DCP] NVIDIA DeepSeek-V3.2 / GLM-5.2 fused attention bypasses DCP handling ## Problem The NVIDIA DeepSeek-V3.2 / GLM-5.2 model override calls its low-level fused norm/RoPE, sparse-indexer, and sparse-MLA operations directly. That bypasses DCP handling present in the corresponding generic forward paths. With DCP greater than one, this creates three correctness problems: 1. The fused norm/RoPE kernel returns when a token has no owner-local KV-cache slot before computing that rank's query R…”
foraxe · GitHub · Jul 28, 2026 - 17
“## `pr-test-amd.yml` — 60 failure(s) **Run started (UTC)**: 2026-08-02 00:29 UTC **Last scanned (UTC)**: 2026-08-03 02:48 UTC **Commits**: sglang [`12eadf8`](https://github.com/sgl-project/sglang/commit/12eadf86f12aec2e6f81a6e38b61b964a4c6b529) ### Summary **Counts**: 60 failures · 6 clusters · 44 infra/network flakes · 2 new today (DeepSeek-V3.2 OOM, Kimi-K2.5 timeout) | # | Cluster | Job (representative) | Test File | Test Function | Status | Hypothesis (confidence) | |---|---------|------|--…”
amd-bot · GitHub · Aug 3, 2026 - 18
“## S3 Card-derived packet Primary topic: `grok-46-day-one` Input mode: map-derived Archetype: practical_review Thesis: Grok 4.6 interesting, not a revolution; do not chase. DeepSeek V4 Pro too slow at launch. Evidence: e1 t=4847, e2 t=4853, e3 t=4883 CTA: kruzhok / `blog_grok_46` Exclude: transcript, chronology, Q&A, catalog, Factory, Opus fourth try, API table. Packet on disk: `/tmp/019-writer-packet.md` (card only, no transcript).”
serejaris · GitHub · Aug 13, 2026 - 19
“### Description All OpenCode Zen models return `AuthError: Request blocked by upstream provider.` — **every single model**, both paid and free. This is NOT a client-side issue. Direct provider API keys (DeepSeek, Anthropic) work perfectly. Steps to reproduce 1. Connect OpenCode Zen via `/connect` with a valid API key 2. Run any Zen model: opencode run "hola" --model opencode/deepseek-v4-pro 3. Or test the Zen API directly with curl: ```bash curl -H "Authorization: Bearer " \ -H "Content-Type: a…”
juanmah98 · GitHub · Jul 31, 2026 - 20
“Thanks for raising this, we will take a look. This looks like an issue where the upstream is not supporting the full OpenAI contract.”
hassiebp · GitHub · Sep 14, 2026 - 21
“## The weights #226 said did not exist are now on Hugging Face [#226](https://github.com/evanwtf/local-llm/issues/226) closed NOT_PLANNED because DeepSeek Flash 4.1 was an API preview with no artifact. As of 2026-09-10 the weights are published: . DeepSeek's own announcement calls it "the smallest model in our new architecture family, with native visual understanding" ( ). ## What it is - **552B backbone**, 8B activated at prefill and 16B at decode. Multimodal MoE. - **A new architecture**, not…”
evandhoffman · GitHub · Sep 10, 2026 - 22
“## Problem `DeepSeek-V4.1-Flash` fails to start on SM120 (RTX PRO 6000 Blackwell) with a `ValueError` from `_sparse_mla_sm120.py` during KV-cache-size probing (`determine_available_memory`), before any request is served. Confirmed across two independent tensor/pipeline-parallel configurations with the same result, ruling out TP degree as the cause. ## Root cause (traced on the vLLM side) `compressed_block_size` is computed as: compressed_block_size = attn_metadata.block_size // self.compress_ra…”
sixtus · GitHub · Sep 18, 2026
Rankings synthesized from community evidence and open benchmarks. See methodology. Not driven by vendor marketing.