ENPT

H2O-Lightning-4B

Last updated:

Open29–38ms$0/M input

Quadrant scores

See the full quadrant

Scored with the public quadrant rubric (maturity × capability; bubble = adoption). Revise as evidence lands.

  • Maturity5.8/10

    Apache-2.0 weights with pinned revisions, a detailed card, serve.sh on stock vLLM plus a small open shim serving /v1/systemone, and a Spaces demo. Company-run but no SLA, no hosted API, no third-party integrations found.

  • Capability8.1/10

    Official JevBench (17 Oct): Intelligence 60.0, calibration 90.0, p50 0.029 s. Choice (2–255, best up to ~16), yes/no, score, text and images. Vendor ECE 0.017 on its own image sets. Intelligence 60 trails 31B models.

  • Adoption12/100

    About 4.1k downloads and 81 likes in 15 days, a Hugging Face-built Spaces demo, a launch post with 8 likes. Backed by an established company. No integrations or production users reported.

Vendor claims

publicado (OK do Fabricio 17/10 BRT). Board audit of 17 Oct 2026: see the “Board audit” section below; where it conflicts with older board statements on this page, it wins.

publicado (OK do Fabricio 17/10 BRT).

Read this first. Three different “versions” are in play. The JevBench board measured v1.1 (HF tag v1.1). The card’s main is v1.3.1, which is not on the board. The “205/231” figure on the card is v1.2.3, a third revision, run one-way by H2O.ai. Only the board number (72.5) is independent; everything else here is UnverifiedClaim.

H2O-Lightning-4B is a 4B decision model from H2O.ai, fine-tuned from Qwen3.5-4B under Apache-2.0. It reads a record (and, since the image build, pictures that come with it) plus typed questions and returns a probability for every option in one forward pass and one output token, so the cost is input tokens only. It runs on unmodified vLLM behind a small open-source shim (h2o_lightning_shim.py) that serves POST /v1/systemone.

Specs

Attribute Value
Company H2O.ai
Weights Apache-2.0 on Hugging Face, 4.66B parameters in BF16 (the card says “4B”, Qwen3.5-4B base with its vision tower)
Repo created 2 Oct 2026; launch post on X 13 Oct 2026
Revisions main = v1.3.1 (two-way reading, recommended); v1.3.0.5; v1.2.3; v1.1 = the JevBench board’s weights
Question types noul (yes/no), choice (2–255 options, “decides best with up to about 16”), score (ordinal)
Input Text and images (JPEG/PNG via data URI in state, up to ~1 MP by default); no video. A box question exists in the image package
Serving serve.sh (stock vLLM 0.30.0 + shim) on http://127.0.0.1:8741/v1/systemone; also an OpenAI Decisions API route (/v1/decisions)
Context 40,960 tokens in the evaluated setup; over-limit inputs return HTTP 422
Latency JevBench measured p50 29 ms on-GPU (RTX 5090, loopback, no network); the card reports 38 ms client p50 on an L40S
Price Self-hosted (compute only). JevBench’s cost estimate: $0.021 per 1,000 decisions, from Qwen3.5-4B market rates and 654 mean input tokens
Demo Spaces demo built by the Hugging Face team (ZeroGPU)

Independent board: JevBench (checked 17 Oct 2026, board v1.7.46)

The result H2O.ai shows on its card is real on the board. It is the open-weights board’s official composite, the equal-weight harmonic mean of Intelligence, Calibration, Speed and Cost.

System Composite Intelligence Calibration Speed Cost $ per 1,000
H2O-Lightning-4B v1.1 (open, #1 official) 72.5 60 90 93 60 $0.021 (est.)
Jev 1.13.0 (hosted reference, unranked) 71.5 64 91 91 55 $0.032
Quyet-1.0-Large (31B) 71.4 73 90 87 51 $0.045 (est.)

Check: 4 / (1/60.0 + 1/90.0 + 1/92.6 + 1/60.3) = 72.5, matching the card’s table. Three things to read with it:

  • Composite is not Capability. On the board’s Capability headline (the mean of Intelligence and Calibration, no speed or cost) H2O-Lightning-4B scores 75.0 and ranks 9th in the open-weights table. The 31B models lead on raw Intelligence (Quyet 73, decisio v0.8.0 70, against 60 here). The 4B wins on speed and cost.
  • One draw, one revision. The row is the v1.1 weights, measured on the v1.6 sealed pool with a labelled cost estimate. The card’s main (v1.3.1) has no board result.
  • Image board. On ImageJevBench v0.3.0 the open-weights “H2O-Lightning-4B (image)” row has Capability 60.4 (Intelligence 44.9, Calibration 76.0), 11th by Capability, official #10, behind several hosted systems (kettei-fat 75.9, Sage 1.4 + images 75.8).

Not on Hugging Face’s Decision Index 0.3 board. H2O.ai’s own run of the public DI 0.3 kit gives v1.3.1 text 57.86 (two-way) / vision 67.13 — a self-run, not a leaderboard result (Jev 1.13 is 60.11 on the board; the two are not comparable).

What the card claims (UnverifiedClaim)

  • 205/231 on JevBench’s public items is the v1.2.3 revision, one-way, on an RTX PRO 4500. The shipped main (v1.3.1) scores 203/231 two-way and 198/231 one-way, and the card’s paired tests find no significant difference between v1.3.1 and v1.2.3 (p 0.75 two-way, 0.12 one-way).
  • Image sets: 87.4%, ECE 0.017 over 1,877 questions the authors built from public datasets’ annotations. Weak spots they list: counting many small objects (63%), reading values off line charts, Rico UI elements (65%).
  • Routing table: at a 0.9 confidence threshold, 53% of questions are escalated and 0.7% slip through confidently wrong, measured by H2O.ai on 3,474 public third-party questions (v1.2.3, one-way).
  • The X post says it “beat Microsoft-Decision-1 and OpenAI’s Decisions API on JevBench” (true on composite: 72.5 against 69.1 and 62.5 on the API board, which the board keeps separate from the open-weights one) and is “faster than Jev and nearly as capable” (72.5 vs 71.5 composite, Intelligence 60 vs 64, p50 29 ms on-GPU vs 0.24 s end to end for Jev 1.13).

Fit / anti-fit

Fit when you need: a small open decision model you can run on one GPU with stock vLLM, in the /v1/systemone shape, with images, and a board result that doesn’t rely on the vendor.

Anti-fit when you need: the highest raw accuracy (31B models score higher on Intelligence), a hosted API with an SLA, many-option choice questions (best up to ~16), or video.

Limits

  • Evaluated mostly in English. Probabilities are calibrated for decision questions, not open-ended text.
  • The board measures v1.1; the weights on main are different and unmeasured by the board.
  • Latency is measured on-GPU over loopback; the board’s own network-inclusive estimate is 0.21 s.
  • Not on the Decision Index 0.3 board or the Artificial Analysis index.

Why it is in the catalog

Open Apache-2.0 weights, a typed /v1/systemone server, a documented card, and an independent board result (#1 open-weights composite). Same inclusion logic as Quyet and Span-02.

Score working (17 Oct 2026)

  • Maturity = availability 8 × 0.30 + docs 8 × 0.25 + integrations 3 × 0.25 + support 3 × 0.20 = 2.40 + 2.00 + 0.75 + 0.60 = 5.75 → 5.8
  • Capability = types 10 × 0.30 + calibration 7 × 0.25 + latency 7 × 0.25 + accuracy 8 × 0.20 = 3.00 + 1.75 + 1.75 + 1.60 = 8.10 → 8.1
  • Adoption = engagement 25 × 0.35 + ecosystem 8 × 0.35 + production 2 × 0.30 = 8.75 + 2.80 + 0.60 = 12.15 → 12

Availability 8: Apache-2.0 weights with pinned revisions (as Lumma-fev); no hosted API, so not 9. Docs 8: a long, reproducible card with serving, calibration and failure modes, but no separate docs site. Integrations 3: /v1/systemone and an OpenAI Decisions route served locally (compatibility = 3). Support 3: company-run, no SLA.

Types 10: choice, yes/no and score, plus images. Calibration 7: the board’s axis is strong (90.0) and the card shows a method and ECE, but the ECE is on H2O.ai’s own sets (precedent: Quyet, Sage). Latency 7, not 8: 29 ms is board-measured but self-hosted loopback with no p99 (the linne-oak precedent); 8 would give capability 8.4. Accuracy 8: independent board result (GLiDE / d1-ultra rule), not 9 because Intelligence 60 is below Jev’s 63.6.

Board audit (17 Oct 2026)

publicado (OK do Fabricio 17/10 BRT). Checked against the official boards on 17 Oct 2026: Decision Index 0.3.2 (data of 16 Oct, 19:00 UTC), JevBench and ImageJevBench live. The boards are separate sources and are never averaged. Older board statements and rankings further down this page are superseded where they conflict with this section.

Board Row Result
JevBench open weights (live 17 Oct) H2O-Lightning-4B v1.1 Capability 75.0; Intelligence 60.0; calibration 90.0; p50 0.029 s; $0.021/1k
ImageJevBench v0.3.0 H2O-Lightning-4B Capability 60.4; Intelligence 44.9; calibration 76.0; p50 0.57 s; $0.078/1k

Board-status statements on this page to recheck against the table (may be superseded):

  • “The card’s main is v1.3.1, which is not on the board.”
  • “The card’s main (v1.3.1) has no board result.”
  • “Not on the Decision Index 0.3 board or the Artificial Analysis index.”

Score update (audit, 17 Oct 2026). Capability 8.1 → 8.1 (unchanged) = types × 0.30 + calibration × 0.25 + latency × 0.25 + accuracy × 0.20 = 10 × 0.30 + 7 × 0.25 + 7 × 0.25 + 8 × 0.20 = 8.10 → 8.1. Types, calibration and latency come from the earlier working (10 / 7 / 7); only accuracy is re-derived from the board (8; was 8): JevBench Intelligence 60.0 ÷ 63.6 × 8 = 7.5, half-up. Maturity and adoption unchanged.

Hugging Face on 17 Oct (h2oai/h2o-lightning-4b): 4,142 downloads, 81 likes, licence tag apache-2.0, last modified 2026-10-17.

H2O Lightning 31B v1.1 on the Decision Index (29 Oct 2026)

publicado (OK do Fabricio 29/10 BRT).

The larger sibling, h2oai/h2o-lightning-31b (full fine-tune of Gemma 4 31B; gated repo, licence tag “other”, last modified 29 Oct), entered the Decision Index 0.4.1 as v1.1 on 28 Oct (08:52 BRT): Full 68.64, #7 of 243, public 67.75, board ECE 0.074, median 100 ms, p95 702 ms. On JevBench, the 28 Oct changelog (v1.7.13) also adds H2O-Lightning-4B v1.4.1 on the new 27 Oct draw, shown unranked for now.

Scores unchanged (5.8 / 8.1 / 12): this page is scored on the 4B, whose accuracy is already 8. The gated 31B is not scored separately.