oMLX 0.6.3rc1

This is the release candidate for oMLX 0.6.3. After a short testing period, this release will be followed by either rc2 or the final release. Thank you, as always, to everyone who has patiently waited for this release!

This release introduces a faster Qwen ANE split tuner, extends ANE prefill to additional quantization formats, and adds DFlash 2 runtime support. It also substantially improves distributed-cluster reliability, restores prefix-cache reuse in long Claude Code sessions, reports Responses API truncation correctly, and adds bit-exact Laguna decode optimizations.

Qwen ANE Tuner V2 and Wider Quantization Support

• Made the ANE tuner much faster. Instead of preparing and testing many complete model configurations, the tuner first measures a few representative layers, predicts the best ANE/GPU split, and then verifies it with the full model. This reduces full-model setup runs from 11 to at most 3 while still basing the final recommendation on real prompt performance. By @onthehub97 in #2891.

• Added affine Q5, Q6, and Q8 ANE prefill support. Eligible Qwen3.5/3.6/3.8 MLP and GDN projections with group sizes 64 or 128 can now use the hybrid ANE/GPU path. Q4 behavior remains unchanged. By @GordoAR and @onthehub97 in #2833 and #2889.

• Improved ANE prefill for longer prompts. Large prompt chunks can now be divided into blocks that fit the compiled ANE program, while any remaining tokens continue on the GPU. This allows the server to keep efficient prompt chunk sizes instead of shrinking every chunk to the ANE block size. #2890.

• Made ANE tuning results easier to trust and diagnose. The tuner no longer recommends a configuration when the ANE was compiled but never actually used. Benchmark logs also show how much work ran on the ANE and GPU, making it easier to understand why a candidate succeeded or failed. By @beaglemoo in #2829.

Local 2,048-token Qwen3.8-27B measurements on M3 Ultra reported:

These are local serial measurements. Performance and the best split depend on the Mac, checkpoint, quantization, and prompt shape.

DFlash 2 Support

• Added end-to-end DFlash 2 support. Compatible checkpoints can use checkpoint-derived sliding windows and per-model block-size overrides. DFlash now matches the regular batched engine's sampling behavior, including min_p and seeded requests. The default sink size is now 0, SSD prefix-cache writes remain functional with that setting, and versioned DFlash2 checkpoints appear in the draft-model picker. Initial support by @liang2kl in #2840, finalized in #2850, with picker support by @williamxie1989 in #2879.

Measured on an M3 Ultra with Qwen3.8-27B-oQ4e-mtp, the z-lab/Qwen3.8-27B-DFlash2 draft, temperature 0.7, and a 128-token generation tail:

Exact repeated prompts continue to use the in-memory prefix cache. SSD snapshot writes also work with the new sink-size default.

Distributed Serving and Cluster Reliability

• Added a persistent cluster incident feed. Activation, staging, and peer-health failures are recorded by the server and remain visible after dashboard refreshes. Incidents have stable sequence numbers, machine-readable guidance codes, severity, source, and explicit dismissal state. By @alytaphoenix in #2866.

• Improved two-Mac activation and planning reliability. Dashboard polling no longer clears a valid plan or changes tensor topology during activation. Cross-user clusters resolve model paths in each peer's own home directory, and unsupported VLM pipeline plans are no longer offered. By @alytaphoenix in #2819.

• Stopped remote rank processes more reliably. Teardown now validates each peer's deployment marker and process identity, sends SIGTERM, escalates to SIGKILL when necessary, and reports whether the rank was actually reaped. Failure markers remain available as diagnostic evidence. By @xunlinkx in #2722.

• Reduced repeated planner work. Cluster autoconfiguration caches model layouts and shard metadata until the relevant files change, avoiding repeated safetensors scans and monkey-patch installation during dashboard polling. By @alytaphoenix in #2883.

• Added tensor-parallel support for quantized Nemotron-H at TP=2. Uneven but group-aligned routed-expert shards, corrected quantized Mamba slicing, and stricter planner checks allow compatible Nemotron-H checkpoints to load and generate across two Macs. By @alytaphoenix in #2844.

• Improved distributed request compatibility and failure handling. Unsupported reasoning_effort values are retried through the same bounded fallback used by local engines. Requests also check rank health before streaming begins, allowing a dead or partially failed cluster to return HTTP 503 instead of an empty HTTP 200 response. By @alytaphoenix in #2869.

• Preserved configured memory-guard settings during remote admission. Cluster probes now initialize local settings before falling back to defaults, so custom memory tiers and ceilings are honored. By @xunlinkx in #2835.

APIs, Agent Workflows, and Admin Fixes

• Restored prefix-cache reuse in long Claude Code sessions. The Anthropic adapter now strips changing <total_tokens> budget markers from system and developer content while preserving user-quoted markers. On the reported 112K-token workload, reused tokens increased from 4,096 to 112,640 and turn latency fell from minutes to seconds. By @Pietroski in #2882.

• Reported Responses API truncation correctly. Requests stopped by max_output_tokens now return status: "incomplete" with incomplete_details.reason: "max_output_tokens". Streaming responses terminate with response.incomplete instead of response.completed, allowing clients to continue truncated reasoning or generation. By @mvdbos in #2817.

• Fixed forwarded arguments in omlx launch. The oMLX forwarding separator is removed without discarding a second separator intended for the launched tool, so flags passed to Claude Code and similar CLIs remain flags instead of becoming prompt text. By @fparrav in #2818.

• Added a token-budget override for external accuracy benchmarks. The optional value acts as a floor over each benchmark's default, preventing thinking models from being truncated before producing an answer without reducing larger benchmark-native budgets. By @williamxie1989 in #2863.

• Fixed default-model state after unsetting a model. Removing the default flag now clears the server-side pointer and updates the dashboard immediately instead of leaving requests routed to the old default until restart. By @alytaphoenix in #2884.

Laguna Decode Performance

• Added default-on, bit-exact compiled fusions for Laguna XS 2.1. Compiled router renormalization, expert combination, per-head softplus gating, and SiLU gate/up work improved single-token decode by 3.96% on an M4 Max, from 66.4 to 69.1 tok/s. The verified greedy trajectory remained token-identical. By @0xClandestine in #2827.

• Experimental routed and shared-expert fusion banks remain opt-in because they were neutral or slower on the tested MLX runtime.

Upgrade Notes

• This is a release candidate. Please report regressions before the final 0.6.3 release.

• DFlash draft sink size now defaults to 0 when unset. Existing explicit non-negative values are preserved.

• Qwen ANE/GPU prefill remains experimental, opt-in, and dependent on private Apple runtime interfaces and the oMLX native custom kernels.

• Keep the configured ANE sequence_length at or below the scheduler's delivered prefill chunk width. With prefix caching enabled, 2,048 remains the safe default because cache boundaries split delivered chunks at 2,048 tokens.

• Streaming Responses API clients should handle response.incomplete as the terminal event when generation reaches max_output_tokens.

• Distributed inference remains experimental and disabled by default.

• No manual settings migration is otherwise required.

Thanks to @liang2kl, @GordoAR, @onthehub97, @beaglemoo, @xunlinkx, @alytaphoenix, @williamxie1989, @Pietroski, @fparrav, @mvdbos, and @0xClandestine for their contributions.

Full Changelog: v0.6.2...v0.6.3rc1

oMLX 0.6.2

oMLX 0.6.2 combines focused hotfixes for regressions and edge cases uncovered in 0.6.1 with a new built-in ANE/GPU split tuner. It fixes the TurboQuant KV + Lightning MTP verification crash, restores exact GDN SSD-cache snapshots as the default, and addresses several distributed, cluster, MCP, memory, and admin issues.

Thanks to an incredible contribution from @onthehub97, oMLX can now benchmark ANE/GPU splits directly on each Mac instead of relying on ratios tuned for another machine. Dedicated NAX GPU suffix kernels also allow M5-family Macs to find and retain measurable ANE/GPU prefill gains.

Built-in ANE Split Tuner and M5 NAX Support

• Added a built-in ANE/GPU split tuner. The macOS app and web dashboard can now benchmark a GPU-only baseline, several MLP-only splits, and combined MLP/GDN splits directly on the current Mac. Candidate settings remain temporary until the user explicitly applies the result. By @onthehub97 in #2814.

• Added NAX GPU suffix kernels for M5-family Macs. Supported Qwen3.5/3.6/3.8 quantized layers use dedicated NAX QMM kernels for the GPU portion of hybrid ANE/GPU prefill. This resolves the regression seen when early M5 testing used split ratios and GPU kernels tuned for previous generations.

• Removed the blanket M5 ANE disable. Experimental ANE prefill remains disabled by default and must still be enabled per model, but M5 systems can now use the tuner to determine whether a local ANE/GPU split outperforms GPU-only execution.

• Kept safe runtime fallbacks. If NAX support or its metallib is unavailable, oMLX falls back to the existing Metal path. OMLX_QWEN35_QMM_NAX=0 remains available as a NAX kill switch.

Community testing on M5 reported approximately a 5% improvement in a practical benchmark, with the best tuner trial approaching 1,000 prompt tokens/s. Results depend on the chip, model, quantization, and prompt shape, so the tuner recommends GPU-only execution when the best ANE candidate is less than 1% faster.

ANE program loading was also hardened for single-die chips by retrying with smaller program banks when the preferred layout exceeds the device window. Benchmark traces now report the actual compiled MLP and GDN layer counts for easier validation.

Qwen and Cache Hotfixes

• Fixed Lightning MTP generation crashing with TurboQuant KV enabled. The optimized Qwen verification-attention path now safely rejects TurboQuant proxy objects it cannot process and falls back to the compatible attention path. This fixes the regression reported in #2778 and #2780. By @aaramos in #2782.

• Restored exact GDN SSD-cache snapshots as the default. GDN sidecar state now uses FP32 unless reduced precision is explicitly selected. Existing settings carrying the v0.6.0 lossy default are migrated back to FP32, preventing reduced-precision cache restoration from unexpectedly changing greedy outputs or breaking bit-level reproducibility. Reported in #2775.

• Fixed boundary-snapshot arrays remaining alive through a Python closure cycle. The scheduler now traverses snapshot state without creating a recursive closure that could retain large Metal-backed arrays. This prevents latent memory growth for reallocating cache implementations, including affected TurboQuant workflows. By @hojin12312 in #2809.

• Fixed VLM MTP profile conflicts during profile application. Applying output-shaping settings such as presence_penalty now preserves the requested settings and disables the incompatible speed-only VLM MTP toggle, matching request-time behavior. Remaining invalid combinations return an actionable HTTP 400 instead of an internal server error. By @jakeuj in #2806.

Distributed Serving and Cluster Management

• Added distributed thinking_budget enforcement. Rank workers now apply the same per-request thinking-budget processor used by single-machine inference, allowing distributed Qwen deployments to place a hard bound on reasoning without disabling it entirely. By @xunlinkx in #2731.

• Made the distributed request read timeout configurable. Long prefills and cold model reloads can use OMLX_DISTRIBUTED_REQUEST_READ_TIMEOUT to override the existing 300-second default. Invalid, non-finite, or non-positive values now fail with a clear configuration error. By @xunlinkx in #2714.

• Stopped idle cluster polling from repeatedly running the planner. The cluster dashboard still creates its initial plan and responds to explicit model or control changes, but no longer posts /plan every polling cycle. By @xunlinkx in #2721.

• Preserved the selected role of the local coordinator. Dedicated coordinator Macs can now remain configured as headless instead of being reset to the workstation role and unnecessarily reserving 32 GiB. By @xunlinkx in #2723.

• Improved compatibility reporting for older or uninitialized workers. Cluster preflight now handles peers predating omlx._version, distinguishes a missing worker runtime from an actual version mismatch, and keeps unverified workers non-ready until compatibility has been established. By @hellodk in #2737 and #2738.

MCP, Memory, and Admin Fixes

• Restored remote MCP Streamable HTTP connections with MCP SDK 2.x. The client now follows the SDK's reader/writer return contract, and package constraints consistently require MCP 2.x. By @nnhanndakara in #2776, addressing #2520.

• Fixed wired-memory limit recommendations on 128 GiB Macs. Backend warnings, the admin API, and the displayed iogpu.wired_limit_mb command now use the same safe whole-MiB value, eliminating an impossible rounding loop. By @jakeuj in #2799.

Upgrade Notes

• Qwen ANE/GPU prefill remains experimental, opt-in, and dependent on private Apple runtime interfaces. It requires the oMLX native custom kernels and may increase model loading time and memory use.

• The ANE split tuner takes several minutes because it performs full prewarm and end-to-end measurements across multiple temporary configurations. Saved model settings do not change until the recommended result is explicitly applied.

• Existing GDN SSD-cache settings using the legacy gdn_sidecar_state_dtype key are reset to exact FP32 storage. Users who intentionally accept reduced precision for smaller SSD snapshots can select it again in the updated settings UI.

• The distributed request timeout remains 300 seconds unless OMLX_DISTRIBUTED_REQUEST_READ_TIMEOUT is configured.

Thanks to @onthehub97, @xunlinkx, @hellodk, @nnhanndakara, @jakeuj, @hojin12312, and @aaramos for their contributions, and to everyone who reported and validated the 0.6.1 issues.

Full Changelog: v0.6.1...v0.6.2

oMLX 0.6.1

oMLX 0.6.1 focuses on Qwen3.8 performance and fixes several compatibility regressions found after 0.6.0. It adds opt-in dual-ANE/GPU prefill on M3 Ultra (+18.9% throughput at 32K context), accelerates Lightning MTP generation (+34.0% decode throughput at 16K context), and restores reasoning-effort compatibility, Qwen vision loading, model discovery, and tool-heavy prefix-cache reuse.

Experimental Qwen3.8 Dual-ANE/GPU Prefill

• Added experimental prompt processing across both ANEs and the GPU. Eligible fixed-size Qwen3.8 MLP and GDN prefill blocks split work across two physical ANEs and Metal, while decode, verification, residual chunks, and unsupported layers continue using the existing GPU path. The feature supports compatible AWQ and oQ4e layouts and is disabled by default. By @onthehub97 in #2756, extended and hardened in #2760.

Measured on an M3 Ultra with Qwen3.8-27B-oQ4e-mtp, Lightning MTP disabled, a 2,048-token ANE block, and a 128-token generation tail:

Longer prompts benefit more because they contain more complete fixed-size blocks. The 4K case contains only one 2,048-token ANE block followed by a 2,047-token GPU tail.

This path uses private Apple runtime interfaces and approximate INT8 weights. On the reference system, enabling it increased peak memory by about 4.15 GB and model loading from 3.35 seconds to approximately 27–29 seconds. Decode remains on the GPU, so this is a prompt-processing optimization rather than a token-generation acceleration.

Qwen3.8 Lightning MTP and Serving Performance

• Sped up Lightning MTP verification and Qwen3.8 prefill. Fused target-verification attention and GDN prework reduce dispatch overhead, while high-memory systems use larger prefill chunks when memory permits. #2751.

Measured on an M3 Ultra with Qwen3.8-27B-oQ4e-mtp, Lightning MTP enabled, and a 128-token generation tail:

• Improved concurrent Qwen3.8 VLM throughput. When several requests are ready together, oMLX now prefers ordinary batching instead of mixing Lightning MTP and batched decoding on the same model. Batch-4 throughput improved from 69.34 to 108.14 tok/s for greedy decoding and from 64.68 to 103.79 tok/s for sampled decoding, while uncontended single-request Lightning MTP remains available. By @DiscoStew6082 in #2752.

• Fused Qwen MoE router top-k selection. The optimized Metal path improved Qwen3.6-35B-A3B decode throughput by 3.9% at 4K context and 2.5% at 16K in the reported workload. #2755.

Qwen3.8 VLM and Model Loading

• Fixed affected Qwen3.8 checkpoints falling back to text-only inference. Channels-first vision patch-embedding weights are normalized during MLX checkpoint loading, preserving full vision support and Lightning MTP. By @frank-beans in #2754.

• Fixed Qwen VLM checkpoints with root mtp.* weights. These weights are now mapped to the attached language-model MTP module instead of failing strict VLM loading and silently disabling vision. Reported in #2732.

• Restored intentionally filtered Hugging Face cache models to discovery. Auxiliary safetensors omitted by filtered downloads no longer cause otherwise loadable models to disappear from the model list. Reported in #2742.

Reasoning and Agent Compatibility

• Normalized reasoning_effort across model chat templates. Common client values are retried with compatible aliases or the model's native default instead of returning HTTP 400 when a template uses a different effort vocabulary. This fixes Qwen3.8 clients sending values such as high. Reported in #2740.

• Added explicit DeepSeek V4 reasoning-effort aliases. OpenAI- and Hermes-style values such as minimal, medium, xhigh, and ultra now map to the closest supported DeepSeek level. By @jonathan308 in #2724.

• Preserved prefix-cache reuse across tool-adjacent system messages. Claude Code system reminders that follow tool results can remain in place when the model template safely supports them, avoiding repeated large prefills in long tool-heavy sessions. Template-specific probing keeps the fallback safe for incompatible templates. By @q-p in #2753, addressing #2483.

Distributed Serving and Benchmarks

• Fixed false distributed-version mismatches. Cluster probes now compare local MLX and MLX-LM versions using the same source reported by the peer, so matching editable or nightly installations no longer fail enrollment because module and package metadata differ. By @hellodk in #2758.

• Improved community intelligence benchmark identity. Uploads now carry the organization-qualified model repository separately from the display name, allowing omlx.ai to distinguish checkpoints with the same leaf name without cluttering the model-name column.

Upgrade Notes

• Qwen ANE prefill remains experimental and disabled by default. It requires compatible native custom kernels and has currently been validated on M3 Ultra.

• Enabling or changing Qwen ANE settings reloads the model because fixed-shape ANE programs are compiled eagerly.

• Existing configurations remain compatible, and no manual settings migration is required.

Thanks to @onthehub97, @DiscoStew6082, @frank-beans, @q-p, @hellodk, and @jonathan308 for their contributions.

Full Changelog: v0.6.0...v0.6.1

oMLX 0.6.0

Compared with 0.6.0rc1, the final release adds community intelligence benchmark publishing and broader Qwen3.8 and Gemma 4 support while improving reasoning controls, embeddings, model discovery, and serving stability.

oMLX 0.6.0 introduces experimental distributed serving, more responsive concurrent inference, lower long-context memory use, new model families, built-in web search, speech-to-text, and community intelligence benchmark publishing.

Distributed Serving

• Added experimental distributed serving across Macs. Models can use tensor or pipeline parallelism with capability-aware planning, selective shard staging, memory guards, and one-click activation. Qwen3.6-27B reached 28.6 tok/s across two Macs versus 16.1 tok/s on one, while a 225 GB MiniMax-M3 checkpoint loaded across 128 GB and 256 GB Macs. By @ashhart in #2423.

• Added heterogeneous Metal and CUDA model pools. Apple Silicon and NVIDIA workers can contribute to one logical pool with memory-aware layer placement, worker enrollment, ConnectX discovery, and NCCL verification. By @ashhart in #2591.

• Added SSD-backed prompt reuse for distributed ranks. Ranks restore only cache prefixes available across the cluster, with incremental storage that stays linear as context grows. #2620.

Distributed serving is included in the DMG but remains experimental and disabled by default. Enable it from Web Dashboard > Global Settings > Advanced > Distributed Inference, then save and restart oMLX.

Community Intelligence Benchmarks

• Intelligence benchmark results can now be published to omlx.ai. Eligible local runs upload a score summary and compressed per-question answers without uploading question text. Small or external-endpoint runs remain private, and both the web dashboard and macOS app show upload status and link to My Submissions. #2693.

Serving Responsiveness and Long-Context Memory

• Decode throughput during concurrent prefill improved by 1.6x to 43x. Prefill now yields GPU time to active decodes and adapts chunk size to a target stall time, while solo prefill performance remains unchanged. #2633.

• Mixed CacheList prefix storage is now linear instead of quadratic. The affected Inkling workload dropped from 282.7 GB at 84K tokens to 15.3 GB at 94K tokens while preserving byte-identical restores. By @Pietroski in #2550.

• GDN recurrent state now uses bounded SSD sidecars by default. The new policy keeps recurrent state separate from ordinary KV storage, with RHT-INT16 reducing storage by 1.93x versus FP32 while restoring in FP32. By @hojin12312 in #2569 and #2644.

• Memory pressure now reclaims MLX buffers before shrinking the hot cache. This avoids repeatedly discarding reusable prompt state while retaining immediate emergency-pressure enforcement. Reported in #2581, fixed by @PhilipJohnBasile in #2635.

Model Support and Performance

• Expanded Qwen3.8 support. oMLX now loads blockwise FP8, embedded MTP, and the mixed ModelOpt NVFP4 unsloth/Qwen3.8-27B-NVFP4 checkpoint directly. NAX prompt throughput for common group-size-128 checkpoints improved from 513 to 925 tok/s in the reported workload. By @scaryrawr, @LKyaha, and @True2456 in #2653, #2659, and #2657.

• DeepSeek V4 prefill is 23–28% faster across tested context lengths. Optimized attention, cache, indexer, and scheduler paths improve prompt throughput, while native handling of unaligned tails reduced peak memory from 142.255 GiB to 105.059 GiB in a 347K-token test. By @DiscoStew6082 and @jonathan308 in #2559, #2562, #2563, #2568, and #2674.

• Added Lightning MTP support for unified Gemma 4 checkpoints. Embedded assistant configurations are preserved, and depth-8 verification now restores rotating caches correctly after rejected drafts. By @djr747 and @adamw3455 in #2683 and #2690.

• Added Ling 3.0 Flash support. FP8, mixed FP4/FP8, oQ/oQe conversion, batching, and SSD prefix-cache restores are supported. The trained SwiGLU clamp improved measured HumanEval accuracy from 71.3% to 88.4%. By @scaryrawr in #2526, #2534, and #2571.

• Added Meta Muse Glimmer 30B VLM support with DFlash. Text, vision, reasoning, ATEM tool calls, prefix caching, and oQ quantization are supported, with about 1.4x decode speedup on the tested oQ4 pair. #2586, #2587, and #2643.

• Added complete Jina Reranker v3.5 support. Projector loading, sliding-window attention, dual matching, block fusion, and reference-score parity are included. By @damascoemi in #2449 and #2494.

• Sentence-transformers checkpoints now honor declared pooling metadata. CLS, last-token, and mean pooling are selected from the checkpoint instead of silently returning incompatible embeddings. By @cedricamram in #2594.

Chat, APIs, Reasoning, and Tool Calling

• Added built-in web search to admin chat. Models can use web_search and fetch_url through DDGS, DuckDuckGo, Brave, or SearXNG, with source cards and redirect-safe URL fetching. External API clients are unchanged. #2596.

• Added file and realtime speech-to-text. Compatible Whisper, Voxtral, and Qwen3-ASR models support file transcription and live microphone input with correct segmentation, language detection, and CJK streaming. #2603.

• Improved reasoning output and controls. Responses API streams prompt-opened thinking through dedicated reasoning events, while the API, web dashboard, and macOS app now preserve free-form or numeric reasoning-effort values such as xhigh and 0.9. #2623, #2675.

macOS App and Updates

• Reworked the self-updater to make bundle replacement crash-safe. Atomic bundle exchange prevents interrupted updates from leaving an incomplete or damaged app. This addresses the macOS 27 failures reported in #2522.

Affected macOS 27 users upgrading from a build older than 0.6.0rc1 should install 0.6.0 manually from the DMG once. The new updater handles subsequent releases safely.

Miscellaneous

• Improved pressure handling, DeepSeek V4 prefill memory estimates, cache cleanup, hot-cache statistics, repeated aborts, and active-engine unload safety. #2512, #2639.

• Fixed DeepSeek V4 generation anchors, tool-turn round trips, parser edge cases, and thinking-mode benchmark extraction. #2510, #2513, #2661.

• Fixed NVFP4 oQ sensitivity analysis, incomplete sharded-model discovery, default macOS cluster node names, and Homebrew installation guidance. #2433, #2432, #2679, #2694.

• Improved accessibility, Chinese typography, localization, network binding, Dock behavior, and Codex/Claude launcher integration. #2542, #2671, #2480.

• Reorganized Global Settings, refreshed model presets, added MCP exposure and chat tool-round controls, honored web-search result limits, and surfaced unenforced Responses API formats. #2560, #2699, #2696, #2702.

• Improved cluster runtime discovery, mixed Python minor-version support, version status reporting, and macOS server restart recovery. #2697, #2704, #2705.

Upgrade Notes

• Distributed inference remains experimental and disabled by default. Enable it under Global Settings > Advanced, save, and restart oMLX.

• Decode fairness is enabled by default. Set decode_fairness=false to restore the previous concurrent-prefill behavior.

• GDN storage defaults to auto with RHT-INT16. Existing explicit storage and FP32 settings are preserved.

• Legacy mixed-CacheList SSD blocks are invalidated automatically because the new linear format is incompatible.

• DeepSeek V4 temperature-0 output may differ at near-tied token choices because optimized kernels change floating-point reduction order.

• Users running 0.6.0rc1 can upgrade directly. No manual settings migration is otherwise required.

New Contributors

Thank you to @Maxim-Mazurok, @damascoemi, @jetztmalehrlich, @Pietroski, @studioburnside, @tannerdsilva, @LXD-8, @luziyi, @DiscoStew6082, @fxd0h, @mvdbos, @ashhart, @PhilipJohnBasile, @mkmsyk, @nazerim, @leepokai, @dawgflymd, @adamw3455, @djr747, @LKyaha, @cedricamram, @cgs4github, and @hellodk for their first contributions in this release line.

Full Changelog: v0.5.7...v0.6.0

oMLX 0.6.0rc1

This is the first release candidate for oMLX 0.6.0. It rolls up every change since 0.5.7, including the 0.5.8 development releases and 0.6.0.dev1.

This release introduces experimental distributed serving, keeps active generation responsive during concurrent prefills, substantially improves long-context cache and memory behavior, adds several new model families, and expands the admin chat with built-in web search and speech-to-text.

Distributed Serving

• Added experimental distributed serving across Macs. Models can be split with tensor or pipeline parallelism, with capability-aware planning, selective shard staging, per-rank memory guards, liveness supervision, and one-click activation from the Cluster dashboard. Qwen3.6-27B reached 28.6 tok/s across two Macs versus 16.1 tok/s on one Mac with byte-identical output. A 225 GB MiniMax-M3 checkpoint that could not fit either machine alone also loaded across 128 GB and 256 GB Macs and passed its distributed canary. By @ashhart in #2423.

• Added heterogeneous Metal and CUDA model pools. Apple Silicon and NVIDIA CUDA workers can contribute to one logical model-memory pool, with memory-aware contiguous-layer placement, short-lived worker enrollment, ConnectX discovery, and NCCL verification. The current compatibility path keeps physical workers in the outer MLX Ring; the hierarchical Ring-to-NCCL gateway remains future work. By @ashhart in #2591.

• Added SSD-backed prompt reuse for distributed ranks. Each rank stores a process-lifetime chain of cache-boundary snapshots and restores only prefixes available on every rank. Incremental segments keep storage linear: a 12K-token GLM-5.2 chain used 1.17 GB instead of roughly 6.5 GB with cumulative copies. #2620.

Distributed serving is included in the DMG but remains experimental and disabled by default. Enable it from Web Dashboard > Global Settings > Advanced > Distributed Inference, then save the settings and restart oMLX.

Serving Responsiveness and Long-Context Memory

• Active decodes now remain responsive during concurrent prefill. Prefill yields GPU time to running decodes, including streams in another engine, and sizes contended chunks by a target stall time. The feature is enabled by default and can be changed live. Across the tested workloads, overlap decode improved by about 1.6x to 43x; contended prefill took 1.15–2.0x longer while solo prefill remained unchanged. #2633.

• Mixed CacheList prefix storage is now linear instead of quadratic. Sliceable KV members are stored per block while small boundary states remain attached to their checkpoints. On the affected Inkling workload, the legacy path used 282.7 GB for an 84K-token session; the new layout used 15.3 GB for 94K tokens and restored byte-identical output. Incompatible legacy blocks are invalidated automatically on first load. By @Pietroski in #2550.

• GDN recurrent state now uses bounded SSD sidecars by default when the SSD cache is available. The new auto | ssd_sidecar | embedded policy separates recurrent state from ordinary KV storage. RHT-INT16 is the new default sidecar codec, measuring 1.93x less storage than FP32 with 0.002477% mean relative L2 error; restored state still runs in FP32. FP32, BF16, INT8, and RHT-INT8 remain selectable. Existing explicit legacy settings and FP32 deployments are preserved. By @hojin12312 in #2569 and #2644.

• Memory pressure now reclaims the MLX buffer pool before shrinking the hot cache. Reclaimable Metal buffers no longer cause reusable prompt state to be repeatedly discarded. Emergency pressure still enforces immediately, while ordinary hard pressure receives a bounded reclaim grace period. Reported in #2581, fixed by @PhilipJohnBasile in #2635.

• Adaptive prefill admission now accounts for recently reclaimed memory, waits briefly for asynchronous cache cleanup before rejecting long requests, and uses DeepSeek V4’s actual hybrid cache layout instead of the generic KV estimate. Hot-cache promotion failures are also exposed through runtime cache statistics.

Model Support and Performance

• Added Qwen3.8-27B FP8 support and official reasoning levels. Blockwise FP8 checkpoints and embedded MTP tensors load through the Qwen3.5-family compatibility path. reasoning_effort in Chat Completions and reasoning.effort in Responses now reach the model without overriding explicit template arguments. By @scaryrawr in #2653.

• Restored Qwen3.8/Qwen3.5 group-size-128 prefill performance on NAX hardware. Common four-bit checkpoints no longer route through a slower classic Metal fallback when stock MLX can use NAX. On the reported Qwen3.8-27B workload, prompt throughput improved from 513 to 925 tok/s and end-to-end time fell from 10.4 to 6.8 seconds. Pre-NAX Macs retain the existing optimized path. By @True2456 in #2657.

• Added Ling 3.0 Flash support. This includes FP8, official mixed FP4/FP8 checkpoints, oQ/oQe conversion, variable-length batching, and SSD prefix-cache restores. Applying the trained per-layer SwiGLU clamp improved measured HumanEval accuracy from 71.3% to 88.4%. Original implementation and FP4 support by @scaryrawr in #2526, #2534, and #2571.

• Added Meta Muse Glimmer 30B VLM support with DFlash speculative decoding. Text, vision, channel-scoped reasoning, ATEM tool calls, prefix/SSD cache, and oQ quantization are supported. DFlash reached about 1.4x decode on the tested oQ4 pair; image requests continue through the VLM fallback. The vendored implementation was also synchronized with upstream reasoning and FP32 numerical behavior. #2586, #2587, and #2643.

• Added complete Jina Reranker v3.5 support. Projector loading, sliding-window attention, dual matching, block fusion, and reference-score parity are included. By @damascoemi in #2449 and #2494.

• DeepSeek V4 prefill is 23–28% faster across tested context lengths. Native attention routing, 2048-token cache blocks, fused windowed and pooled attention, skipped intermediate lm_head work, in-place pooled-cache appends, and a fused indexer mask improve prompt throughput without changing decode speed. By @DiscoStew6082 and @jonathan308 in #2559, #2562, #2563, and #2568.

• DeepSeek V4 unaligned final prefill tails stay on the native indexer path. A 347,929-token validation reduced full-model MLX peak memory from 142.255 GiB to 105.059 GiB by avoiding a large FP32 fallback workspace. The fix handles partial Metal tiles directly rather than globally splitting scheduler chunks. Reported in #2627, fixed in #2674.

• Added MTPLX sidecar MTP import for compatible Qwen3.6 checkpoints, fixed DeepSeek V4 affine AWQ expert bias/zero-point loading, and removed a per-token host synchronization from grammar-constrained decoding.

Chat, APIs, Reasoning, and Tool Calling

• Added built-in web search to admin chat. Models can use web_search and fetch_url with keyless DDGS metasearch or optional DuckDuckGo, Brave, and SearXNG providers. Results remain visible as source cards across tool rounds. URL fetching rechecks private and loopback destinations after every redirect and caps content before conversion. External API clients are unchanged because only the chat UI advertises these tools. #2596.

• Added file and realtime speech-to-text to chat. Audio STT models switch the composer into transcription mode, while compatible Whisper and Voxtral models expose live microphone input over WebSocket. Whisper file streaming now uses bounded segments with the full anti-hallucination path and per-segment language detection. Qwen3-ASR uses cumulative decoding so multi-token CJK characters remain intact. #2603.

• Fixed streaming Responses reasoning for prompts that already open <think>. DeepSeek V4 reasoning now streams through dedicated reasoning events, creates a separate reasoning item, and reports reasoning token usage instead of leaking thought text through response.output_text.delta. Confirmed with a real checkpoint on Apple Silicon. Reported in #2584, fixed by @leepokai in #2623.

• Hardened tool-call parsing. Tool arguments containing literal close markers are preserved, deeply nested model output now fails closed instead of escaping as RecursionError or SyntaxError, and a malformed call no longer drops a valid neighboring call. #2544, #2593.

• Fixed VLM and Muse Glimmer tool-call paths. VLM requests now carry tool schemas through to protocol parser sessions, preserving JSON-looking values declared as strings. Muse Glimmer also keeps tool calls when a turn opens directly with one instead of returning an empty response. Reported in #2646, fixed in #2645 and #2599.

• Thinking budgets now count only accepted/emitted tokens during both VLM and DSpark MTP speculative decoding. Gemma 4 no longer leaks reasoning into visible content after tool responses, and Ling and Inkling tool protocol variants are parsed without marker leakage.

macOS, UI, and Integrations

• Reworked the macOS self-updater to make bundle replacement crash-safe. A one-shot launchd worker now performs an atomic bundle exchange, so interruption cannot leave /Applications/oMLX.app missing its executable or signature. This addresses the “damaged or incomplete” failures reported on macOS 27 in #2522.

• Important for affected macOS 27 users: install 0.6.0rc1 manually from the DMG once. An older build still runs its old updater while installing this release; the crash-safe updater takes effect for subsequent updates after 0.6.0rc1 is installed.

• Added an Enhanced Readability option for the web admin UI, raising low-contrast helper text and enforcing a 12px minimum without changing KaTeX sizing. By @LXD-8 in #2542.

• Fixed duplicate Dock instances when multiple oMLX app bundles are registered, and changed “All networks” mode to a dual-stack IPv6 wildcard while retaining an explicit IPv4-only option.

• omlx launch codex no longer modifies persistent Codex configuration. omlx launch claude --cross-session now enables Claude Code cross-session messaging explicitly while keeping the privacy-oriented default and unrelated service opt-outs intact. #2519, #2601.

• Hugging Face cache checkpoints now appear as oQ quantization sources under their canonical repository identity, and /v1/models/status exposes favorite and hidden state for external model-management clients.

Upgrade Notes

• This is a release candidate. Please report regressions before the final 0.6.0 release.
• Distributed inference remains experimental and is disabled by default. It is included in the DMG and can be enabled from Web Dashboard > Global Settings > Advanced > Distributed Inference. Save the settings and restart oMLX after enabling it.
• Decode fairness is enabled by default. Set decode_fairness=false to restore the previous concurrent-prefill behavior.
• GDN snapshot storage defaults to auto with RHT-INT16 for configurations without an explicit legacy choice. Existing explicit split/embedded and FP32 settings are preserved.
• Legacy mixed-CacheList SSD blocks are invalidated automatically because their storage layout is incompatible with the new linear format.
• DeepSeek V4 temperature-0 output can differ from earlier builds because the new kernels change floating-point reduction order. The differences occur at near-tied token choices and are not considered a quality regression.
• No manual settings migration is otherwise required.

New Contributors

Thank you to @Maxim-Mazurok, @damascoemi, @jetztmalehrlich, @Pietroski, @studioburnside, @tannerdsilva, @LXD-8, @luziyi, @DiscoStew6082, @fxd0h, @mvdbos, @ashhart, @PhilipJohnBasile, @mkmsyk, @nazerim, @leepokai, and @dawgflymd for their first contributions in this release line.

Full Changelog: v0.5.7...v0.6.0rc1

This development release introduces opt-in distributed model serving across Macs and mixed Metal/CUDA nodes, adds built-in web search and streaming speech-to-text to chat, keeps active generation responsive during concurrent prefills, and brings SSD-backed prompt reuse to distributed ranks.

• Added experimental distributed serving across Macs. One model can now be split across multiple Apple Silicon machines using tensor or pipeline parallelism, with capability-aware planning, selective shard staging, per-rank memory guards, liveness supervision, and one-click activation from the Cluster dashboard. Qwen3.6-27B tensor parallel reached 28.6 tok/s across two Macs versus 16.1 tok/s on one Mac, with byte-identical output; a 225 GB MiniMax-M3 checkpoint that could not fit either machine alone also loaded across 128 GB and 256 GB Macs and passed its distributed canary. The feature is disabled by default under Settings > Advanced while it remains an experimental source-build preview. By @ashhart in #2423, with follow-up fixes for SSH failure handling, clock-safe liveness, MLA/MTP memory accounting, and load admission.

• Added heterogeneous Metal and CUDA model pools. Apple Silicon and NVIDIA CUDA workers can contribute to one logical model-memory pool, with automatic contiguous-layer placement based on usable memory, short-lived GUI-generated worker enrollment, ConnectX discovery and NCCL verification, and cluster diagnostics in the dashboard. The current compatibility path keeps every physical worker in the outer MLX Ring; verified CUDA pairs are placed together, but the future hierarchical Ring-to-NCCL gateway is not implemented yet. By @ashhart in #2591.

• Added decode fairness during concurrent prefill. Prefill now yields GPU time to active decodes, including streams running in another engine, and sizes contended chunks by a target stall time instead of a fixed token count. The feature is enabled by default and can be changed live from the dashboard. On an M3 Ultra, overlap decode improved by about 1.6x to 43x across the tested model pairings, while contended prefill took 1.15-2.0x longer and solo prefill remained unchanged. By #2633.

• Added process-lifetime SSD prompt caching for distributed ranks. Each rank now stores a chain of cache-boundary snapshots and restores the longest prefix held by every rank when the in-memory prompt cache misses. Incremental KV segments keep storage growth linear instead of copying the full prefix at every boundary; a 12k-token GLM-5.2 chain used 1.17 GB instead of about 6.5 GB. Snapshots are scoped to the current rank process and are cleaned up on restart or teardown. By #2620.

• Added built-in web search to the admin chat. The chat composer can expose web_search and fetch_url tools to the selected model, with keyless DDGS metasearch by default and optional DuckDuckGo, Brave Search API, or SearXNG providers. Results stay visible as source cards across tool rounds, while fetched URLs are checked against private and loopback destinations on every redirect and capped before MarkItDown conversion. External API clients are unchanged because only the chat UI advertises the built-in tools. By #2596.

• Added file and realtime speech-to-text to chat. Selecting an audio STT model switches the composer into ASR mode for streamed audio or video transcription, and compatible models also expose live microphone input over WebSocket. Whisper file streaming now uses bounded segments with the full anti-hallucination path and per-segment language detection, while Qwen3-ASR uses cumulative token decoding so multi-token CJK characters no longer turn into replacement characters. Multipart request logging also skips binary bodies instead of buffering and printing them. By #2603.

• Improved Meta Muse Glimmer and DFlash. Long Muse Glimmer oQ4 prefill now routes quantized projections through the native QMM tile, measuring about 4.5% higher prompt throughput on a cold 51k-token prompt with bit-exact output. DFlash responses now preserve parser-emitted tool calls instead of dropping them during speculative generation.

• Fixed MTP performance state after cross-engine prefill contention. The adaptive loop-tax probe could sample a handoff while another engine was prefilling, latch the resulting slowdown onto the model, and suppress MTP performance until restart. Contaminated measurements are now skipped or discarded, and stale high measurements decay back toward the default margin.

• Fixed DeepSeek-V4 affine AWQ checkpoint loading. Per-expert bias and zero-point tensors are now stacked with weights and scales during sanitize, so externally quantized affine checkpoints no longer fail strict weight loading with thousands of leftover *.biases keys. By @True2456 in #2598.

• Improved oQ discovery for Hugging Face cache models. Original, unconverted checkpoints inside the Hugging Face snapshot cache now appear as quantization sources with their canonical repository identity instead of a snapshot hash. By @monroewilliams in #2605, with a follow-up identity-preservation fix.

• Exposed model-management state through the API. /v1/models/status now includes is_favorite and is_hidden, allowing API clients to reproduce dashboard filtering without an additional model-list request. By @monroewilliams in #2602.

New Contributors

• @ashhart made their first contribution in #2423.

This development release adds Meta Muse Glimmer 30B with DFlash speculative decoding, makes DeepSeek V4 prefill 23-28% faster across context lengths, persists GDN recurrent state as bounded SSD sidecars, and fixes Ling accuracy, grammar decoding overhead, and tool-call parsing edge cases.

• Added Meta Muse Glimmer 30B VLM support with DFlash speculative decoding. The model implementation (hybrid SWA/NoPE text backbone, ViT-G/14 encoder) is vendored from the upstream mlx-vlm port by @Blaizzy (Blaizzy/mlx-vlm#1838), plus the quantized embedding-norm fix by @johntdavies (Blaizzy/mlx-vlm#1839). Thank you both for the quick upstream turnaround. On the oMLX side this adds channel-scoped output parsing with ATEM tool calls, prefix and SSD cache integration, and oQ quantization (#2586), and DFlash speculative decoding is wired in #2587, reaching about 1.4x decode on the oQ4 pair.
• Faster DeepSeek V4 prefill, 23-28% across context lengths. Layered across four changes following the ablation in #2558: ratio-128 layers now route through the native attention kernel (#2559), the MXFP4 MoE block threshold matches 2048-token chunks (#2562), and redundant all-pooled indexer scoring is skipped (#2563), all by @DiscoStew6082. DeepSeek V4 cache blocks now target 2048 tokens so those kernels engage with the prompt cache enabled. On top of that, a fused windowed+pooled prefill kernel, skipping the discarded lm_head projection on intermediate chunks, in-place pooled cache appends, and a fused indexer mask, by @jonathan308 in #2568. Measured on an M3 Ultra (V4-Flash 4-bit, default server config, cold prompts, prompt processing tok/s):

The kernel PRs need 2048-token prefill chunks to pay off, which is why the middle row is flat: the block-alignment change is what unlocks them under the default cache-enabled config. Decode speed is unchanged at every context length. Note that temperature-0 outputs can differ from 0.5.8.dev2 for identical prompts: the new kernels change floating-point reduction order, which only flips tokens where the model is effectively indifferent, and quality is unaffected.

• Reduced long-context memory for GDN models when the SSD cache is enabled. GDN recurrent state is now persisted as bounded SSD sidecars instead of retaining one full in-memory state per historical boundary, making long-context Qwen3.6 serving practical on unified-memory Macs. Opt-in and backward-compatible. By @hojin12312 in #2569.
• Fixed Ling-3.0-flash code accuracy, +17pp on HumanEval. Its late layers were running without the trained SwiGLU clamp shipped in config.json; applying it raises HumanEval from 71.3% to 88.4% at no runtime cost. By @True2456 in #2571.
• Removed a per-token host sync from grammar-constrained decoding. Token acceptance is deferred to the top of the next step with bit-identical output, so the forward dispatch never blocks on host work. By @fxd0h in #2561.
• Improved adaptive prefill memory estimates at high context. The estimator now remembers process footprint released after a chunk and accounts for it when sizing the next one, closing an over-admission gap. By @mvdbos in #2573.
• Fixed tool calls whose arguments contain a literal close marker. These were dropped with raw markup leaking into the response; payload boundaries are now located by span scanning with JSON-aware decoding, in both the batch parser and the streaming filter. By @richgoodson in #2544, with follow-up fixes for prose around malformed envelopes at end of stream and for JSON array payloads.

New Contributors

• @DiscoStew6082 made their first contribution in #2559.
• @fxd0h made their first contribution in #2561.
• @mvdbos made their first contribution in #2573.

This development release fixes quadratic mixed CacheList storage and MTP request head-of-line blocking, hardens Ling and Inkling tool-call parsing, and improves Codex launch isolation, dual-stack networking, model compatibility, and localization.

• Fixed quadratic SSD and memory growth when storing long-context prefixes for models with mixed CacheList layers. oMLX now stores sliceable KV members per block while preserving small boundary states, with validation and upgrade handling for legacy cache entries. Original implementation by @Pietroski in #2550.
• Fixed head-of-line blocking when requests arrive during single-stream MTP decoding. Late arrivals can now join promptly through a safe drain handoff instead of waiting for the active generation to finish. Reported by @anicaise-ai in #2515.
• Fixed Ling role and XML tool-protocol markers leaking into visible output, while preserving structured tool-call parsing and request-scoped tool schema validation. By @scaryrawr in #2528.
• Fixed Inkling tool calls using OpenAI-style arguments payloads or truncated nested JSON. The parser now accepts supported payload variants and repairs incomplete braces only after strict JSON parsing fails. By @studioburnside in #2555.
• Fixed omlx launch codex modifying persistent Codex configuration. Provider settings and the model context window are now passed only to the launched process. By @wolfyy970 in #2519.
• The macOS app's “All networks” listen mode now uses the dual-stack IPv6 wildcard, allowing both IPv4 and IPv6 clients while retaining an explicit IPv4-only option. By @tannerdsilva in #2552.
• Localized the Runtime Cache Observability panel in the admin dashboard and added matching strings to every supported locale catalog. By @LXD-8 in #2540.
• Added MTPLX side-car MTP import for compatible Qwen3.6 checkpoints, including contract validation, normalization to model-mtp.safetensors, and a one-click dashboard import path. Original implementation by @rsnow in #2497.
• Added complete Simplified Chinese localization for the macOS app and filled the remaining Chinese translations in the admin dashboard. By @luziyi in #2554.
• Updated MarkItDown to 0.1.7, bringing upstream PPTX conversion performance and compatibility fixes. #2466.

New Contributors

• @Pietroski made their first contribution in #2550.
• @studioburnside made their first contribution in #2555.
• @tannerdsilva made their first contribution in #2552.
• @LXD-8 made their first contribution in #2540.
• @luziyi made their first contribution in #2554.

This development release adds Ling 3.0 Flash and Jina Reranker v3.5 support, enforces thinking budgets during VLM MTP, and fixes long-context memory admission, cache cleanup, reasoning parsing, and model reload behavior.

• Added Ling 3.0 Flash support for FP8, official mixed FP4/FP8 checkpoints, oQ/oQe conversion, variable-length batching, and SSD prefix-cache restores. Original implementation and FP4 support by @scaryrawr in #2526 and #2534.
• Added thinking-budget enforcement during VLM MTP speculative decoding for supported models. By @Maxim-Mazurok in #2456.
• Added Jina Reranker v3.5 support, including its projector format, sliding-window attention, dual matching, and block fusion. By @damascoemi in #2449 and #2494.
• Fixed Gemma 4 reasoning leaking into visible content after tool responses when the prompt had already opened a thought channel. By @jetztmalehrlich in #2533.
• Fixed Hermes context handling to use the context length reported by oMLX and refuse launch when the available context is below 64K. Reported by @dovil in #2538.
• Fixed false long-context HTTP 400 responses when the previous request's KV cache was still held during asynchronous SSD storage. oMLX now waits briefly for scheduler cleanup and checks memory again. Reported by @dovil in #2538.
• Fixed DeepSeek V4 prefill memory estimates by accounting for its hybrid attention and cache layout instead of applying the generic KV formula. Reported by @Letreguilly in #2521.
• Admin benchmarks no longer write generated KV caches to tiered storage, avoiding long cache-write delays after large-context runs. Reported by @zviratko in #2539.
• Changing load-time model settings now clears a cached load failure, so options such as trust_remote_code can be retried without restarting oMLX. Reported by @zviratko in #2532.

oMLX 0.5.7

Hotfix history since 0.5.4

• 0.5.5: DeepSeek V4 prompt-tail visibility, tiered-cache compatibility, and sparse-prefill fallback cache safety.
• 0.5.6: DeepSeek V4 CacheList and MTP/PoolingCache integrity, retained-reasoning prompt consistency, long-context indexer safety, and macOS idle-timeout controls.
• 0.5.7: Official DeepSeek V4 Flash 0731 prompt encoding and safe Claude Code tail-system handling.

Sorry for the unusually frequent hotfix releases since 0.5.4. DeepSeek V4 Flash 0731 exposed several interacting edge cases across prompt encoding, MTP, PoolingCache, tiered cache, and long-context inference. These updates were necessary to complete and stabilize DeepSeek V4 support. Thank you for your patience and understanding while this work was completed.

oMLX 0.5.7 rolls up every fix shipped in 0.5.5 and 0.5.6 and completes the DeepSeek V4 Flash 0731 prompt-format integration. Users upgrading from 0.5.4 can move directly to 0.5.7.

0.5.7 Highlights

• DeepSeek V4 now uses the official Flash 0731 reference encoder. Tool schemas, DSML tool calls, <tool_result> history, thinking mode, reasoning effort, role transitions, and generation boundaries now match the published encoding.
• Claude Code tail-system context uses <｜latest_reminder｜>. A system message appended after a user turn is moved into new reminder context immediately before the user it qualifies, instead of being rendered as an undelimited prompt tail.
• Prompt construction no longer leaks DSML or tool-calling markup. The corrected role transition restores the assistant generation anchor and prevents malformed closing tags or internal instructions from appearing in model output.
• Tool definitions are injected exactly once. Requests with multiple system contexts no longer duplicate the full tool schema.
• Unsupported system placements remain conservative. Ambiguous, developer, and partial-message placements use the existing user_note_safe or strict-consolidation fallback instead of assuming unsupported model semantics.
• Mid-system capability detection is explicit. DeepSeek V4 no longer passes the generic placement probe merely because raw text happened to remain in marker order.

Included Fixes from 0.5.6

• DeepSeek V4 CacheList signatures survive rotating-tip cleanup. Reused tiered-cache entries no longer fail compatibility checks after cache trimming. (#2493)
• MTP cache boundaries remain intact. Prefill snapshots and decode transitions preserve aligned cache positions instead of crossing unsafe boundaries. (#2493)
• Pooling-cache deltas are copied out of parent buffers. Cache snapshots no longer retain mutable parent storage or observe later writes. (#2500)
• Retained-reasoning prompts remain append-only. Repeated agent requests can reuse stable prompt prefixes without rewriting historical thinking context. (#2501)
• Long-context indexer fallback is safe beyond int32 limits. DeepSeek V4 avoids unsupported index arithmetic when context positions exceed the accelerated path's range. (#2502)
• macOS idle timeout can be disabled reliably. The app and admin API preserve explicit zero/disabled timeout settings. (#2498)

Included Fixes from 0.5.5

• DeepSeek DSpark MTP row-wise attention is restricted to safe decode states. Unsafe prefill and unaligned batch shapes fall back to the standard path. (#2490)
• Incompatible tiered-cache blocks are replaced instead of reused. Runtime cache composition changes no longer leave stale blocks attached to a request. (#2487)
• Sparse-prefill failures invalidate prefix-cache hits. A failed sparse path can no longer continue with partially reconstructed cached state. (#2484)

Upgrade Notes

• 0.5.7 is the recommended release for DeepSeek V4 Flash 0731.
• No configuration migration is required when upgrading from 0.5.4, 0.5.5, or 0.5.6.

New Contributors

Thank you to @nigelks for the macOS idle-timeout fix and to @jonathan308 for the DeepSeek V4 PoolingCache, retained-reasoning, and long-context indexer fixes included in this hotfix line.

Full Changelog: v0.5.4...v0.5.7