Skip to content

Releases: huggingface/transformers

Release list

Release: v5.15.0

Release: v5.15.0 Latest
Latest

Choose a tag to compare

View all tags
@LysandreJik LysandreJik released this 10 Aug 10:28
v5.15.0
5eddc12
This commit was signed with the committer’s verified signature.
Cyrilvallez Cyril Vallez
SSH Key Fingerprint: O743lyKbz6cYCDq5uTRdzyLHAo4JjdtgAF9Q3yVhAQA
Verified
Learn about vigilant mode.

Release v5.15.0

New Model additions

Meta Muse Glimmer

Muse Glimmer, released today, is Meta’s new multimodal model, especially designed for agentic use cases. Distilled from Muse to 30B parameters, and released under the Apache 2.0 license, it can be deployed to local setups for privacy-aware applications such as coding, document analysis, personal assistants, Claw- or Hermes-like setups.

Muse Glimmer is a dense 30B parameter model consisting of:

  • 2B ViT-style encoder for vision (Perception Encoder)
  • 28B parameter text decoder

We're covering it in the following blogpost: http://hf.co/blog/muse-glimmer

image

GraniteMoeSWA & GraniteSWA

image

Links: Documentation

Links: Documentation

A.X-K1 & A.X-K2

image

Links: Documentation

Links: Documentation

Cosmos3 Edge

image

Links: Documentation

Breaking changes

Kernels are now opt-in rather than mandatory for linear attention models (Mamba, GDN, Conv-only, etc.), so users who relied on automatic kernel selection must explicitly enable kernels to maintain previous behavior.

  • 🚨 [Kernels] Refactor all linear attn models & native kernels fallback (#47630) by @vasqu

The cache cropping API now only accepts negative values (relative offsets) instead of absolute sizes, so users calling crop methods directly must update their code to pass negative values accordingly.

T5 and its model family (MT5, LongT5, etc.) now support SDPA and other attention backends via ALL_ATTENTION_FUNCTIONS, meaning the default attention implementation may change and users relying on the previous eager-only path should explicitly set attn_implementation="eager" if needed.

  • 🚨 Enable SDPA (and other attention backends) for T5 and propagate to the T5 family (#47014) by @jiqing-feng

Several small private helper functions (e.g., _is_url, _build_image_tokens) have been removed from multimodal processor files, so users or downstream libraries that imported these private functions directly must remove or replace those references.

Attention

This release includes several attention fixes and improvements, including correcting Multi-Head Latent Attention (MLA) cache compression, optimizing Flash Attention max sequence length computation in vision models, and fixing bugs in CTRL flex-attention and SDPA prefill with position bias. Additional changes refactor linear attention models for better maintainability, make Gemma 4's heterogeneous attention config explicit, and improve MPS support via metal-flash-sdpa integration.

Vision

Vision improvements in this release include performance optimizations such as faster image preprocessing for vision-language models (GLM4V, MiniMaxM3-VL, and others) by eliminating redundant tensor copies, and more efficient Flash Attention variable-length paths by precomputing maximum sequence lengths once per forward pass. Several bug fixes were also applied, including correcting dtype alignment in Kosmos2/Kosmos2_5 embedding merges, fixing a position-embedding initialization fallback in Phi4Multimodal, resolving PIL resize parity in Hunyuan-VL, and patching stop-sequence handling in the image-text-to-text pipeline.

Generation

Several generation improvements and bug fixes were made, including enabling batched audio generation for Qwen2.5/3-Omni, allowing sliding window cache layers to work with speculative decoding, and fixing memory overhead from static cache persistence across generate() calls. Multiple model-specific bugs were also resolved, including crashes in KyutaiSpeechToText, MusicgenForCausalLM, CTRL flex-attention, and assisted decoding for EncoderDecoder cache and OlmoHybrid models.

Cache

Several cache-related bugs were fixed, including correcting NemotronH's missing "mlp" layer-type mapping, resolving recurrent-layer padding masks being skipped during chunked prefill and cache continuation for hybrid models, and fixing assisted decoding for models with EncoderDecoderCache and OlmoHybrid. Additional improvements include aligning OlmoHybrid to use a native cache, enabling sliding window layers to support speculative decoding rollback, and stopping the static cache from being stored as a model attribute to reduce unexpected memory overhead.

Kernels

⚠️ The kernels python package will very likely be a required dependency for transformers[torch] in the near future. This will help us deliver maximum performance to all users; kernels will only be downloaded from trusted publishers manually approved by the HF team. Please let us know of any issues you're facing beforehands so that we may solidify our integration.

Improved robustness of the kernels integration by refactoring function handling to use layer repos, fixing CI EROFS fallback patches for kernel downloads via HfApi, resolving a positional argument collision in causal_conv1d_fn, and bumping the FP8 kernels version to prevent NaNs.

Read more

Contributors

kashif, tarekziade, and 70 other contributors
Assets 2
Loading

Patch release: v5.14.1

Patch release: v5.14.1

Choose a tag to compare

View all tags
@Cyrilvallez Cyrilvallez released this 16 Jul 09:41
v5.14.1
a08ace4
This commit was signed with the committer’s verified signature.
Cyrilvallez Cyril Vallez
SSH Key Fingerprint: O743lyKbz6cYCDq5uTRdzyLHAo4JjdtgAF9Q3yVhAQA
Verified
Learn about vigilant mode.

Patch release v5.14.1

This patch solves a few issues which appeared when integrating Inkling model, most notably an issue affecting models using EncoderDecoderCache during assisted generation. It also fixes an issue that could appear during prefill with StaticCache and sdpa without padding for Inkling which uses a position_bias.
It contains the following commits:

Contributors

IlyasMoutawwakil, Cyrilvallez, and vasqu
Loading

Release v5.14.0

Release v5.14.0

Choose a tag to compare

View all tags
@ArthurZucker ArthurZucker released this 15 Jul 19:02
v5.14.0
2fa8c19

Release v5.14.0

New Model additions

Inkling (fresh from Thinking Machines): 975B total, 41B active

image

Inkling is a general-purpose multimodal model that accepts text, image and audio inputs and
generates text outputs. It is intended for use in English and other languages, and across
multiple coding languages. The model is designed to be used by developers building AI-
powered applications, including agentic and tool-use systems, coding assistants, chatbots, and
retrieval-augmented generation systems, and is suitable for general-purpose conversational
use, instruction-following, and other natural language and multimodal tasks. It is released with
open weights to support research, fine-tuning and integration into third-party products by
downstream developers.

TIPSv2

image

Links: Documentation

TIPSv2 DPT

image

Links: Documentation

🚨 Breaking changes

GPTNeoX now remaps embed_out to lm_head and GPTBigCode has _supports_attention_backend = True enabled for vLLM compatibility; users relying on the previous weight naming or attention backend behavior for these models should update their code accordingly.

  • 🚨 Fix GPTBigCode and GPTNeoX for the Transformers modelling backend for vLLM (#47198) by @hmellor

Kernels

Several kernel-related fixes and improvements were made, including pinning the kernels dependency to a compatible version in the benchmark workflow, removing a deprecated package_name argument from LocalLayerRepository, and making the DeepGEMM Triton fallback more robust when CUDA_HOME is unset or misconfigured. Additionally, SDPA prefill was updated to leverage the FlashAttention kernel with StaticCache, yielding significant performance gains (up to 260% faster for large input sizes).

Generation

Generation improvements include adding Multi-Token Prediction (MTP) decoding support, static ensemble verification for speculative decoding to improve draft token acceptance rates, and a fix for crashes in greedy assisted generation with different tokenizers. A misleading double-negative warning message for synced_gpus in continuous batching mode was also corrected.

Performance

Fixed a Flash Attention performance regression affecting models like Qwen3-VL and resolved a MoE decode optimization bug where the grouped-to-batched matrix multiplication switch was not applied to experts residing in submodels (e.g., VLMs with a nested text config).

Cache

Cache dispatch logic was simplified by introducing explicit layer-type mappings for sliding and static layers, reducing complexity in cache routing. Additionally, fixes were made for read-only cache failures in CPU CI environments and for MPS graph cache growth during variable-length batch training on Apple Silicon.

Bugfixes and improvements

Read more

Contributors

tarekziade, andreasgoulas, and 43 other contributors
Loading

Patch release v5.13.1

Patch release v5.13.1

Choose a tag to compare

View all tags
@ArthurZucker ArthurZucker released this 11 Jul 09:15
v5.13.1
4626421

Patch release v5.13.1

This patch is focused on enabling transformers for the latest release of vllm!

  • Be more defensive with remap_legacy_layer_types for custom models (#47245) from @hmellor
  • Fix custom code which doesn't know about the new linear layer type names (#47174) from @hmellor
  • Fix case where _LazyAutoMapping.register is passed a str key (#47148) from @hmellor

Contributors

hmellor
Loading

Release v5.13.0

Release v5.13.0

Choose a tag to compare

View all tags
@vasqu vasqu released this 03 Jul 16:06
v5.13.0
6af945f

Release v5.13.0

New Model additions

KimiK 2.5, 2.6, and 2.7

image

This release includes the architecture for Kimi 2.5 which is used by 2.5-2.7:

Kimi K2.5 is an open-source, native multimodal agentic model that advances practical capabilities in long-horizon coding, coding-driven design, proactive autonomous execution, and swarm-based task orchestration. The model was proposed in Kimi K2.5: Visual Agentic Intelligence and further improved in [Kimi K2.6: Advancing Open-Source Coding](Kimi K2.5: Visual Agentic Intelligence).

Kimi K2.5 achieves significant improvements on complex, end-to-end coding tasks, generalizing robustly across programming languages (Rust, Go, Python) and domains spanning front-end, DevOps, and performance optimization. The model is capable of transforming simple prompts and visual inputs into production-ready interfaces and lightweight full-stack workflows, generating structured layouts, interactive elements, and rich animations with deliberate aesthetic precision.

Links: Documentation

MiMo-V2-Flash

image

MiMo-V2-Flash is a Mixture-of-Experts (MoE) language model developed by the Xiaomi MiMo team. Designed to establish a new balance between long-context modeling capabilities and inference efficiency, the model is built for strong performance in complex reasoning and agentic tasks. Trained on 27T tokens with native 32k sequence lengths, MiMo-V2-Flash seamlessly supports an extended 256K context window while significantly reducing KV-cache storage compared to standard global attention models.

Links: Documentation

Nemotron 3.5 ASR

image

Nemotron 3.5 ASR is a 600M-parameter multilingual speech recognition model from NVIDIA, built for high-quality transcription in both low-latency streaming and high-throughput batch settings, with native punctuation and capitalization. For streaming, it offers configurable chunk sizes—80ms, 160ms, 560ms, and 1120ms, letting users trade off latency against accuracy to suit their application. Its cache-aware FastConformer-RNNT architecture is central to this capability: unlike traditional buffered streaming, which repeatedly reprocesses overlapping audio windows, the model processes only each new incoming chunk while reusing cached encoder context from prior chunks. This eliminates redundant computation, significantly improves efficiency, and minimizes end-to-end delay without sacrificing accuracy, making it well suited to real-time transcription workloads.

Links: Documentation

NemotronAsrStreaming

Nemotron ASR Streaming is a 600M-parameter English speech recognition model from NVIDIA, built for high-quality transcription in both low-latency streaming and high-throughput batch settings, with native punctuation and capitalization. For streaming, it offers configurable chunk sizes—80ms, 160ms, 560ms, and 1120ms, letting users trade off latency against accuracy to suit their application. Its cache-aware FastConformer-RNNT architecture is central to this capability: unlike traditional buffered streaming, which repeatedly reprocesses overlapping audio windows, the model processes only each new incoming chunk while reusing cached encoder context from prior chunks. This eliminates redundant computation, significantly improves efficiency, and minimizes end-to-end delay without sacrificing accuracy, making it well suited to real-time transcription workloads.

Links: Documentation

Qwen3 ASR

image

Qwen3 ASR is an automatic speech recognition model from Alibaba's Qwen team that combines a Whisper-style audio encoder with a Qwen3 language model decoder for speech-to-text transcription. The model supports automatic language detection and multilingual transcription.

A forced aligner model is also included. It can be used to timestamp a provided transcript and its audio. It uses the same audio encoder model with a classification head that predicts a word's length. This model can be used with the transcript from any ASR model (see the example below with Parakeet CTC).

Links: Documentation

ZAYA

image

ZAYA1 is a 760M active / 8.4B total parameter MoE language model trained by Zyphra. It combines Compressed
Convolutional Attention (CCA), a nonlinear ZAYA1 router, and residual scaling.

Links: Documentation

VideoPrism

The VideoPrism model was proposed in the paper VideoPrism: A Foundational Visual Encoder for Video Understanding by Google DeepMind (blog post).

VideoPrism is a general-purpose video encoder that tackles diverse video understanding tasks with a single frozen model. The model is pretrained on a large-scale heterogeneous corpus containing 36M high-quality video-caption pairs and 582M video clips with noisy parallel text (e.g., ASR transcripts). The pretraining approach improves upon masked autoencoding through global-local distillation of semantic video embeddings and a token shuffling scheme, enabling the model to focus primarily on the video modality while leveraging text associated with videos. VideoPrism achieves state-of-the-art performance on 31 out of 33 video understanding benchmarks across four broad task groups, from web video question answering to computer vision for science.

Links: Documentation

RADIO

RADIO (Reduce All Domains Into One) is a family of vision foundation models from NVIDIA trained by multi-teacher distillation (e.g. CLIP, DINOv2, SAM) into a single ViT backbone. It produces both an image-level summary embedding and dense spatial features, and supports variable input resolutions through a Cropped Position Embedding (CPE) patch generator.

Links: Documentation

MiniCPM3

MiniCPM3 is the third-generation MiniCPM dense language model from OpenBMB. The 4B variant
(openbmb/MiniCPM3-4B) outperforms many 7B–9B open
models on standard benchmarks while remaining lightweight enough for on-device usage.

MiniCPM3 combines several architectural ideas:

  • Multi-head Latent Attention (MLA) from DeepSeek-V2, which compresses the key/value cache
    into a low-rank latent representation while still using rotary embeddings on a portion of the
    query/key heads.
  • A standard SwiGLU MLP (no MoE).
  • Three scalar scaling factors that govern signal flow:
    • scale_emb — scales input embeddings.
    • scale_depth / sqrt(num_hidden_layers) — scales residual connections.
    • hidden_size / dim_model_base — scales hidden states before the language model head.

Links: Documentation

Breaking changes

A broad set of modeling changes have been made to standardize layer declarations, mask/cache construction, and hybrid-attention handling, making many models cleanly exportable (ONNX, torch.export, ExecuTorch) and fullgraph-compilable — users relying on internal modeling APIs may need to update their code accordingly.

Attention masking for image tokens in Gemma 3/4 models has been fixed to correctly respect sliding window boundaries in local layers, which changes model behavior and may affect reproducibility of previous results.

  • 🚨 [gemma 3/4] Fix bidirectional attention masking crossing sliding window boundaries (#46850) by @douglas-reid

The Expert Parallelism (E...

Read more

Contributors

kurtmckee, tarekziade, and 68 other contributors
Loading

Patch release v5.12.1

Patch release v5.12.1

Choose a tag to compare

View all tags
@vasqu vasqu released this 15 Jun 17:29
v5.12.1
ddb849a

Patch release v5.12.1

Updated the lower bound for PEFT and a fix for auto tokenizer to properly resolve the mistral tokenizer (when mistral-common is installed). This is similar to v.5.10.3 minus the fixes that were already included in the main release - vLLM will first target 5.10.3 🤗

Full Changelog: v5.12.0...v5.12.1

Contributors

hmellor and itazap
Loading

Patch release v5.10.4

Patch release v5.10.4

Choose a tag to compare

View all tags
@vasqu vasqu released this 15 Jun 17:29
v5.10.3
89eb876

Patch release v5.10.4

Update: Note that on pypi 5.10.3 doesn't exist and this this saved under 5.10.4 (so essentially a minor version skipped). Sorry about that, that's on me. Just wanted to clarify to make this less confusing!

A few fixes needed for vLLM to sync with transformers 🤗

Full Changelog: v5.10.2...v5.10.3

Contributors

hmellor, itazap, and 2 other contributors
Loading

Release v5.12.0

Release v5.12.0

Choose a tag to compare

View all tags
@vasqu vasqu released this 12 Jun 14:39
v5.12.0
e0e7504

Release v5.12.0

New Model additions

MiniMax-M3-VL

image

MiniMax-M3-VL is the vision-language member of the MiniMax-M3 family that pairs a CLIP-style vision tower with 3D rotary position embeddings with the MiniMax-M3 text backbone. It uses a mixed dense/sparse Mixture-of-Experts decoder with SwiGLU-OAI gated experts and a lightning indexer for block-sparse attention. The model processes images through a Conv3d patch embedding system and includes specialized components for efficient multimodal understanding and generation.

Links: Documentation

PP-OCRv6: update documentation and slow tests (#46576)

image

The official weights for PP-OCRv6 are out: PP-OCRv6 is a lightweight OCR system that combines architectural innovation with data-centric optimization. It redesigns the backbone, detection neck, and recognition neck around a unified MetaFormer-style building block with structural reparameterization. Three model tiers (medium, small, tiny) share the same block primitives, covering deployment scenarios from server to edge.

  • PP-OCRv6: update documentation and slow tests (#46576) by @ zhang-prog

Add Parakeet-RNNT (#46331)

ParakeetForRNNT: a Fast Conformer Encoder + an RNN-T (RNN Transducer) decoder

  • RNN-T Decoder: Standard neural transducer:
    • LSTM prediction network maintains language context across token predictions.
      • Joint network combines encoder and decoder outputs.
      • Greedy transducer decoding for inference: a blank emission advances the encoder frame by one, a non-blank emission stays on the same frame.

Bugfixes and improvements

Significant community contributions

The following contributors have made significant changes to the library over the last release:

Contributors

kashif, tarekziade, and 14 other contributors
Loading

Release v5.11.0

Release v5.11.0

Choose a tag to compare

View all tags
@vasqu vasqu released this 10 Jun 16:32
v5.11.0
e7b5b96

Release v5.11.0

New Model additions

DiffusionGemma

image

DiffusionGemma is engineered to reduce the sequential bottlenecks of standard causal language models by employing an encoder-decoder architecture specifically optimized for inference speed. During inference, DiffusionGemma leverages multi-canvas sampling, where rather than generating one token at a time, the model iteratively denoises a full block of tokens using a diffusion sampler. This block-autoregressive approach facilitates text generation at higher speeds compared to traditional sequential generation methods.

Links: Documentation

DeepSeek-V3.2

image

DeepSeek-V3.2-Exp is an experimental model from DeepSeek-AI that introduces DeepSeek Sparse Attention (DSA), a trainable, fine-grained sparse attention mechanism designed to improve training and inference efficiency in long-context scenarios. Built on top of DeepSeek-V3.1-Terminus with a 685B-parameter Mixture-of-Experts backbone, it reduces the quadratic cost of attention over long sequences by attending only to a selected subset of past tokens while maintaining virtually identical benchmark performance. The work was extended in DeepSeek-V3.2 which pairs DSA with scalable reinforcement learning and achieves gold-medal level results on competition math and competitive programming benchmarks.

Links: Documentation | Paper

Kernels

The KernelConfig API was extended to support n-to-1 module fusion and parameter transformation, simplifying how custom kernels are integrated with Transformers modules. Additional fixes include resolving a dtype mismatch in the Mamba2 CUDA kernel path for NemotronH/Zamba2, adding fine-grained fp8/fp4 Triton kernel support, and correcting the FalconMamba fast-path warning to recommend pip install kernels instead of mamba-ssm.

Parallelization

Fixed model parallel beam search bugs in the Qwen2-VL, Qwen2.5-VL, and Qwen3-VL MoE model families, and added documentation for tensor parallelism support with continuous batching.

Bugfixes and improvements

Significant community contributions

The following contributors have made significant changes to the library over the last release:

Contributors

NicolasHug, csantosbh, and 33 other contributors
Loading

Patch release v5.10.2

Patch release v5.10.2

Choose a tag to compare

View all tags
@vasqu vasqu released this 04 Jun 18:43
v5.10.2
0dad7b8

Patch release v5.10.2

There was a big bug in the model conversion of models related to clip, this affected models like sam3 and others. Please make sure to update 🙏

Full Changelog: v5.10.1...v5.10.2

Contributors

zucchini-nlp
Loading