TUTTI: Toward generalizable audio-to-score transcription via fully synthesized data

This could be an interesting, new paper by Maosong Sun and his team! Tutti frutti!

What about bidirectional, i.e. from Score-to-Audio (S2A)?

From the abstract:

"Generalizable Audio-to-Score (A2S) transcription is fundamentally constrained by the severe scarcity of high-quality, real-world paired data.

Relying solely on existing human-annotated datasets often restricts the generalization of A2S models, limiting their efficacy primarily to single-instrumentation domains.

To break this dependency on scarce real-world data, we introduce TUTTI (Transformer for Unified audio-To-score Transcription trained on Synthetic multi-Instrumentation Data), a pre-training paradigm driven by a purely synthetic, large-scale dataset.

Rather than using human-composed scores, we leverage a symbolic music generation model to generate a massive, highly scalable multi-instrumentation corpus and create audio-score pairs with expressive acoustic characteristics. Capitalizing on the generated data, we employ a standard Transformer encoder-decoder architecture.

We empirically demonstrate that pre-training a unified attention-based model on generated, multi-instrumentation data yields a consistently stronger foundational representation than single-instrumentation training.

When fine-tuned with downstream real-world datasets, TUTTI outperforms previous approaches, establishing new overall state-of-the-art results across various A2S baselines.

Notably, TUTTI shows remarkable cross-instrument transferability, effectively adapting to unseen instruments with highly competitive performance. ..."

[2609.00640] TUTTI: Toward generalizable audio-to-score transcription via fully synthesized data (preprint, open access, only 8 pages)

Hugging Face Open ASR Leaderboard tracks speech-to-text in Hindi, its first Asian language

Good news! However, only phone conversations?

"added its first Indic language sets: Monsoon en-IN and Monsoon hi-IN. The leaderboard previously covered only European languages.

The data comes from unscripted, dual-channel phone conversations recorded by 4,888 speakers across hundreds of Indian districts, each tagged with 12 attributes including age, occupation, device model, and native district.

Each language has a public split for self-scoring and a private, held-out split so models can’t be tuned to the test data;

Hindi references use a lattice of accepted spelling variants rather than one fixed transcript, since no normalizer can resolve Hindi’s spelling variations.

Prior research cited here found ASR error rates ranging from about 4% to 44% across different districts, a gap that standard aggregate word-error-rate scores don’t show. The results show that South Asian languages require different assessment tools, but show promise as ASR extends to billions of speakers on the subcontinent."

OpenAI cuts off Cursor

The Open ASR Leaderboard Adds Its First Global South Language "Voice Arena and Hugging Face partner to launch open ASR evaluation for Hindi and Indian English"

Microsoft AI Introduces MAI-Transcribe-1.5: 2.4% WER and Up to 5x Faster Long-Audio Transcription

Impressive! Really polyglott!

WER = word error rate

"... It transcribes a full hour of audio in under 15 seconds. Best-in-class on FLEURS accuracy. Leads the accuracy-speed Pareto frontier. ..."

"... The model handles 43 languages with a single system. It is optimized for diverse accents, dialects, and real-world acoustic conditions. ..."

Microsoft AI Introduces MAI-Transcribe-1.5: 2.4% WER on Artificial Analysis, Best-in-Class FLEURS Accuracy, and Up to 5x Faster Long-Audio Transcription - MarkTechPost "Microsoft's Superintelligence team has shipped MAI-Transcribe-1.5, a production-focused speech-to-text model with expanded language coverage, domain-aware keyword biasing, and faster long-form inference."

MAI-Transcribe-1.5 (official website)

IBM launches Granite Speech 4.1 2B with bidirectional, multilingual capabilities

Good news! Big Blue tackles ML & AI!

"Why: Most open ASR [automatic speech recognition] models force a hard tradeoff — you either get accuracy or speed, not both. Existing systems struggle with multilingual transcription, domain-specific jargon, and real-time latency requirements without bloating to 10B+ parameters.

What: IBM has released Granite Speech 4.1 2B, an open speech-language model that scores a 5.33 mean WER on the Open ASR Leaderboard — outperforming many models several times its size — while supporting multilingual ASR across 6 languages, bidirectional speech translation, and keyword list biasing for names, acronyms, and technical terms. Licensed under Apache 2.0.

How: The model uses a 16-layer Conformer encoder trained with dual-head CTC (graphemic + BPE outputs), a 2-layer Q-Former projector that downsamples audio to a 10Hz embedding rate, and a fine-tuned granite-4.0-1b-base LLM backbone. A companion variant — Granite Speech 4.1 2B-NAR — replaces autoregressive decoding with non-autoregressive transcript editing in a single forward pass, achieving an RTFx of ~1820 on a single H100 GPU. A third variant, Granite Speech 4.1 2B-Plus, adds speaker-attributed ASR and word-level timestamps. Trained on 174,000 hours of audio. ..."

"... At the heart of Granite 4.1 is a new generation of dense, decoder‑only language models, offered in 3B, 8B, and 30B parameter base and instruct model sizes. Across weight classes, the models significantly outperform similarly sized Granite 4.0 language models. The team found, for example, that the new Granite 4.1 8B instruct model consistently matches or outperforms the Granite 4.0 32B Mixture‑of‑Experts model, while using a simpler — and therefore more flexible — architecture for fine tuning for downstream tasks. ..."

What's Inside: OpenAI's Privacy Filter, Qwen's FlashQLA and more...

Introducing the IBM Granite 4.1 family of models (original news release) "IBM’s most expansive model release to date covers new language, vision, speech, embedding, and guardian models — tailored for enterprise workloads."

On Lost in Translation, Found in Embeddings: Sign Language Translation and Alignment

This could be an interesting paper by Andrew Zisserman and his team!

How many people are really deaf? According to Google about 11 million Americans or 3.6% of the population.

Personally, every time I see a human sign language interpreter next to e.g. a politician in a video it distracts me and drives me nuts to watch them wildly gesticulating (maybe I suffer from ADHD or autism).

Hopefully, machine learning & AI can mediate here very soon so that these  gesticulations are only displayed to those viewers that are deaf or those viewers who enjoy watching those sign language interpreters.

From the abstract:

"Our aim is to develop a unified model for sign language understanding, that performs sign language translation (SLT) and sign-subtitle alignment (SSA). Together, these two tasks enable the conversion of continuous signing videos into spoken language text and also the temporal alignment of signing with subtitles -- both essential for practical communication, large-scale corpus construction, and educational applications.
To achieve this, our approach is built upon three components:
(i) a lightweight visual backbone that captures manual and non-manual cues from human keypoints and lip-region images while preserving signer privacy;
(ii) a Sliding Perceiver mapping network that aggregates consecutive visual features into word-level embeddings to bridge the vision-text gap; and
(iii) a multi-task scalable training strategy that jointly optimises SLT and SSA, reinforcing both linguistic and temporal alignment.
To promote cross-linguistic generalisation, we pretrain our model on large-scale sign-text corpora covering British Sign Language (BSL) and American Sign Language (ASL) from the BOBSL and YouTube-SL-25 datasets.
With this multilingual pretraining and strong model design, we achieve state-of-the-art results on the challenging BOBSL (BSL) dataset for both SLT and SSA. Our model also demonstrates robust zero-shot generalisation and finetuned SLT performance on How2Sign (ASL), highlighting the potential of scalable translation across different sign languages."

[2512.08040] Lost in Translation, Found in Embeddings: Sign Language Translation and Alignment

Figure 1. A unified sign language understanding model. Givensigning data, our model performs both SLT and SSA, guided by textual prompts. For both tasks, a 500-frame (20s at 25 fps) video is used as input.
In SLT mode, the model receives the sign video with frame-level timestamps specifying the region of interest (not shown for clarity), and generates a spoken language translation for that segment.

In SSA mode, the model takes the sign video, a target sentence along with its audio-aligned timestamps (if available),

and predicts the timestamps where the sentence is signed.

Google Translate brings real-time speech translations to any headphones for over 70 languages

Good news! The Tower of Babel resurrected!

"Google Translate’s latest update brings live speech translations, originally available only on the Pixel Buds, to any headphones you want, with support for over 70 languages. It’s rolling out today [12/12/2025] in beta and just requires a compatible Android phone with the Translate app (unlike Apple’s similar feature, which requires AirPods). ..."

"... Starting today the beta is rolling out in the Translate app on Android in the U.S., Mexico, and India, works with any pair of headphones, and supports more than 70 languages. And we’ll be bringing it to iOS and more countries in 2026. ..."

Google Translate brings real-time speech translations to any headphones | The Verge

Bringing state-of-the-art Gemini translation capabilities to Google Translate (original news release) "We’re bringing Gemini’s most powerful translation capabilities to Google Translate for text, launching a beta experience for live speech-to-speech translations with headphones, and adding new languages to the app for practice and skill building."

Credits: Last Week in AI

On the Runway world model

Very recommendable! However, the models have not yet been officially released.

I sure like the name runway world model! 😀 But it seems to be an impressive model family!

"Runway’s GWM-1 family of video-generation models respond to user input in real time while producing scenes that remain consistent regardless of the camera’s position.

What’s new: Runway introduced GWM-1, a trio of “general world models” that were trained to understand how scenes behave, not just how scenes appear.

GWM Worlds generates scenes,

GWM Robotics produces synthetic data for training and testing robots, and GWM Avatars generates conversational characters with facial expressions and lip-synced speech. (In addition, the company added audio generation, audio editing, and multi-shot video editing capabilities to Gen-4.5, its flagship video generator.) 

Architecture: Autoregressive diffusion model based on Gen-4.5

Input/output: Text and images in, video out (up to 2 minutes, 1280x720-pixel resolution, 24 frames per second)

Availability: The models will be available in “coming weeks.” GWM Worlds and GWM Avatars will be available via web interface, GWM Robotics software development kit by request.

Undisclosed: Parameter count, training data and methods, pricing, release dates, performance metrics

How it works: Unlike typical diffusion models that generate an entire video simultaneously by removing noise progressively over a number of steps, GWM-1 generates one frame at a time based on past frames and control inputs.
This autoregressive approach enables the model to respond to control input in real time. Runway built each GWM-1 model by post-training Gen-4.5 on domain-specific data. The models take still images and text as input.

GWM Worlds generates a video simulation as the user navigates through the scene by issuing text commands. Users prompt the system to define an agent, physics, and world (such as a person walking through a city or a drone flying over mountains). The model maintains space and geometry consistently as objects come in and out of view, so objects remain in place as they shift in and out of the camera’s view.

GWM Robotics was trained on unspecified robotics data to generate sequences of frames that show how a scene changes, from a robot’s point of view, depending on its actions. Developers can explore alternative robot motions or directions of travel by modifying the simulated actions and observing the output.

GWM Avatars is intended for conversational applications. Users select a voice and enter a portrait and/or text, and the model generates a character with realistic facial expressions, voices, lip sync, and gestures that will interact conversationally. Characters can be photorealistic or stylized.

Behind the news: Until recently, world models, or models that predict the future state of an environment given certain actions taken within that environment, reflected fairly limited worlds. Upon its launch in early 2024, OpenAI’s Sora 1 generated video output that was impressive enough to inspire arguments over whether it qualified as a world model of the real world. Those arguments were premature, since Sora 1’s output, however photorealistic it was, was not consistent with real-world physics, for instance. But they presaged models like Google Genie 2, which produces 3D video-game worlds that respond to keyboard inputs in real time, and World Labs [Marble], which generates persistent, editable, reusable 3D spaces from text, images, and other inputs.

Why it matters: Runway is among several AI companies that are racing to build models that simulate coherent worlds including objects, materials, lighting, fluid dynamics, and so on. Such models have huge potential value in entertainment and augmented reality but also in industrial and scientific fields, where they can help to design new products and plan for future scenarios. GWM Robotics (aimed at robotics developers) and GWM Avatars (which may be useful in applications like tutoring or customer service) show that Runway’s ambitions extend beyond entertainment.

We’re thinking: The world-model landscape is dividing between models that produce videos with real-time control (Runway GWM Worlds, Google Genie 3, World Labs RTFM) and those that make exportable 3D spaces (World Labs Marble). These approaches target different applications: Real-time interactivity enables training loops in which agents could learn from immediate feedback, while exportable 3D assets feed activities like game development, in which developers may refine and reuse assets across projects."

OpenAI's Answer to Gemini 3, Runway's Interactive Worlds, Disney's Alliance With OpenAI, Adapting LLMs for Low-Data Domains

Runway product

Dual-Process Image Generation (preprint published in June 2025, open access, one of the latest research paper were an employee of Runway is a coauthor)

This is one of the images Google AI returned when I Googled runway world model. Hallucination? Sycophancy? Incompetence? Not up to date? 😊

On Omnilingual ASR: Open-Source Multilingual Speech Recognition for 1600+ Languages

Amazing stuff! Can we add a few hundred more languages please! 😊 A modern Tower of Babel!

This model is extensible so you can add another language if you like!

From the abstract:

"Automatic speech recognition (ASR) has advanced in high-resource languages, but most of the world's 7,000+ languages remain unsupported, leaving thousands of long-tail languages behind.

Expanding ASR coverage has been costly and limited by architectures that restrict language support, making extension inaccessible to most--all while entangled with ethical concerns when pursued without community collaboration.

To transcend these limitations, we introduce Omnilingual ASR, the first large-scale ASR system designed for extensibility.

Omnilingual ASR enables communities to introduce unserved languages with only a handful of data samples. It scales self-supervised pre-training to 7B parameters to learn robust speech representations and introduces an encoder-decoder architecture designed for zero-shot generalization, leveraging a LLM-inspired decoder.

This capability is grounded in a massive and diverse training corpus; by combining breadth of coverage with linguistic variety, the model learns representations robust enough to adapt to unseen languages. Incorporating public resources with community-sourced recordings gathered through compensated local partnerships, Omnilingual ASR expands coverage to over 1,600 languages, the largest such effort to date--including over 500 never before served by ASR. Automatic evaluations show substantial gains over prior systems, especially in low-resource conditions, and strong generalization.

We release Omnilingual ASR as a family of models, from 300M variants for low-power devices to 7B for maximum accuracy.

We reflect on the ethical considerations shaping this design and conclude by discussing its societal impact. In particular, we highlight how open-sourcing models and tools can lower barriers for researchers and communities, inviting new forms of participation. Open-source artifacts are available at this https URL."

[2511.09690] Omnilingual ASR: Open-Source Multilingual Speech Recognition for 1600+ Languages

Meta AI speech processing now recognizes 1600 languages including over 500 never before served by any ASR system

Very impressive! The Tower of Babel is becoming reality!

"Omnilingual ASR recognizes speech for over 1,600 languages

Meta’s Fundamental AI Research team launched Omnilingual ASR, a suite of models that transcribes speech in more than 1,600 languages, including 500 low-resource languages never before transcribed by AI. The system uses a 7 billion parameter wav2vec 2.0 speech encoder paired with two decoder variants, achieving character error rates below 10 percent for 78 percent of supported languages.

Users can extend the system to new languages using just a few audio-text sample pairs through in-context learning, eliminating the need for large training datasets or specialized expertise. ..."

"Takeaways:

• We’re introducing Meta Omnilingual Automatic Speech Recognition (ASR), a suite of models providing automatic speech recognition capabilities for more than 1,600 languages, achieving state-of-the-art quality at an unprecedented scale.
• Omnilingual ASR was designed as a community-driven framework. People around the world can extend Omnilingual ASR to new languages by using just a few of their own samples.
• We’re also releasing the Omnilingual ASR Corpus, an extensive collection of transcribed speech in 350 underserved languages; Omnilingual wav2vec 2.0, a scaled up massively multilingual speech representation model; and a language exploration demo people can explore languages covered by the model.

 ..."

From the abstract:

"While automatic speech recognition (ASR) systems have made remarkable progress in many high resource languages, most of the world’s 7,000+ languages remain unsupported, with thousands of long-tail languages effectively left behind.

Expanding ASR coverage has long been regarded as prohibitively expensive and of limited benchmark value, further hampered by architectures that restrict language coverage to a fixed set that make extension inaccessible to most communities—all while entangled with ethical concerns when pursued without community collaboration.

To transcend these limitations, this article introduces Omnilingual ASR, the first large-scale ASR system designed for extensibility. More specifically, Omnilingual ASR enables communities to introduce unserved languages with only a handful of their own data samples.

On the modeling side, Omnilingual ASR scales self-supervised pre-training to 7B parameters to learn robust speech representations and introduces an encoder–decoder architecture designed for zero-shot generalization, leveraging a large language model-inspired decoder to effectively exploit these representations.

This capability is grounded in a massive and diverse training corpus; by combining breadth of coverage with linguistic variety, the model learns representations robust enough to adapt to previously unseen languages. Incorporating public resources with community-sourced recordings gathered through compensated local partnerships, Omnilingual ASR expands coverage to more than 1,600 languages, the largest such effort to date—including over 500 never before served by any ASR system. Automatic evaluations show substantial gains over prior systems, especially in extreme low-resource conditions, and strong generalization to languages never encountered during training.

Crucially, Omnilingual ASR is released as a family of models ranging from compact 300M variants for low-power devices to large 7B models for maximum accuracy.

Throughout the paper, we reflect on the ethical considerations shaping this design and conclude by discussing its broader societal impact. In particular, we highlight how open-sourcing models and tools can lower barriers for researchers and communities alike, inviting new forms of participation without requiring onerous expertise or heavy compute.  ..."

Data Points: Meta AI now recognizes 1600 languages

Omnilingual ASR: Advancing Automatic Speech Recognition for 1,600+ Languages (original news release)

Omnilingual ASR: Open-Source Multilingual Speech Recognition for 1600+ Languages (paper)

NVIDIA Open Sources Parakeet TDT 0.6B: Achieving a New Standard for Automatic Speech Recognition ASR and Transcribes an Hour of Audio in One Second

This seems to be quite impressive!

Apparently, the Parakeet ASR models were first released by NVIDIA in January 2024. It published a research paper on it in September of 2023 (see below)

"NVIDIA has unveiled Parakeet TDT 0.6B, a state-of-the-art automatic speech recognition (ASR) model that is now fully open-sourced on Hugging Face. With 600 million parameters, a commercially permissive CC-BY-4.0 license, and a staggering real-time factor (RTF) of 3386, this model sets a new benchmark for performance and accessibility in speech AI.

Blazing Speed and Accuracy

At the heart of Parakeet TDT 0.6B’s appeal is its unmatched speed and transcription quality. The model can transcribe 60 minutes of audio in just one second, a performance that’s over 50x faster than many existing open ASR models. On Hugging Face’s Open ASR Leaderboard, Parakeet V2 achieves a 6.05% word error rate (WER)—the best-in-class among open models. ..."

NVIDIA Open Sources Parakeet TDT 0.6B: Achieving a New Standard for Automatic Speech Recognition ASR and Transcribes an Hour of Audio in One Second - MarkTechPost

Investigating End-to-End ASR Architectures for Long Form Audio Transcription (open access)

Figure 2. Architecture of the NVIDIA Parakeet encoder with blocks of downsampling and subsampling, conformer encoder blocks with limited context attention (LCA), and global token (GT)

New AI Audio Model by Nvidia Synthesizes Sounds That Have Never Existed

Amazing stuff! 

Maybe this is exactly what e.g. contemporary music needs since it has become so boring for many years! A new sound machine!

"Nvidia’s newly revealed ‘Fugatto’ model looks to go a step further, using new synthetic training methods and inference-level combination techniques to “transform any mix of music, voices, and sounds,” including the synthesis of sounds that have never existed.

While Fugatto isn’t available for public testing yet, a sample-filled website showcases how Fugatto can be used to dial a number of distinct audio traits and descriptions up or down, resulting in everything from the sound of saxophones barking to people speaking underwater to ambulance sirens singing in a kind of choir. While the results on display can be a bit hit or miss, the vast array of capabilities on display here helps support Nvidia’s description of Fugatto as ‘a Swiss Army knife for sound.’”"

"... Fugatto is a foundational generative transformer model that builds on the team’s prior work in areas such as speech modeling, audio vocoding and audio understanding.

The full version uses 2.5 billion parameters and was trained on a bank of NVIDIA DGX systems packing 32 NVIDIA H100 Tensor Core GPUs.

Fugatto was made by a diverse group of people from around the world, including India, Brazil, China, Jordan and South Korea. Their collaboration made Fugatto’s multi-accent and multilingual capabilities stronger. ..."

From the abstract:

"Fugatto is a versatile audio synthesis and transformation model capable of following free-form text instructions with optional audio inputs. While large language models (LLMs) trained with text on a simple next-token prediction objective can learn to infer instructions directly from the data, models trained solely on audio data lack this capacity. This is because audio data does not inherently contain the instructions that were used to generate it. To overcome this challenge, we introduce a specialized dataset generation approach optimized for producing a wide range of audio generation and transformation tasks, ensuring the data reveals meaningful relationships between audio and language. Another challenge lies in achieving compositional abilities -- such as combining, interpolating between, or negating instructions -- using data alone. To address it, we propose ComposableART, an inference-time technique that extends classifier-free guidance to compositional guidance. It enables the seamless and flexible composition of instructions, leading to highly customizable audio outputs outside the training distribution. Our evaluations across a diverse set of tasks demonstrate that Fugatto performs competitively with specialized models, while ComposableART enhances its sonic palette and control over synthesis. Most notably, we highlight our framework's ability to execute emergent sounds and tasks -- sonic phenomena that transcend conventional audio generation -- unlocking new creative possibilities. \href{https://fugatto.github.io/}{Demo Website.}"

New AI Audio Model Synthesizes Sounds That Have Never Existed - Human Progress

Nvidia’s new AI audio model can synthesize sounds that have never existed "What does a screaming saxophone sound like? The Fugatto model has an answer..."

Now Hear This: World’s Most Flexible Sound Machine Debuts (company blog post) "Using text and audio as inputs, a new generative AI model from NVIDIA can create any combination of music, voices and sounds."

Fugatto 1: Foundational Generative Audio Transformer Opus 1 (open access; blind submission to ICLR 2025)