Good news! It is getting crowded in the space of protein models! Here is another one! Expect rapid advances in medicine and biology!
The emphasis here seems to be on modeling the 3D structure of proteins.
"[Protein Model] Anthrogen Introduces Odyssey: A 102B Parameter Protein Language Model that Replaces Attention with Consensus and Trains with Discrete Diffusion. Odyssey is Anthrogen’s multimodal protein language model family that fuses sequence tokens, FSQ structure tokens, and functional context for generation, editing, and conditional design, it replaces self attention with Consensus that scales as O(L) and reports improved training stability, it trains and samples with discrete diffusion for joint sequence and structure denoising, it ships in production variants from 1.2B to 102B parameters, it claims about 10x data efficiency versus competing models in matched evaluations, and API access is opening for external users to test real design workflows"
"... Odyssey is a frontier, multimodal protein model family that learns jointly from sequence, 3D structure, and functional context. It supports conditional generation, editing, and sequence + structure co-design. We scale our production-ready models from 1.2B to 102B parameters.
At input, Odyssey treats proteins as more than strings. Amino acid sequences are used as usual, while 3D shape is turned into compact structure tokens using a finite scalar quantizer (FSQ)—think of it as a simple alphabet for 3D geometry so the model can “read” shapes as easily as letters.
Alongside these, we include light-weight functional cues—domain tags, secondary-structure hints, orthologous group labels, or short text descriptors—so the model can reason about what a region does, not just what it looks like. The three streams are embedded separately, then fused, so local sequence patterns and long-range geometric relationships end up in one shared representation. ..."
From the abstract:
"We present Odyssey, a family of multimodal protein language models for sequence and structure generation, protein editing and design.
We scale Odyssey to more than 102 billion parameters, trained over 1.1 × 1023 FLOPs. The Odyssey architecture uses context modalities, categorized as structural cues, semantic descriptions, and orthologous group metadata, and comprises two main components:
a finite scalar quantizer for tokenizing continuous atomic coordinates, and
a transformer stack for multimodal representation learning.
Odyssey is trained via discrete diffusion, and characterizes the generative process as a time-dependent unmasking procedure.
The finite scalar quantizer and transformer stack leverage the consensus mechanism, a replacement for attention that uses an iterative propagation scheme informed by local agreements between residues.
Across various benchmarks, Odyssey achieves landmark performance for protein generation and protein structure discretization. Our empirical findings are supported by theoretical analysis."
Anthrogen introduces Odyssey, the world's largest and most powerful protein language model. (original news release)
Figure 2: Overall architecture schematic of Odyssey.
Figure 2: Illustration of self-consensus for d = 2, depicting the local neighborhood principle and the operative steps of self-consensus.
No comments:
Post a Comment