Huge Memory AI Server Aims to Shatter the Memory Wall with up to 128 TB of DRAM per server

This is huge! Mind boggling!

"Memory is arguably the most serious constraint on modern AI large language models (LLMs). According to one influential paper, LLM token generation is an inherently memory-bound task, meaning the rate at which models output text is limited by how quickly data can be read in from memory. The severity of this bottleneck grows with model size. This creates a “memory wall” that holds back LLM inference performance.

AI hardware startup Majestic Labs is taking a direct—and comprehensive—approach to solving this problem. It’s developing a new AI server, Prometheus, with up to 128 terabytes of memory. That’s over 60 times more than Nvidia’s DGX B300 server, a cutting-edge AI processing rack. ..."

Huge Memory AI Server Aims to Shatter the Memory Wall - IEEE Spectrum "Majestic Labs’ Prometheus packs up to 128 TB of DRAM per server"

Source