New AI tool compresses data without losing critical details

Good news!

Data compression has been a fundamental subject for computer science and data science for several decades. How to preserve critical details is one of the major challenges.

"In brief

• The next generation of science experiments will produce vast amounts of data, challenging capacity for storage and analysis.
• SLAC researchers developed a new method that uses AI to compress data without erasing subtle details that are valuable for experiments but lost in conventional file-compression methods.
• The method separates features of a dataset by size, then shrinks and maps these features onto a neural network to preserve them.

...

Conventional data-compression methods can erase some of the fine details in measurements that correspond to valuable scientific information. For example, the tiny speckles in X-ray images of molecules can contain important information about how materials transform. ..."

From the abstract:

"Scientific data acquisition continues to outpace storage and analysis capabilities, making voxel-based representations increasingly intractable.

Implicit neural representations (INRs) offer a promising solution by encoding signals through coordinate-based neural networks, serving as surrogates of data, with computational and storage requirements scaling with network complexity rather than data dimensionality.

However, smaller INRs struggle to faithfully represent multiscale structures, high-frequency information and fine textures that constitute a large proportion of scientific measurements.

We propose WIEN-INR, a theoretically guided hierarchical INR framework that distributes modelling across resolution scales and enables improved representation capacity through a novel enhancement network to recover subtle details.

This multiscale architecture allows smaller networks to retain the full spatial-frequency content of the signal as well as preserve training efficiency and lower storage cost.

Evaluated on distinct raw experimental measurements across scales and complexities, WIEN-INR represents a practical step towards a broader adoption of neural representations in scientific workflows, delivering compact, robust and high-fidelity representations."

New AI tool compresses data without losing critical details | Stanford Report "A neural network developed by SLAC researchers shrinks massive experimental datasets while retaining all crucial information, enabling faster discovery across fields."

Multi-resolution enhancement for full-spectrum neural representations (open access)

The AI-based method uses neural networks to reduce the overall file size while allowing users to control what information is kept. The neural network encodes features of the measurement at different scales in a compact form. At decoding, users can select a small region of interest to decode, and this can be done at different scales and resolutions, allowing finer features to be recovered when needed.

Fig. 1: Efficient INRs preserve HF and physics-related details.

Fig. 2: Schematic of the WIEN-INR architecture.