Documents
Presentation Slides
ndzip: A High-Throughput Parallel Lossless Compressor for Scientific Data
- Citation Author(s):
- Submitted by:
- Fabian Knorr
- Last updated:
- 3 March 2021 - 1:34am
- Document Type:
- Presentation Slides
- Document Year:
- 2021
- Event:
- Presenters:
- Fabian Knorr
- Categories:
- Log in to post comments
Exchanging large amounts of floating-point data is common in distributed scientific computing applications. Data compression, when fast enough, can speed up such workloads by reducing the time spent waiting for data transfers. We propose ndzip, a high-throughput, lossless compression algorithm for multi-dimensional univariate regular grids of single- and double-precision floating point data. Tailored towards efficient implementation on modern SIMD-capable multicore processors, it compresses and decompresses data at speeds close to main memory bandwidth, significantly outperforming existing schemes. We evaluate this novel method using a representative set of scientific data, demonstrating a competitive trade-off between compression effectiveness and throughput.