Bit Grooming: statistically accurate precision-preserving quantization with compression, evaluated in the netCDF Operators (NCO, v4.4.8+)
Tóm tắt
Abstract. Geoscientific models and measurements generate false precision (scientifically meaningless data bits) that wastes storage space. False precision can mislead (by implying noise is signal) and be scientifically pointless, especially for measurements. By contrast, lossy compression can be both economical (save space) and heuristic (clarify data limitations) without compromising the scientific integrity of data. Data quantization can thus be appropriate regardless of whether space limitations are a concern. We introduce, implement, and characterize a new lossy compression scheme suitable for IEEE floating-point data. Our new Bit Grooming algorithm alternately shaves (to zero) and sets (to one) the least significant bits of consecutive values to preserve a desired precision. This is a symmetric, two-sided variant of an algorithm sometimes called Bit Shaving that quantizes values solely by zeroing bits. Our variation eliminates the artificial low bias produced by always zeroing bits, and makes Bit Grooming more suitable for arrays and multi-dimensional fields whose mean statistics are important. Bit Grooming relies on standard lossless compression to achieve the actual reduction in storage space, so we tested Bit Grooming by applying the DEFLATE compression algorithm to bit-groomed and full-precision climate data stored in netCDF3, netCDF4, HDF4, and HDF5 formats. Bit Grooming reduces the storage space required by initially uncompressed and compressed climate data by 25–80 and 5–65 %, respectively, for single-precision values (the most common case for climate data) quantized to retain 1–5 decimal digits of precision. The potential reduction is greater for double-precision datasets. When used aggressively (i.e., preserving only 1–2 digits), Bit Grooming produces storage reductions comparable to other quantization techniques such as Linear Packing. Unlike Linear Packing, whose guaranteed precision rapidly degrades within the relatively narrow dynamic range of values that it can compress, Bit Grooming guarantees the specified precision throughout the full floating-point range. Data quantization by Bit Grooming is irreversible (i.e., lossy) yet transparent, meaning that no extra processing is required by data users/readers. Hence Bit Grooming can easily reduce data storage volume without sacrificing scientific precision or imposing extra burdens on users.
Từ khóa
Tài liệu tham khảo
Burtscher, M. and Ratanaworabhan, P.: FPC: A high-speed compressor for double-precision floating-point data, IEEE T. Comput., 58, 18–31, https://doi.org/10.1109/TC.2008.131, 2009.
Caron, J.: Compression by scaling and offset, available at: http://www.unidata.ucar.edu/blogs/developer/entry/compression_by_scaling_and_offfset (last access: 13 September 2016), 2014a.
Caron, J.: Compression by bit shaving, available at: http://www.unidata.ucar.edu/blogs/developer/entry/compression_by_bit_shaving (last access: 13 September 2016), 2014b.
Collet, Y.: LZ4 lossless compression algorithm, available at: http://lz4.org (last access: 13 September 2016), 2013.
Dennis, J. M., Edwards, J., Evans, K. J., Guba, O., Lauritzen, P. H., Mirin, A. A., St-Cyr, A., Taylor, M. A., and Worley, P. H.: CAM-SE: A scalable spectral element dynamical core for the Community Atmosphere Model, Int. J. High Perform. C., 26, 74–89, https://doi.org/10.1177/1094342011428142, 2012.
Deutsch, L. P.: DEFLATE compressed data format specification version 1.3, Tech. Rep. IETF RFC1951, Internet Engineering Task Force, Menlo Park, CA, USA, https://doi.org/10.17487/RFC1951, 1996.
Eaton, B., Gregory, J., Drach, B., Taylor, K., and Hankin, S.: NetCDF Climate and Forecast (CF) metadata conventions, available at: http://cfconventions.org/cf-conventions, last access: 13 September 2016.
Gailly, J.-L. and Adler, M.: zlib documentation, available at: http://zlib.net (last access: 13 September 2016), 2000.
Gregory, J.: The CF metadata standard, CLIVAR Exchanges, 8, 4, available at: http://cfconventions.org/Data/cf-documents/overview/article.pdf (last access: 13 September 2016), 2003.
HDF Group: HDF5: API Specification Reference Manual, The HDF Group, Champaign-Urbana, IL, USA, 2015.
IEEE: IEEE standard for floating-point arithmetic, Tech. Rep. ISO/IEC/IEEE 60559 (IEEE Std 754-2008), IEEE Computer Society, Piscataway, NJ, USA, 2008.
Isenburg, M., Lindstrom, P., and Snoeyink, J.: Lossless compression of predicted floating-point geometry, Comput. Aided Design, 37, 869–877, https://doi.org/10.1016/j.cad.2004.09.015, 2005.
Krotkov, N. A., McClure, B., Dickerson, R. R., Carn, S. A., Li, C., Bhartia, P. K., Yang, K., Krueger, A. J., Li, Z., Levelt, P. F., Chen, H., Wang, P., and Lu, D.: Validation of SO2 retrievals from the Ozone Monitoring Instrument over NE China, J. Geophys. Res., 113, D16S40, https://doi.org/10.1029/2007JD008818, 2008.
Liu, S., Huang, X., Ni, Y., Fu, H., and Yang, G.: A high performance compression method for climate data, in: IEEE International Symposium on Parallel and Distributed Processing with Applications, 26–28 August 2014, Milan, Italy, 68–77, https://doi.org/10.1109/ISPA.2014.18, 2014.
Rew, R., Hartnett, E., and Caron, J.: NetCDF-4: Software implementing an enhanced data model for the geosciences, in: Proceedings of the 22nd AMS Conference on Interactive Information and Processing Systems for Meteorology, 24–28 January 2006, p. 6.6, American Meteorological Society, AMS Press, Boston, MA, USA, 2006.
Rew, R., Davis, G., Emmerson, S., and Davies, H.: The NetCDF Users' Guide, Version 3.6.1, University Corporation for Atmospheric Research, Boulder, CO, USA, available at: http://www.unidata.ucar.edu/software/netcdf/docs/user_guide.html, last access: 13 September 2016.
Rienecker, M. M., Suarez, M. J., Gelaro, R., Todling, R., Bacmeister, J., Liu, E., Bosilovich, M. G., Schubert, S. D., Takacs, L., and Kim, G.-K.: MERRA: NASA's modern-era retrospective analysis for research and applications, J. Climate, 24, 3624–3648, 2011.
Salomon, D. and Molta, G.: Handbook of Data Compression, 5th ed., Springer-Verlag, London, UK, 2010.
Seward, J.: bzip2 documentation, available at: http://bzip.org (last access: 13 September 2016), 2007.
Silver, J. D. and Zender, C. S.: Finding the Goldilocks zone: Compression-error trade-off for large gridded datasets, Geosci. Model Dev. Discuss., https://doi.org/10.5194/gmd-2016-177, in review, 2016.
Zender, C. S.: Analysis of self-describing gridded geoscience data with netCDF Operators (NCO), Environ. Modell. Softw., 23, 1338–1342, https://doi.org/10.1016/j.envsoft.2008.03.004, 2008.
Zender, C. S.: NCO User Guide, available at: http://nco.sf.net/nco.pdf, last access: 13 September 2016a.
Zender, C. S.: netCDF Operators (NCO), version 4.6.1, Zenodo, https://doi.org/10.5281/zenodo.61341, 2016b.
Zender, C. S. and Mangalam, H. J.: Scaling properties of common statistical operators for gridded datasets, Int. J. High Perform. C., 21, 458–498, https://doi.org/10.1177/1094342007083802, 2007.
Zender, C. S., Bian, H., and Newman, D.: Mineral Dust Entrainment And Deposition (DEAD) model: Description and 1990s dust climatology, J. Geophys. Res., 108, 4416, https://doi.org/10.1029/2002JD002775, 2003.
Ziv, J. and Lempel, A.: A universal algorithm for sequential data compression, IEEE T. Inform. Theory, 23, 337–343, 1977.