An Information-Theoretic Approach to Analog-to-Digital Compression

doi:10.1017/9781108616799.003

2 - An Information-Theoretic Approach to Analog-to-Digital Compression

Published online by Cambridge University Press: 22 March 2021

Alon Kipnis,

Yonina C. Eldar and

Andrea J. Goldsmith

Edited by

Miguel R. D. Rodrigues and

Yonina C. Eldar

Show author details

Miguel R. D. Rodrigues: Affiliation:
University College London
Yonina C. Eldar: Affiliation:
Weizmann Institute of Science, Israel

Book contents

Get access

Summary

Processing, storing, and communicating information that originates as an analog phenomenon involve conversion of the information to bits. This conversion can be described by the combined effect of sampling and quantization. The digital representation in this procedure is achieved by first sampling the analog signal so as to represent it by a set of discrete-time samples and then quantizing these samples to a finite number of bits. Traditionally, these two operations are considered separately. The sampler is designed to minimize information loss due to sampling based on prior assumptions about the continuous-time input. The quantizer is designed to represent the samples as accurately as possible, subject to the constraint on the number of bits that can be used in the representation. The goal of this chapter is to revisit this paradigm by considering the joint effect of these two operations and to illuminate the dependence between them.

Keywords

analog-to-digital conversion digital-to-analog conversion sampling distortion rate rate distortion Nyquist sampling sub-Nyquist sampling aliasing ADX

Information

Type: Chapter
Information: Information-Theoretic Methods in Data Science , pp. 44 - 71
DOI: https://doi.org/10.1017/9781108616799.003 [Opens in a new window]
Publisher: Cambridge University Press
Print publication year: 2021

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Book purchase

Temporarily unavailable

References

Eldar, Y. C., Sampling theory: Beyond bandlimited systems. Cambridge University Press, 2015.Google Scholar

Gray, R. M. and Neuhoff, D. L., “Quantization,” IEEE Trans. Information Theory, vol. 44, no. 6, pp. 2325–2383, 1998.Google Scholar

Shannon, C. E., “Communication in the presence of noise,” IRE Trans. Information Theory, vol. 37, pp. 10–21, 1949.Google Scholar

Landau, H., “Sampling, data transmission, and the Nyquist rate,” Proc. IEEE, vol. 55, no. 10, pp. 1701–1706, 1967.Google Scholar

Shannon, C. E., “A mathematical theory of communication,” Bell System Technical J., vol. 27, pp. 379–423, 623–656, 1948.Google Scholar

Shannon, C. E., “Coding theorems for a discrete source with a fidelity criterion,” IRE National Convention Record, vol. 4, no. 1, pp. 142–163, 1959.Google Scholar

Berger, T., Rate-distortion theory: A mathematical basis for data compression. Prentice-Hall, 1971.Google Scholar

Walden, R., “Analog -to-digital converter survey and analysis,” IEEE J. Selected Areas in Communications, vol. 17, no. 4, pp. 539–550, 1999.Google Scholar

Candy, J., “A use of limit cycle oscillations to obtain robust analog-to-digital converters,” IEEE Trans. Communications, vol. 22, no. 3, pp. 298–305, 1974.Google Scholar

Oliver, B., Pierce, J., and Shannon, C., “The philosophy of PCM,” IRE Trans. Information Theory, vol. 36, no. 11, pp. 1324–1331, 1948.Google Scholar

Neuhoff, D. L. and Pradhan, S. S., “Information rates of densely sampled data: Distributed vector quantization and scalar quantization with transforms for Gaussian sources,” IEEE Trans. Information Theory, vol. 59, no. 9, pp. 5641–5664, 2013.Google Scholar

Matthews, M., “On the linear minimum-mean-squared-error estimation of an undersampled wide-sense stationary random process,” IEEE Trans. Signal Processing, vol. 48, no. 1, pp. 272–275, 2000.Google Scholar

Chan, D. and Donaldson, R., “Optimum pre- and postfiltering of sampled signals with application to pulse modulation and data compression systems,” IEEE Trans. Communication Technol., vol. 19, no. 2, pp. 141–157, 1971.Google Scholar

Dobrushin, R. and Tsybakov, B., “Information transmission with additional noise,” IRE Trans. Information Theory, vol. 8, no. 5, pp. 293–304, 1962.Google Scholar

Kolmogorov, A., “On the Shannon theory of information transmission in the case of continuous signals,” IRE Trans. Information Theory, vol. 2, no. 4, pp. 102–108, 1956.Google Scholar

Lapidoth, A., “On the role of mismatch in rate distortion theory,” IEEE Trans. Information Theory, vol. 43, no. 1, pp. 38–47, 1997.Google Scholar

Kontoyiannis, I. and Zamir, R., “Mismatched codebooks and the role of entropy coding in lossy data compression,” IEEE Trans. Information Theory, vol. 52, no. 5, pp. 1922–1938, 2006.Google Scholar

Zemanian, A. H., Distribution theory and transform analysis: An introduction to generalized functions, with applications. Courier Corporation, 1965.Google Scholar

Wolf, J. and Ziv, J., “Transmission of noisy information to a noisy receiver with minimum distortion,” IEEE Trans. Information Theory, vol. 16, no. 4, pp. 406–411, 1970.Google Scholar

Witsenhausen, H., “Indirect rate distortion problems,” IEEE Trans. Information Theory, vol. 26, no. 5, pp. 518–521, 1980.Google Scholar

Landau, H., “Necessary density conditions for sampling and interpolation of certain entire functions,” Acta Mathematica, vol. 117, no. 1, pp. 37–52, 1967.Google Scholar

Beurling, A. and Carleson, L., The collected works of Arne Beurling: Complex analysis. Birkhäuser, 1989, vol. 1.Google Scholar

Beutler, F. J., “Sampling theorems and bases in a Hilbert space,” Information and Control, vol. 4, nos. 2–3, pp. 97–117, 1961.Google Scholar

Kipnis, A., Eldar, Y. C., and Goldsmith, A. J., “Fundamental distortion limits of analog-to-digital compression,” IEEE Trans. Information Theory, vol. 64, no. 9, pp. 6013–6033, 2018.Google Scholar

Bennett, W., “Statistics of regenerative digital transmission,” Bell Labs Technical J., vol. 37, no. 6, pp. 1501–1542, 1958.Google Scholar

Kipnis, A., Goldsmith, A. J., and Eldar, Y. C., “The distortion rate function of cyclostationary Gaussian processes,” IEEE Trans. Information Theory, vol. 64, no. 5, pp. 3810–3824, 2018.Google Scholar

Kipnis, A., Goldsmith, A. J., Eldar, Y. C., and Weissman, T., “Distortion rate function of sub-Nyquist sampled Gaussian sources,” IEEE Trans. Information Theory, vol. 62, no. 1, pp. 401–429, 2016.Google Scholar

Kipnis, A., “Fundamental distortion limits of analog-to-digital compression,” Ph.D. dissertation, Stanford University, 2018.Google Scholar

Kipnis, A., Goldsmith, A. J., and Eldar, Y. C., “The distortion-rate function of sampled Wiener processes,” in IEEE Transactions on Information Theory, vol. 65, no.1, pp. 482-499, Jan. 2019. doi: 10.1109/TTT.2018.2878446Google Scholar

Kipnis, A., Reeves, G., and Eldar, Y. C., “Single letter formulas for quantized compressed sensing with Gaussian codebooks,,” in 2018 IEEE International Symposium on Information Theory (ISIT), 2018, pp. 71–75.Google Scholar

Kipnis, A., Reeves, G., Eldar, Y. C., and Goldsmith, A. J., “Compressed sensing under optimal quantization,,” in 2017 IEEE International Symposium on Information Theory (ISIT), 2017, pp. 2148–2152.Google Scholar

Accessibility standard: Unknown

Why this information is here

This section outlines the accessibility features of this content - including support for screen readers, full keyboard navigation and high-contrast display options. This may not be relevant for you.

Accessibility Information

Accessibility compliance for the PDF of this chapter is currently unknown and may be updated in the future.