Part VII: Modern Applications
Information Theory in the Modern World
Shannon's theoretical framework, developed in 1948, now underpins virtually every digital technology. Data compression reduces the bandwidth and storage required for everything from text files to streaming video. Cryptographic systems exploit information-theoretic principles to guarantee security. And machine learning is increasingly understood through the lens of information theory.
This final part surveys three major application domains: compression algorithms from LZ77 to modern video codecs, the information bottleneck framework connecting deep learning to rate-distortion theory, and the information-theoretic foundations of cryptography and network coding.
Chapters in This Part
Chapter 19: Data Compression
Lempel-Ziv (LZ77/LZ78), DEFLATE (ZIP), JPEG (DCT + quantization), MP3 psychoacoustic model, modern video codecs. Achieving near-entropy compression.
Chapter 20: Information Bottleneck & ML
Tishby's information bottleneck, deep learning as compression, mutual information in neural networks, PAC-Bayes bounds, the information plane.
Chapter 21: Cryptography & Network Information Theory
One-time pad (perfect secrecy), Shannon's secrecy capacity, public-key cryptography, network coding, and Slepian-Wolf distributed source coding.
Key Results in This Part
LZ complexity: Lempel-Ziv algorithms achieve entropy rate for ergodic sources asymptotically
Information bottleneck: \( \min_{p(t|x)} I(X;T) - \beta I(T;Y) \) — compressed representation \(T\) of \(X\) preserving info about \(Y\)
Perfect secrecy: \( H(M|C) = H(M) \) requires key length \(\geq\) message length
Slepian-Wolf: Distributed sources \(X,Y\) can be compressed to rates \(R_X + R_Y \geq H(X,Y)\)