ECE 563 - Information Theory

Summer 2009 | Fall 2009 | Spring 2010 | Summer 2010
Section Type Times Days Location Instructor
A DIS 0800 - 0920 T R   163 Everitt Lab  Richard Blahut

Web Page http://courses.ece.uiuc.edu/ece563
Official Description Mathematical models for channels and sources; entropy, information, data compression, channel capacity, Shannon's theorems, and rate-distortion theory. Same as CS 578 and STAT 563. Prerequisite: One of ECE 534, MATH 464, MATH 564.
Hours 4 hours.
Subject Area Communications
Course Prerequisites Credit in MATH 464 or MATH 564 or ECE 534
Course Directors Richard E Blahut
Description Mathematical models for channels and sources; entropy, information, data compression, channel capacity, Shannon's theorems, rate-distortion theory.
Notes Same as: CS 578 and STAT 563
Credit 4 hours
Topics
  • Entropy, relative entropy, mutual information
  • Asymptotic equipartition property
  • Entropy rates of a stochastic process
  • Lossless data compression (Huffman, Ziv-Lempel, Arithmetic, Shannon-Fano codes): Kraft inequality, Shannon's source coding theorem
  • Channel capacity: jointly typical sequences, Fano's inequality, Shannon's channel coding theorem and its converse
  • Differential entropy
  • Gaussian channels
  • Rate-distortion theory: Shannon's source coding theorem relative to a fidelity criterion
Course Prerequisites One of MATH 464, MATH 564, or ECE 534.
Texts T. Cover and J. Thomas, Elements of Information Theory, Wiley, 1991.