ECE 751 / ECE 886- Spring 2008

INFORMATION THEORY

Professor Yariv Ephraim

STII Room 221

 

 

Time:                   Wednesday 4:30-7:10 pm 

 

Place:                        STII   260 

 

Final Exam:               Take home exam to be assigned on Thursday 30 April and due on Thursday 7 May by 4:30pm in my office.

 

Office Hours:             Monday: 6:00-7:00 pm 

Wednesday 3:15-4:15 pm 

Other time by appointment

 

To contact me please use yephraim@gmu.edu

 

Course Description:

 

Information theory provides the theoretical basis of communications. It defines information and channel capacity, and provides coding theorems for sources and channels. These theorems set limits on the achievable communication rates in terms of the source entropy and channel capacity. In this course we define the basic information measures, prove source and channel coding theorems, study practical algorithms for data compression, and go over some related statistical aspects of information theory such as hypothesis testing.

 

 

 

Course Outline:

 

  1. Introduction and review (Chap 1, week 1)
  2. Measures of information: entropy, relative entropy and mutual information (Chap 2, weeks 2,3)
  3. The asymptotic equipartition property and the Shannon-McMillan-Breiman Theorem (Chap 3, week 4)
  4. Entropy rate and the Source-Coding Theorem  (Chap 4, week 5)
  5. Huffman code, Arithmetic code and Lempel-Ziv code  (Chap 5, weeks 6,7)
  6. Take home mid-term (week 8)
  7. The method of Types (Chap 11, week 9)
  8. Channel capacity and the channel-coding theorem. (Chap 7, weeks 10-11)
  9. Joint source-channel coding theorem (Chap 7, week 11)
  10. Differential entropy (Chap 8, week 11)
  11. The Gaussian channel (Chap 9, week 12)
  12. Rate distortion theory and quantization    (Chap 10, week 13)
  13. Hypothesis testing   (Chap 11, week 14)                       

 

 

Text Books:

 

  1. T. M. Cover and J. A. Thomas, Elements of Information Theory. John Wiley & Sons, Inc., New York, 2006. (Required)
  2. R. G. Gallager, Information Theory and Reliable Communication. John Wiley & Sons, Inc., New York, 1968.

 

Prerequisite:

 

ECE-528 or instructor permission

 

Communication:

 

Course announcements, assignments and homework solutions will be emailed to you. I will use your email addresses which are on file at the GMU Registrar. If you wish to have your course material delivered to another email address, you may include a .forward command in your GMU directory. It is important that your mail box does not reach its capacity at any time during the semester.

 

Grading:

 

Homework 15%, Mid-term 40%, Final 45%