Information Theory: Part I: An Introduction to the Fundamental Concepts

Description

This book is about the definition of the Shannon measure of Information, and some derived quantities such as conditional information and mutual information. Unlike many books, which refer to the Shannon's Measure of information (SMI) as "Entropy," this book makes a clear distinction between the SMI and Entropy. In the last chapter, Entropy is derived as a special case of SMI. Ample examples are provided which help the reader in understanding the different concepts discussed in this book. As with previous books by the author, this book aims at a clear and mystery-free presentation of the central concept in Information theory — the Shannon's Measure of Information. This book presents the fundamental concepts of Information theory in a friendly-simple language and is devoid of all kinds of fancy and pompous statements made by authors of popular science books who write on this subject. It is unique in its presentation of Shannon's measure of information, and the clear distinction between this concept and the thermodynamic entropy. Although some mathematical knowledge is required by the reader, the emphasis is on the concepts and their meaning rather on the mathematical details of the theory. Contents Preface Acknowledgements Chapter 0: Elements of Probability Theory Chapter 1: Introduction, Definition, and Interpretations of Shannon's Measure of Information Chapter 2: Properties of Shannon's Measure of Information Chapter 3: Conditional and Mutual Information Chapter 4: Multivariate Mutual Information Chapter 5: Entropy and the Second Law of Thermodynamics Appendix A: Proof of an Equivalent Markovian Property Appendix B: Proof of the Uniqueness of the Function H Appendix C: The SMI for the Continuous Random Variable Appendix D: Functional Derivatives and Functional Taylor Expansion Appendix E: Some Inequalities for Convex Functions Appendix F: Distribution Functions in 1D Models Appendix G: Entropy Change in an Expansion Process in a Gravitational Field Notes References and Suggested Reading Index