Overview
Sometimes the best description of a system is one that acknowledges our ignorance. Maximum entropy methods provide a coherent framework for doing this in a rigorous fashion, and their success has been felt in everything from physics to ecology to machine learning.
In this tutorial, Simon DeDeo introduces MaxEnt through a series of examples, taking students from the basics of the subject up to its applications in science and engineering. Basic comfort with the use of probabilities, and familiarity with exponentials and logarithms, is required. By the end of the course, students will have a tool for modeling complex systems, and a new set of concepts for thinking about what models are meant to do in the first place.
Note that Complexity Explorer tutorials are meant to introduce students to various important techniques and to provide illustrations of their application in complex systems. A given tutorial is not meant to offer complete coverage of its topic or substitute for an entire course on that topic.
This tutorial is designed for more advanced math students. Math prerequisites for this course are an understanding of calculus, partial derivatives, shannon entropy; basic probability.
Read Simon DeDeo’s Q&A about his Maximum Entropy tutorial here.
Syllabus
- Maximum Entropy Methods (Simon DeDeo)
- A Simple Example: Waiting for a Taxicab
- The Maximum Entropy Method
- MaxEnt Applied to the Taxicab Example, Part 1
- MaxEnt Applied to the Taxicab Example, Part 2
- Review of MaxEnt
- A Real-World Example: Modeling the Open Source Ecosystem, Part 1
- Modeling the Open Source Ecosystem, Part 2
- Modeling the Open Source Ecosystem, Part 3
- A Second Real-World Example: Modeling Sears-Roebuck Catalog Prices, Part 1
- Modeling Sears-Roebuck Catalog Prices, Part 2
- Conclusion
- Homework