Exponential Families & Information Maximization

Exponential families are natural statistical models. In physics they are used since their elements maximize the entropy subject to constrained expectation values of a fixed set of associated observables. An important subclass are the graphical and hierarchical (log linear) models that are used to model interactions between different random variables. They also appear in information geometry and algebraic statistics due to their nice structural properties.

The information distance from an exponential family has an interpretation as information loss through a projection onto that family. Mutual information, conditional mutual information and multi-information allow for such a geometric interpretation. In this project we analyze the maximization of the distance from exponential families. This problem is motivated by principles of information maximization known from theoretical neuroscience. The project aims at identifying natural models of learning systems that are consistent with information maximization and, at the same time, display high generalization ability. In this context, topological closures of exponential families turn out to be essential. Geometrically they are equivalent to polytopes and display a rich combinatorial structure.

Felice, D. ; Mancini, S. and N. Ay: Canonical divergence for measuring classical and quantum complexity. Entropy, 21 (2019) 4, 435 Bibtex [DOI] [ARXIV]

Langer, C. and N. Ay: Comparison and connection between the joint and the conditional generalized iterative scaling algorithm. Proceedings of the 11th workshop on uncertainty processing WUPES '18, June 6-9, 2018 / V. Kratochvíl (ed.). MatfyzPress, 2018. - P. 105-116 Bibtex [FREELINK]

Amari, S. and N. Ay: Standard divergence in manifold of dual affine connections. Geometric science of information : second international conference, GSI 2015, Palaiseau, France, October 28-30, 2015, proceedings / F. Nielsen... (eds.). Springer, 2015. - P. 320-325 (Lecture notes in computer science ; 9389) Bibtex [DOI]

Kahle, T. ; Rauh, J. and S. Sullivant: Positive margins and primary decomposition. Journal of commutative algebra, 6 (2014) 2, p. 173-208 Bibtex [DOI] [ARXIV]

Montúfar, G. and J. Rauh: Scaling of model approximation errors and expected entropy distances. Kybernetika, 50 (2014) 2, p. 234-245 Bibtex [DOI] [ARXIV]

Montúfar, G. ; Rauh, J. and N. Ay: Maximal information divergence from statistical models defined by neural networks. Geometric science of information : first international conference, GSI 2013, Paris, France, August 28-30, 2013. Proceedings / F. Nielsen... (eds.). Springer, 2013. - P. 759-766 (Lecture notes in computer science ; 8085) Bibtex MIS-Preprint: 31/2013 [DOI] [ARXIV]

Rauh, J.: Optimally approximating exponential families. Kybernetika, 49 (2013) 2, p. 199-215 Bibtex MIS-Preprint: 73/2011 [ARXIV] [FREELINK]

Montúfar, G. and J. Rauh: Scaling of model approximation errors and expected entropy distances. Proceedings of the 9th workshop on uncertainty processing WUPES '12 : Marianske Lazne, Czech Republik ; 12-15th September 2012 Academy of Sciences of the Czech Republik / Institute of Information Theory and Automation, 2012. - P. 137-148 Bibtex [ARXIV] [FREELINK]

Matúš, F. and J. Rauh: Maximization of the information divergence from an exponential family and criticality. IEEE international symposium on information theory proceedings (ISIT) 2011 : July 31-August 5, 2011 in St. Petersburg, Russia IEEE, 2011. - P. 903-907 Bibtex [DOI]

Rauh, J.: Finding the maximizers of the information divergence from an exponential family. IEEE transactions on information theory, 57 (2011) 6, p. 3236-3247 Bibtex MIS-Preprint: 82/2009 [DOI] [ARXIV]

Rauh, J.: Finding the maximizers of the information divergence from an exponential family. Dissertation, Universität Leipzig, 2011 Bibtex [FREELINK]

Rauh, J. ; Kahle, T. and N. Ay: Support sets in exponential families and oriented matroid theory. International journal of approximate reasoning, 52 (2011) 5, p. 613-626 Bibtex MIS-Preprint: 28/2009 [DOI] [ARXIV]

Kahle, T.: Neighborliness of marginal polytopes. Beiträge zur Algebra und Geometrie, 51 (2010) 1, p. 45-56 Bibtex MIS-Preprint: 57/2008 [ARXIV] [FREELINK]

Kahle, T. ; Wenzel, W. and N. Ay: Hierarchical models, marginal polytopes, and linear codes. Kybernetika, 45 (2009) 2, p. 189-207 Bibtex MIS-Preprint: 30/2008 [ARXIV] [FREELINK]

Wennekers, T. ; Ay, N. and P. Andras: High-resolution multiple-unit EEG in cat auditory cortex reveals large spatio-temporal stochastic interactions. Biosystems, 89 (2007) 1/3, p. 190-197 Bibtex [DOI]

Ay, N. and A. Knauf: Maximizing multi-information. Kybernetika, 42 (2006) 5, p. 517-538 Bibtex MIS-Preprint: 42/2003 [ARXIV]

Kahle, T. and N. Ay: Support sets of distributions with given interaction structure. 7th Workshop on Uncertainty Processing : WUPES'06 ; Mikulov, Czech Republik ; 16-20th September 2006 Academy of Sciences of the Czech Republik / Institute of Information Theory and Automation, 2006. - P. 52-61 Bibtex MIS-Preprint: 94/2006 [FREELINK]

Wennekers, T. and N. Ay: A temporal learning rule in recurrent systems supports high spatio-temporal stochastic interactions. Neurocomputing, 69 (2006) 10/12, p. 1199-1202 Bibtex [DOI]

Wennekers, T. and N. Ay: Finite state automata resulting from temporal information maximization and a temporal learning rule. Neural computation, 17 (2005) 10, p. 2258-2290 Bibtex [DOI]

Wennekers, T. and N. Ay: Stochastic interaction in associative nets. Neurocomputing, 65 (2005), p. 387-392 Bibtex [DOI]

Ay, N. and T. Wennekers: Dynamical properties of strongly interacting Markov chains. Neural networks, 16 (2003) 10, p. 1483-1497 Bibtex MIS-Preprint: 107/2001 [DOI]

Ay, N. and T. Wennekers: Temporal infomax leads to almost deterministic dynamical systems. Neurocomputing, 52 (2003) 4, p. 461-466 Bibtex [DOI]

Matúš, F. and N. Ay: On maximization of the information divergence from an exponential family. Proceedings of 6th workshop on uncertainty processing : Hejnice, September 24-27, 2003 Oeconomica, 2003. - P. 199-204 Bibtex MIS-Preprint: 46/2003

Wennekers, T. and N. Ay: Temporal Infomax on Markov chains with input leads to finite state automata. Neurocomputing, 52 (2003) 4, p. 431-436 Bibtex [DOI]

Wennekers, T. and N. Ay: Spatial and temporal stochastic interaction in neuronal assemblies. Theory in biosciences, 122 (2003) 1, p. 5-18 Bibtex [DOI]

Ay, N.: An information-geometric approach to a theory of pragmatic structuring. The annals of probability, 30 (2002) 1, p. 416-436 Bibtex MIS-Preprint: 52/2000 [FREELINK]

Ay, N.: Locality of global stochastic interaction in directed acyclic networks. Neural computation, 14 (2002) 12, p. 2959-2980 Bibtex MIS-Preprint: 54/2001 [DOI]

Wennekers, T. and N. Ay: Information-theoretic grounding of finite automata in neural systems. Bibtex MIS-Preprint: 52/2002

Ay, N.: Aspekte einer Theorie pragmatischer Informationsstrukturierung. Dissertation, Universität Leipzig, 2001 Bibtex