Exponential Families & Information Maximization

Exponential families are natural statistical models. In physics they are used since their elements maximize the entropy subject to constrained expectation values of a fixed set of associated observables. An important subclass are the graphical and hierarchical (log linear) models that are used to model interactions between different random variables. They also appear in information geometry and algebraic statistics due to their nice structural properties.

The information distance from an exponential family has an interpretation as information loss through a projection onto that family. Mutual information, conditional mutual information and multi-information allow for such a geometric interpretation. In this project we analyze the maximization of the distance from exponential families. This problem is motivated by principles of information maximization known from theoretical neuroscience. The project aims at identifying natural models of learning systems that are consistent with information maximization and, at the same time, display high generalization ability. In this context, topological closures of exponential families turn out to be essential. Geometrically they are equivalent to polytopes and display a rich combinatorial structure.

Amari, S. and N. Ay: Standard divergence in manifold of dual affine connections. Geometric science of information : second international conference, GSI 2015, Palaiseau, France, October 28-30, 2015, proceedings / F. Nielsen... (eds.). Springer, 2015. - P. 320-325 (Lecture notes in computer science ; 9389) Bibtex [DOI]

Kahle, T. ; Rauh, J. and S. Sullivant: Positive margins and primary decomposition. Journal of commutative algebra, 6 (2014) 2, p. 173-208Bibtex [DOI] [ARXIV]

Montúfar, G. and J. Rauh: Scaling of model approximation errors and expected entropy distances. Kybernetika, 50 (2014) 2, p. 234-245Bibtex [DOI] [ARXIV]

Montúfar, G. ; Rauh, J. and N. Ay: Maximal information divergence from statistical models defined by neural networks. Geometric science of information : first international conference, GSI 2013, Paris, France, August 28-30, 2013. Proceedings / F. Nielsen... (eds.). Springer, 2013. - P. 759-766 (Lecture notes in computer science ; 8085) Bibtex MIS-Preprint: 31/2013 [DOI] [ARXIV]

Rauh, J. : Optimally approximating exponential families. Kybernetika, 49 (2013) 2, p. 199-215Bibtex MIS-Preprint: 73/2011 [ARXIV] [FREELINK]

Montúfar, G. and J. Rauh: Scaling of model approximation errors and expected entropy distances. Proceedings of the 9th workshop on uncertainty processing WUPES '12 : Marianske Lazne, Czech Republik ; 12-15th September 2012 Academy of Sciences of the Czech Republik / Institute of Information Theory and Automation, 2012. - P. 137-148Bibtex[ARXIV] [FREELINK]

Matus, F. and J. Rauh: Maximization of the information divergence from an exponential family and criticality. IEEE international symposium on information theory proceedings (ISIT) 2011 : July 31-August 5, 2011 in St. Petersburg, Russia IEEE, 2011. - P. 903-907Bibtex [DOI]

Rauh, J. : Finding the maximizers of the information divergence from an exponential family. IEEE transactions on information theory, 57 (2011) 6, p. 3236-3247Bibtex MIS-Preprint: 82/2009 [DOI] [ARXIV]

Rauh, J. : Finding the maximizers of the information divergence from an exponential family. Dissertation, Universität Leipzig, 2011Bibtex[FREELINK]

Rauh, J. ; Kahle, T. and N. Ay: Support sets in exponential families and oriented matroid theory. International journal of approximate reasoning, 52 (2011) 5, p. 613-626Bibtex MIS-Preprint: 28/2009 [DOI] [ARXIV]

Kahle, T. : Neighborliness of marginal polytopes. Beiträge zur Algebra und Geometrie, 51 (2010) 1, p. 45-56Bibtex MIS-Preprint: 57/2008 [ARXIV] [FREELINK]

Kahle, T. ; Wenzel, W. and N. Ay: Hierarchical models, marginal polytopes, and linear codes. Kybernetika, 45 (2009) 2, p. 189-207Bibtex MIS-Preprint: 30/2008 [ARXIV] [FREELINK]

Wennekers, T. ; Ay, N. and P. Andras: High-resolution multiple-unit EEG in cat auditory cortex reveals large spatio-temporal stochastic interactions. Biosystems, 89 (2007) 1/3, p. 190-197Bibtex [DOI]

Ay, N. and A. Knauf: Maximizing multi-information. Kybernetika, 42 (2006) 5, p. 517-538Bibtex MIS-Preprint: 42/2003 [ARXIV]

Kahle, T. and N. Ay: Support sets of distributions with given interaction structure. 7th Workshop on Uncertainty Processing : WUPES'06 ; Mikulov, Czech Republik ; 16-20th September 2006 Academy of Sciences of the Czech Republik / Institute of Information Theory and Automation, 2006. - P. 52-61Bibtex MIS-Preprint: 94/2006 [FREELINK]

Wennekers, T. and N. Ay: A temporal learning rule in recurrent systems supports high spatio-temporal stochastic interactions. Neurocomputing, 69 (2006) 10/12, p. 1199-1202Bibtex [DOI]

Wennekers, T. and N. Ay: Finite state automata resulting from temporal information maximization and a temporal learning rule. Neural computation, 17 (2005) 10, p. 2258-2290Bibtex [DOI]

Wennekers, T. and N. Ay: Stochastic interaction in associative nets. Neurocomputing, 65 (2005), p. 387-392Bibtex [DOI]

Ay, N. and T. Wennekers: Dynamical properties of strongly interacting Markov chains. Neural networks, 16 (2003) 10, p. 1483-1497Bibtex MIS-Preprint: 107/2001 [DOI]

Ay, N. and T. Wennekers: Temporal infomax leads to almost deterministic dynamical systems. Neurocomputing, 52 (2003) 4, p. 461-466Bibtex [DOI]

Matus, F. and N. Ay: On maximization of the information divergence from an exponential family. Proceedings of 6th workshop on uncertainty processing : Hejnice, September 24-27, 2003 Oeconomica, 2003. - P. 199-204Bibtex MIS-Preprint: 46/2003

Wennekers, T. and N. Ay: Temporal Infomax on Markov chains with input leads to finite state automata. Neurocomputing, 52 (2003) 4, p. 431-436Bibtex [DOI]

Wennekers, T. and N. Ay: Spatial and temporal stochastic interaction in neuronal assemblies. Theory in biosciences, 122 (2003) 1, p. 5-18Bibtex [DOI]

Ay, N. : An information-geometric approach to a theory of pragmatic structuring. The annals of probability, 30 (2002) 1, p. 416-436Bibtex MIS-Preprint: 52/2000 [FREELINK]

Ay, N. : Locality of global stochastic interaction in directed acyclic networks. Neural computation, 14 (2002) 12, p. 2959-2980Bibtex MIS-Preprint: 54/2001 [DOI]

Wennekers, T. and N. Ay: Information-theoretic grounding of finite automata in neural systems. Bibtex MIS-Preprint: 52/2002

Ay, N. : Aspekte einer Theorie pragmatischer Informationsstrukturierung. Dissertation, Universität Leipzig, 2001Bibtex