Sparse coding and efficient sensing

For being able to act autonomously in its world, an agent has to learn appropriate internal representations. In the brain we find simple and complex cells and the concept of sparse coding. Sparse coding seems to mirror the structure of natural scenes and signals, and the combination of simple and complex cells can provide invariant representations. We will give an introduction to sparse coding and show how the state of a sparsely encodeable world can be sensed very efficiently, e.g. by compressive sensing or adaptive hierarchical sensing. We will show how abstract and generalized simple and complex cells lead to representations which are invariant to a large class of transformations, which is the basis for the success of deep convolutional networks. We will see that deep convolutional networks with their simple and complex cells favor sparse codes.