Fibers and Singularities of ReLU Networks via Toric Path Lifting

I revisit the path-lifting factorization for ReLU networks showing that it carries a toric-geometric structure and helps describe the fibers and singularities of the realization map. The lift maps the weights to their products along input-output paths and is invariant under the positive rescaling symmetries of ReLU networks. The readout then aggregates the active paths and is linear on each fixed activation-pattern branch. Consequently, for a finite dataset, the lifted fiber is a union of activation-restricted affine sections of the toric variety. This factorization also distinguishes two sources of singular behavior: degeneracies on the boundary of the toric lift and non-clean intersections between the lifted variety and the affine readout constraints. This provides a common geometric framework for studying parameter redundancy and singularities and ultimately their role in optimization and generalization in overparameterized networks.