x) = uv1 x1 + uv2 x2 x1 x2 θ = (u, v1 , v2 ) h(θ) = v2 1 + v2 2 − u2 Conserved function: Neural network 2D 1D, 1 hidden neuron: → Independent conservation laws hk,k′  (U, V) = ⟨uk , uk′  ⟩ − ⟨vk , vk′  ⟩ Linear networks ReLu networks σ(s) = max(s,0) σ(s) = s hk (U, V) = ∥uk ∥2 − ∥vk ∥2 How many? Determine them? (h1 , …, hK ) conserved ⟹ Φ(h1 , …, hK ) conserved Independence: ∀θ, (∇h1 (θ), …, ∇hK (θ)) are independent g(θ, x) := Uσ(V⊤x) = ∑ k uk σ(⟨x, vk ⟩) Example: θ = (U, V) σ σ 5 /15