A study of representation learning without encoders

Latent variables models have long been used in dimensionality reduction and representation learning. By far, the most common method is based on the Autoencoder, which is formed by an encoder-decoder pair. Recently, a model was introduced called the Gradient Origin Network (GON) which replaces the encoder with a gradient descent step. While being shown to be similarly performant, not much else is known about how it reaches representations. This work extends known results applicable to autoencoders such as its convergence to PCA solutions, while trying to establish representational similarity. As GONs have a variational allegory, we establish the presence of posterior collapse and a polarised regime in variational GONs (VGONs). Finally, we extend the definition of the polarised regime to better encapsulate a wider variety of models beyond just variational models based on a Gaussian prior.