[Paper Review] Dimensionality Reduction as Probabilistic Inference

1. Probabilistic Dimensionality Reduction

The Dimensionality Reduction is a field to make an intractable high-dimensional data into tractable row-dimensional data by extracting essential information. By shrinking the size of the data, we can expect following advantages.

• Shrinking computational complexity
• Figuring out the essential factors of the data
• Visualizing the high-dimensional data

There is a lot of framework to approach the dimension reduction, but most of them share common sense that there are hidden factors deep down in the data and the things we can observe is only the result of manipulated, noised or reproduced information of the factors.

By setting hypothesis about the process of the hidden factors realization, we can reversely follow the process and estimate the hidden factors using the observable data. Because there are various points of view regarding the process, various algorithms have been made. In the probabilistic view, they set the process as follow. \(\begin{align*} X & \sim p(\cdot) \\ Y & \sim exponential(f(X)) \end{align*}\) where $X$ is hidden random vector and $Y$ is observable random vectors. $p$ is just unknown process that making $X$ and $f(\cdot)$ is the way that the $X$ is manipulated into $Y$. The exponential means any exponential distribution family which can make thing easier.

Below list is the most famous dimension reduction algorithms using the probabilistic framework and I handled most of them in the post.

• PCA (Principal Component Analysis)
• FA (Factor Analysis)
• GMMs (Gaussian Mixture Models)
• NMF (Nonnegative Matrix Factorization)
• LDA (Latent Dirichlet Allocation)
• ICA (Independent Component Analysis)

Above algorithms are rather traditional. In these days, a lot of machine learning based methods ,not based on the probabilistic view, are made and they have showed a lot amazing results in the processing big data. The most popular machine learning based algorithms are SNE, t-SNE and UMAP. However, the algorithms have some weakness in that they cannot handle some “uncertainty” and hard to explain the process.

The paper “Aditya Ravuri and Neil D.Lawrence, 2023, Dimensionality Reduction as Probabilistic Inference” target to understand the non-probablistic DR in to probabilistic framework. By using the various statistical methods, it reconstruct the famous ML based dimension reduction algorithms including SNE, t-SNE, UMAP, CMDS, Isomap, Kernel PCA, MVU, LE, Spectral Embeddings, LLE and Diffusion Maps.

They used Variational Inference, GPLVMs and Matern-$\nu$ graph Gaussian Process to reconstruct the DR methods. Therefore, before understanding the ProbDR, let’s figure out each of the fields.

1. Learning Based DR - SNE,t-SNE,UMAP
2. Kernel based DR - Multi Dimension Scaling (MDS), Isomap, kernel PCA, MVU (Maximum Variance Unfolding), Diffusion Maps
3. Other DR algorithms - Laplacian Eigenmaps , Spectral Embedding, Locally Linear Embedding (LLE)
4. Variational Inference
5. Gaussian Process Latent Variable Models (GPLMVs)
6. Matern-$\nu$ Graph Gaussian Process

---

This post is ongoing. After I made all of the post of above lists, I’ll finished it.