What is the goal of unsupervised machine learning?

arXiv:2610.11697 · cs.LG · Submitted 2026-10-08 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "What is the goal of unsupervised machine learning?".

Tom: The gist The unsupervised learning field is heterogenous and serves several different goals, including estimating distributions, generating new data points, extracting features for downstream tasks, and understanding the data.

Jane: First, who's behind it and why it matters.

Paper summary: Tom: So wrapping up this discussion on What is the goal of unsupervised machine learning? The authors propose four different goals: estimating the distribution, generating new data points, extracting features for downstream tasks, and understanding the data.

Jane: And they make a strong point that these goals can be quite separate from each other; they aren't all about one thing.

Lu: It seems like the main implication is that instead of chasing a single unified goal for unsupervised learning, we should focus on which goal is most relevant to our specific problem first.

Meng: So if I were building an AI system, I wouldn't try to make it do all four things at once; I'd decide if feature extraction or distribution estimation is the immediate priority.

Tom: Exactly. The paper suggests that we have a few different directions to explore, and that the field of unsupervised learning will keep pushing these different goals in orthogonal ways going forward.

Jane: It’s a reminder that what we consider "understanding" data is something very deep, and it requires us to think about how human brains process those results.

Lu: The paper suggests we should be careful not to get stuck on the idea that there's one single answer to this question.

Tom: That’s right. We have this list of four goals, and they are distinct objectives that need separate investigation, which is what What is the goal of unsupervised machine learning? sets out.

Conclusion: Tom: So, we’ve been looking at this paper now titled "What is the goal of unsupervised machine learning?". It really asks what exactly we're trying to achieve when we do unsupervised learning.

Jane: Yeah, it breaks down these different aims into four distinct categories. Estimating the distribution, generating new data points, extracting features for other tasks, and understanding the underlying data structure itself.

Lu: The authors suggest these goals aren't all the same thing; they're quite orthogonal objectives we should look at separately.

Meng: I’m thinking about how this applies practically. If we just try to do everything at once, it might get messy when you’re trying to build something real.

Lalam: From a modeling standpoint, it seems like the paper is arguing that there isn't one single objective for unsupervised learning.

Tom: Exactly. It’s not about finding one magic answer; it’s about recognizing these different paths we can take depending on what we need to do next.

Jane: It makes you wonder which of these goals is the most important for the specific problem you’re tackling right now.

Lu: And that leads into how different methods are used to tackle each of those four things, like density estimation versus sampling from a learned distribution.

Meng: So, it's not just about the math; it’s about choosing which tool fits which job in your AI project.

Tom: Right. It frames unsupervised learning less as one monolithic pursuit and more as several different directions we can pursue simultaneously or sequentially.

Jane: It gives us a clearer way to think about what success looks like in this field, beyond just getting a single metric to go up.

Lu: This is interesting because it opens up a whole new space for research where you explore these distinct objectives independently.

Tom: And that means the next step is figuring out how these different goals actually connect and conflict with each other in practice.

Aapo Hyv¨arinen

Department of Computer Science, University of Helsinki

cs.LG

Submitted: 2026-10-08

Updated: 2026-10-08

License: http://creativecommons.org/licenses/by-nc-nd/4.0/

Importance score: 80/100

The gist: The gist The unsupervised learning field is heterogenous and serves several different goals, including estimating distributions, generating new data points, extracting features for downstream tasks,

Key concepts

Estimating the distribution: This fundamental goal is learning the probability density function (pdf) of the input data. Researchers use various methods like kernel density estimators or normalizing flows to learn this underlying probability structure from a given dataset.
Generating new data points: This involves sampling new data points that look like they belong to the original dataset's distribution. Techniques such as diffusion models and GANs are used for this, although measuring how well a generated point matches the original distribution is challenging.
Extracting features for downstream tasks: This focuses on learning a useful representation of the data, often called representation learning. Methods like autoencoders and contrastive learning are employed to create features that improve performance on subsequent supervised tasks.
Understanding the data: This goal seeks to uncover the underlying structure and phenomena within the data, often through finding identifiable components or factors. This aims to provide human-understandable insights into what is being measured.

Summary

The gist The unsupervised learning field is heterogenous and serves several different goals, including estimating distributions, generating new data points, extracting features for downstream tasks, and understanding the data.

Proposed list of goals for unsupervised learning

The author tentatively specifies four different goals for unsupervised learning: 1. Estimating the distribution 2. Generating new data points 3. Extracting features for downstream tasks and 4. Understanding the data <ref:2610.11697#pg3>. The discussion applies to self-supervised learning, which uses supervised algorithms to accomplish these goals <ref:2610.11697#pg3>.

Goal 1: Estimating the distribution

The most fundamental goal is learning the probability distribution of the data, which typically involves learning the probability density function px or its logarithm from a data set X <ref:2610.11697#pg4>. This estimation can be done using methods like kernel density estimator, reproducing kernel Hilbert spaces, Energy-Based Modelling, Score-Based Modelling, and Normalizing flows <ref:2610.11697#pg4>. Validation involves computing the objective function on a held-out test data set or using the learned distribution to generate new data by sampling <ref:2610.11697#pg5>.

Goal 2: Generating new data points

This goal is formalized as sampling a data point from the pdf of the data px, often conditioned by an input u, i.e., sampling from pxu <ref:2610.11697#pg6>. Methods for this include Langevin methods, diffusion models, and Generative Adversarial Networks (GANs) <ref:2610.11697#pg6>. Validation is challenging because There is no simple obvious measure that would tell how closely a generated data point, or even a set of generated points, follows the distribution of the original data set X <ref:2610.11697#pg6>.

Goal 3: Extracting features for downstream tasks

Feature extraction means learning a function f: R n → R m such that the features in z = f(x) are useful, which is a special case of representation learning <ref:2610.11697#pg7>. Methods include autoencoders and contrastive learning, where Contrastive learning approaches have also been used in the context of nonlinear ICA <ref:2610.11697#pg7>. Validation is straightforward in the semi-supervised learning framework by measuring performance on the downstream task using a conventional measure such as classification accuracy <ref:2610.11697#pg8>.

Goal 4: Understanding the data

This goal focuses on furthering the understanding of the structure of the data and the phenomena being measured, often through finding components or factors as a basis for understanding, with Identifiability is a key concept with this goal <ref:2610.11697#pg9>. Methods include latent variable models, such as linear factor analysis and Independent Component Analysis, which provide an identifiable alternative to methods like principal component analysis <ref:2610.11697#pg9>. Validation is difficult because the “understanding” that is the goal here should happen in the brains of humans who receive the results of the unsupervised learning as input <ref:2610.11697#pg9>.

Connections between the goals

Some goals are related, such as an approximation of the pdf learned in Goal 1 being used for sampling in Goal 2 <ref:2610.11697#pg10>. However, some goals are orthogonal; for instance, Goals 1 and 2 need a nonlinear function approximator, and this approximator might not enable understanding the data as in Goal 4 <ref:2610.11697#pg9>. Furthermore, there can be contradictions where achieving one goal makes another more difficult, particularly regarding the goal of understanding the data <ref:2610.11697#pg9>.

Criticism of this list and possible further goals

The list of four goals is not exhaustive, as Data compression could also be seen as a separate goal <ref:2610.11697#pg10>. Additionally, causal learning has many different aspects some of which may not fit the four goals given here, suggesting causal discovery could be a separate goal <ref:2610.11697#pg10>. The paper concludes that unsupervised learning does not have a single goal but rather several orthogonal ones <ref:2610.11697#pg10>.

Conclusion

The author proposes that unsupervised learning has different, often orthogonal, goals and suggests the list can be extended in future work <ref:2610.11697#pg10>. The paper emphasizes that the practical relevance of these goals is not comparable, as density estimation might only be an intermediate goal <ref:2610.11697#pg9>.

Conflict of Interest Statement

The Author has no conflicts of interest to declare <ref:2610.11697#pg10>.

References

[2] Hyv¨arinen, A.: Estimation of non-normalized statistical models using score matching. J. of Machine Learning Research 6, 695–709 (2005).

[3] Vincent, P.: A connection between score matching and denoising autoencoders. Neural computation 23(7), 1661–1674 (2011) <ref:2610.11697#pg6>.

[4] Song, Y., Ermon, S.: Generative modeling by estimating gradients of the data distribution. In: Advances in Neural Information Processing Systems, pp. 11895–11907 (2019) <ref:2610.11697#pg5>.

[5] Sugiyama, M., Suzuki, T., Kanamori, T.: Density Ratio Estimation in Machine Learning. Cambridge University Press (2012) <ref:2610.11697#pg4>.

[6] Gutmann, M.U., Hyv¨arinen, A.: Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. J. of Machine Learning Research 13, 307–361 (2012) <ref:2610.11697#pg6>.

[7] Song, Y., Kingma, D.P.: How to train your energy-based models. arXiv preprint arXiv:2101.03288 (2021) <ref:2610.11697#pg4>.

[8] Chen, Y.-C.: A tutorial on kernel density estimation and recent advances. Biostatistics & Epidemiology 1(1), 161–187 (2017) <ref:2610.11697#pg4>.

[9] Sriperumbudur, B., Fukumizu, K., Gretton, A., Hyv¨arinen, A., Kumar, R.: Density estimation in infinite dimensional exponential families. J. of Machine Learning Research 18, 1–59 (2017) <ref:2610.11697#pg4>.

[10] Wenliang, L., Sutherland, D.J., Strathmann, H., Gretton, A.: Learning deep kernels for exponential family densities. In: Proceedings of the 36th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 97, pp. 6737–6746 (2019) <ref:2610.11697#pg4>.

[11] Papamakarios, G., Nalisnick, E., Rezende, D.J., Mohamed, S., Lakshminarayanan, B.: Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research 22(57), 1–64 (2021) <ref:2610.11697#pg4>.

[12] Hyv¨arinen, A.: A noise-corrected Langevin algorithm and sampling by halfdenoising. Transactions on Machine Learning Research (2025) <ref:2610.11697#pg6>.

[13] Croitoru, F.-A., Hondru, V., Ionescu, R.T., Shah, M.: Diffusion models in vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023) <ref:2610.11697#pg6>.

[14] Yang, L., Zhang, Z., Song, Y., Hong, S., Xu, R., Zhao, Y., Zhang, W., Cui, B., Yang, M.-H.

Improvements for AI systems

A system can be developed that learns to estimate the distribution of observed data by utilizing deep learning methods like score-based modeling or normalizing flows, enabling a probabilistic understanding of the underlying data structure. This allows for more rigorous modeling than simple density approximators, potentially leading to more robust downstream applications.
A generative AI system can be created that samples from learned distributions conditioned on specific inputs (prompt or similar input u), moving beyond static sampling by learning the conditional pdf pxu directly from the data set X alone. This enables high-fidelity data generation tasks, such as realistic image synthesis, without requiring an analytical formula for px.
A feature extraction module can be implemented that learns a function f: R n -> R m based on unlabelled data to create features z such that they are much easier to perform the 'downstream' supervised learning task of predicting y from x by using the features z instead of the original x. This directly supports semi-supervised learning scenarios where labelled data is scarce.
An interpretability layer can be added to feature extraction methods, focusing on Identifiability to ensure learned features are unique and well-defined, preventing them from being merely up to a linear transformation. This enables human investigators to gain insights into the structure of the data by interpreting components or factors.
A causal discovery module can be integrated that learns the causal connections and directions between the variables, which is useful for designing interventions in reinforcement learning or general action selection, moving beyond simple correlation to understanding system mechanisms.

Abstract

Unsupervised learning is one of the main branches of machine learning. Here I argue that unlike the other branches of machine learning (supervised and reinforcement learning), unsupervised learning is a rather heterogenous field that can serve several different goals. It seems futile to try to define one single goal for unsupervised learning. I identify four different goals for unsupervised learning: 1) Estimating the distribution, 2) Generating new data points, 3) Extracting features for downstream tasks, and 4) Understanding the data.