hierarchical Dirichlet Processes

If we can put a prior on random partition and use likelihood distribution to model data points, we can use the Bayesian framework to learn the latent dimension, which is the main idea of Dirichlet process mixture model. When it comes to hierarchical clustering problem, we usually assume some information is shared between groups. One natural proposal about hierarchical clustering problem is each group i is modeled by a Dirichlet process mixture model DP(G_0(i))and all base measure G_0(i) are related to a parametric form G_0(). However, if the G_0() is continuous, this proposal generally cannot model shared information between groups. One idea is to make G_0() become discrete by limiting the choice of G_0().

In this paper, we simplify the problem by assuming groups are conditional independently drawn from G_0, which means G_0() becomes G_0. The main idea of this paper is to use any base measure H and let G_0 which is drawn from other Dirichlet process DP(H). Note that G_0 is discrete with probability one due to the fact of Dirichlet process.

1. Introduction

It is a common practice to tune the latent dimension K in order to get the best performance of a model. One weakness of this practice is that the corpus is static and unchanged, which means it is generally difficult to do inference given new unseen data points. In that case, we may either re-train the model in the whole corpus including these unseen data points or use some algebraic/heuristic fold-in technique to do inference. If we can come out some prior on the latent dimension and likelihood distribution on data points, we learn the latent dimension K on the fly from the corpus based on the Bayesian framework. This is a important property when it comes to online stream mining.