Epistatic Clustering: A Model-Based Approach for Identifying Links Between Clusters

Most clustering methods assume that the data can be represented by mutually exclusive clusters, although this assumption may not be the

case in practice. For example, in gene expression microarray studies, investigators have often found that a gene can play multiple functions

in a cell and may, therefore, belong to more than one cluster simultaneously, and that gene clusters can be linked to each other in certain

pathways. This article examines the effect of the above assumption on the likelihood of finding latent clusters using theoretical calculations

and simulation studies, for which the epistatic structures were known in advance, and on real data analyses. To explore potential links

between clusters, we introduce an epistatic mixture model which extends the Gaussian mixture by including epistatic terms. A generalized

expectation-maximization (EM) algorithm is developed to compute the related maximum likelihood estimators. The Bayesian information

criterion is then used to determine the order of the proposed model. A bootstrap test is proposed for testing whether the epistatic mixture

model is a significantly better fit to the data than a standard mixture model in which each data point belongs to one cluster. The asymptotic

properties of the proposed estimators are also investigated when the number of analysis units is large. The results demonstrate that the

epistatic links between clusters do have a serious effect on the accuracy of clustering and that our epistatic approach can substantially reduce

such an effect and improve the fit.