This paper considers likelihood-based testing of the null hypothesis of m0 components against the alternative of m0+1 components in a finite mixture model. The number of components is an important parameter in the applications of finite mixture models. Still, testing the number of components has been a long-standing challenging problem because of its non-regularity.
We develop a framework that facilitates the analysis of the likelihood function of finite mixture models and derive the asymptotic distribution of the likelihood ratio test statistic for testing the null hypothesis of m0 components against the alternative of m0+1 components. Furthermore, building on this framework, we propose a likelihood-based testing procedure of the number of components. The proposed test, extending the EM approach of Li, Chen and Marriott (2009), does not use a penalty term and is implementable even when the likelihood ratio test is difficult to implement because of non-regularity and computational complexity.