Feature Informativeness, Curse-of-Dimensionality and Error Probability in Discriminant Analysis

Tatjana Pavlenko

Centre for Mathematical Sciences
Mathematical Statistics
Lund University,
2001

ISBN 91-628-4775-9
LUNFMS--1012-2001

Abstract:

This thesis is based on four papers on high-dimensional discriminant analysis. Throughout, the curse-of-dimensionality effect on the precision of the discrimination performance is emphasized. A growing dimension asymptotic approach is used for assessing this effect and the limiting error probability are taken as the performance criteria.

In the first paper, the asymptotic distribution of the discriminant function is established and the limiting expressions for the error probabilities are then obtained. Using the latter, it is show that the performance of discrimination is severely inflicted by the curse-of-dimensionality and a consistent approximation of the likelihood based discriminant function is proposed.

The second paper discusses the performance of discrimination from the point of view of feature discriminating power. A concept of informativeness is introduced as a feature evaluation tool. By means of a weighted discri-minant function, which distributes weights among features according to their informativeness, the impact of the latter into the error probability is evaluated in a high-dimensional setting. An optimal, in a sense of minimum error probability, type of weight function is established. The weighting scheme is illustrated by some examples which justify the appropriateness of the developed approach and show that the discrimination performance can be improved upon by a suitably chosen weight function.

In the third paper, the weighting technique is elaborated by using an estimation procedure in the feature evalua-tion. The presence of high-dimensional features is shown to lead to overestimation of their informativeness, which increases the error probability thereby reflecting the curse-of-dimensionality effect. The explicit form of the weight function, which provides the minimum of the limiting error probability when weighting by estimates, is found.

In the fourth paper, a threshold based feature selection procedure is introduced in high-dimensional discriminant analysis. This is incorporated into the discriminant function by means of an inclusion-exclusion factor, which eliminates the sets of features whose informativeness does not exceed a given threshold. The relationship between the fraction of selected features and the selection threshold is established. Combined effect of feature selection and curse-of dimensionality on the error probability is evaluated.

Key words:

.