Abstract

Brain decoding, a paradigm that consists of predicting stimuli or mental states from concurrent functional brain data, is steadily gaining popularity in neuroimaging based research. While the machine learning and pattern recognition techniques involved in classification-based analysis are getting more and more sophisticated, the statistical methods used for the evaluation of the obtained results are not developing at the same speed. This represents a problem because inadequately evaluated results can be misleading and further claims based on them might be unfunded. This PhD thesis presents a sound procedure for the evaluation of classification results from brain decoding experiments overcoming some limitations of current common practice. An evaluation method within the Bayesian hypothesis testing framework, namely Bayesian test of independence, is presented, which recasts the question whether there is evidence that the classifier has learned from data to discriminate the classes or not as a test of independence between predicted and true class labels. Within the multi-class setting a classifier can learn the complete discrimination of all classes or only subsets of classes. The Bayesian test for partial independence is derived for the latter case. This approach closes a gap as common practice methods do not allow for analysis of potential subsets except for additional binary analysis. Data from simulated and real experiments are considered when comparing the novel methods to common practice. The proposed procedure is robust with imbalanced datasets, it reduces potential issues that arise when having a single null-hypothesis, there is the possibility to incorporate prior knowledge for the learning case, it is compatible with small sample size test sets, and it prevents the misleading interpretation of the estimated prediction accuracy. Experimental evidence shows, that the proposed Bayesian test of independence not only addresses the question whether learning from data has taken place or not but also that it can be successfully used in settings where common practice tests are stressed to their limits.