Research Group of Prof. Dr. C. Sminchisescu
Mathematical Sciences

Twin Gaussian Processes for Structured Prediction

Liefeng Bo and Cristian Sminchisescu


We describe twin Gaussian processes (TGP) [1], a generic structured prediction method that uses Gaussian process (GP) priors [2] on both covariates and responses, both multivariate, and estimates outputs by minimizing the Kullback-Leibler divergence between two GP modeled as normal distributions over finite index sets of training and testing examples, emphasizing the goal that similar inputs should produce similar percepts and this should hold, on average, between their marginal distributions. TGP captures not only the interdependencies between covariates, as in a typical GP, but also those between responses, so correlations among both inputs and outputs are accounted for. TGP is exemplified, with promising results, for the reconstruction of 3d human poses from monocular and multicamera video sequences in the recently introduced HumanEva benchmark, where we achieve 5 cm error on average per 3d marker for models trained jointly, using data from multiple people and multiple activities. The method is fast and automatic: it requires no hand-crafting of the initial pose, camera calibration parameters, or the availability of a 3d body model associated with human subjects used for training or testing.


Description: TGP is a package for Twin Gaussian Processes.

Requirement: Matlab 7.6.0.

Download: [code]. [hog features for HumanEva-I]. This package is free for academic use only. You run it at your own risk.


  1. Liefeng Bo and Cristian Sminchisescu, Twin Gaussian Processes for Structured Prediction, International Journal of Computer Vision (IJCV), vol. 87, pp. 28-52, 2010. [PDF].