What can two images tell us about a third one?

This paper discusses the problem of predicting image features in an image from image features in two other images and the epipolar geometry between the three images. We adopt the most general camera model of perspective projection and show that a point can be predicted in the third image as a bilinear function of its images in the first two cameras, that the tangents to three corresponding curves are related by a trilinear function, and that the curvature of a curve in the third image is a linear function of the curvatures at the corresponding points in the other two images. Our analysis relies heavily on the use of the fundamental matrix which has been recently introduced (Faugeras et al, 1992) and on the properties of a special plane which we call the trifocal plane. Though the trinocular geometry of points and lines has been very recently addressed, our use of the differential properties of curves for prediction is unique. We thus completely solve the following problem: given two views of an object, predict what a third view would look like. The problem and its solution bear upon several areas of computer vision, stereo, motion analysis, and model-based object recognition. Our answer is quite general since it assumes the general perspective projection model for image formation and requires only the knowledge of the epipolar geometry for the triple of views. We show that in the special case of orthographic projection our results for points reduce to those of Ullman and Basri (Ullman and Basri, 1991). We demonstrate on synthetic as well as on real data the applicability of our theory.