Fisheye Calibration Basics

Camera calibration is the process of computing the extrinsic and intrinsic parameters of a camera. Once you calibrate a camera, you can use the image information to recover 3-D information from 2-D images. You can also undistort images taken with a fisheye camera.

Fisheye cameras are used in odometry and to solve the simultaneous localization and mapping (SLAM) problems visually. Other applications include, surveillance systems, GoPro, virtual reality (VR) to capture 360 degree field of view (fov), and stitching algorithms. These cameras use a complex series of lenses to enlarge the camera's field of view, enabling it to capture wide panoramic or hemispherical images. However, the lenses achieve this extremely wide angle view by distorting the lines of perspective in the images

Because of the extreme distortion a fisheye lens produces, the pinhole model cannot model a fisheye camera.

Fisheye Camera Model

The Computer Vision Toolbox™ calibration algorithm uses the fisheye camera model proposed by Scaramuzza [1]. The model uses an omnidirectional camera model. The process treats the imaging system as a compact system. In order to relate a 3-D world point on to a 2-D image, you must obtain the camera extrinsic and intrinsic parameters. World points are transformed to camera coordinates using the extrinsics parameters. The camera coordinates are mapped into the image plane using the intrinsics parameters.

Extrinsic Parameters

The extrinsic parameters consist of a rotation, R, and a translation, t. The origin of the camera's coordinate system is at its optical center and its x- and y-axis define the image plane.

The transformation from world points to camera points is:

Intrinsic Parameters

For the fisheye camera model, the intrinsic parameters include the polynomial mapping coefficients of the projection function. The alignment coefficients are related to sensor alignment and the transformation from the sensor plane to a pixel location in the camera image plane.

The following equation maps an image point into its corresponding 3-D vector.

are the ideal image projections of the real-world points.
represents a scalar factor.
are polynomial coefficents described by the Scaramuzza model, where .
is a function of (u,v) and depends only on the distance of a point from the image center: .

The intrinsic parameters also account for stretching and distortion. The stretch matrix compensates for the sensor-to-lens misalignment, and the distortion vector adjusts the (0,0) location of the image plane.

The following equation relates the real distorted coordinates (u'',v'') to the ideal distorted coordinates (u,v).

Fisheye Camera Calibration in MATLAB

To remove lens distortion from a fisheye image, you can detect a checkerboard calibration pattern and then calibrate the camera. You can find the checkerboard points using the detectCheckerboardPoints and generateCheckerboardPoints functions. The estimateFisheyeParameters function uses the detected points and returns the fisheyeParameters object that contains the intrinsic and extrinsic parameters of a fisheye camera. You can use the fisheyeCalibrationErrors to check the accuracy of the calibration.

Correct Fisheye Image for Lens Distortion

Open Live Script

Remove lens distortion from a fisheye image by detecting a checkboard calibration pattern and calibrating the camera. Then, display the results.

Gather a set of checkerboard calibration images.

images = imageDatastore(fullfile(toolboxdir('vision'),'visiondata', ...
    'calibration','gopro'));

Detect the calibration pattern from the images.

[imagePoints,boardSize] = detectCheckerboardPoints(images.Files);

Generate world coordinates for the corners of the checkerboard squares.

squareSize = 29; % millimeters
worldPoints = generateCheckerboardPoints(boardSize,squareSize);

Estimate the fisheye camera calibration parameters based on the image and world points. Use the first image to get the image size.

I = readimage(images,1); 
imageSize = [size(I,1) size(I,2)];
params = estimateFisheyeParameters(imagePoints,worldPoints,imageSize);

Remove lens distortion from the first image I and display the results.

J1 = undistortFisheyeImage(I,params.Intrinsics);
figure
imshowpair(I,J1,'montage')
title('Original Image (left) vs. Corrected Image (right)')

J2 = undistortFisheyeImage(I,params.Intrinsics,'OutputView','full');
figure
imshow(J2)
title('Full Output View')

References

[1] Scaramuzza, D., A. Martinelli, and R. Siegwart. "A Toolbox for Easy Calibrating Omnidirectional Cameras." Proceedings to IEEE International Conference on Intelligent Robots and Systems, (IROS). Beijing, China, October 7–15, 2006.

Documentation