Every pixel in a sensor image is the output of a red, green OR blue sensor. To display the image, we must create an image that has a red, green
and blue pixel at each location. We do this by interpolating the missing sensor values. Methods for interpolating the missing sensor values are referred to as “demosaicing”.
Demosaicing has become one of the most popular image processing exercises in digital signal processing. The
information describing demosaicing algorithms is overwhelming. Fortunately, demosaicing algorithms can be grouped into a few categories. At the low end there are simple “fill-up the data” algorithms; in the middle are linear interpolation algorithms; at the high end there are algorithms that choose an interpolation rule based on image contents (adaptive).
There are two more distinctions to make between demosaicing algorithms. First, some algorithms interpolate using information within a color type. For example, unknown green values are filled in only by known green pixels, and similarly for red and blue. The bilinear algorithm uses this approach. Other algorithms try to take advantage of the high correlation between color types. In these algorithms, the green pixels are interpolated using all of their spatial neighbors. The Laplacian algorithm interpolates using data from multiple color types classes.
Most newer digital cameras acquire images using a single image sensor overlaid with a CFA, so demosaicing is part of the processing pipeline required to render these images into a viewable format.
Many newer digital cameras can save images in a raw format allowing the user to demosaic it using software, rather than using the camera's built-in firmware.
The aim of a demosaicing algorithm is to reconstruct a full color image (i.e. a full set of color triples) from the spatially undersampled color channels output from the CFA. The algorithm should have the following traits:
Avoidance of the introduction of false color artifacts, such as chromatic aliases, zippering (abrupt unnatural changes of intensity over a number of neighboring pixels) and purple fringing
Maximum preservation of the image resolution
Low computational complexity for fast processing or efficient in-camera hardware implementation
Amenability to analysis for accurate noise reduction
Color filter array
Main article: Color filter array
The Bayer arrangement of color filters on the pixel array of an image sensor. Each two-by-two cell contains two green, one blue, and one red filter.A color filter array is a mosaic of color filters in front of the image sensor. Commercially, the most commononly used CFA configuration is the Bayer filter illustrated here. This has alternating red (R) and green (G) filters for odd rows and alternating green (G) and blue (B) filters for even rows. There are twice as many green filters as red or blue ones, exploiting the human eye's higher sensitivity to green light.
Since the color subsampling of a CFA by its nature results in aliasing, an optical anti-aliasing filter is typically placed in the optical path between the image sensor and the lens to reduce the false color artifacts (chromatic aliases) introduced by interpolation.
Since each pixel of the sensor is behind a color filter, the output is an array of pixel values, each indicating a raw intensity of one of the three filter colors. As a result, an algorithm is needed to estimate for each pixel the color levels for all color components, rather than a single component.
To reconstruct a full color image from the data collected by the color filtering array, a form of interpolation is needed to fill in the blanks. The mathematics here is subject to individual implementation, and is called demosaicing.
In this example, we use Adobe Photoshop's bicubic interpolation to simulate the circuitry of a Bayer filter device such as a digital camera.
The image below simulates the output from a a Bayer filtered image sensor; each pixel has only a red, green or blue component. The corresponding original image is shown alongside the demosaiced reconstruction at the end of this section.
Bayer filter samples
Red Green Blue
A digital camera typically has means to reconstruct a whole RGB image using the above information. The resulting image could be something like this:
The reconstructed image is typically accurate in uniform-colored areas, but has a loss of resolution (detail and sharpness) and has edge artifacts (for example, the edges of letters have visible color fringes and some roughness).
These algorithms are examples of multivariate interpolation on a uniform grid, using relatively straightforward mathematical operations on nearby instances of the same color component. The simplest method is nearest-neighbor interpolation which simply copies an adjacent pixel of the same color channel. It is unsuitable for any application where quality matters, but can be useful for generating previews given limited computational resources. Another simple method is bilinear interpolation, whereby the red value of a non-red pixel is computed as the average of the two or four adjacent red pixels, and similarly for blue and green. More complex methods that interpolate independently within each color plane include bicubic interpolation, spline interpolation, and Lanczos resampling.
Although these methods can obtain good results in homogenous image regions, they are prone to severe demosaicing artifacts in regions with edges and details when used with pure-color CFAs. However, linear interpolation can obtain very good results when combined with a spatio-spectral (panchromatic) CFA.
Pixel correlation within an image
More sophisticated demosaicing algorithms exploit the spatial and/or spectral correlation of pixels within a color image. Spatial correlation is the tendency of pixels to assume similar color values within a small homogeneous region of an image. Spectral correlation is the dependency between the pixel values of different color planes in a small image region.
These algorithms include:
Variable Number of Gradients interpolation computes gradients near the pixel of interest and uses the lower gradients (representing smoother and more similar parts of the image) to make an estimate. It is used in first versions of dcraw, and suffers from color artifacts.
Pixel Grouping uses assumptions about natural scenery in making estimates. It has fewer color artifacts on natural images than the Variable Number of Gradients method; it was introduced in dcraw from rel. 8.71 as "Patterned Pixel Grouping".
Adaptive homogeneity-directed interpolation selects the direction of interpolation so as to maximize a homogeneity metric, As a result typically minimizing color artifacts. It has been implemented in recent versions of dcraw.
It has recently been shown that super-resolution and demosaicing are two faces of the same problem and it is reasonable to address them in a unified context. Note that both these problems address the aliasing issue. Therefore, especially in the case of video (multi-frame) reconstruction, a joint super-resolution and demosaicing approach provides the optimal solution.
Various commercial products implement proprietary estimation methods about which little is publicly known, and which may or may not be similar to publicly known algorithms.
Some methods may produce better results for natural scenes, and some for printed material, for instance. This reflects the inherent problem in estimating pixels that we do not really know for certain. Naturally there is also the ubiquitous tradeoff of speed versus quality of estimation.
If you have a RAW image, you can use different demosaicing from that which is built into the camera, sometimes yielding higher quality.