With digital cameras and camcorders, "image processing" refers to the techniques used to convert raw digital data from the image sensor to a video-compatible signal, which is then ready to be compressed or sent to a display. In addition, the image processing section--also called the image pipeline--computes meaningful statistics to assist in auto-exposure, auto-focus and auto-white balance. Basic "image processing" consists of three classes: sensor data processing, color conversion and noise reduction.

Sensor data processing
Three types of color sensors are used in the industry, each with different strengths and weaknesses. The most prevalent sensor in digital still cameras (DSCs) is the RGB sensor; R, G and B stand for the three primary colors. In RGB sensors, a color filter array is placed on top of the sensor; the most common color pattern is called Bayer pattern and is illustrated in Fig. 1.

Fig 1.: RGB Bayer Pattern

In camcorders, two other types of sensors are commonly used: CMYG sensors and 3-CCD sensors. The CMYG sensor uses the complementary primary colors: cyan, yellow and magenta and green. The advantage of this sensor is that is has increased light sensitivity but at the expense of diminished color sensitivity. Therefore, the CMYG sensor is used primarily for low-end cameras with very small sensors and is usually considered unsuitable for still image capture. A 3-CCD sensor, combined with optical beam splitters, is used in high-end camcorders and combines light sensitivity and color sensitivity but at the expense of system cost, size and power. Therefore, 3-CCD sensors are usually limited to small optical sizes and are not typically used in still photography.

With Ambarella's focus on hybrid cameras that are uncompromised in both still and video quality, RGB sensors were, naturally, the sensor of choice. RGB Bayer sensors are theoretically capable of recovering most of the resolution in the sensor. Ambarella’s sensor processing stage uses all of the modern digital signal processing (DSP) techniques to recover as much of the spatial and color information using classical multi-dimensional signal processing, directional interpolation and multi-channel techniques (sometimes called super-resolution).

Color Conversion
The raw data coming from the sensor is a linear function of the photon count in a sensor, while the final video signal must conform in its colorimetry to the international standard for video, including Gamma correction. Advanced algorithms for rendering true brilliant colors from the raw data are used in Ambarella’s image processing pipeline.   

Noise Reduction
Clearly, low-light performance is a very important issue in digital photography. In low light, the low photon count per sensor cell results in a low electrical signal. The influence of both electrical noise and photonic noise results in an extremely noisy image. While the ideal solution is to increase the sensor cell size or the number of available photons (i.e., use a flash) to reduce the noise level, visual impact and interaction with compression algorithms can be done with advanced signal processing techniques: edge-preserving noise reduction and spatio-temporal filtering. Ambarella’s use of these techniques greatly improves the low-light performance for both video and still images.

Auto-White Balance, Auto-Focus and Auto-Exposure
While professional users might want to use manual tuning to achieve a personal trade-off, it is the expectation that today’s digital cameras and camcorders will have automatic functioning that is not only truly automatic but also extremely accurate. This means achieving often better results than all but the most trained professionals can achieve using tedious manual controls.  Extensive statistics are calculated at very high speeds by the image processing section and are then made available to the central processor. The end result is that excellent results can be achieved, while preserving the ability to address the experiences, and tastes, of various camera manufacturers. With Ambarella’s processor, both high-resolution and high-frame-rates are achieved at the same time. As a result, the camera is able to process data very quickly.

Conclusion
Ambarella’s image processing capabilities provide technologically advanced sensor data processing, color conversion, noise reduction and automatic functionality in order to support a superior experience and image quality for the user.