%% Low Light Enhancement
%
% This example shows how to enhance low-light images using an algorithm suitable for FPGAs.
%
% Low-light enhancement (LLE) is a pre-processing step for applications in
% autonomous driving, scientific data capture, and general visual
% enhancement. Images captured in low-light and uneven brightness
% conditions have low dynamic range with high noise levels. These qualities 
% can lead to degradation of the overall performance of computer vision algorithms
% that process such images. This algorithm improves the visibility of the
% underlying features in an image.
%
% The example model includes a floating-point frame-based algorithm as a 
% reference, a simplified implementation that reduces division operations, 
% and a streaming fixed-point implementation of the simplified algorithm 
% that is suitable for hardware. 
%
%% LLE Algorithm
%
% This example performs LLE by inverting an input image and then
% applying a de-haze algorithm on the inverted image. After
% inverting the low-light image, the pixels representing non-sky region
% have low intensities in at least one color channel. This characteristic is
% similar to an image captured in hazy weather conditions [1]. The 
% intensity of these dark pixels is mainly due to scattering, or airlight, 
% so they provide an accurate estimation of the haze effects. To improve
% the dark channel in an inverted low-light image, the algorithm modifies
% the airlight image based on the ambient light conditions. The airlight 
% image is modified using the dark channel estimation and then refined with
% a smoothing filter. To avoid noise from over-enhancement, the example 
% applies non-linear correction to better estimate the airlight map. Although
% this example differs in its approach, for a brief overview of low-light 
% image enhancement, see <docid:images_ug#example-ex57643726 Low Light Enhancement>(Image Processing Toolbox).
%
% The LLE algorithm takes a 3-channel low-light RGB image as input. This 
% figure shows the block diagram of the LLE Algorithm.
%
% <<../LLEHDLblockdiagram.png>>
%
% The algorithm consists of six stages.
% 
% 1. _Scaling and Inversion_: The input image $I^c(x,y), c\ \epsilon\
% [r,g,b]$ is converted to range [0,1] by dividing by 255 and then
% inverting pixel-wise.
%
% $$I^c_{scal}(x,y) = \frac{I^c(x,y)}{255}$$
%
% $$I^c_{inv}(x,y) = 1 - I^c_{scal}(x,y)$$
% 
% 2. _Dark Channel Estimation_: The dark channel is estimated by finding
% the pixel-wise minimum across all three channels of the inverted image
% [2]. The minimum value is multiplied by a haze factor, $z$, that 
% represents the amount of haze to remove. The value of $z$ is between
% 0 and 1. A higher value means more haze will be removed from the image.
%
% $$ I_{air}(x,y) = z \times \min_{c\ \epsilon\ [r,g,b]} I^c_{inv}(x,y)$$
% 
% 3. _Refinement_: The airlight image from the previous stage is refined by
% iterative smoothing. This smoothing strengthens the details of the image
% after enhancement. This stage consists of five filter iterations with a 
% 3-by-3 kernel for each stage. The refined image is stored in 
% $I_{refined}(x,y)$. These equations derive the filter coefficients, $h$, 
% used for smoothing. 
%
% $$I_{refined(n+1)}(x,y) = I_{refined(n)}(x,y) * h,\  n = [0,1,2,3,4]\  \&
% \ I_{refined(0)} = I_{air}$$
% 
% $$ where \ h = \frac{1}{16}
%   \left[ {\begin{array}{ccc}
%    1 & 2 & 1\\ 2 & 4 & 2\\ 1 & 2 & 1\\
%   \end{array} } \right] $$
%
% $$Let\ I_{refined(5)}(x,y) = I_{refined}(x,y)$$
%
% 4. _Non-Linear Correction_: To reduce over-enhancement, the refined image
% is corrected using a non-linear correction equation shown below. The
% constant, $m$, represents the mid-line of changing the dark regions of
% the airlight map from dark to bright values. The example uses an
% empirically-derived value of $m=0.6$.
%
% $$ I_{nlc}(x,y) = \frac{[I_{refined}(x,y)]^4}{[I_{refined}(x,y)]^4+m^4} $$
%
% 5. _Restoration_: Restoration is performed pixel-wise across the three
% channels of the inverted and corrected image, $I_{nlc}$, as shown:
%
% $$ I^c_{restore}(x,y) = \frac{I^c_{scal}(x,y)-I_{nlc}(x,y)}{1-I_{nlc}(x,y)} $$
% 
% 6. _Inversion_: To obtain the final enhanced image, this stage inverts 
% the output of the restoration stage, and scales to the range [0,255]. 
%
% $$ I^c_{enhanced}(x,y) = 255 \times (1 - I^c_{restore})$$
%% LLE Algorithm Simplification
%
% The scaling, non-linear correction, and restoration steps involve a divide
% operation which is not efficient to implement in hardware. To reduce the
% computation involved, the equations in the algorithm are simplified by
% substituting the result of one stage into the next stage. This
% substitution results in a single constant multiplication factor rather
% than several divides. 
%
% Dark channel estimation without scaling and inversion is given by
%
% $$I_{air}(x,y) = \frac{z}{255}I'_{air}(x,y)$$ where $$I'_{air}(x,y) = 255
% - \min_{c\ \epsilon\ [r,g,b]}I^c(x,y)$$
%
% The result of the iterative refinement operation on $I_{air}$ is
%
% $$I_{refined}(x,y) = \frac{z}{255}I'_{refined(5)}(x,y)$$
%
% where
%
% $$I'_{refined(n+1)}(x,y) = I'_{refined(n)}(x,y) * h,\  n = [0,1,2,3,4]\
% \& \ I'_{refined(0)}(x,y) = I'_{air}(x,y)$$
%
% Substituting $I_{refined}$ into the non-linear correction equation gives
%
% $$I_{nlc}(x,y) = \frac{z^4[I'_{refined}(x,y)]^4}{z^4[I'_{refined}(x,y)]^4
% + (255 \times m)^4}$$
%
% Substituting $I_{nlc}$ into the restoration equation gives
%
% $$I^c_{restore}(x,y) = 1 - \frac{I^c(x,y)}{255} -
% \frac{I^c(x,y)}{255}\frac{z^4}{(255 \times m)^4}[I'_{refined}(x,y)]^4$$
%
% Subtracting $I^c_{restore}$ from 1 and multiplying by 255
% gives
%
% $$I^c_{enhanced}(x,y) =  I^c(x,y)\times \left(1 +
% \Big[\frac{z}{255 \times m}I'_{refined}(x,y)\Big]^4\right)$$
%
% With the intensity midpoint, $m$, set to 0.6 and the haze factor, 
% $z$, set to 0.9, the simplified equation is 
%
% $$ I^c_{enhanced}(x,y) = I^c(x,y) \times \left( 1 +
% \Big[\frac{1}{170}I'_{refined}(x,y)\Big]^4 \right)$$
% 
% In the equation above, the factor multiplied with $I^c(x,y)$ can be
% called the Enhancement Factor. The constant $\frac{1}{170}$ can be 
% implemented as a constant multiplication rather than a divide. Therefore, 
% the HDL implementation of this equation does not require a division block.
% 
%% HDL Implementation
%
% The simplified equation is implemented for HDL code generation by 
% converting to a streaming video interface and using fixed-point data
% types. The serial interface mimics a real video system and is efficient 
% for hardware designs because less memory is required to store pixel data 
% for computation. The serial interface also allows the design
% to operate independently of image size and format, and makes it more 
% resilient to video timing errors. Fixed-point data types use fewer
% resources and give better performance on FPGA than floating-point types.
%
open_system('LLEExample');
%% 
% The location of the input image is specified in the _LowLightImage_
% block. The _LLEBehavioural_ subsystem computes the enhanced image using
% the raw equations as described in the LLE Algorithm section. The
% _LLESimplified_ subsystem computes the enhanced image using the
% simplified equations. The _simpOutputViewer_ shows the output of the
% _LLESimplified_ subsystem.
% 
% The _LLEHDL_ subsystem implements the simplified equation using a 
% steaming pixel format and fixed-point blocks from Vision HDL Toolbox. 
% The _Input_ subsystem converts the input frames to a pixel stream of 
% |uint8| values and a |pixelcontrol| bus using the Frame To Pixel block.
% The _Output_ subsystem converts the output pixel stream back to image 
% frames for each channel using the Pixel To Frame block. The resulting
% frames are compared with result of the _LLESimplified_ subsystem. The 
% _hdlOutputViewer_ subsystem and _inputViewer_ subsystem show the 
% enhanced output image and the low-light input image, respectively.
%
open_system('LLEExample/LLEHDL');
%%
% The _LLEHDL_ subsystem inverts the input |uint8| pixel stream by 
% subtracting each pixel from 255. Then the _DarkChannel_ subsystem 
% calculates the dark channel intensity minimum across
% all three channels. The _IterativeFilter_ subsystem smooths the airlight
% image using sequential <docid:visionhdl_ref#bt4qqzr-1 Image Filter> blocks.
% The bit growth of each filter stage is maintained to preserve the precision.
% The Enhancement Factor is calculated in EnhancementFactor area. The 
% constant $\frac{1}{170}$ is implemented using a Gain block. The 
% <docid:visionhdl_ref#bvkomp_ Pixel Stream Aligner> block aligns the 
% input pixel stream with the pipelined, modified stream. The aligned 
% input stream is then multiplied by the modified pixel stream. 

%% Simulation and Results 
%
% The input to the model is provided in the _LowLightImage_ (Image From
% File) block. This example uses a 720-by-576 pixel input image with RGB
% channels. Both the input pixels and the enhanced output pixels use |uint8|
% data type. The necessary variables for the example are initialized in
% InitFcn callback.
%
% The _LLEBehavioural_ subsystem uses floating-point Simulink blocks to
% prototype the equations mentioned in the LLE Algorithm section. The 
% _LLESimplified_ subsystem implements the simplified equation in 
% floating-point blocks, with no divide operation. The _LLEHDL_ subsystem
% implements the simplified equation using fixed-point blocks and streaming 
% video interface. The figure shows the input image and the enhanced output 
% images obtained from the _LLESimplified_ subsystem and the _LLEHDL_ subsystem.
%
% <<../LLEHDLResults.JPG>>
%
% The accuracy of the result can be calculated using the percentage of
% error pixels. To compute the percentage of error pixels in the output
% image, the difference between the pixel value of the reference output
% image and the _LLEHDL_ output image should not be greater than one, for
% each channel. The percent of pixel values that differ by more than 1 is
% computed for the three channels. The _simpError_ subsystem compares the
% result of the _LLEBehavioural_ subsystem with the result of the
% _LLESimplified_ subsystem. The _hdlError_ subsystem compares the result
% of the _LLEHDL_ subsystem with the result of the _LLESimplified_
% subsystem. The error pixel count is displayed for each channel. The table
% shows the percentage of error pixels calculated by both comparisons.
%
% <<../LLEpercentageErrorPixels.jpg>>
% 
% You can generate HDL code for the LLEHDL subsystem. An HDL Coder(TM)
% license is required to generate HDL code. This design was synthesized for
% the Intel(R) Arria(R) 10 GX (115S2F45I1SG) FPGA. The table
% shows the resource utilization. The HDL design achieves a clock rate
% of over 250 MHz.
%
% <<../LLEResource.jpg>>
%% References
%
% [1] X. Dong, G. Wang, Y. Pang, W. Li, and J. Wen, "Fast efficient
% algorithm for enhancement of low lighting video" IEEE International
% Conference on Multimedia and Expo, 2011.
%
% Copyright 2018 The MathWorks, Inc.



