How to process DNG raw image to sRGB colourspace?

[raviy0807]

Hi

I am trying to develop raw dng image of DeepISP dataset (https://www.kaggle.com/knn165897/s7-isp-dataset). As I want to process raw image with min. processing and want to learn the pipeline of raw space conversion to image what can be viewed without introducing non-linearity (sRGB without gamma and tone).

Following are the steps that I took :

1. Dcraw Param : Initial dcraw used in lib are (-D -W -4 -t 0 -T), but it does not avoid the default WB therefore I changed to (-r 1 1 1 1 -j -D -W -4 -o 0 -t 0 -T). -r will ensure that any channel is not provided with any gain.
2. Read tiff file produced in previous step by dcraw.
3. Demosaic : Just to keep simple, I tried using bilinear but color-demosaic lib producing demosaic image out of range. This seems to be not possible with bilinear. Therefore switch to opencv for demosaicing and normalize the output again.
4. Cam_to_XYZ : taking motivation from the dng example HERE , used camera_neutral_to_xy and camera_space_to_XYZ_matrix function to get the cam_to_xyz matrix. After multiplying the matrix with image, everytime image is produce way beyond the [0,1] range. This seems to be fine and illumination after adaptation if by default D50 as it is DNG
5. XYZ_to_sRGB : after having image in XYZ coordinate, it is required to convert into well known color space for visualization. As sRGB is widely used for the task. Continue same motivation from tutorial, I tried to use XYZ_to_sRGB function with same setting except switching off cat.
6. After this conversion, i understand i can get negative values due to limitation of color space gamut. However i am getting value in the range of [-0.2 , 3.4]

The upper limit is way beyond the required values. If the upper limit is near 1 like 1.02/1.1, it make sense to clip it. But now, this value is way beyond the upper limit. I don't want to do gamma at this point to preserve the linearity.

Is there any issue with therotation matrix calculation of cam_to_xyz or xyz_to_srgb? If Library is not handling these cases, how can we handle it?
@KelSolaar @MichaelMauderer

[raviy0807]

Also, for the dng case, it make sense to use -r in getting intermediate image, otherwise we would be performing the WB twice raw WB (once dcraw will perform taking raw WB mulpilier ) and 2nd time, the Cam_to_XYZ matrix technique explained by Adobe DNG file interpolate and use WB in its matrix.

[KelSolaar]

Hi @raviy0807,

It would be useful to send us the image after step 1, are you working with linear floating-point data upon reading the image? We are working only and exclusively with linear floating-point data so you need to make sure that the DCRaw tiff file is read as a linear file and in the range [0, 1]. As for point 6, I'm not surprised and it is actually quite expected, the image would help though as it is hard to comment on anything atm.

Cheers,

Thomas

[raviy0807]

Hi thomas,

I have attached the raw dng image as well as stage1 processed image in following link

To answer your question, after stage 1, i am performing demosaicing with opencv and then convert image in range [0,1]

I am curious to know, why image after stage 6 is expected to have large value outside the range (0,1). Based on my understanding slight high value make sense, but getting value almost 3 times is unexpected.

Following are the question:

1. Is the above mentioned process good enough? Is there any hidden details that are present in library implementation?
2. How to deal with such scenario when value are much larger than the range? Downscaling will result in false color and too much of white region.

[raviy0807]

@KelSolaar, any suggestion on this?

[KelSolaar]

Yes!

Here is a notebook with some conversion steps, I did not double-checked but it seems correct: https://colab.research.google.com/drive/1RQSL92andpY9RBrGnXRtPDdsU0YJL-J3?usp=sharing

Mosaiced Image (Normalised for Display)

Demosaiced Image

Demosaiced Image Normalised using White Level

XYZ D50 Image

sRGB Image

It is entirely normal to have values above 1 and under 0 because the various matrices scale the channels, it is not a problem and actually a good thing because you preserve the entire dynamic range of the image without any clipping. You might normalise the image to taste by adjusting its exposure.

Note that here I did not recover the highlights so they will be pinkish. After recovery you will want to apply a S-Curve / Contrast Curve to your image to make it pleasing.

Hope that helps!

Cheers,

Thomas

[AndreasKarge]

Hi, just some notes to (3) and (4) related to question 1:
3: maybe some total variation minimization algorithm can be used using bilinear only as inital value
4: as I understood you map raw patch signals directly to expected tristimuli, it might be better to use spectral data to estimate the matrix,

[raviy0807]

@AndreasKarge could you explain your '4' point?

[KevinJW]

I think the suggestion is that using spectrally measured data for the colour filters/lens and or the scene conditions would allow you to better estimate the tristimulus of the scene that the camera is sensing versus using the matrix in the DNG file which is likely not optimised for your exact conditions, and is likely based on the manufacturer's own test unit or other specific camera, with their lighting and training patches

Whilst this is almost certainly the case, I'd see this level of improvement as near the end of your refinement process vs the other issues you mention.

[raviy0807]

Thanks @KevinJW for the explanation.

Based on my understanding, the current implementation does not directly use the DNG color matrix for conversion into XYZ space. The Color library interpolate the scene illumination and then based on the illumination, color matrix is interpolated with the help of 2 provided color matrix in dng file. It is performing as written in chapter 5-6 of DNG specification : Here

This matrix works after demosaicing. Are you suggesting to perform color conversion before demosaicing i.e. directly on bayer pattern image?

[KevinJW]

no, I was only saying that relying on the DNG supplied matrices is an approximation (possibly a very good one) that can be improved if you have spectral data for your specific use case, if you don't - then using the matrices is the right thing to do

[raviy0807]

@KelSolaar , I am glad that my steps are in right direction for raw image development. Slight suggestion on changing the implementation of dcraw command : -r 1 1 1 1 option should be used at it ensures that dcraw will not WB based on dng metadata.

As you said, it is fine to have values over 1 (in above example its 3) for sRGB color space as it preserve the range. But it highlights the saturation area (Pinkish region), how can we deal with this saturation in color library ?

Normally, libraries perform clipping, which is not true representation of gamut and we are loosing information.

• How do we recover highlights? People talk about analyzing the histogram and identify the saturation range and then again use dcraw at identified saturation. This appraoch is just good enough to understand a particular image, but not the common practice to adopt in libraries. Interesting Example : HERE
• For visualization, the notebook used cctf. Can we visualize with linearity? How to save image with linearity? To save the image in png/tiff so that it can be read by other libraries, do we need to apply any other function?
• Idea here is that, I don't want to introduce non-linearlity with complex method. I am creating the pipeline which can be used in reverse direction as well. So for a given rgb with metadata, i want to go back to bayer pattern as well. This library contains all required steps. But before moving to reverse direction, is all steps good enough to call linear sRGB of good quality? Definately due to saturation/ pink highlights it is not worth of calling good image.

[KevinJW]

Yes, you basically have 2 options if you wish to preserve linearity and remove any skew introduced by the sensor reaching saturation:

1. clipping/scaling the output based on the sensor levels - yes this will loose information from the other channels that are not clipped, but will result in "equal" values for your peak output levels.
2. You try hallucinate the missing data using the saturation level to mask which pixels in each channel are potentially distorted due to the clipping. This would have to be seen as an area of the image that you have less confidence in terms of linearity - but in some way the distortion introduced in the clipping is already non-linear, so your hope is to generate something less distorted.

A lot of the existing tools perform option 2 by adjusting other pixels down at the same time as filling in the detail resulting in less linearity, if you have an "unbounded" mechanism that allows the filled in pixels to "go above 1.0" then it should preserve the linearity of the unclipped parts of the image.

In terms of saving the data to a PNG/TIFF the key would be not applying any functions to avoid any transfer functions being applied when you save the data. This should be lossless if you save a 16bit integer image as there are not many raw files that need more than this, (at which point switch to an appropriate size of floating point).

reversing the process is still possible for both the above options as long as you preserve the pixel mask for the image that way you can always clip those pixels you made up

[KelSolaar]

Slight suggestion on changing the implementation of dcraw command : -r 1 1 1 1 option should be used at it ensures that dcraw will not WB based on dng metadata.

-W        Don't automatically brighten the image

Is actually better because not only white balance should not occur but the image is not scaled at all which gives you a chance to rescale yourself by either using the EXIF white and black levels OR compute them yourself by analyzing the black stripes around the image that you get with non-cropped raw files, i.e. -d Document mode (no color, no interpolation):

To extend on @KevinJW point on highlight recovery, there are actually two definitions to do it in colour-hdri:

• :func:colour_hdri.highlights_recovery_blend
• :func:colour_hdri.highlights_recovery_LCHab

I'm using the second one with relatively good success, keep in mind that it is a) slow b) destructive so you won't be able to round-trip to the original image.