Conversion between RGB and YCbCr may be wrong

[galenlynch]

As far as I understand the documentation, the RGB color space being used by both this package and ColorTypes.jl is the sRGB color space. I do not understand whether a specific YCbCr color space is being used by this package, i.e. it's not clear if YCbCr values in this package represent colors according to Rec. 601, Rec. 709, or perhaps none of the above. However, it seems that converting between sRGB and YCbCr requires a specific YCbCr color space.

It seems that the code in this package that converts between sRGB and YCbCr uses coefficients to convert between Rec. 601 YCbCr and RGB values in that same color space. However, sRGB uses the color primaries of Rec. 709, not Rec. 601. The same wikipedia page I linked to above mentions that in this case, people often use Rec. 709 color space when converting between sRGB and YCbCr, producing YCbCr values in the Rec. 709 color space.

It seems that if my understanding that Colors.jl uses RGB values in the sRGB color space is right, then this conversion is wrong. The fix may be simple: if the input color space is sRGB, and the output color space is YCbCr values in the Rec. 709 color space, then it's simply a matter of changing the coefficients.

Either way I think it would be helpful to document which YCbCr color space is used by Colors.jl. Even if the RGB and YCbCr values could just be different ways of representing colors in any color space, being able to convert one into the other (I think) requires a defined color space for each.

[kimikage]

related to #372

[galenlynch]

Happy to put together a PR if there is consensus about whether color types such a RGB represent a particular color space, e.g. sRGB.

If that's the case, then all that would be required is deciding what color space YCbCr represents (I vote Rec 709), and changing the conversion accordingly.

If that's not the case, then I think some additional information about the color space should be required to convert between color representations.

[kimikage]

RGB is interpreted as sRGB. I don't really know who uses YCbCr and how they use it. Personally, I have no objection to using Rec 709, but there is some confusion with gamma in the "so-called" Rec 709.

Another problem with YCbCr is that its components are not normalized in [0,1] or [-0.5,0.5]. (cf. JuliaGraphics/ColorTypes.jl#10)

PR is welcome, and I would like to review it, but I can't make a decision about which definition is appropriate. So, it might be a good idea to announce the PR on Discourse before merging.

[galenlynch]

YCbCr is used in most digital video files, and video cameras. That's a good question of what color space people would "expect," but since Rec 709 is both closely matched to sRGB, and also very widely used, I think it makes sense for this package. I'm not sure that YCbCr not being between [0,1] is a problem. Currently, YCbCr in Colors.jl have the same values as the standard "mpeg" range, which is widely adopted, which seems reasonable to me. The transfer characteristics of Rec. 709 for encoding is well defined, so I'm not sure what you mean that there's confusion about its gamma. Are you talking about the fact that its display transfer function is not defined? If so, that shouldn't be a problem for converting between color spaces. By Rec. 709 I mean Recommendation ITU-R BT.709-6.

Sure, happy to talk about this on discourse instead. Is there a specific channel to talk about it?

As it stands the conversion code is demonstrably wrong for converting to and from sRGB (though maybe not by much). Without defining which YCbCr color space that is being used, any conversion to or from other color spaces is ill-defined. I think the easiest fix would be to pick any YCbCr color space and then make sure the conversion functions are accurate for that space. Towards this, I think ITU-R BT.709-6 is a reasonable choice for YCbCr. I'll post on discourse to make sure people agree. However, I think any choice is better than the current inaccurate conversion.

[KelSolaar]

The various Y'CbCr flavours do not mandate that a specific RGB colourspace must be transported by the encoded data. To take an example, it is perfectly fine to save a Display P3 Jpeg file and while doing so, the colour data is encoded as Y'CbCr values using the ITU-R BT.601 weights as per ITU-T T.871. As annoying as it is, and often, the encoded data can be in a different RGB colourspace than the weights used to encode it.

We have a quite complete Python implementation that you could use here: https://github.com/colour-science/colour/blob/develop/colour/models/rgb/rgb_colourspace.py along with some yet-to-be-added code to generate the matrices there: colour-science/colour#486

[galenlynch]

If I understand you correctly, you're saying that YCbCr color spaces can be made for any RGB color space. The issue here is that the RGB color space is sRGB, but the color matrix being used to convert from RGB to YCbCr is the color matrix for ITU-R BT.601, which is not the same as the one you would use for sRGB. Does that agree with what you said?

Thanks for the link! WIll take a look.

Unless I'm misunderstanding something, I think the color matrix used to convert between sRGB and YCbCr here is wrong (since it was made with the ITU-R BT.601 color space RGB primaries, and not the sRGB primaries). So the solution is either use a color matrix made with the sRGB primaries to convert to a YCbCr color space that is matched to the sRGB (not commonly used, as far as I know), or convert to another commonly used YCbCr color space that is pretty close (ITU-R BT.709). However, I am by no means an expert in color science, so perhaps I am misunderstanding something?

[KelSolaar]

If I understand you correctly, you're saying that YCbCr color spaces can be made for any RGB color space.

Unfortunately, yes!

but the color matrix being used to convert from RGB to YCbCr is the color matrix for ITU-R BT.601, which is not the same as the one you would use for sRGB.

Does not really matter, commonly BT.601 weights are used for SD Video, which can use sRGB/BT.709 primaries while the BT.709 weights are used for HD Video which can also use the sRGB/BT.709 primaries. Practically speaking though HD is certainly more common than SD, so if a default was to be chosen, it would certainly make sense to use the BT.709 weights.

[galenlynch]

commonly BT.601 weights are used for SD Video, which can use sRGB/BT.709 primaries

But how do you then succinctly describe how to interpret YCbCr values that are made with the sRGB/BT.709 primaries and transformation characteristic, but used the BT.601 color matrix to convert from RGB to YCbCr instead of the BT.709 color matrix? You can't say that the resulting YCbCr values are in the BT.601, BT.709, or any other standaradized color space, and retrieve the original absolute color (tristimulus value), i.e. the CIE XYZ color, can you? You would need to specify the primaries, transformation characteristic, and color matrix to correctly interpret those YCbCr values.

My understanding is that for any RGB color space, you can use the primaries for that color space to make a "color matrix" that transforms RGB values in that color space to a YCbCr values in the same color space, (which is the first matrix here). Because standardized color spaces, e.g. BT.709, specify their color primaries then you can use those primaries to make a color matrix for that particular color space. The same is true of BT.601. When you get a YCbCr value and know which standardized color space it is in, you can use the color matrix, primaries, and transformation characteristic of that color space to interpret what absolute color the YCbCr value represents. So a YCbCr value of (128, 128, 128) in the BT.709 color space can unambiguously be mapped onto a particular color in CIE XYZ.

So if you do what is currently being done in this package and take a sRGB value, which represents a particular CIE XYZ color, and simply transformed it to YCbCr with the BT.601 color matrix, then it is not enough to say that it belongs to any standardized color space to retrieve the encoded color. If you were to interpret that YCbCr value as a point in the BT.601 color space, you would mistakenly decode a different CIE XYZ color than the original color.

Is that correct? Sorry if I'm being dense.

[galenlynch]

I see, I guess that's what you were saying here:

As annoying as it is, and often, the encoded data can be in a different RGB colourspace than the weights used to encode it.

Well, I think we should change the conversion between RGB and YCbCr so that the YCbCr values are in a standardized color space, e.g. BT.709, instead of a non-standard color space. However, the point is well taken that this non-standard color space may also be valid, and may also be used in many digital images and videos (hopefully with the right ICC profile), but just requires a more extensive description of how to interpret it.

[galenlynch]

Or alternatively, Colors.jl should describe what color space its YCbCr values are in, and keep its current non-standard color space, and leave it up to the user to sort it out?

I was just trying to use Colors.jl to transform RGB images into YCbCr images, and then encode them into a video with libx264 via FFmpeg. In this case (and likely most cases) it's much easier to have the output YCbCr color space be a standardized one, instead of trying to specify an arbitrary YCbCr color space like the one currently being used and figure out how to add that to the video's metadata via FFmpeg. It's also not at all clear that different video players will honor this custom color space when playing back a video.

[galenlynch]

Also @KelSolaar your Colours Python package is quite impressive, though I've just skimmed the documentation.

[kimikage]

After the in-progress ColorTypes v0.11 development, we are planning to make some major changes in ColorTypes v0.12. Also, ColorTypes v0.12 will be paired with Colors v0.13.

I would like to modify the documentation in ColorTypes v0.12 and Colors v0.13 to weaken the connection between the gamma, gamut, white point and other contexts and color types. In other words, I would like to make it clear that the management of the contexts is eventually the responsibility of the downstream packages or users.
This means that ColorTypes.jl is not going to provide a different color type for each compliant standard. Instead, we aim to make it easy for downstream packages or users to create their own color types.

Then it becomes a matter of how to define the default system. This should be discussed purely based on the rationality of the definition, compatibility, and processing efficiency. So, it would be good to have materials to discuss the quantitative or visual differences caused by the proposed changes.

[kimikage]

As a first step, I created a draft PR to update README.md in ColorTypes.jl.
xref: JuliaGraphics/ColorTypes.jl#250

In terms of compatibility with sRGB, there is also sYCC. The name is somewhat minor, but it is not uncommon to find cases of sYCC-equivalent as a result of misuse of Rec.601. 😅
https://www.color.org/sYCC.pdf

[kimikage]

I added the candidate list. JuliaGraphics/ColorTypes.jl#250 (comment)

@galenlynch, If you still have the energy to face this chaos, please leave a comment.

cc: @KelSolaar

[galenlynch]

Sorry I missed this, happy to help

[Myndex]

To rephrase what Thomas (KelSolaar) said, it's simply important to stick by the standard for the particular color space. The standards for sRGB to YCbCr indicate using the 601 matrix. From what I've gathered this was done to make it easier to transcode the substantial body of NTSC and PAL material.

Thing is, a YʹCbCr Signal must be decoded into component RʹGʹBʹ in order for it to be displayed on a physical monitor. 601 is therefore "close enough" as it is the reverse of the same 601 transform to get the signal back to sRGB.

Remember that sRGB was developed in the 1990s and while Rec709 HDTV was around as a standard, it wasn't what everybody had in their home yet. TV shows are still being produced in NTSC and PAL, and there was a massive material that would need to eventually get transferred.

It has a result, we now have these confusions come up, because the BT.709 uses the .21 .71 .07 coefficients, and BT.601 uses the .30 .59 .11 (rounding for ease of reading).

BT.601 and BT.709 both go from RʹGʹBʹ to YʹCbCr without linearizing, in other words without removing the gamma. You go from BT.601 or BT.709 to sYCC and remain gamma or TRC encoded, though 601 and 709 have different transfer curves or gamma.

TL;DR - SD and HD conventions

As a result of the foregoing, BT.601 is associated with standard definition and BT.709 is associated with high definition. So in practical terms, the convention is to use BT.601 when encoding anything with a resolution of 720 x 486 and lower for NTSC, 720 x 576 for PAL, and use BT.709 for higher resolutions.

[Artoria2e5]

FWIW, HDMI copes with the matrix ambiguity by explicitly naming the matrix in the "YCC" names. So you get sYCC₆₀₁, Adobe YCC₆₀₁, xvYCC₆₀₁, and xvYCC₇₀₉. The same might be applicable to the automatic space-naming cnvt stuff, although to churn out type names automagically it might also need to say "601full" (0-255 e.g. sYCC, JPEG) and "601clamped" (16-235/16-240; xv allows for 1-254 but the linear relationship is the same).

[Myndex]

Yes, it's okay to use discordant coefficients so long as you use the same ones going in as used for going out, and vice versa.

BUT: it should be mentioned that these encodings as they stand should not be used for colormetric purposes. In other words, not used in places where you would use proper "luminance" not converted to LAB etc.

The purpose for these YCC types is the efficient encoding of image data, and not uniform colorimetric conversions per se.