how to remove/change/transparent in fragment shader?

[lv2lrn4lf]

i been searching on a way to do what i can do in javscript and the canvas context 2d, change the pixels color or delete them. I can't figure it out. I have search all over the internet and can't come across some code. I have read the website but still have a hard time wrapping my head around webgl.

https://developer.mozilla.org/en-US/docs/Web/API/Canvas_API/Manipulating_video_using_canvas

in the above link is what i want to do in webgl. that exactly what i want to do in webgl, how do i write the logic in the fragment shader to do this?

[greggman]

You copy the video every frame into a texture. Also supply the texture you want to mix into the same shader

One example is here

https://webglfundamentals.org/webgl/lessons/webgl-2-textures.html

which is a followed up to the articles about image processing

Just change the images to videos and continue drawing each frame. Example:

https://jsgist.org/?src=91467d82afe81c41743f27eec22dd301

[lv2lrn4lf]

I want to remove a color from a image in fragment shader, how do i do that? I can do it in plain JavaScript with the canvas 2d context as shown in that link i link to. But i can't find any example that filter a color or make a certain color transparent. How would I write this logic in shader? How would i do this with webgl?

[greggman]

You need translate the code from the article

    for (let i = 0; i < length; i += 4) {
      const red = data[i + 0];
      const green = data[i + 1];
      const blue = data[i + 2];
      if (green > 100 && red > 100 && blue < 43) {
        data[i + 3] = 0;
      }
    }

Into GLSL

That is basically

 vec4 color = texture2D(sampler2D, texcoord);
 if (color.g > 100.0 / 255.0 && color.r > 100.0 / 255.0 && color.b < 43.0 / 255.0) {
    color = vec4(0);
 }

https://jsgist.org/?src=9a6ff91dfa2f72df3ac3f98dd5327a33

But that is specific to that one example video

[lv2lrn4lf]

That link has resource that cannot be used. so i just threw the code in my project shader language and wrote a function in the shader language to pass it a vec2 and return a vec4 from texture2D and it does not produce the result i want... i want to remove the color black and it remove about as much as if i did it with context 2d with a notice performance boost... but it does not remove all the black.
here my function:

vec4 checkForFilterColor(vec2 textCoord){
  //create texture with color coordinate info
  vec4 color = texture2D(tex, texCoord);
  //convert rgb to uv coolor coordinate 100.0/255.0 - 0.0 /255.0
  if (color.g < 10.0/255.0 && color.r < 10.0/255.0 && color.b < 10.0/255.0) color = vec4(0) 
    //transparent color
    color = vec4(0);
   return color;
}

I don't want to lose your focus by showing you all the shader code i want to hear your suggestion on this peace of shader language code and why do i still see some black around the glowing solid color image?

In the mean time I'll try to increase the 10.0 up to 30.0 to cover more shades of black i guess... but i think when i did something similar with the context 2d api, but it also remove some of the color of the object which cause things to change how they look...

also thank you for code! I was hoping for code.

[greggman]

That link has resource that cannot be used

I don't know what you mean? Are you saying the code doesn't run?

Screen.Recording.2022-05-02.at.10.19.29.AM.mov

Or are you saying something else?

The code you posted seems to have a typo

this

  if (color.g < 10.0/255.0 && color.r < 10.0/255.0 && color.b < 10.0/255.0) color = vec4(0) 
    //transparent color
    color = vec4(0);
   return color;

Is really this

  if (color.g < 10.0/255.0 && color.r < 10.0/255.0 && color.b < 10.0/255.0) {
     color = vec4(0) 
  }

  //transparent color
  color = vec4(0);
  return color;

So it always returns black

If it was me I'd try to do something more generic. Examples

(1) Use a uniform

  uniform vec3 minColor;

  ...

  if (color.g < minColor.g && color.r < minColor.g && color.b < minColor.b) {
     color = vec4(0) 
  }

Then you can pass in the color

gl.uniform3fv(minColorLocationYouLookedUpPreviously, [10 / 255, 10 / 255, 10 / 255]);

Maybe 10 is the wrong value and you need 20 or 30 or 55 or whatever

(2) Use more GLSL

contining from 1 above

   if (all(lessThan(color.rgb, minColor))) {
     color = vec4(0);
   }

(3) Compute a perceptual luminance.

I'm not entirely sure what the correct values are but looking at this it looks like maybe

 float lim = color.r * 0.2126 + color.g * 0.7152 + color.b * 0.0722;
 if (lim < 0.1) {   // replace this 0.1 with a uniform
   color = vec4(0);
 }

Of course you can achieve the same result but just using different values for r,g,b in (1) above.

[lv2lrn4lf]

okay i was using it in firefox but it work in chrome. I acutally came across another code that I alter that does this green thing... but even with the link yuou link me too i see that there is still green around him... so i guess no real way to remove the green with out getting more complicated. Here the code i was using until i replace it with users:

  precision mediump float;
  
  uniform sampler2D tex;
  uniform float texWidth;
  uniform float texHeight;

  uniform vec3 keyColor;
  uniform float similarity;
  uniform float smoothness;
  uniform float spill;

  vec2 RGBtoUV(vec3 rgb) {
    return vec2(
      rgb.r * -0.169 + rgb.g * -0.331 + rgb.b *  0.5    + 0.5,
      rgb.r *  0.5   + rgb.g * -0.419 + rgb.b * -0.081  + 0.5
    );
  }

  vec4 ProcessChromaKey(vec2 texCoord) {
    vec4 rgba = texture2D(tex, texCoord);
    float chromaDist = distance(RGBtoUV(texture2D(tex, texCoord).rgb), RGBtoUV(keyColor));

    float baseMask = chromaDist - similarity;
    float fullMask = pow(clamp(baseMask / smoothness, 0., 1.), 1.5);
    rgba.a = fullMask;

    float spillVal = pow(clamp(baseMask / spill, 0., 1.), 1.5);
    float desat = clamp(rgba.r * 0.2126 + rgba.g * 0.7152 + rgba.b * 0.0722, 0., 1.);
    rgba.rgb = mix(vec3(desat, desat, desat), rgba.rgb, spillVal);

    return rgba;
  }

  vec4 checkForFilterColor(vec2 texCoord){
    //create texture with color coordinate info
    vec4 color = texture2D(tex, texCoord);
    float shadeOff = 60.0;
    //convert rgb  to vu color coordinate 100.0 / 255.0 - 0.0 / 255.0
    if (color.g < shadeOff/255.0 && color.r < shadeOff/255.0 && color.b < shadeOff/255.0)
      //transparent color
      color = vec4(0);
    return color;
  }

  void main(void) {
    vec2 texCoord = vec2(gl_FragCoord.x/texWidth, 1.0 - (gl_FragCoord.y/texHeight));
    gl_FragColor = checkForFilterColor(texCoord);//ProcessChromaKey(texCoord);
  }

it doesn't rid of all the image... is there away for webgl or the canvas api period to read a media stream? the original source has no black background but a transparent background but when it is read my the video element put a black background behind it. So i first use regular canvas 2d to try to remove it and than i use webgl to try to remove it... alot of stuff in my head like why can't i filter out the black color using composition, and how to remove all the color of a image.

And not to go against you about the syntax but this shader code does remove the black background and even the code that comment out do... but it doesn't remove all visual back I see... what are those black? different shades of black to gray? why are t hey so visual black but not being remove by the shader... maybe a color range from black to not so black but that what i thought i did with color.g < 60.0/255.0

i ones visit webgl it seem like magic i came back for performance and i see that the language is similar to JavaScript syntax but it strongly type. And the real barrier to getting in to webgl is that for the most part it is just math of telling what position, what color, and what color position to render at in a world of points, lines, and triangles.

---

You wrote a lot which isn't something i hate i'm just reading as a i reply. So i see you did provide some logic to how to handle the range of color via

if (all(lessThan(color.rgb, minColor))) {
     color = vec4(0);
   }

I check out the link and saw the pretty pixel art if you scroll down showing data of shown pixel.... I also read some of the replies. I see another solutions you also listed as well:

float lim = color.r * 0.2126 + color.g * 0.7152 + color.b * 0.0722;
 if (lim < 0.1) {   // replace this 0.1 with a uniform
   color = vec4(0);
 }

which look to calculate if the light or lightness of the area is dark or black. I'll try both solutions out and give a feedback.

Thanks for the reply.

[greggman]

One where you can set the minColor

https://jsgist.org/?src=398b5b5505ad1938a8cae6201f9e5d28

Screen.Recording.2022-05-02.at.10.46.03.AM.mov

[lv2lrn4lf]

interesting so, the green behind him is so specified, that even if it off by a little the program might not pick up the green screen at all... and things that you wouldn't think would be process by the program is.... but why is there a tint of green.... i guess the same as to why this color green is use that it is so specified that any offness will not be pick up, but at the same time little shadow or lights may not affect it... so you would need a another filter of shades of green....

I try both solutions you provided which make for three solution in total and they all do what i want more good than other but none restore the image to it's original appearance as it was before being transfer with a transparent background and it turn black when put back on the video element as a meidastream which than i have to take it and try to remove the black through some type of filter.

Thank you for the solutions you provided which all are actually solutions put does not actually provide what i want. i will mark them as answers none the less....

And if you are interested in figuring out what i'm talking about just take a transparent image of a red neon 2d sign with a glow effect to it and transparent background than at a black background and try to remove the black background with these filter solutions and you will see the original transparent background image is different from the filter one cause of the different in color being mix in so it red black and not black.... the color that is on it need to be contrast away from it i guess that why people do black and light contrast and then try to filter to help pick up different if the color is very contrast.... anyway thanks if you have more to say or other solutions to try i'm up to hear it.