Sitrec Video Effects (FLIR1)

Mick West

Mick West

Administrator
Staff member
Rendering a 3D plane is okay, but it's not what's on the video. So I spent a little time writing a variety of custom shaders to try to duplicate what's going on under the hood with the video pipeline - from initial focus, sensor noise, gamma points, NTSC encoding, downsampling and re-encoding.


Source: https://www.youtube.com/watch?v=3R7ApBIaK2U


Still a work in progress, but coming along.
https://www.metabunk.org/sitrec/?sitch=flir1


Use "Video Zoom" to see it up close.
 
Some tweaks:
  • Initial rendering is now anti-aliased with 4x sampling. This gives a more accurate initial image, but at the same 484x484 resolution (previously it had random pixels).
  • Added a fake compression artifact, very slight effect
  • Added a camera shake - not copying the video shake, but using a physically-based system to get something similar.
The idea here is to try to make everything based on actual physics/optics/algorithms and not just try to match what it looks like. Of course there's limits.

The basic pipeline is:
  1. Add camera shake (Physics: air turbulence bouncing the camera around)
  2. Render the scene with anti-aliasing at 484x484 resolution (i.e recording it on the sensor in native resolution, this probably should be 480, but I use 484 to match the video which is 242x242)
  3. Apply "defocus" blur for assume out-of-focus ATFLIR camera (Optics: the camera seems out of focus)
  4. Add noise. (Physics: normal sensor noise)
  5. Convert to greyscale: (The camera is greyscale)
  6. Invert if needed (Here's a problem. The IR mode is just inverted visible light, needs a simulation of heat+glare)
  7. Adjust levels and gamma. (ATFLIR BH vs. WH is not a simple inversion. WH has grey as the inverse of black, so we apply a levels adjustment - different for IRW and TV, there's no IRB footage in FLIR1)
  8. Add a digital zoom if needed. This zooms in on the 484x484 2x keeping the same resolution
  9. Add blur for the conversion to NTSC video. The is more in the horizontal direction, as seen by examining the lines in the UI. (Note the UI has no processing in Sitrec, yet)
  10. Add fake compression artifacts. This just skews each 8x8 block a little, creating discontinuities between blocks. I actually do it on 16x16 blocks because of the next step
  11. Rescale the image to 242x242, which I sort of fake by just averaging out each 2x2 block of pixels into a single color. This is so it matches the actual FLIR1 video we have, which is half the size of the (lost) original. This step would have introduced some blur, but that's covered by step 9
 
Couldn't
5. Convert to greyscale: (The camera is greyscale)
be done first, if only to speed up the rest of the pipeline?

I'd also hope that If you:
4. Add noise. (Physics: normal sensor noise)
with the right spectrum, then you'll get some of
10. Add fake compression artifacts.
for free, specifically the quantisation artefacts, rather than the ME ones you're currently doing. For that you'd want more energy in the lower frequencies, as that's where the quantisation's grainier.

One other idea that might overcomplicate things is for
3. Apply "defocus" blur for assume out-of-focus ATFLIR camera (Optics: the camera seems out of focus)
to have more smear in one parameterised and time-dependent angle than the orthogonal angle. Yes I'm thinking GIMBAL...
Code: 
So in a simplified case where the smeared angle is 0o from horizontal, rather than a kernel of
1 2 1
2 4 2
1 2 1
You'd have something more like
1 4  6 4 1
2 8 12 8 2
1 4  6 4 1
 
FatPhil said:
Couldn't
5. Convert to greyscale: (The camera is greyscale)
be done first, if only to speed up the rest of the pipeline?
Click to expand...
I don't think it would make any difference to speed as this is all done in shaders which operate on the R,G,B,A channels simultaneously. You can see them all here:

https://github.com/MickWest/sitrec/tree/main/src/shaders

e.g. the greyscale conversion:
C: 
    fragmentShader: /* glsl */`

        uniform float opacity;

        uniform sampler2D tDiffuse;

        varying vec2 vUv;

        void main() {

           vec4 color = texture2D(tDiffuse, vUv);
            // // set rgb to the average of r,g,b
            // float average = (color.r + color.g + color.b) / 3.0;
            // gl_FragColor.r = average;
            // gl_FragColor.g = average;
            // gl_FragColor.b = average;

            float grey = 0.299 * color.r + 0.587 * color.g + 0.114 * color.b;
            gl_FragColor = vec4(vec3(grey), opacity);


        }`

FatPhil said:
I'd also hope that If you:
4. Add noise. (Physics: normal sensor noise)
with the right spectrum, then you'll get some of
10. Add fake compression artifacts.
for free, specifically the quantisation artefacts, rather than the ME ones you're currently doing. For that you'd want more energy in the lower frequencies, as that's where the quantisation's grainier.
Click to expand...
I'm not sure how that would work. Noise is added per-pixel, essentially adding a random value to each sensor pixel. This is a fixed range, but really should vary with gain (more gain, more noise). I don't do any quantization (reducing the resolution) of the output value.
FatPhil said:
3. Apply "defocus" blur for assume out-of-focus ATFLIR camera (Optics: the camera seems out of focus)
to have more smear in one parameterised and time-dependent angle than the orthogonal angle. Yes I'm thinking GIMBAL...
Click to expand...
The spread of the glare in gimbal is different from the out-of-focus TV mode in FLIR1. But yes, it would ideally be some kind of convolution to represent the point spread function. That might be complicated - it's high dynamic range
 
You must log in or register to reply here.

Latest posts

Back
Top