Audio and video codecs

[DemiMarie]

Being able to decode audio and video would be nice. Audio and video decoders are a well-known source of security holes.

One problem is that many existing implementations resort to hand-written assembler code. I am not sure if Wuffs can hope to match their performance.

[nigeltao]

Audio would be nice. FLAC and MP3 is already listed on https://github.com/google/wuffs/blob/main/doc/roadmap.md

Video is a lot more work, but, yeah, it would be nice to have something like ffmpeg with much more memory safety guarantees.

As for hand-written assembler code, Wuffs' PNG decoder already uses CPU-arch specific intrinsics, which is roughly equivalent.

[DemiMarie]

As for hand-written assembler code, Wuffs' PNG decoder already uses CPU-arch specific intrinsics, which is roughly equivalent.

Depends on the situation; there are cases where the compiler will do a bad job at stuff like instruction scheduling or register allocation, but I agree that asm should be a last resort. That said, making assembler code safe would be amazing, if perhaps out of scope.

[ratchetfreak]

Has there been any consideration for the API for these decoders.

Audio should be pretty simple, a config for communicating the sample rate, amount of channels, and bit depth. And perhaps an extension on the IO buffers to let them be used as ring buffers.

For video I don't think it's as simple as the image API augmented with audio. But I have no real experience with movie clip APIs to know what one should expect from them. Especially when you want to be able to use GPU acceleration for playback.

[DemiMarie]

For video I don't think it's as simple as the image API augmented with audio. But I have no real experience with movie clip APIs to know what one should expect from them. Especially when you want to be able to use GPU acceleration for playback.

I consider GPU acceleration to be out of scope for this.

[nigeltao]

Has there been any consideration for the API for these decoders.

I haven't done any API design work yet. I was going to do it at the same time I started on an actual FLAC or MP3 decoder, but there's other work to do first.

[fire]

In my research on video codex's I have found mention of this NUT mux format from ffmpeg. Maybe it is a small enough problem to be useful.

Other possible candidates for video is https://www.jonolick.com/home/mpeg-video-writer (256 lines of code!)

References

godotengine/godot-proposals#4710 (has some nut details at the end)