Optimization for TV shows

[pannal]

Hey,

I've been experimenting with Whisper AI and stable-ts (and modifying auto_subtitle, which uses Whisper AI, to use stable-ts).

My current findings are for English (R***ard H**mond's Workshop S01E03):
* results_to_sentence_srt produces better results than whisper's default without stable-ts
* model "base.en" > model "medium.en" > model "tiny.en"
* the above is a little weird to me, but base just produced more correct words and sentences
* using temperature=0, beam_size=3, best_of=3 for transcribe and start_at_first_word=True for results_to_sentence_srt seems to be the sweet spot between ultra long sentences, ultra tiny sentences (Whisper AI non-cli-default) and somewhat acceptable timing/pacing

Please post your results!

[pannal]

OK, I've cut an 8 minute bit out of a particular episode and compared three models with several settings.

Still testing, but it seems like the default settings actually are the best.

The biggest factor is fp32 vs fp16 when it comes to repetition. Theory: use fp32 for TV series and beam_size=5.

[pannal]

If anyone's interested, here's how this went
model.en, 32/16, beam_size, compression_ratio_threshold*10, no_speech_threshold*10:

        tiny, fp32, d, d, d: occasional errors, long lines - bad
        tiny, fp32, 2, 16, 4: rare errors, long lines, pretty accurate, better dialog 
        tiny, fp32, 3, 16, 4: occasional errors, long lines, better dialog
        tiny, fp32, d, 16, 4: occasional errors, long lines, better dialog
        tiny, fp16, d, 16, 4: no errors, all long lines, bad dialog - bad
        
        base, fp32, d, d, d: no errors, good dialog, repetition!  - bad
        base, fp32, d, 16, 4: rare errors, good dialog, sync rarely ooo - meh
        base, fp32, 3, 16, 4: no errors, all long lines, bad dialog
        base, fp16, d, 16, 4: no errors, good dialog, massive repetition! - bad
        
        small, fp32, d, d, d: very rare errors, good dialog, sync rarely ooo - OK
        small, fp32, d, d, d: (start_at_first_word) ^same, slightly late subs
        small, fp32, d, d, d: (start_at_first_word/end_at_last_word) no diff
        small, fp32, d, 16, 4: rare errors, good dialog, sync rarely ooo - meh
        small, fp32, 3, 16, 4: more errors, ...
        small, fp32, 3, d, d: loss of complete phrases
        small, fp32, 3, 20, 6: loss of complete phrases
        small, fp32, 5, d, d:  rare errors, good dialog, best sync - good
        small, fp16, d, 16, 4: no errors, good dialog, repetition! - bad

Result (for now): small, fp32, beam_size=5. Only viable option.

All of this has been tested with results_to_sentence_srt on CPU.

[pannal]

I will test fp32 vs fp16 vs cuda soon.

[pannal]

d=default (API, not CLI)

[vogelcodes]

I've been trying to do the same, but I'm not sure how to pass these parameters in the script. Could you paste a example of the script you're using?

[pannal]

Well, to be honest, I've abandoned this for now. Whisper itself needs optimizations for TV/Movie subtitles to properly be generated and timed.

My tests yielded somewhat usable subs, but not nearly good enough.