Use VideoTimestamps

[moi15moi]

I replaced some functions in convert.py to use VideoTimestamps. Now, the functions ms_to_frame and frame_to_ms are frame-perfect.

There are breaking changes:

• frame_to_timedelta has been replaced with frame_to_ms.
• timedelta_to_frame has been replaced with ms_to_frame.

This PR will require extensive testing.

---

[moi15moi]

Wrong timetype

Suggested change

	print(f"{name}: {format_timedelta(time)} | {ms_to_frame(int(time.total_seconds() * 1000), TimeType.EXACT, fps)}")
	print(f"{name}: {format_timedelta(time)} | {ms_to_frame(int(time.total_seconds() * 1000), TimeType.START, fps)}")

[moi15moi]

I realized that the ms_to_frame function is not perfectly accurate for chapters.

VideoTimestamps performs all its calculations in milliseconds. That's perfect for SRT, WebVTT, ASS, and any format that has a maximum precision in milliseconds. However, for any format that exceeds millisecond precision, it won't be perfect when the file is not an MKV.

It will work well with MKV files because their precision is (99% of the time) in milliseconds. So, if we have a time in nanoseconds, there is a way to properly handle it in milliseconds.

But for non-MKV files (e.g., M2TS), the precision can be "infinite," making it impossible to properly convert a time, let's say in nanoseconds, to milliseconds.

However, I talked with cubicibo (who has the Blu-ray spec), and he told me that a chapter needs to be exactly the frame time (a.k.a. TimeType.EXACT). In this case, we can safely assume that if we truncate the time to milliseconds, it will still be displayed on the same frame.

As for subtitles, they should technically never exceed millisecond precision since ASS, WebVTT, SRT, etc., cannot represent those values. Therefore, if we truncate the timedelta, we don't lose any information.

If you ever need a timedelta_to_ms function, here is how it would look:

def timedelta_to_ms(
    time: timedelta, time_type: TimeType, rounding_method: RoundingMethod = RoundingMethod.ROUND
) -> int:
    """
    Converts a timedelta to milliseconds without losing information.

    :param time:                The timedelta.
    :param time_type:           The time type.
    :param rounding_method:     If you want to be compatible with mkv, use RoundingMethod.ROUND else RoundingMethod.FLOOR.
                                For more information, see the documentation of [timestamps](https://github.com/moi15moi/VideoTimestamps/blob/683a8b48ad394d60ced0deda0ddb87b70e0bfa83/video_timestamps/timestamps.py#L14-L29)
    :return:                    The resulting frame number.
    """

    if rounding_method == RoundingMethod.ROUND:
        time_ms = time.total_seconds() * 1000  # total_seconds returns a float, which can have imprecision.

        if time_type in (TimeType.START, TimeType.END):
            ms = ceil(time_ms)
        elif time_type == TimeType.EXACT:
            ms = floor(time_ms)
        else:
            raise ValueError(f"The TimeType {time_type} isn't supported.")
    elif rounding_method == RoundingMethod.FLOOR:
        # It is impossible to ensure precision for the FLOOR method because it is impossible
        # to convert, for example, nanoseconds to milliseconds without losing information.
        # It works well for the ROUND method because the maximum precision of video timestamps
        # is in milliseconds, so we don't lose any information in the conversion process.
        # Note: For chapters, the specification says that it should always be TimeType.EXACT, 
        # so we can simply floor it.
        raise ValueError("It is impossible to ensure precision for the FLOOR method.")
    else:
        raise ValueError(f"The RoundingMethod {rounding_method} isn't supported.")

    return ms

Edit (08-18-2024):
Since we cannot always convert a timedelta to milliseconds, you could think that VideoTimestamps should simply use a higher precision (ex: instead of milliseconds, use nanoseconds), but actually, it would only make things worse.

In the BD spec (thanks again cubicibo), for a $fps= {24000 \over 1001}$, $N_{ticks} = 3753.75$.
But, actually, it will trunc $N_{ticks}$ to create the PTS.

Exemple:

$$\begin{gather} \text{Frame 0 PTS} : \lfloor 0 \times 3753.75 \rfloor = 0 \\\ \text{Frame 1 PTS} : \lfloor 1 \times 3753.75 \rfloor = 3753 \\\ \text{Frame 2 PTS} : \lfloor 2 \times 3753.75 \rfloor = 7507 \\\ \text{Frame 3 PTS} : \lfloor 3 \times 3753.75 \rfloor = 11261 \\\ \end{gather}$$

In nanoseconds, it gives:

$$\begin{gather} \text{Frame 0 Time} : \lfloor 0 \times 3753.75 \rfloor \times {1\over 90000} \times 10^9 = 0 \text{ns} \\\ \text{Frame 1 Time} : \lfloor 1 \times 3753.75 \rfloor \times {1\over 90000} \times 10^9 = 41700000 \text{ns} \\\ \text{Frame 2 Time} : \lfloor 2 \times 3753.75 \rfloor \times {1\over 90000} \times 10^9 = 83411111.\overline{1} \text{ns} \\\ \text{Frame 3 Time} : \lfloor 3 \times 3753.75 \rfloor \times {1\over 90000} \times 10^9 = 125122222.\overline{2} \text{ns} \\\ \end{gather}$$

If we compare it with the equation we use to approximate the time with an $fps$, we can clearly see that we lose a lot of precision:

$$\begin{gather} \text{Frame 0 Time} : 0 \times {1 \over {24000 \over 1001}} \times 10^9 = 0 \text{ns} \\\ \text{Frame 1 Time} : 1 \times {1 \over {24000 \over 1001}} \times 10^9 = 41708333.\overline{3} \text{ns} \\\ \text{Frame 2 Time} : 2 \times {1 \over {24000 \over 1001}} \times 10^9 = 83416666.\overline{6} \text{ns} \\\ \text{Frame 3 Time} : 3 \times {1 \over {24000 \over 1001}} \times 10^9 = 125125000 \text{ns} \\\ \end{gather}$$

[moi15moi]

A new version of VideoTimestamps is available.
I will try to use it which should resolve all the problem of the PR.

[moi15moi]

@Vodes I just realised that the structure of muxtools doesn't allow to know what is the video timescale (it is the inverse of time_base). It is a requirement to create FPSTimestamps or TextFileTimestamps objects.
This create a need to refactor a lot of code which I don't see myself do.
In my opinion, you should consider to remove the convert module and simply directly use VideoTimestamps. The timedelta idea isn't wrong, but the fact that timedelta.total_seconds return a float is really wrong. float aren't precise which introduce errors. The best is to conserve all the time in int and then, call time_to_frame by specifying the input unit. For example, for xml chapter, it would be 9 (because it is in nanoseconds), for ogg chapter, it would be 3 (because it is in milliseconds) and for ass subs, it would be 2 (because it is centiseconds).

[Vodes]

I don't quite understand what that timescale is supposed to be.
If that implies that every conversion needs to reference a video file directly then I may close this and scrap the whole idea if no one intervenes.
I don't care enough about accuracy to introduce such a massive usability hindrance/annoyance.

As for timedeltas:
I don't really care about a float representation for anything either way and it's not needed for construction.
I may or may not need a timedelta in and out because the ass lib wants that.

[moi15moi]

To understand what a timescale is, you need to understand how each frame time is represented in a video. Each frame has a PTS (Presentation timestamp) assigned to it and the video has a timescale.
So, to know what is the time of a specific frame, you need to calculate this: $time\_of\_i\_frame= {pts\_of\_i\_frame \over timescale}$

$pts\_of\_i\_frame$ and $timescale$ are generated by the tool you use to create the video (ffmpeg, mkvmerge, etc).

Even if the $pts$ is generated internally by the tool, you can know what is the value for a CFR video.
To know what is the $pts\_of\_i\_frame$, you need to calculate:
$ticks = {timescale \over fps}$
$pts\_of\_i\_frame = \text{roundingMethod}(i\_frame \times ticks)$ (the $\text{roundingMethod}$ may be $\text{floor}$ or $\text{round}$ depending on the tool and the video format)

Example

For mkv file, the $timescale$ is usually 1000, so, it would give
$ticks = {1000 \over {24000 \over 1001}} = {1001 \over 24}$
And, let's say we are looking for the pts of the frame 10 and the $\text{roundingMethod}$ is $\text{round}$, it would give us:
$pts\_of\_10\_frame = \text{round}(10 \times {1001 \over 24}) = 417$

And then, from what, we can find the time: $time\_of\_10\_frame= {417 \over 1000} = 0.417 \text{ seconds}$

Note that for .m2ts file, the timescale is always 90000. If you calculate if, you will see that it doesn't give exactly the same time has with the timescale 1000 which is why you need to precise it when you create a FPSTimestamps or TextFileTimestamps objects

[moi15moi]

Someone asked my in pm why I opened a PR.
I opened one because the timedelta_to_frame and frame_to_timedelta and frame_to_ms from the convert module doesn't always return the good result because of rounding error.

Here is a test that compare muxtools converter and VideoTimestamps one:

import os
from datetime import timedelta
from fractions import Fraction
from pathlib import Path
from muxtools.utils.convert import timedelta_to_frame, frame_to_timedelta, frame_to_ms
from video_timestamps import FPSTimestamps, RoundingMethod, TimeType, TextFileTimestamps

dir_path = Path(os.path.dirname(os.path.realpath(__file__)))


def test_frame_to_ms():
    fps = Fraction(24000, 1001)

    timestamps = FPSTimestamps(RoundingMethod.ROUND, Fraction(1000), fps)
    frame = 18
    assert frame_to_ms(frame, fps) == timestamps.frame_to_time(frame, TimeType.EXACT, 3)


def test_timedelta_to_frame():
    fps = Fraction(24000, 1001)

    timestamps = FPSTimestamps(RoundingMethod.ROUND, Fraction(1000), fps)
    ms = 124
    assert timedelta_to_frame(timedelta(milliseconds=ms), fps) == timestamps.time_to_frame(ms, TimeType.EXACT, 3)


def test_with_timestamps_file():
    fps = dir_path.joinpath("timestamps.txt")

    timestamps = TextFileTimestamps(fps, Fraction(1000), RoundingMethod.ROUND)
    ms = 501
    assert timedelta_to_frame(timedelta(milliseconds=ms), fps) == timestamps.time_to_frame(ms, TimeType.EXACT, 3)

Here is what timestamps.txt contains:

# timestamp format v2
0
42
83
125
167
209
250
292
334
375
417
459

Here is a zip file (proof.zip) with the py and txt files.

All the tests currently fails.
Also, muxtools doesn't allow to specify the TimeType in the conversion process which is a bit weird.

[Vodes]

So... From what I gather you just want a way to provide the timescale thingy and it would already solve a lot of problems?

[moi15moi]

Yes, if you are able to have the timescale everywhere where you call timedelta_to_frame, frame_to_timedelta and frame_to_ms, we could use VideoTimestamps which would resolve all the problem.

[Vodes]

Okay I'll try to think of something on the weekend

Edit: I caught a pretty bad flu on Thursday so this may be delayed

[astiob]

The code you posted produces 0 for 42 ms in test_with_timestamps_file using video_timestamps. The muxtools call instead produces 1. In the timestamps.txt, 42 ms clearly corresponds to the second frame, so muxtools is right and video_timestamps is wrong, surely?

[astiob]

Meanwhile, in test_timedelta_to_frame they both give the same result to me, that is, the test passes.

[moi15moi]

The code you posted produces 0 for 42 ms in test_with_timestamps_file using video_timestamps. The muxtools call instead produces 1. In the timestamps.txt, 42 ms clearly corresponds to the second frame, so muxtools is right and video_timestamps is wrong, surely?

My bad, I didn't write the tests correctly. I edited my message to fix it.

Meanwhile, in test_timedelta_to_frame they both give the same result to me, that is, the test passes.

Are you sure you are on muxtools master branch?

[astiob]

My bad, I didn't write the tests correctly. I edited my message to fix it.

Indeed, this seems to have fixed 42 ms. Thanks.

But the new test uses a timestamp beyond the end of the file. video_timestamps seems to extrapolate the timestamps endlessly (e. g. 601 gives 14), but I don’t see how this case is relevant or correct, given that no timestamps beyond the end of the file are known.

Are you sure you are on muxtools master branch?

You’re right; I wasn’t. But the difference is explicitly intentional:

def timedelta_to_frame(
    ..., allow_rounding: bool = True
) -> int:

    :param allow_rounding:      Use the next int if the difference to the next frame is smaller than 0.01.
                                This should *probably* not be used for subtitles. We are not sure.

(This says 0.01, but the actual threshold was raised to 0.03 in d72c8a1.)

[astiob]

To be clear, with allow_rounding=False, the test again passes.

Finally, frame_to_ms seems bugged indeed, because it calls frame_to_timedelta with the default rounding=True, which rounds the timestamp to whole centiseconds. But this might be intentional. If it isn’t, then editing it to pass rounding=False makes it return 750.75 in this test, which seems correct and fits its declared return type of float.

(As the test is written, this is more precise than the whole-millisecond timestamp returned by FPSTimestamps, but I assume this is because you explicitly configured it to round timestamps to whole milliseconds, and you could remove that and make them match exactly.)

[moi15moi]

But the new test uses a timestamp beyond the end of the file. video_timestamps seems to extrapolate the timestamps endlessly (e. g. 601 gives 14), but I don’t see how this case is relevant or correct, given that no timestamps beyond the end of the file are known.

VideoTimestamps try to extrapolate the time like aegisub does.
If you think it shouldn't, open a issue on the repos.

To be clear, with allow_rounding=False, the test again passes.

allow_rounding is clearly just a hack. For example, with ms=83, it fails again with allow_rounding=False.

[astiob]

As for video timebase/scale, I’m confused because:

• If you’re telling muxtools to use an FPS, it’s because you don’t have or want a video file. So you don’t have a timescale, either.
• Conversely, if you want to match a video exactly, your generally have to use timestamps/timecodes. Using an FPS is safe if it evenly divides the video timescale, but then you don’t need the timescale, either. For any other timescale, you can never be sure what rounding the video encoder/muxer applied to its timestamps, and you need the exact timestamp file.
• If you’re using timestamps from a Matroska file, they’re always exact in decimal form. I’m not sure what you want to use the timescale for.
• If your video is non-Matroska, that’s a tricky case indeed. But I’d rather say it calls for supporting timestamp inputs that aren’t Matroska’s v2 timestamp files, rather than an extra timescale argument.

If you think it shouldn't, open a issue on the [VideoTimestamps] repos.

I don’t know what goals VideoTimestamps has and why. In this PR, you’re suggesting a muxtools change, and it seems muxtools already handles timestamp files just fine.

allow_rounding is clearly just a hack. For example, with ms=83, it fails again with allow_rounding=False.

You know, all your examples just show that the behaviour of muxtools is different from VideoTimestamps, with a very specific configuration at that, but they do nothing to explain why VideoTimestamps results are better—to demonstrate the actual problem you’re trying to solve—which would be more useful to see. Like, why do you think 83 is supposed to return 2 and 124 is also supposed to return 2?

I’ll try to infer from your examples. Are you asserting that the default FPS handling in muxtools targets typically-rounded integer-millisecond Matroska timestamps as a common use case, and the current calculations don’t match this specific target?

For example, 2 frames at 24M fps are about 83.417 ms, which gets rounded to 83, so you expect 83 to map to 2 because that’s what it would be mapped to in a standard MKVToolNix-produced millisecond-precision Matroska file. And 3 frames are (exactly) 125.125 ms, which gets rounded to 125, so you expect 125 to map to 3 and 124 to 2, because in the same Matroska file 124 would still be within frame 2. Is this correct?

[astiob]

How do you expect frame_to_ms to work on files with sub-millisecond precision?

[astiob]

Oh, and if my guess about your assumptions is correct, then I think the timebase thing you mentioned is just that this currently-assumed millisecond precision should be configurable? Is this right?

[moi15moi]

• If you’re telling muxtools to use an FPS, it’s because you don’t have or want a video file. So you don’t have a timescale, either.

If you want to convert a time to a frame or vice-versa, you necessarily have a video or a idea of how the video would be (like, you know the fps of your video, so with this logic, you also know the timescale of your video).

• Conversely, if you want to match a video exactly, your generally have to use timestamps/timecodes. Using an FPS is safe if it evenly divides the video timescale, but then you don’t need the timescale, either. For any other timescale, you can never be sure what rounding the video encoder/muxer applied to its timestamps, and you need the exact timestamp file.

Yes, you are totally right when you say you can never be sure what rounding the video encoder/muxer applied to its timestamps. Except for mkv file, I don't know any encoder/muxer that use round. All of them use floor.
And for mkv, some round them (ffmpeg, mkvmerge) and some floor (makemkv, but technically, it simply copy the PTS of the m2ts which are floored).

A timestamps file may be rounded by mkvmerge depending on the timescale you supply with --timestamp-scale factor (warning, timestamps-scale doesn't directly mean timescale, but there is a way to convert from one to the other), so it is not true that a timestamp file represent perfectly the PTS what will be generated by mkvmerge.

• If you’re using timestamps from a Matroska file, they’re always exact in decimal form. I’m not sure what you want to use the timescale for.

Yes, in general, the timescale for mkv is 1000, so the maximum precision about the PTS are in milliseconds. But, for example, for m2ts file, the timescale is 90000, so there isn't a decimal form available.

allow_rounding is clearly just a hack. For example, with ms=83, it fails again with allow_rounding=False.

You know, all your examples just show that the behaviour of muxtools is different from VideoTimestamps, with a very specific configuration at that, but they do nothing to explain why VideoTimestamps results are better—to demonstrate the actual problem you’re trying to solve—which would be more useful to see. Like, why do you think 83 is supposed to return 2 and 124 is also supposed to return 2?

I don't have a example with a $fps = {24000 \over 1001}$, but for a $fps = 30$, you can read the first part that show a example how the TimeType.EXACT, TimeType.START and TimeType.END works: https://github.com/moi15moi/VideoTimestamps/blob/c4a7b4c620b1db6e190686e307cce3ede3aaf191/docs/Algorithm%20conversion%20explanation.md

Here is a updated proof.zip with the fps 30 (also I changed the TimeType.EXACT to TimeType.START in the test because it seems that muxtools want this behaviour).

Are you asserting that the default FPS handling in muxtools targets typically-rounded integer-millisecond Matroska timestamps as a common use case, and the current calculations don’t match this specific target?

Yes, but note that any other use case like m2ts file currently aren't properly supported. You cannot ignore the timescale.

For example, 2 frames at 24M fps are about 83.417 ms, which gets rounded to 83, so you expect 83 to map to 2 because that’s what it would be mapped to in a standard MKVToolNix-produced millisecond-precision Matroska file. And 3 frames are (exactly) 125.125 ms, which gets rounded to 125, so you expect 125 to map to 3 and 124 to 2, because in the same Matroska file 124 would still be within frame 2. Is this correct?

You cannot calculate like this. Please, read this message.

How do you expect frame_to_ms to work on files with sub-millisecond precision?

Depending on the TimeType, it will floor, round or ceil.

Oh, and if my guess about your assumptions is correct, then I think the timebase thing you mentioned is just that this currently-assumed millisecond precision should be configurable? Is this right?

Yes, we could say that. By default, the timescale parameter could be 1000 (which is milliseconds).

[astiob]

You cannot calculate like this. Please, read this message.

I just did calculate exactly how your message describes.

[moi15moi]

You cannot calculate like this. Please, read this message.

I just did calculate exactly how your message describes.

I tought you rounded the time, not the PTS.

For example, 2 frames at 24M fps are about 83.417 ms, which gets rounded to 83, so you expect 83 to map to 2 because that’s what it would be mapped to in a standard MKVToolNix-produced millisecond-precision Matroska file. And 3 frames are (exactly) 125.125 ms, which gets rounded to 125, so you expect 125 to map to 3 and 124 to 2, because in the same Matroska file 124 would still be within frame 2. Is this correct?

It depend on which TimeType you are talking about.
But, let's say we have these timestamps, it would give:

$$\begin{gather} Frame_0 : 0 \text{ ms} \\\ Frame_1 : 42 \text{ ms} \\\ Frame_2 : 83 \text{ ms} \\\ Frame_3 : 125 \text{ ms} \\\ Frame_4 : 167 \text{ ms} \\\ Frame_5 : 209 \text{ ms} \end{gather}$$

EXACT

$$\begin{gather} Frame_0 : [0, 41] \text{ ms} \\\ Frame_1 : [42, 82] \text{ ms} \\\ Frame_2 : [83, 124] \text{ ms} \\\ Frame_3 : [125, 166] \text{ ms} \\\ Frame_4 : [167, 208] \text{ ms} \end{gather}$$

START

$$\begin{gather} Frame_0 : 0 \text{ ms} \\\ Frame_1 : [1, 42] \text{ ms} \\\ Frame_2 : [43, 83] \text{ ms} \\\ Frame_3 : [84, 125] \text{ ms} \\\ Frame_4 : [126, 167] \text{ ms} \end{gather}$$

END

$$\begin{gather} Frame_0 : [1, 42] \text{ ms} \\\ Frame_1 : [43, 83] \text{ ms} \\\ Frame_2 : [84, 125] \text{ ms} \\\ Frame_3 : [126, 167] \text{ ms} \end{gather}$$

[astiob]

I tought you rounded the time, not the PTS.

With a time base of 1 ms, PTS is time, expressed in milliseconds.

It depend on which TimeType you are talking about.

This makes sense to me. But weren’t we talking about timedelta_to_frame? It doesn’t have a TimeType.

[moi15moi]

It depend on which TimeType you are talking about.

This makes sense to me. But weren’t we talking about timedelta_to_frame? It doesn’t have a TimeType.

Is has one, it is just not properly stated. It seems to be TimeType.EXACT.

[astiob]

Ah, it looks like you’re adding TimeType parameters to muxtools’ own APIs in this PR. OK, that explains some things. (Again, I think this deserved an explicit mention.)

IIUC, this should be good for frame rates up to 1000 fps, which is probably mostly sufficient for muxtools.

Matroska chapters have nanosecond precision, so they would benefit from frame_to_ns when working with Matroska files that have sub-millisecond precision of video timestamps. (Dunno about other formats.)

And subtitles might possibly benefit from frame_to_cs to avoid double-rounding, but it’s possible that it doesn’t matter.

I suggest that both of these be investigated after dealing with the current stuff. Chapters need other fixes anyway (e. g. to stop rounding them to centiseconds for a start).

[moi15moi]

In my opinion, a frame_to_time with a output_unit parameter (like I did it here) would be better then to have frame_to_cs, frame_to_ms, frame_to_ns, etc...