Standardizing the subscription export format between all *Tube projects

[SamantazFox]

This Issue is a direct follow up to iv-org/invidious#2897.

As proposed by @TheAssassin, I've created a dedicated repository (and org) to develop and maintain a uniformized specification.

---

Key points from the discussions that took place in the previous issue:

• The format should not be limited to Youtube. Many projects support more than one service, so the format should be flexible enough for all of them.
• A versioning scheme and a compatibility policy has to be determined
• The exact amount of data exported has yet to be determined, based on each projects' need, while also ensuring that a portable format is created.

[absidue]

• The data for each service should be clearly marked as coming from that service, that way applications can easily ignore data for services that they don't support.
• I think it also makes sense to tie the list of supported services to the version scheme of the format, if a new service needs to be added, it needs to be added in a version bump. Doing it this way means that the format of the data for that specific service will need to be standardised. Does this slow down the process of an application adding support for a service? Yes, but it also avoids the situation where other applications would encounter unexpected data.

[ChunkyProgrammer]

I would also like to add FreeTube has support for "Subscription Profiles" (NewPipe has support for this too with "Channel Groups"). It allows you to organize subscriptions into groups so you can easily find subscriptions or only load some channels in your subscription feed.

[ghost]

think it also makes sense to tie the list of supported services to the version scheme of the format, if a new service needs to be added, it needs to be added in a version bump.

Depends if the version number only increases with a breaking change, if semantic versioning is used, or some entirely different approach is taken.

Does this slow down the process of an application adding support for a service? Yes, but it also avoids the situation where other applications would encounter unexpected data.

Any parser can just throw away objects that do not support a frontend it supports. But I agree that the names should be standardized as not to create conflicts.

[ChunkyProgrammer]

Should a separate issue be made for playlists or is the plan to keep them in the same export file?

It also might be a good idea to post the initial design into git as a working draft to be expanded on and changed 😅

[SamantazFox]

Should a separate issue be made for playlists or is the plan to keep them in the same export file?

Separate issue: That's a good idea! Playlists are probably the biggest part of the spec!
Separate file: Maybe? But that mean we'd have to support something like zip archives to export everything in a portable format.

It also might be a good idea to post the initial design into git as a working draft to be expanded on and changed

Done here: #2

[FireMasterK]

We should also designate a common format for watch history. In Piped, we currently have a PR to add support for it.

[opusforlife2]

Will service exports (e.g. Google Takeout) be taken into consideration while designing the spec? Or is that a non-goal?

[absidue]

I think that shouldn't be part of the spec. Designing a spec around Google or SoundCloud or another service's GDPR exports is going to be a nightmare, especially if one of them changes their format. This should just be for interfacing between different *Tube projects and any other project that wants to adopt the standard.

[opusforlife2]

True, the spec doesn't have to conform to their formats. Instead, there could be a generic data converter, which takes in all such exports and turns them into a Pipe-friendly format. Which easily allows the spec to take whatever shape it needs.

What about Fediverse (ActivityPub) websites? I haven't checked, but do they have a standardised format across services that could be studied?

[SamantazFox]

True, the spec doesn't have to conform to their formats. Instead, there could be a generic data converter, which takes in all such exports and turns them into a Pipe-friendly format. Which easily allows the spec to take whatever shape it needs.

A generic data converter is a possible future project for this org, but right now we want to focus on the common user data format.

What about Fediverse (ActivityPub) websites? I haven't checked, but do they have a standardised format across services that could be studied?

Not that I'm aware of.

[TheRedCyclops]

Could a groups field be added to subscriptions? Freetube and NewPipe support this feature