Some ambiguities in the CLAP standard to discuss

[colugomusic]

I'm working on adding CLAP support to my DAW. I am quite early in development and my knowledge of CLAP is still limited. This is a bunch of things that I have run into so far that I think are ambiguous, open to interpretation, or not specified. Most of these discussions started on Discord and we felt it would be a good idea to bring it here so that they can be discussed in a more structured way. Since I am so new to CLAP some of my comments may very well be based on misinterpretations, misunderstandings, ignorance or just basic stupidity, so be patient with me.

ITEM 1 - Documentation vs Standard

This first item is a more general one but it has an effect on everything else here.

What are the comments in the C headers? Are they:

1. The working draft of the CLAP standard.
2. Just documentation.

Since I'm not sure the answer to this yet I will just refer to them as "C header comments".

A distinction to keep in mind is that in documentation, ambiguous and badly-defined language can often be very helpful. The writer can draw on analogies and existing concepts to help the reader understand the API. Whereas in a standard, language like that is bad. It is OK to give suggestions for best practices, but details shouldn't be open to interpretation. So whatever the C header comments are, they cannot be both documentation, and also a standard.

ITEM 2 - SLEEP 😴

The word "sleep" is used several times in the C header comments, but is not explicitly defined anywhere. There is ambiguity around what it means for a plugin to "fall asleep" and "wake up". It is ambiguous whether the state of "asleep" is:

1. A explicit state which a plugin enters as a result of explicit interaction with the CLAP API, (e.g. the plugin returning CLAP_PROCESS_SLEEP from process, or the host calling stop_processing), or,
2. A state which the plugin is simply implied to be in, in certain situations.

process.h - Plugins can return CLAP_PROCESS_SLEEP from process.

• Is the plugin then considered to be "asleep" until the next call to processing, or until the next call to start_processing, or neither?
• Is it required for the host to call stop_processing in response to CLAP_PROCESS_SLEEP, or is it merely a suggestion?

plugin.h - "Call stop_processing before sending the plugin to sleep."

This language suggests that:

1. Before calling stop_processing, the plugin is not considered to be logically "asleep"
2. The host has the ability to "send the plugin to sleep"
3. "Asleep" could perhaps be interpreted as "processing is no longer being called at regular intervals."

start_processing, stop_processing and processing

The C header comments imply to me that start_processing should be called before processing, and when the host no longer wants to calling processing at regular intervals, it should call stop_processing. So a potential API trace may look something like:

start_processing();
processing();
processing();
processing();
processing();
stop_processing();

However, since the term "sleep" is left unspecified, a fair interpretation of the C header comments may lead to something more like:

start_processing();
processing();
stop_processing();
start_processing();
processing();
stop_processing();
start_processing();
processing();
stop_processing();
start_processing();
processing();
stop_processing();

i.e. the host developer simply wraps any call to processing in a start_processing/stop_processing pair.

It is ambiguous whether this usage of the API would be good or bad, or neither good nor bad. I can't suggest how to improve the C header comments around this because I don't actually know what the point is of the start_processing and stop_processing functions, or what the plugins developers are expected to do in these calls. (In the plugins in the clap-plugins repository, these functions are no-ops outside of validation checks.)

My general proposal to consider is to remove the words "sleep" and "wake" entirely from the standard, and reserve those concepts for the documentation. This is why I wanted to start the discussion with the distinction between "standard" and "documentation". These words are potentially helpful analogies to use in the documentation. In the standard they are open to interpretation, unless they are explicitly defined ahead of their usage.

If the concept of "sleep" is to be included in the standard then a proposed definition of "asleep" might be something like:

A plugin is considered to be asleep if calling process would have no audible effect on the output signal, and no other observable side-effects.

That is probably still ambiguous so could be improved further. Again I am not a CLAP expert so I'm probably not the one who should be writing this kind of thing yet. Personally I can't see what purpose a definition like this would actually serve in the standard which is why I instead propose removing the word "sleep" entirely, and just being explicit about when and where the functions start_processing and stop_processing can and should be called.

ITEM 3 - thread-safe and realtime-safe

The [thread-safe] tag is used to denote functions that are safe to call from any thread. "Safe" is ambiguous here. Is it safe to call from a realtime processing context, or merely safe from data races?
If thread-safe also means "realtime-safe", that should be stated somewhere.
If thread-safe does not necessarily mean "realtime-safe", then is a new tag, perhaps something like [realtime-safe], required? Or is this information better expressed some other way?

ITEM 4 - clap_host functions

Are plugins allowed to call get_extension during process?

Currently unspecified.

Are plugins allowed to call request_restart if they are not currently active?

Currently unspecified.
Proposed answer from baconpaul via Discord:

While not definitive I think the answer is “no” and we should mark this method [active].

Are plugins allowed to call request_process if they are not already processing?

Currently unspecified.
Proposed answer from baconpaul via Discord:

I think the answer should be yes and we should indicate in the doc that such a request is a no-op

[baconpaul]

So my thoughts. Again these aren't definitive just reactions. I'm happy to take the result of this convo with our collaborators and turn it into a set of PRs.

Item 1 - what's the spec / standard

The standard is the c code. The documentation of the standard is the comments in the c code. We try to make the comments unambiguous and useful. But the only documents that exist are the code and comments in headers, so the code comments need to iterate towards unambiguous.

Item 2 - sleep

I propose

• a plugin is asleep if it is activated and does not have process called for a given block.
• It is correctly asleep if that skipping of process has no effect on the output of the overall host program.
• Two parties may know whether a plugin "can" be asleep. The plugin (if it knows it has no more rendering to do absent external input) or the host (if it knows its output would be unchanged independent of the state of the plugin)
• Plugin side
  • A plugin advertises it is done and can be slept by returning CLAP_PROCESS_SLEEP
  • The host can choose to honor this, and if it does, calls stop_processing on the plugin and no longer calls process unless the host has some reason to do so (event or audio aimed at the plugin for instance).
• Host questions
  • You are correct that either start/proc/proc/proc/stop or start/proc/stop/start/proc/stop are valid in the spec but only if the plugin returns CLAP_PROCESS_SLEEP. Otherwise the host has to obey the semantics which are briefly described in process.h, namely CLAP_PROCESS_CONTINUE the host should not call stop, and so forth. We should expand this documentation clearly. Most plugins use start processing to set up state and stop processing to tear it down.
• Reaction
  • We should define sleep clearly in the documentation
  • We should expand the documentation on process.h as to expectations of host
  • We should clarify that a host which calls stop_processing on a plugin which did not return CLAP_PROCESS_SLEEP may get truncated audio. It can do it of course and may be indifferent to that, but there's no guarantee that a plugin produces a well behaved continuous audio stream over a call to start/stop processing

Item 3 - Thread Safe

To date I have interpreted this as (1) can be called from any thread (2) including the audio thread. And the audio thread is realtime safe.

de-facto hosts generally implement these as something akin to toggling a std::atomic or dropping an item in a lock free queue.

We should update thread.h to make this clear.

Item 4

• Plugins should not call get_extension during process. A host which receives a call in such a context would be well behaved if it returned nullptr. We should mark get_extension thusly in the doc
• My answers to the other two are still the answers to the other two :)

[colugomusic]

Again I'm trying to think about this from the POV of writing a technical specification, not just API usage documentation. So that's why I'm being so autistic about it. I think in terms of usage documentation, you are right that what is there already is OK, because developers can use their common sense to interpret it. But it doesn't serve as a technical specification yet.

The docs in thread-check.h states:

audio-thread: This thread is used for realtime audio processing.

I think we are agreed that this is not technically true, but that developers will use their common sense to understand the interactions with the render extension.

Ultimately the host developer knows whether or not they are running the plugin in a realtime thread, so they know whether or not they are allowed to lock a mutex or allocate memory or whatever, and that knowledge is going to override whatever the CLAP documentation says.

If CLAP had a real technical specification defining the standard, and not just API usage documentation, then the host developer would have to take that more seriously, and not just ignore or mentally reinterpret the parts that don't make sense.

So I think what I'm saying is that on the plugin side of things, I see no reason not to mandate (i.e. formally mandate, in a technical specification) that audio-tread/thread-safe be realtime-safe. However on the host side, I think that same mandate actually falls outside the scope of the CLAP standard.

[baconpaul]

So thinking about it last night I think the most accurate statement is

for a plug-in

• if it does not implement the render extension audio thread means realtime safe
• If it does implement the render extension and returns true when set to realtime mode using the api audio thread means realtime safe
• If it does implement the render extension and implements has hard requirement audio thread means realtime safe
• Else audio thread is simply a threading indicator for which thread (vs ui thread) the method is intended for

For a plug-in thread safe means can be called from any thread including the audio thread so above constraint applies

for a host

• functions it provides to a plugin marked audio thread are functions the plugin can call from the audio thread
• Therefore the plugin must assume that the host meets the same Audio thread constraint it provides
• Specifically if a host does not provide realtime constraint for its audio thread functions it should not expect consistent rendering from a plugin which either does not support render, which returns false from render in offline, or returns true for hard realtime requirement

That’s the full state space and all the cases right?

[colugomusic]

Yes I think this is very good and probably much better than anything I could write!

[baconpaul]

OK cool I will cleaning up and put together as a PR for the thread checker header at least.

[baconpaul]

#415 there's this convo summarized as a PR

[colugomusic]

To be honest I think producing a real technical specification defining the CLAP standard, in addition to the existing documentation in the C headers, is not a particularly monumental task, so once I am further along with my support for CLAP in my DAW and have more knowledge under my belt I may try to pursue that (perhaps as one giant markdown file.) I think the only really complex part of the API right now is the params extension.

On Discord someone mentioned there were concerns about the difficulty of keeping that kind of thing in sync when changes are made but I think it is worth it. I'm still not clear on how much work has already been done towards something like that, whether there is anything started and then abandoned.

[baconpaul]

Membership on the clap team is basically: do work that garners approval from other members of the clap team. So you would be more than welcome to join the team and take a swing at that.

In sync is a concern and also inconsistency between the code docs and the external markdown. I think having a bit of a chat before you start that in earnest is smart. But like I mentioned on discord, cleaner documentation, web presence, etc... is something I think would greatly benefit the project indeed.

[defiantnerd]

Regarding audio-thread: this implies that all functions that are being called within this context:

• MUST BE lock free
• MUST BE without memory allocation

This goes for plugin and host functions. This is also being true in a offline rendering context - there is no reason why this should not be behave the same.

[baconpaul]

Ok! Well the host callbacks are ones basically everyone implements as write to a non locking fifo or toggle an atomic and they’ve been able to make that work to date. If you come across one which doesn’t let us know!

[colugomusic]

I doubt there is anything that can't be achieved in a realtime safe manner with enough engineering effort, and the clap_host callbacks in particular would not be much of a challenge. But my point is less pragmatic - just that mandating that the host must use realtime-safe algorithms and data structures in a non-realtime thread is nonsense.

[baconpaul]

yes you've made that very clear, including using some strong language to do so! That's why i endeavored to expand what the actual constraint is above.

[colugomusic]

Apologies for the strong language, I think it was unnecessary so I will try to catch myself in the future.

[baconpaul]

No worries! We all get heated sometimes (including me!) but I mentioned it because I would hope you would do same to me if i suggested 'nonsense' as opposed to 'doesn't make sense'. Text based collaboration is hard also, as we all know.

Anyway seems like we are at a good point on the realtime issue! What else is outstanding? I guess "someone other than me and you discuss the other points" :)

[baconpaul]

Well gosh I just went and read the doc for get_extension

[plugin|host] extension say

   // Query an extension.
   // The returned pointer is owned by the [plugin|host]
   // It is forbidden to call it before plugin->init().
   // You can call it within plugin->init() call, and after.
   // [thread-safe]
   const void *(CLAP_ABI *get_extension)(const struct clap_plugin *plugin, const char *id);

that raises a few questions and i'd be interested in @defiantnerd @abique and others view

1. What does 'owned by the [plugin|host]' mean exactly? I think what that sentence means is 'the [plugin|host] guarantees returned extension, if not null, is valid for the life of the plugin, until such a time as plugin destroy is called'
2. 'thread-safe' is clear but is this realtime safe? Should we mark it [thread-safe & !audio-thread]? Or should we indicate that the host and plugin has to provide this in a realtime safe way?

Since this function almost always just returns a static object reference, realtime safe (implied in thread safe) is not a high bar. But we should make sure it is what we want.

[colugomusic]

1. What does 'owned by the [plugin|host]' mean exactly? I think what that sentence means is 'the [plugin|host] guarantees returned extension, if not null, is valid for the life of the plugin, until such a time as plugin destroy is called'

When I saw this, I assumed it was referring to who owns the memory, as in you should not call delete/free on a pointer returned from get_extension.

Also I think it's not actually specified anywhere that null can be returned from get_extension to indicate that the extension is not supported.

[colugomusic]

I have some more thoughts related to this. I think this is the only function in the CLAP interface which explicitly states "the returned pointer is owned by the [plugin|host]", which as you said, implies that the returned pointer can safely be stored by the [host|plugin] without having to worry about it being invalidated.

Since no other functions in the API are documented this way then I assume it should be inferred that any other pointers passed between plugin and host should never be stored, and if the plugin/host needs to make a copy of the passed-in data, it should always take a copy of the entire data structure, not the pointer. The main reason I'd like to infer that is that it obviously makes everything dramatically easier for both the host and the plugin if they do not have to keep track of every pointer they pass to make sure the pointed-to data is stored somewhere where its lifetime can exceed the function call.

This may seem obvious but it's not explicitly stated anywhere so it has to be inferred simply from the fact that the "owned by the plugin|host" line of documentation only appears in one place.

[colugomusic]

Some more answered and unanswered questions from my excursion today.

clap_process nullability

• Can clap_process.in_events be null?
• Can clap_process.out_events be null?

I assume the answer in both cases is "yes" but it is not currently specified unless I missed it.

clap.tail extension

There is currently no explanation of what "tail" is or what the host is supposed to do in response to CLAP_PROCESS_TAIL.

Are plugins allowed to cancel the tail (by returning something other than CLAP_PROCESS_TAIL)?

Not specified, but the answer from Timo via Discord is "Yes".

Are plugins required to call changed in that situation (since the tail has technically "changed")?

Not specified, but the answer from Timo is "No".

How should the host interpret the return value of the final call to process, where the ending samples of the tail were processed?

According to Timo plugins may return CLAP_PROCESS_SLEEP here.
I assume the answer is: The host sends the plugin to sleep after this final process call unless either CLAP_PROCESS_CONTINUE or CLAP_PROCESS_CONTINUE_IF_NOT_QUIET is returned.

[defiantnerd]

the host counts these samples after sending its last event.

• Assume the host processes 1024 sample blocks.
• Effect plugin announces 2048 samples tailsize
• In block n0 sample 512 the host has its last non-silent (=non-null) audio word.

Then the host must render blocks 'n1' and 'n2' (1024 samples for n1 at least 512 samples for n2 and must use all delivered samples.

It is safe to assume that the last tailsize announced before n0 will be rendered, does not change. Why should it?

[colugomusic]

What does it mean if a plugin calls clap_host_tail.changed() during process(), but did not return CLAP_PROCESS_TAIL from the previous call to process()? Should the host ignore this (erroneous?) call? Should it make a note of the tail length and use it the next time CLAP_PROCESS_TAIL is returned (assuming changed() is not called again.)

[defiantnerd]

What does it mean if a plugin calls clap_host_tail.changed() during process(), but did not return CLAP_PROCESS_TAIL from the previous call to process()? Should the host ignore this (erroneous?) call? Should it make a note of the tail length and use it the next time CLAP_PROCESS_TAIL is returned (assuming changed() is not called again.)

Yes. In this case it is up to the host to decide. And remember: the plugin does not call changed() very often and it will not call it without having any interactions like parameter changes (-> algo select) or loading a preset (-> algo change).

[colugomusic]

What does it mean if a plugin calls clap_host_tail.changed() during process(), but did not return CLAP_PROCESS_TAIL from the previous call to process()? Should the host ignore this (erroneous?) call? Should it make a note of the tail length and use it the next time CLAP_PROCESS_TAIL is returned (assuming changed() is not called again.)

Yes. In this case it is up to the host to decide.

I'm not sure what you mean. Is your answer "Yes" (Yes what?). Or is your answer "It is up to the host to decide"? (Decide what?) If the host and the plugin are not in agreement about what that call to changed() means then the wrong result is going to be produced.

And remember: the plugin does not call changed() very often and it will not call it without having any interactions like parameter changes (-> algo select) or loading a preset (-> algo change).

I don't undestand why it matters how often changed() is being called.?

[colugomusic]

the host counts these samples after sending its last event.

• Assume the host processes 1024 sample blocks.
• Effect plugin announces 2048 samples tailsize
• In block n0 sample 512 the host has its last non-silent (=non-null) audio word.

Then the host must render blocks 'n1' and 'n2' (1024 samples for n1 at least 512 samples for n2 and must use all delivered samples.

It is safe to assume that the last tailsize announced before n0 will be rendered, does not change. Why should it?

Are you saying that the tail does not start from the end of the audio block that was just processed, but rather, the tail starts at the last non-silent sample of the audio block that was just processed?

[colugomusic]

Further clarification on clap_host.request_process

The current documentation is:

// Request the host to activate and start processing the plugin.
// This is useful if you have external IO and need to wake up the plugin from "sleep".
// [thread-safe]

"Activate and start processing" suggests that an inactive plugin can call this to schedule the host to activate it (in the main thread). I'm pretty sure clap-host does not actually do this. Instead it only schedules the plugin to wake up. The scheduled wakeup will not happen until the plugin is activated by some other means.

So I don't know if that is a bug in clap-host or a bug in the documentation. clap-host also appears to be full of data races so it's probably in the wrong about a lot of things.

[baconpaul]

But that writing certainly makes me thing request processing does not require is active

[colugomusic]

I wonder if that is a colloquial vs technical use of the word activate

Yes I was also wondering the same thing

[colugomusic]

What does the wrapper do may be another interesting question.

Do you mean clap-wrapper? Looks like this is a no-op
https://github.com/free-audio/clap-wrapper/blob/main/src/clap_proxy.cpp#L531

Are all request_* functions allowed to be no-ops?

[defiantnerd]

The clap-wrapper also does not care about PRORCESS_SLEEP, since that concept is not known to VST3, therefore this will work in 99.999% of all cases. In doubt, the wrapper just continues to pass render calls from the host.

[defiantnerd]

But that writing certainly makes me thing request processing does not require is active

setprocessing must be in active state, since getting "active" is allocating buffers.

Activation is a thing that can take time and might now be realtime safe.
Processing is a thing that should be lightweight and shall only be used to shake things into order before processing.

Both state changes have their value and task.

[colugomusic]

More clap_process stuff

// Audio buffers, they must have the same count as specified
// by clap_plugin_audio_ports->count().
// The index maps to clap_plugin_audio_ports->get().
// Input buffer and its contents are read-only.
const clap_audio_buffer_t *audio_inputs;
clap_audio_buffer_t       *audio_outputs;
uint32_t                   audio_inputs_count;
uint32_t                   audio_outputs_count;

This is assuming that at the very least, the plugin supports the CLAP_EXT_AUDIO_PORTS extension. I think it should be stated that if the audio ports extension is not supported, audio_inputs_count and audio_outputs_count should both be set to zero (I assume this is the correct behavior?) Or is the plugin simply not allowed to read those fields at all?

I assume also that audio_inputs and audio_outputs still need to point to valid structs (because of the general rule that pointers passed by the host are never null unless explicitly specified), but those structs may not contain any valid data so they shouldn't be read by the plugin if it doesn't support CLAP_EXT_AUDIO_PORTS.

[baconpaul]

My read:

If a plugin implements audio-ports and not audio-ports-config, then it advertises the shape it needs, and a host which sends a different shape should have no expectation that the plugin is well behaved.

If a plugin implements audio-ports-config then multiple config are available.

While I write my plugins defensively in the first case, a host which passes a mono channel to a plugin with audio ports which asks for stereo input and which receives a SEGV has a host bug, even thought it has too-trusting a plugin.

[colugomusic]

Ok that does make sense. I have not learned the audio-ports-config extension just yet so I am a bit behind.

I think it would be a good idea to specify explicitly that the host is well behaved in this regard, and must provide a sufficient buffer for the port. Interestingly that makes the channel_count field of clap_audio_buffer pointless since the plugin is already aware of its own port configuration and clap_audio_buffer is not used anywhere else. I guess maybe the struct was future-proofed for use in future extensions?

[baconpaul]

it is correct that count is redundant given the extensions we have today and the semantic implied above.

::process existed before audio-ports and way way before audio-orts-config is part of the history
but also: channel in the process data vs expecting the plugin to keep its state that it has in current config is a book-keeping nicety

I can easily imagine a different world where audio-ports-config was more fundamental and the process data structure contained an audio-port-config data structure rather than this subset of it as the metadata. Makes logical sense, but isn't where we are, and isn't where we can go with a 1.x obviously.

[defiantnerd]

Please note that audio-ports-config is not suitable for checking all configs. It's only use is to provide a list of configs the user(!) csn select.

You want to look at configurable-audio-ports extension.

[baconpaul]

Oh right thanks timo!

[colugomusic]

What is a .clap file?

This is pretty important but isn't specified anywhere as far as I know. I don't know if there is anything I am missing or misunderstanding but in my DAW I am going to be making the following assumptions:

On Linux and Windows

It's either a folder or it's a renamed shared library (.so/.dll).
If it's a folder then the folder contains a shared library with exactly the same name inside, e.g.

/CLAP/PluginName.clap/
    /PluginName.clap

On macOS

It's an app bundle which contains a shared library with exactly the same name inside Contents/MacOS, but without the ".clap" e.g.

/CLAP/PluginName.clap/
    /Contents/
        /MacOS/
            /PluginName

The clap-info tool is currently inconsistent between platforms on what it will report as the actual 'plugin'. I think ideally it should always return the full path to the actual shared library which is what it does on Windows, whereas on macOS it will return the path to the app bundle, which is less useful if you are using it to generate a list of shared libraries to open.
I haven't tried clap-info on Linux yet so I don't know what the behavior is there.

[colugomusic]

Ok I managed to find where some of this is specified (in entry.h) and I think I was confused about two things:

• Some of the test plugins I found on the internet consist of a "PluginName.clap" folder containing a "PluginName.clap" shared library + other resources files, but this is just a coincidence. The actual CLAP plugin is just the shared library and the name of the containing folder is irrelevant.
• The internal structure of the macOS bundle isn't relevant because I should not be loading CLAP plugins using some cross-platform DSO library (e.g. Boost.DLL), because '.clap' plugin files are not necessarily shared libraries. On macOS I should use some platform-specific way of loading the plugin entry point (i.e. CFBundleCreate as the clap-info tool does.)

[baconpaul]

Yes. On operating systems with bundles it is the os bundle with the clap extension and on operating systems without it is a dso with a clap extension

[colugomusic]

Thanks for the clarification! And I think the existing docs are actually fine here, I was just making wrong assumptions based on other things.