JIT plugin format

[Photosounder]

Last year on KVR there was some discussion about the possibility of a CLAP JIT format. The idea was essentially that the idea of making one binary for each architecture and platform had run its course (it's inconvenient for the developer and terrible for futureproofing, as we've seen with old unupdated 32-bit plugins tragically fading into oblivion) and that we need a much more 2020s option of having one file that can potentially work on any platform, any OS or CPU architecture (even virtual ones like WebAssembly), even platforms that don't yet exist. I was told back then that this was under consideration, but I see nothing about this. Has this idea been dropped?

[luciascarlet]

I agree this is a good idea; it might not be suitable for every plugin necessarily, but it would be great to have. I feel like the CLAP format itself is generic enough that it might not really require any modifications to run under WASM, but in that case, there should be some generic and standardised method of providing a GUI that can run across all platforms. HTML and JavaScript seem like the natural solution for this.

[Photosounder]

HTML? Do many hosts render HTML and execute Javascript? I mean I don't hate the idea of having that option, but a more traditional and generic way to provide graphics is through a framebuffer that is updated by callback, that alone is good enough for most developers. But yes as long as your plugin sticks to portable code without strictly depending on platform-specific APIs then you could have one file for all platforms (which would be great for Linux as making Linux builds of plugins was never a popular choice despite the lack of obstacles).

[abique]

I've thought about this, but it is a lot of work.
You'd need to standardize the usage of WASI and also provide a canvas or cairo-like interface maybe, and maybe even some kind of windowing system abstraction.

For such a project to succeed, we'd need a full support library on the clap side. Making that one to be super reliable and well designed so big player will consider using it, isn't a part time job.

[Photosounder]

I agree that WASI is the superior approach, although I don't know how stable it's gonna be, I think it's only a matter of time before it matures and stabilises (for instance they recently added some form of thread support). I'm reading a lot about it and it seems to have pretty much everything we need.

I'm gonna work in that direction but the way I envision it it will have little to do with CLAP, mostly due to my framebuffer philosophy and also so I don't have to worry about all of CLAP's features. We'll see after I get anything working if we can do anything to reconcile approaches or if anyone wants to adapt the idea for anything else but for me WASM plugins that can be hosted on an open source IPlug2 bridge plugin would satisfy what I'm looking for and I don't need much network effect (I don't need either DAW makers nor plugin markers to adopt my envisioned approach) beyond maybe a wide array of builds of the bridge plugin.

have everything working: font rendering, painting, interaction events, windowing, open file dialogs, use network sockets, ...

That's why it will be better if I start with my own approach, my requirements are lower, my plugins already do font rendering and other rendering/GUI on their own, that's why I only need a framebuffer + mouse/keyboard inputs. We can worry about making a library full of sophisticated features after we have a proof of concept working. I don't work full time on plugins anymore though so it might take me some time.

[defiantnerd]

Regarding the topic UI: The lack of a consistent UI system for C++ programmers is a result of the versatility of C++. There is not one fits it all solution. And today, things like HiDPI and platform restrictions make things even harder.

It's very simple guys, I don't know why you want to make it more complicated than it needs to be, it's like I said, you provide a framebuffer.

And then the complicated things arrive: HiDPI, font rendering, anti-aliasing, bitmaps or vector formats, mouse events, keyboard events (some hosts don't let you do that), focus issues, rendering with 60Hz, accessibility, shaders... you can scale complexity at all dimensions.

A summary of current solutions can be found here: https://philippegroarke.com/posts/2018/c++_ui_solutions/
It also shows the problems of each UI framework.

I am currently developing my own approach to this, where several things will be different and the target range is indeed from embedded up to full scale applications. It is at a very early stage and I am still developing on the interface/infrastructure design. It has a different architecture than other frameworks, allowing to use the same code on a client but having different abilities on the seat side, but I think this will be an advantage. The communication between client and seat is also designed to be run over a bidirectional connection like a SPI interface etc.

[baconpaul]

Yeah it’s an interesting problem

I think the three questions are interesting: can a blind user use your ui, can a Japanese user use it, and can it render the preset name “synth 🌊 for 🐰” help define the difference between a frame buffer and a toolkit (as you know, defiantnerd, because we’ve discussed this). None of the commonly used ui frameworks used in plugins can do all three today.

[defiantnerd]

Indeed. You nailed it @baconpaul: accessibility and full utf8 support would be king and I think it is possible to cover them.

[baconpaul]

Oh and to your question upstream the clap-saw-demo at https://github.com/surge-synthesizer/clap-saw-demo is a standalone clap only MIT licensed synth which uses VSTGUI not QT to make a UI. There's an initial port of that to imgui also but it's relatively display driver independent.

And if you want something even simpler https://github.com/baconpaul/clap-c99-distortion is a no-ui C99 implementation of a (really bad) distortion plugin which doesn't even use the helpers and so on.

[Photosounder]

Just to let you guys know that I did reach the first significant and exciting step of getting this done which is making a WASM "plugin" that updates a framebuffer provided by the host (the simple C file in the screenshot below, compiled to WASM by Emscripten's emcc, the resulting WASM "plugin" is only 276 bytes), then made a "host" that loads the WASM plugin, calls the expected function to update a 512x512 framebuffer every frame and then displays that framebuffer. From here I should be able to move quickly.

[abique]

How do you plan to get mouse events? Change the mouse cursor, control the position of the mouse cursor during a mouse interaction?

[abique]

Here is an idea:

• You'll need more than what clap provides: a full windowing system abstraction (and it's OK), because drag and drop, copy paste, shaders, ...
  • in a similar way to WASI, you'll provide a runtime that is complementary to WASI to provide the extra GUI/Desktop features
• We can bring this technology without modifying existing hosts, so it is accessible to all

Approach:

• create the wasm plugin's "support interfaces/runtime", this runtime has to be versioned and the plugin explicitly mention their min version requirement; version N+1 should be backward compatible with version N
• create a wasm bridge plugin which will proxy and host wasm clap plugin:
  • this plugin is linked with a wasm runtime, will do the 1:1 translation layer from wasm to native as well as providing all the support interfaces to the wasm plugin
  • the user installs this wasm bridge, and all its hosts will be then able to load wasm plugins
  • the wasm bridge plugin will take care of scanning all the wasm plugins and merge the factories together
• it could be possible to create a wasm host support library, basically only three public functions are needed: clap_wasm_dlopen(), clap_wasm_dlsym(), clap_wasm_dlclose()? But this should not be recommended because the user may not be able to update the version of the wasm support lib that is linked with the host

Conclusion:

• with this approach we can get wasm for free in every hosts which supports native clap plugins
  • the deployment and adoption of this tech from the host side is resolved
• we can have as much "GUI & Desktop" support library as we want in the plugin proxy
  • if this interface already includes a vector rendering, font rendering, etc... it is then much less work for the plugin manufacturer to jump on wasm, because I can predict that if you ask people to build freetype, harfbuzz, cairo, libpng, and all the minimum requirement to make descent interactive GUI then you'll loose 95% on the way.
  • we can start with very few interfaces, and add more step by step:
    1. create window and paint buffer
    2. get keyboard and mouse events
    3. get touch and pen events (touch screen)
    4. access desktop copy paste buffer
    5. drag and drop
    6. maybe add a vector rendering library (font rendering, image loading and rendering, cairo like?)
    7. etc...

[Photosounder]

How do you plan to get mouse events? Change the mouse cursor, control the position of the mouse cursor during a mouse interaction?

For my IPlug2 bridge plan, just like I did with SplineEQ and Spiral (I have a good habit of avoiding relying on features specific to libraries so I did everything to make all such things be handled my own way, although not entirely, I still relied on invisible IPlug controls for knobs): have one control that represents the whole window, then in IControl::OnMouseDown() and the likes call the WASM function to receive mouse events. Everything is just about taking things from where you'd normally get them then passing them on to the WASM plugin, you can basically copy what you do with IPlug2. That goes for any similar question, if IPlug2 takes drag-and-drop then I can pass them on to the WASM plugin, if it doesn't, then it's an IPlug2 problem, but I'm confident that IPlug2 has everything truly necessary. Shaders aren't truly needed, I already addressed this, there's no truly portable and universal way to do this that I know of that will still work in 30 years, if you want shaders you want ephemeral platform-specific builds, how else would you currently have shaders on macOS? Of course I could offer a layer of abstraction for drawing commands of primitives, but that's not part of my initial goal.

One way to look at the initial lack of such features is that the IPlug2 host (or any other maintained host) will always be at least as recent as any WASM plugin, so if anything becomes implemented in the host, provided that it was done by someone wise enough to not implement something that will stop working in 10 years, then you can start taking that new feature for granted. So let's say in a few years WebGPU becomes the true universal eternal standard (doubtful), then I could add this as something a plugin could use and start taking for granted.

If I were to make a CLAP host then I guess I would go the low effort route of using SDL/rouziclib (my library that has a GUI system based on SDL and it does windows, raster graphics, OpenCL graphics, of course all the SDL stuff, keyboard, mouse, touch (though I haven't tried that myself), string input (I even made some pretty advanced text editor as a basic control type), drag-and-drop, clipboard, my library also has built-in image and audio file saving and loading and OS-specific file dialogs), but then you'd need to ship SDL along with this. Maybe that would be better for me than using IPlug2 actually. Btw I really don't know why you guys insist on talking about offering things like font rendering, image loading or vector rendering or any of that when you don't even offer a window and framebuffer, however you want to do those things is the plugin developer's problem and using WASM doesn't prevent them for doing whatever they want, like you want to load PNGs, just add stb_image to your WASM plugin, I don't see why anyone would want to depend on what the plugin framework offers for rendering when it will cause trouble if they want to ditch the framework and move on to something else.

Btw it looks like I'm gonna have to deal with many small technical problems, all that WASM stuff is still a bit rough around the edges. I'm already dealing with inconsistencies in whether or not host initialisation works depending on how the host is launched, the main SDK compiler produces invalid binaries, and more incredibly, in a move that deserves an award for idiocy in design, dynamic allocation on WASM very often returns 0 pointers which are valid buffers, as in you can have a very valid buffer that you can use but is effectively NULL, and there's no clear way to get around this that works... 😂 But at least I was able to pass a dynamically allocated string in a call to a host function, so that's good, now I can call functions and pass data both ways.

[abique]

To continue with my proposal above, what if this wasm plugin bridge was providing a full html browser context, so you can access webgl, canvas, mouse events, etc...?
This bridge would be a bit heavy but probably lighter than your web browser, and you'd not have to worry about designing interfaces and maintaining backward compat

[Photosounder]

To continue with my proposal above, what if this wasm plugin bridge was providing a full html browser context, so you can access webgl, canvas, mouse events, etc...? This bridge would be a bit heavy but probably lighter than your web browser, and you'd not have to worry about designing interfaces and maintaining backward compat

An intriguing idea, but I'm not sure how that would work, like, how would the WASM plugin send graphical data, in what form. You're free to try of course.