Sound decomposition and transfer

[zqevans]

As someone who is closely involved in the electronic music production community, I'm fascinated by the potential of DDSP-based models for new kinds of audio processing.

The example of reverb transfer in the original paper is already something that would be incredibly impactful among producers if it can be done reliably. As I understand it, the way that works is that you train your model with the Reverb effect at the end, and then you can take that trained reverb model and apply it to other signals. Is that training an IR that could be loaded into other reverb modules in other programs? Can it only be applied to other signals that are generated through DDSP?

Also, is this concept of "train the model on one source and apply it to another" possible with other effects? For example, a lot of sound design in electronic music is achieved through lots of modulation of the cutoff frequency of various filters. Could a DDSP model be trained to capture that filter movement and transfer it to another signal? Or perhaps "de-filter" the signal to show what it would could sounded like without that filter movement?

My machine learning knowledge and DSP knowledge are a bit more intuitive than academic, so I'd love a more concrete understanding of what's possible with DDSP to know what I should be pursuing in my learning to implement things like this.

[jesseengel]

Hey, Fyi, You can apply the learned IR responses just like any other IR response, so you would just need to save the learned variable of the reverb module as a wav file and load it as an IR in your favorite convolutional reverb. There's no code to do that out of the box, but it should just be a line or two.

[zqevans]

Awesome, that's what I was hoping. Do you have any insight on some of the other stuff I'm thinking about? Like transfer for other kinds of effects such as filters?

At some point I imagine I'm getting into the realm of ongoing research and it's going to be harder to find people to talk to about that outside of the offices of Google or other big companies.

[jesseengel]

Sorry for the delay in follow up. The models in the repo at the moment are generative models, so they're only really able to perform "disentanglement" (decomposing into the individual components) as well as they can completely model the audio signal. I think you could find some success applying that technique to other components such as filters/chorus etc, but how much would depend a lot on how well the model fits the data. Also, it can be difficult to disentangle a filter because the generative model itself (harmonic distribution) can be thought of as a filter, so without additional constraints, it could just as easily use the sinusoids or a filter to achieve similar effects.

[zqevans]

Yeah, I realized that it could express the time-varying amplitude in the harmonics using the distribution so it would be pretty arbitrary without more constraints.

Good to know. Thanks for the follow-up.