1. Being able to use a GAN to be able to give a synth an input sound (i.e. a reece sound from a dnb track) and have it generate a patch that matches it.
2. Being able to do 1 but apply it to a whole Ableton project. The interesting thing about Ableton is that they're actually XML files under the covers.
Even if you couldn't get close, it would be fascinating to see how an AI would design synth patches for subtractive synths.
There is a long tradition of using AI and AI-ish techniques to try to generate FM synthesis patches, for example [1] genetic algorithms (1993) to [2] neural networks (2017). See the "previous work" section in the latter paper for more references.
Disclosure: I wrote the original sound engine in Dexed. I love seeing stuff like this!
Hey dexed is awesome! I was the coauthor for [2], it for was my 3rd year undergraduate dissertation. A lot of the work went into creating a Python VST host capable of generating huge datasets of random patches.
I'm looking at some of the fancy new ML tricks that have come out since that paper was wrote and have been thinking there would be significant improvements over the LSTM we used.
1. For sure, I was thinking something along the lines of a multiview VAE that gets as input either `f(z|audio, midi)` or `f(z|dx7_parameters)` and must produce as output `f(audio|midi,z)` or `f(dx7_parameters|z)`
2. Yea, I have tried to pick apart Ableton files in the past but the format is a bit of a nightmare, it might be easier to use source separation like https://github.com/deezer/spleeter to build your dataset!
I am not an ML expert so not sure of the details on how it works, but I believe Google’s DDSP work is similar to 1: https://magenta.tensorflow.org/ddsp
As I understand it, their system can take an audio input and “learn” how to control the parameters of a synth to reproduce that sound. I had the pleasure of seeing a demo of it from someone who worked on it and thought it was impressive!
Synths have existed for a while which can use “resynthesis” to take a sound and recreate it using additive synthesis (e.g. Hartmann Neuron, Alchemy), but I wouldn’t probably call this AI/ML.
It's AI/ML if you capture sounds at different levels/frequencies, run them all through FFTs, and then use AI/ML to model how the FFTs change with level/frequency.
The Neuron used to work like this - but it's not a cheap way to synthesise sounds, and it never sounded all that awesome.
This works in the DX7's parameter space and tries to find sweet spots within that space - which are hard to discover, even if you know what you're doing.
I'm not convinced it's much more successful than a plain old constrained parameter randomiser. But it is much easier to use.
>The interesting thing about Ableton is that they're actually XML files under the covers.
Are .alp files also XML? How would I go about viewing the contents of one? I had a quick go at it once but couldn't get anything not looking like binary.
The reason I ask is because the native alp files and instruments have preview sounds, and I'm curious if there's any way to add previews to your own created instruments.
I wonder how far we are away from the following:
1. Being able to use a GAN to be able to give a synth an input sound (i.e. a reece sound from a dnb track) and have it generate a patch that matches it.
2. Being able to do 1 but apply it to a whole Ableton project. The interesting thing about Ableton is that they're actually XML files under the covers.
Even if you couldn't get close, it would be fascinating to see how an AI would design synth patches for subtractive synths.