We started building our first ML-powered plugin Deprox in June of 2023, hitting the market in late September. In this article, I'll present everything we learned along the way in the most compact form possible. Each bit can be expanded to its own article, but I'd like to write this general overview while the experience is still fresh.

Data

To train a machine learning (ML) model, you need data. Bad data = no plugin, so we obsessed early on about designing a data-collection mechanism that would yield several different plugins from a single data set.

I see a lot of audio plugin developers using either (a) a prefabbed dataset; or (b) scraping the web for data. The first strategy is ok to test out a model in an academic paper, and the second works well for a specific type of plugin that is aiming to do generative audio, similar to what ChatGPT is doing for text and Midjourney for images. There's a third way, though: make your own dataset. That's what we did.

Building your own dataset takes time, but it can be incredibly rewarding. This section contains some tips and recommendations about data.

Work with students

No knock against the pros, but student musicians are amazing.

Without exception, the ones we hired could basically do any technique we asked for.
Students are not yet at a point in their career where they're maximizing their earnings, so they're usually ok with industry-standard rates set by the union. Given the amount of data you need to collect, and especially if you're a small team that doesn't have much money, this can be the difference between an algorithm with noticeable artefacts (not enough data) and pristine sound.
Most importantly, students are always practicing. Some of them practice 8-10 hours a day. Our recording sessions, which ranged from 1 to 3 hours, felt like a practice session for them. They all brought whatever rep they were working on in school.

Other than that, students have great ideas, are overwhelmingly kind, and look to form collaborative relationships that can potentially last for years. This is true for pros as well, of course. So the unique advantage of working with students is that the needs of your project almost always align well with their needs to practice and make a bit of cash on the side.

Microphones and space

Studio wizardry and the internet can solve 95% of problems that you come up against. Ran out of busses? Patch in a spare mixer. Vintage compressor not working? Download a reasonable digital facsimile. Ran out of disk space? Google Drive break.

What a studio can't solve is when you run out of microphones and can't configure the space for a clean take.

Most studios have several really good microphones and 1-2 good rooms to record in. But they are not equipped to run your experiment, and in the beginning, you may not fully know what you want to run either. Really quickly, as you get up and running, you'll want to try a bunch of different microphones and room configurations. In our case, we sometimes had up to thirty microphones recording a single singer.

If you don't reserve enough microphones and make sure there are enough dividers and foam to carve up a space, you'll likely hit a hard limit of what you can achieve. You can usually rent 10-15 microphones and 4-5 dividers for less than 500 euros. You can even get discounts for weekends and if you're able to work with less popular models. So as you're lining up your musicians, make sure to line up your gear well in advance.

The million rule

As a rule of thumb, with real-time audio, you need a million data points to make something decent and ten-million to make something great. Meaning that given the size of the models and the number of parameters (more on this later), more than ten-million data points likely won't result in more learning: the model's simply not big enough.

If you're recording mono audio at 44100 samples per second, and if a single slice of data is 2048 samples long, you need roughly 13 hours of audio to get a million samples. You can reduce the number of hours through data augmentation or if you allow samples to overlap a bit, aka "stride". For 10 million, we're at 130 hours. While that may sound like a lot, that's a month in the studio. A month in the studio is not that bad. Many bands spend a month in the studio making an album.

As you're scaling up your operations, you can build a different data set every month. At that point, congrats! 🥳🎉👏🤝👍🌷 You're a normal, boring company. Your average supermarket, construction site, and bookstore has a bunch of products that come in weekly to fill the shelves. It's a supply chain. If you treat data collection like a supply chain, it simplifies a lot the work and becomes more predictable.

One session, many plugins

Before you start recording, try to imagine the maximum number of plugins you can squeeze out of a recording session. At Finnvox, for example, we have Byzantine bus setups for every project with all sorts of vintage compression and EQ on the tracks. We also have more microphones than we're currently using in our training. We've used about 5% of the data so far, but we plan on using all of it one day. If we hadn't collected it, we'd be in a much less secure place.

Record at 96k

Downsample later. But it's totally worth it to 2x your storage costs and have everything at 96k in case you need it. Almost every time we work with 96k data, we get a better result on 44.1k data.

Data is everywhere

We work mostly with live musicians, but there's no reason you can't apply this way of thinking to people working with electronic instruments. For example, if you're doing DDSP, you can ask a local studio for access to their projects over a month-long window and use the control data + audio for training. Or if you know a DJ, work with them on a script that can generate millions from a core set of ideas. Mostly, what I'm trying to convey is that if you're interested in achieving an outcome using machine learning, there's likely a source of data closer than you think that can be the basis of your dataset.

Training

After you've collected all that data, you're gonna want to train it. A lot of these tips are for any company training ML models, but there is some audio-specific stuff as well.

Don't start by designing a new model

In 2023, there's a high chance the sound you're targeting is achievable using one of the currently-popular audio models. They are, in no particular order:

MelUNet (we use this)
(micro)TCN (we use this)
WaveGlow
WaveNet
Tacotron 2

If you can get close to your target from any of these models after a few training iterations, even if it's grainy or garbly, chances are that you can get a decent sound after a few weeks through a combination of training and post-processing.

It could be that the sound you want to achieve can't be approximated by these models. If this happens, one really good strategy that's worked for us is pinging the authors of the least-bad option, explaining the problem, and seeing if they're willing to help out. You'd be amazed how far you can get just by doing this.

Also, if you can't get the sound you want in real-time, consider using a variant of WaveNet or a transformer that does single-sample predictions. At that point, you're no longer building a real-time plugin, but that's ok. If someone likes the sound your plugin makes, they'll use it, even if it's not real-time. And in less than a year, ARA support will be rolled out across more DAWs, which will allow you to preprocess audio and create the illusion of real-time.

Transfer learning isn't as important as you'd think

Conventional wisdom in machine learning is that you should start off using transfer learning from a set of pre-trained weights. We've tried this, but it's never moved the needle for us. We're almost always altering something that slightly changes the topology of the base model we're using, so the efficiency takes a hit, and given how small the real-time models are, they are actually pretty quick to train (like 6-7 hours on an older GPU).

Post-processing is your secret weapon

One of my favorite papers in audio-ML these days is Neural Vocoder Is All You Need for Speech Super-resolution. Here's a part that made my eyes pop out when I read it:

We observe that even replacing neural network methods with a simple replication-padding method without learning, NVSR can still significantly outperform existing speech SR methods.

In layman's terms, what the authors are saying is that they tried a version of their industry-leading method with just the post-processing layer and it still outperformed several state-of-the-art ML models. It takes a lot of chutzpah to say this: basically, you're telling the world that in spite of all the fancy machine learning in the article (which is what most of them get paid to do), a simple method they found still outperforms some ML-based methods.

When I first read the article, I replicated the above result (which one can do less than an hour as you don't need to train an ML model) and their claim is 100% true. Since then, we've been huge fans of applying very simple post-processing hacks to otherwise fancy models. We may not win the Turing Award for it, but we get a better product to market faster.

Choose the grungiest bare-metal cloud service you can find

I got a call from our Google Cloud rep reminding me recently that we had nearly 100k of free credits still available in their startup program that we hadn't used. At the same time, we gladly pay out-of-pocket for runpod.io even though maybe, theoretically, we could do the same thing on Google Cloud. Why?

Cloud providers often provide lots of incentives to get you building on their cloud, but the fine-print often winds up in charges to your credit card in ways you don't expect and often have lots of identity-management and setup steps between you and your GPU. You can see this right now: Google "Google for Startups - Free Credit Machine Learning" and you'll find no easy, three-step guide to paying for a GPU box with free credits.

Furthermore, a lot of times they GPU you want is just plain not available, but you don't know until you're 50 clicks into their UI. It's basically as bad as a University cluster, but more expensive and behind the veneer of corporate jargon.

On the other hand, services like RunPod are, for lack of a better word, hardcore. They have a fixed number of GPUs with 0 ambiguity about what's available. You have to wake up early and fight to get the one you want. But, it's super cheap and laser focused on getting you training your models as fast as possible. The pods just work and topping up is a breeze. If you run out of money, they'll just kill your job. They won't charge your credit card automatically unless you ask them (don't set up auto-charge early on, it will just burn through money accidentally).

There are some haters on Reddit that complain that the service has degraded over the years, and I wasn't there before 2023, but we've tried lots of services and Runpod has come out on top. There are several companies like it. In general, if a cloud provider has a Discord server where developers negotiate about who can use what GPU, you should consider using it.

Keep track of your experiments

We've run thousands of experiments at this point, and if we didn't have a way to keep track of the trained weights, loss curves, and our subjective impressions of the models, we'd slow down to a crawl. In the early days, we even accidentally demoed the wrong model to a group of musicians that was accidentally trained with the input set as the output. Yes, we showed off an identity function. And we did it because we weren't keeping track of our experiments.

Nowadays, we use Weights & Biases (wandb), and we're pretty happy with the result. It gives us everything we need (storage, clean UI) and is a one-liner to set up in a python script.

Do listening tests earlier than you're comfortable with

Unless things get better in the training space, the first results of your training will sound compressed, noisy, and garbly, probably with a low-pass filter to boot. This is normal, and even though you know it will sound like crap, you should still listen to it. This is partially to motivate yourself to make it sound better, but even more importantly, you can always hear how a model gets better over time, and this will guide your intuition about how to tweak your model.

Even when you see a loss function that is nowhere near competitive, you should listen to the output instead of throwing it away and moving to something else. When I've failed to do this, I may arrive at a better loss function quickly, but I will have no idea how the sound improved as I got there, which takes away an opportunity to improve the plugin.

Do hyperparameter sweeps at the real-time boundary

Most services, including wandb and Sagemaker, have some sort of hyperparameter sweep functionality where they can kill off underperforming models early on and help settle on the parameters that produce the best loss functions. For real-time audio programming, this is especially important as many parameter configurations will result in unusable models because they are too slow or require too much data, which causes an audible delay.

A good rule of thumb is to aim for 100k weights and assume that anything bigger than this will sound clicky. For those of you that follow the world of LLMs, yes, this size is 0.0065% the size of something like Google Switch. So it's puny, but it's often all you need, and in any case, it's all you can afford given the constraints of real-time audio (more on that later).

These technologies are getting faster all the time, so I'm sure 100k will be obsolete by the time I click publish, but you can always run a test by generating noise with a model and just listening for the clicks. This will quickly establish where the boundary is in your era (if you're reading this in the year 2358, my assumption is that you are a computer with multiple real-time systems, so perhaps this hits even closer to home).

Have listening sessions with engineers you trust

Once you churn out a few models that you're happy with, do a blind listening test with engineers. Attach no qualifiers to your sounds: just play the input, the output, and the target in random order. If you're doing generative AI, then it's just the target.

For example, when we were building Deprox, we had a very specific experimental method we used to build our dataset, and our choice of model aligned with what we were seeking to extract from the data. However, during a listening session at Finnvox, the overwhelming consensus was that Deprox gave a cleaner baseline for mixing the majority of vocal tracks.

"Automatic, mix-ready vocals" was certainly not what we wanted to build going into the project. In a way, it sounds a bit hand-wavy, and our more precise hypotheses about how we were acting on the proximity effect and plosives felt more secure to articulate. However, sometimes you need need to realize that the thing you've built might not quite be the thing you set out to build, but it still does something cool, and that's ok.

Another important element of these sessions is that they speak hard truths. We brought four plugins to our session with Finnvox and had ranked them internally in terms of how useful we thought they were. If our ranking was 1. A, 2. B, 3. C, 4. D, theirs was 1. A, 2. D, 3. C, and B didn't rank. Meaning it was so bad that they didn't even think it merited further development. All of them consistently zeroed-in on one feature of the plugin that we hadn't noticed and that proved to be unsolvable. So we canned it.

These listening sessions are not a guarantee of quality: all they mean is that people you paid for a couple hours gave you honest and candid feedback. Even if the feedback was promising, the very same engineers still may not use your product two or three months out. So while these listening sessions are helpful, don't let them create a false sense of security. You still need to trust your gut on when your product is ready, even with stuff that "went well" when you showed it off to the pros.

When to stop

There are two competing theories here. Standard startup/scrappy wisdom dictates "ship first, ask questions later." On the other hand, there are many stories of successful companies, like Oeksound, where the founder built something for three years and only released it when he felt it was basically perfect.

I'm in the second camp. Take your time. Before shipping, you can run all sorts of test with your plugin, give it for free to studios to get feedback, and even charge a bit for early access on Patreon. But resist the urge to ship a half-baked plugin just to get it out there.

In machine-learning, the urge to ship is especially strong because making incremental tweaks is so hard: sometimes, you have to collect more data, retrain a model, or just scrap the thing entirely. Another way to put it is that, on the heels of feedback or a brilliant idea, you can't buy a pack of Redbull, dive into a cave, and code for 12 hours. Instead, you are diving into WhatsApp, calling the studio to book time, and asking singers to come back in two weeks. For people like me that are biased towards engineering, this can be frustrating.

In spite of this, don't release your plugin to global fanfare until it's basically perfect. By perfect, I mean that you are using it regularly in a rotation with the other plugins you use to mix tracks. If you're currently not mixing anything, dig up old projects and redo them with your plugin.

This perfection does not just exude from the plugin itself. It exudes from you. For example, one thing I did when we built Deprox was remixed all an entire album of a cappella Wizard of Oz covers using it. It sounds ridiculously better, and I've shared this with most of my friends that use the plugin. They hear the difference immediately.

So, as painful as it is, wait. Keep building, and hit the market when you're truly satisfied. Like good food, there's no substitute for taking time. It doesn't matter if, in the interim, ten plugins come out that sort of do what yours does. It's ok to be the eleventh, as long as it's the best. And if you have investors or other external pressure that creates a deadline-y feeling, you can still communicate progress by telling them what you're learning while going slow. Anyone who has backed you in good faith will understand that you're playing a long game and will keep supporting you.

Inference

ML-based audio processing is still the Wild West. For all of the advances in ML over the past decade, the companies that support the most ambitious machine learning frameworks and datasets (Baidu, Alibaba, Facebook, Microsoft, Google) are mostly concerned with reasonably-quick inference that's done mostly on a server. Even the fastest commercial inference rigs powering Baidu's and Google's self-driving cars run at a relatively-modest 60 FPS. Granted, each frame has roughly 1 million pixels, but these pixels are predicted in parallel on an insanely powerful GPU.

Audio processing knows no such luxury: in almost any DSP algorithm of repute, each sample is calculated based on at least one previous sample, which means that it's very challenging to do computations in parallel. Furthermore, there's no time to send the audio data to the GPU, so all of the computations need to be done on a single processing thread. Of course, hardware may change, but in 2023, ML-based audio DSP is CPU-bound, mostly sequential, and must work on 44100 samples (!) a second.

Here are some tips to follow as you build your inference rig. They'll probably go stale in a matter of weeks, but at the very least it'll create a pang of nostalgia for those who harken back to the good old days of real-time audio machine learning in late 2023.

Use ONNX

This is your best option. It used to get a bad rep for real-time processing because of its unpredictable memory allocations, but it has improved significantly over the past couple years. While it is still not real-time safe because of these allocations, it is statistically safe, meaning that if you tune your model to a certain size (see the hyperparameter sweep section above), you'll get an underflow in less than 1/10,000 runs.

Engineers won't use your plugin if it crashes all the time, but if it causes an underflow once a month, that's fine as long as you make it known (a) you're aware of it; (b) you're sorry about it; (c) it's due to a constraint (ONNX) out of your control; and (d) you're actively working on an alternative framework to mitigate this (RTNeural, see below).

Outside of this limitation, ONNX is super easy to use. At first, I was pretty intimidated because of the name - I get spooked by anything without enough vowels and that has an X, so double-whammy. But the documentation is clear and, if you copy the relevant boilerplate, you can go from PyTorch to ONNX c++ in a matter of minutes. We've open sourced our solution on this gist to get you up and running.

Use JUCE

As unfun as C++ is in 2023, JUCE is still by far your best bet to build an audio plugin. It takes a bit of time to get the sheen and feel of something professional, but JUCE is an assemblage of hundreds of hard-earned wins against the vagaries of poorly-documented audio frameworks and operating-system weirdness.

We build a first version of our plugin using ONNX + Juce in two days. I can't stress enough how important this is: even though it was nowhere near market ready, having a crappy first version of a plugin that you can iterate on incrementally can make the difference between slideware and software. Otherwise, your UI, ML models and platform constraints risk drifting too far from each other. We made this mistake with one plugin idea and wound up in a situation where we had basically designed something that couldn't be achieved in real-time. Starting from a real plugin and building on it (instead of, say, a PyTorch-based Discord bot that will one day become a plugin) gives you the best shot at operating with in the laws of techno-feasibility.

Check out RTNeural

We're really excited about RTNeural and we're actively contributing to its development. Until hardware vendors start making dedicated chips for real-time audio inference, projects like RTNeural are our best hope for fast, real-time inference in real-time situations.

For those of you who are targeting Apple Silicon, you may be able to take advantage of Core ML to achieve even better real-time performance. We tried, but in spite of our efforts, we couldn't get it to beat ONNX. We've heard the same thing from other teams. I'm sure that Apple will improve its performance over time, but the lack of transparency coupled with the cross-platform issues make RTNeural a more promising option in the near-term.

Sales and marketing

Yay! You've built and shipped your first ML-based plugin. Chances are that it will create a lot of value for a lot of people, and you likely want to redeem some of that value in the form of money.

The good news is that the audio community will always pay for the best plugins. People may take issue with something's price or level of service, and you should address these issues, but I've never heard of piracy tanking a project, nor have I heard of great projects failing to earn enough income to support a small team.

Don't get me wrong: plugin development is hard. There are lots of really good plugins that don't make money, and somewhat unfairly, there are mediocre plugins that manage to gain short-term traction. But great plugins always create a sustainable economy around themselves. This idea used to make me feel lousy because my hit-rate was pretty over a 15-year period, but I've come to really appreciate it. Success never feels arbitrary: it is deeply rewarding, and you can hear it in the music people make.

Distance yourself from the experiment

When we first shipped Deprox, I decided to do a major Deprox detox. I stopped all data science and went into full-blown marketing and sales mode. This created a jarring impression from me, as I went from something with lots of momentum to lots of downtime. The downtime is of course just an illusion: an absence of sales and marketing means that you have everything left to do, but at first, it feels disorienting.

As unproductive as it sounds, I spent a few days with a really light workload just trying not to code. I mostly read books and articles. That gave my brain the time to unwind out of coding mode, and while I still had the immense task of sales and marketing ahead of me, coding no longer felt as comfortable or familiar. So I was faced with two, not one, uncomfortable options, which made it slightly easier to choose the one that I actually needed to do (sales and marketing).

You already have a community

Prior to building a ML-based plugin, a lot of my plugin development was quite solitary. In machine learning, you have a community whether you like it or not:

The authors that wrote the papers whose models you're using.
The engineers who wrote the frameworks you're using.
The folks that helped create the dataset you're using.
The early-adopters that will try anything AI-related just to get a sense of how it stacks up against older tools.

Not all of these folks necessarily use plugins, but many of them will gladly try something if you say "hey, thanks for writing this paper, my newest plugin uses one of the algorithms you developed." This gets word of mouth going, which will ultimately reach your target audience if people are enthusiastic about what you've done.

Don't market your plugin as AI

As mentioned above, there are folks that will try your plugin just because it is based on machine learning. This is entirely adjacent to a hype cycle: it is born of genuine curiosity. You probably fall into this category as well: if you are considering developing an ML-based plugin, you've probably tried some out to get a sense of how they sound.

That relationship with early adopters is where your AI-marketing should begin and end. Don't plug your plugin as an AI-based tool on your website. Obviously this is my opinion and it bucks current trends, but how many plugins do you use on a daily bases that shout from the rooftops "FFT-based!" or "digital!"?

Marketing a new plugin as AI-based cheapens it because it detracts from its most important quality: the way it helps people make great sound. Early on, mentioning ML or AI makes sense only insofar as it builds trust, the same way that an artisanal baker may advertise how they make bread. But using it as a one-off buzzword usually comes from a place of either insecurity or money-making. The second one (money-making) is legit for larger plugins: I have friends who market plugins as "AI-based" because Google Analytics shows that this increases sales by X%. But at that point, you're already confident in how awesome your plugin is, as are your core users. You can afford to place your faith in data because you're dealing with tens of thousands of downloads. In the beginning, though, avoid using AI-related wording in your main marketing.

Have a free demo

We made the mistake early on of diffusing Deprox plugin without a 14-day free trial. That's a terrible idea. Always include a fully-unlocked free trial, even if it means taking two days to build a hacky custom DRM solution.

Here's why:

People that make it to the end of the 14 days and still want to use your plugin will buy it.
People that don't buy it will gladly tell you why not if you ask them (you can build this into your plugin as a feedback button).
Anything less than this won't give people a full impression of your plugin, and if folks lack this impression, how can you expect them to know if it's right for them?
A 14-day free trial oozes trust and confidence in your plugin.

Another mistake we made early on was releasing our plugin as a subscription. Subscriptions are very common for machine-learning projects because models are always getting updated, so there's an idea that you're paying for monthly model updates. While this makes sense in theory, in the audio-plugin world, subscription-based pricing models will subject you to a form of cold fury that is traditionally reserved for kitten thieves and those who put green peas in guac. Don't do it. Go with a 14-day free trial from day one.

The rule of 100

Any plugin developer can get to 100 users purely organically. The announcement recipe is well known and is achievable in a day: Gearspace (formerly Gearslutz), KVR, a blog post somewhere, messages on your social media, and organic recommendations of people who followed you as you were building it.

If, a few weeks or after your launch, there aren't around ~100 people using your plugin at least semi-regularly, it probably sucks. Sorry. This is in spite of the fact that you love it and that others may love it too. We say this as developers that have made several sucky plugins, most of which we love.

It doesn't necessarily mean that the sound sucks or the UI sucks (although it could mean this). In many cases, it means your plugin doesn't work as a plugin: it doesn't solve enough people's problem or inspire enough people musically/creatively. This could be because it's too specific, too general, not impactful enough, or it's really a feature in plugin's clothing.

If you're still attached to elements of the plugin's sound or UI, it's fine to take this plugin back to the drawing board and iterate again until it's substantially improved or extract bits and pieces into a new plugin. The worst thing you can do is build an ad machine that keeps your plugin alive through paid search, 48-hour expiring offers and FOMO. You may sell plugins this way, but it'll dry up after a few months and you'll be back to the drawing board, minus those precious months when you could have been working on something cool instead of convincing people to buy something they didn't really want anyway.

If your plugin sucks, don't worry. You don't suck, your ideas don't suck, and you're still talented. You just made a bad plugin. Time to make the next one.

Updates

Plugin aficionados don't take kindly to botched updates. They bought your plugin because they like the sound, and they've gotten used to a workflow to achieve that sound at a quick pace. If a plugin changes the workflow or, worse, changes the sound - even for the better - your users will rise up in fury.

In the machine-learning world, this can be frustrating because we're retraining and tweaking models all the time, and due to the black-boxy nature of models, it'll sometimes lead to noticeable differences in sound at a regular basis. To mitigate this, make sure to always make older versions of your model accessible via the UI. In fact, the best thing is to make the version a parameter in your JUCE AudioProcessorValueStateTree. That way, the automation preserves the versioning, and people can opt-into the new versions of models if and when they so choose.

Conclusion

Making ML-based plugins is incredibly rewarding on multiple levels.

They solve problems and enable sounds that are impossible to achieve with traditional methods.
You feel like you're on the cutting edge of something, which for me at least is exhilarating.
You build bridges with all sorts of communities and projects thanks to the deeply collaborative nature of the community.
While not as bad as quantum computing, ML-based approaches (especially generative ones) are notoriously hard to get right, so people are understanding and accepting of failure.

If you're building a ML-based plugin, don't hesitate to join our Discord. While we have private channels where we talk about our corporate plans to harvest humans as energy sources that feed into a giant supercomputer that shapes reality, on our public channels, we mostly discuss beautiful sound and ways that machine learning can or can't help us get there. I hope to see you there 👋🎙️🎧🤘

0 comments

Deprox Q&A

Hello Deprox!

See all posts from Meeshkan