When we set out to build audio plugins in Meeshkan, we made a list of "really hard things" and vowed to check them off one by one in the form of delicious plugins for your aural degustation. Deprox was towards the top of that list, and it came from a problem that several of us have faced over our many, many years of recording singers.

While microphones are in none of the four food groups, singers sure do like to eat them, us included. There are many reasons for this. Getting cozy with a microphone often triggers a feeling of being close to a musical subject, be it the song itself, an audience, or one's imagination. Also, intimate recordings pick up loads of beautiful, subtle tones and inflections, and they tend not to have too many room reflections.

However, with this intimacy comes challenges. The spectrum of recordings often gets subtly warped, and this warping leads to the audio sounding harsh or strident. You can hear it right away, but it's very hard to pick up in spectrograms or adaptive EQ because it's fleeting and often appears in the transients. Listen to this recording as an example. While it doesn't clip, it is too rich in proxy frequencies, which makes it difficult to mix.

Click to listen

One way to correct this is just to move the microphone back 10cm. But often times, we can't redo the perfect take, nor would we want to. In spite of its harsh proximity frequencies, close recordings have eerie closeness and beauty that touches us deeply. Sometimes that beauty is not from "sing-y" sounds at all, but rather swallows, rattles, and fleeting breaths. Let's listen to the same recording again, but in its context, and you'll hear what I mean.

Click to listen

So we want to keep the whole shebang, but we wind up doing pirouettes to fix harsher frequencies in the mix. Even high-end Grammy-winning recordings with massive mixing budgets suffer from too-hot close-mic'd mixes because it's so damn hard to fix.

Hard, did someone say? That sounds like a job for (da, da da da, da da, daaaaa)...

Deprox

The heart of Deprox is a 326-hour dataset, recorded at Finnvox studios, of singers on over 20 microphones arranged at various distances from a singer. We were fortunate to work with some of Finland's best singers, including students and graduates of Finland's Pop and Jazz Conservatory. We were even a bit gobsmacked when none other than Aku Keikonen, who charmed a whole country on The Voice of Finland, agreed to lend his voice to the project.

Isn't his voice dreamy? But all of our singers were so damn good, and if Deprox doesn't pan out, our next plan is to release a "Live at Finnvox" recording with all of the music they laid down.

Here is one of the original recording rigs we used when making Deprox. It evolved over time, but the idea was to record signers from various different angles and distances, and also have the singers move from time to time, in order to have as much material as possible to learn from.

Armed with the world's most nuanced, beautiful, poetic dataset, we unleashed a torrent of machine learning models on our recordings. Ultimately, we gravitated towards a paper called Efficient neural networks for real-time modeling of analog dynamic range compression because it had good realtime performance and compressors, like our plugin, don't alter the spectrum of a sound too much. After all, the whole point of Deprox is to restore the perfect take, so it should have a mega-clean spectrum that provides a solid basis for downstream processing.

That being said, no neural network was quite cut out for what we are doing, so we had to experiment a bunch and use many different variations inside of the plugin. Many-a-GPU were burned through, but golly did we make some really nice loss functions.

Benchmarks

The first thing I'd like to point out is that the final plugin, while it makes a different-looking waveform, does not touch the spectrum on a macro level. You can check out the screenshots below to see what I mean. In the first one, we see that there are noticeable changes in amplitude:

Whereas the frequency spectrums are nearly identical:

That said, the plugin kicks it into overdrive on the microlevel. The visualization below shows the "spectral diff" at any given moment between the original sound and the rendered audio. It is making micro adjustments all the time. So while on the macro-level it has the behavior of a compressor (which is what we want when we're tracking a singer), on the micro level it behaves like an equalizer.

Let's hear it in action! Compare the excerpt above to the following one. It preserves the beauty of the more intimate sections and retains the power of the more expressive ones without saturating.

Click to listen to the original

Click to listen to the Deprox'd

The gain of the processed graph is definitely less intense, but it's not quite what you'd see with a compressor, and certainly not what you'd see with EQ.

Demo

You can try Deprox out at this link. The version on the site uses the "aggresive mids" setting, which is the most popular way to use Deprox. The site accepts mono audio files less than ten seconds long. We're confident that you'll love the result! When you're ready to use the plugin in your DAW, you can mosey on over to our Gumroad sales page.

What's in store

Like most AI-powered products (Chat GPT, Midjourney, etc), we're constantly tweaking our models and training them on more and more data. We will blast y'all periodically with news of new releases. Your subscription pays for our team-of-two's continued development of the product.

You can also help influence how we train our models, from the data we collect to the algorithms we choose, on our Discord! Here's a link to join: https://discord.gg/BCKmGrnBt.

0 comments

Everything We Learned While Building an Audio Plugin Using Machine Learning

Deprox Q&A

See all posts from Meeshkan