Zero Shot (any sample; esp. instruments)

https://github.com/RetroCirce/Zero_Shot_Audio_Source_Separation
(as USS Bytedance came out now, zero shot can be regarded as obsolete now, although zero-shot might is rather better for single instruments than for SFX)

You provide e.g. sample of any trumpet or any other instrument, and AI returns it from a song you choose to separate.

Guide and troubleshooting for local installation (get Discord invitation in footer first if necessary).

Google Colab troubleshooting and notebook (though it may not work at times when GDrive link resources are out of download limit, also it returns some torch issues after Colab updates in 2023).

Check out also this Colab alternative:

https://replicate.com/retrocirce/zero_shot_audio_source_separation

It's faster (mono input required).

Also available on https://mvsep.com/ in a form of 4 stems without custom queries, and it’s not better than Demucs in this form.

"Zero shot isn't meant to be used as a general model, that's why it accelerates on a specific class of sounds with some limitations in mind.... It mostly works the best when samples match the original input mixture, of course there are limitations"

"You don’t have to train any fancy models to get decent results [...] And it’s good at not destroying music". But it usually lefts some vocal bleeding, so process the result using MDX to get rid of these low volume vocals. Zero-shot is also capable of removing crowd from recordings pretty well.

As for drums separation, like for snares, it’s not so good as drumsep/FactorSynth/RipX, and it has cutoff.

"I did zero shot tests a week or two ago, and it was killing it, pulling harmonica down to -40dB, synth lines gone, guitars, anything. And the input sources were literally a few seconds of audio.

I've been pulling out whole synths and whistles and all sorts.

Knocks the wind model into the wind, zero shot with the right sample to form the model backbone works really well

The key is to give it about 10 seconds of a sample with a lot of variation, full scales, that kinda thing"

Dango.ai

Custom stem separation feature (paid, 10 seconds for free)

Expensive

Special method of separation by viperx (ACERVO DOS PLAYBACK) edited by CyberWaifu

Process music with Demucs to get drums and bass.

Process music with MDX to get vocals.

Separate left and right channels of vocals.

Process vocal channels through Zero-Shot with a noise sample from that channel.

Phase invert Zero-Shot's output to the channel to remove the noise.

Join the channels back together to get processed vocals.

Invert the processed vocals to music to get the instrumental.

Separate left and right channels of instrumental.

Process instrumental channels through Zero-Shot with a noise sample from that channel.

Phase invert Zero-Shot's output to the channel to remove the noise.

Join the channels back together to get processed instrumental.

Process instrumental with Demucs to get other.

Combine other with drums and bass to get better instrumental.

So it sounds like Zero-Shot is being used for noise removal.

As for how Zero-Shot and the noise sample works…