AudioSep
“I decided to try AudioSep: https://github.com/Audio-AGI/AudioSep on MultiSong Dataset.
I used prompt 'vocals'. I was sure it would be bad, but I didn't think it's so bad.
https://mvsep.com/quality_checker/entry/8408
I also tried it on the Guitar dataset - it's even worse - negative SDR. Maybe I'm doing something wrong. But I tried the example with cat from the demo page, and it worked the same as in there. So I think I have no errors.”
sdr: 0.33
si_sdr: -2.39
l1_freq: 17.62
log_wmse: 6.72
aura_stft: 3.66
aura_mrstft: 5.55
bleedless: 9.29
fullness: 16.58
Colab on GH probably gives unpickiling issue. You might be able to fix it be executing:
!pip install torch==2.5.0
After you execute all the installation-related cells.
Since then, probably something more about dependencies is also needed, like it ‘s coded now in the inference Colab.
Medley Vox (different vocalists)
Use already separated vocals as input (e.g. by Roformers, vox_ft or MDX23C fullband a.k.a. 1648 in UVR or 1666 on MVSEP).
Local installation video tutorial by Bas Curtiz:
https://youtu.be/VbM4qp0VP8
(NVIDIA GPU acceleration supported, or perhaps CPU - might be slow)
Cyrus version of MedleyVox Colab with chunking introduced, so you don't need to do chunking manually:
https://colab.research.google.com/drive/10x8mkZmpqiu-oKAd8oBv_GSnZNKfa8r2?usp=sharing (07.02.25 fork with fairseq fix and GDrive integration)
Currently, we have a duet/unison model 238 (default in Colab),
and main/rest 138 to uncomment in Colab.
Recommended model is located in vocals 238 folder (non ISR-net one).
While:
“The ISR_net is basically just a different type of model that attempts to make audio super resolution and then separate it. I only trained it because that's what the paper's author did, but it gives worse results than just the normal fine-tuned.”
MedleyVox is also available on MVSEP, but it has more bleeding and “doesn't work as well as the Colab iteration with duets”. (Isling/Ryanz)
The "duet/unison model 238" will be used by default.
``and main/rest 138 to uncomment in Colab`` if you need it.
Then go to the first cell again. To "uncomment" means to delete the "#" from the beginning of the line before the "!wget" so the line will be used to download the model files.
Do it for both pth and json lines
(you might be asked whether to replace existing pth and json files by the alternative model you just downloaded in the place of the previous one)
``Recommended model is located in vocals 238 folder (non ISR-net one).``
That's the model used in the Colab by default. You can ignore that information. It's for users using the MV on their own machine.
The output for 238 model is 24kHz sample rate (so 12kHz model in Spek).
You might want to upscale the results using e.g. AudioSR or maybe even Lew’s vocal enhancer location further below the linked section.
The output is mono.
You might want to create a "fake stereo" as input by copying the same channel over the two, then do the same with another channel, and then create the stereo result from both channels processed separately in dual mono with MV.
The AI will create a downmix from both input channels instead of processing channels separately.
Be aware that “dual mono processing with AI can often create incoherencies in stereo image (like the voice will be recognized in some part only in left channel and not the other, as they are processed independently)” jarredou
"The demos sound quite good (separating different voices, including harmonies or background [backing] vocals)"
It's for already separated or original acapellas.
The model is trained by Cyrus. The problem is, it was trained with 12kHz cutoff… “audiosr does almost perfect job [with upscaling it] already, but the hugging page doesn’t work with full songs, it runs out of memory pretty fast”.
It was possible at some point that later stages of the training, looking like over fitting were responsible for higher frequency output.
It’s sometimes already better than BVE models, and the model has already similar to demo results on their site.
Sadly, the training code is extremely messy and broken, but a fork by Cyrus with instructions is planned, with releasing datasets including the one behind geo-lock. Datasets are huge and heavy.
Original repo (Vinctekan fixed it - the video at the top contains it)
https://github.com/jeonchangbin49/medleyvox
____
Outdated
https://colab.research.google.com/drive/1StFd0QVZcv3Kn4V-DXeppMk8Zcbr5u5s?usp=sharing (pip issues fixed 29.08.24, defunct as of 06.02.25)
(outdated instructions, current Colab explains everything)
“Run the 1st cell, upload song to folder infer_file, run the 2nd cell, get results from folder results = profit”
Further explanations how to use the Colab:
``Run the 1st cell``
So press the first "play" button then you load the Colab
``upload song to folder infer_file``
Looks like the folder for the input file has changed from infer_file to input in newer Colabs.
So, once you started the first cell, and it finished, open Colab file manager (folder icon on the left) and go to /content/MedleyVox/input\
Now paste your song there and wait till it's done.
``run the 2nd cell``
So the next play button below the first one once you scroll down a bit. Now it will start separation
``get results from folder results``
Go to file manager again and find /content/MedleyVox/results
right-click on the result file and download it. Wait till it's done.
``Currently, we have a duet/unison model 238 (default in Colab)``
So you don't have to change anything in the Colab to separate using it.
Old info
https://media.discordapp.net/attachments/900904142669754399/1050444866464784384/Screenshot_81.jpg (dead)
Colab old
https://colab.research.google.com/drive/17G3BPOPBPcwQdXwFiJGo0pKrz-kZ4SdU
Older Colab
https://colab.research.google.com/drive/1EHJFBSDd5QJH1FQV7z0pbDRvz8yXQvhk
(The same one, but here you need to change the .ckpt, .json and .pth files there from Cyrus [more details in the video above].)
____________________________________________________________________
About other services:
Check this chart by Bas Curtiz to check what AIs use various (also online) services, plus their pricing.
At this point everything mentioned above this link for at least instrumentals, vocals, 4-6 stems is better than below, (with exceptions for some single stems described at the top) commonly known services:
Spleeter
and its implementation in:
Izotope RX-8/9/10
which just uses 22kHz models instead of 16kHz in the original Spleeter. There is no point in using these anymore. The same goes to most AIs described below (or only for specific stems):
voiceremover.org, lalal.ai,
phonicmind
melody.ml
RipX, Demix,
ByteDance Ripple/CapCut
For reference, you can check a comparison chart on MVSEP.com,
or results of demixing challenge from Sony (kimberley_jensen there is 9.7 MDX-UVR model for vocals - 2nd best on the time)
and watch this comparison.
To hear 4 stems models comparison samples you can watch this video comparison (December 2022).
It all also refers to new:
real-time
AI separation tools like
Serato
and
Stems 2.0
tensorflow model (which can be found in newer Virtual DJ 2023 versions, now free for home users - better than Serato and Spleeter implementations)
- they all do not perform better than the best offline solutions at the very top of the document. But “Esp. since it's on-the-fly [...] results are more than decent (compared to others).”
Acon Digital Remix
(Vocals, Piano, Bass, Drums, and Other)
“Just listened to the demo, not great [as for realtime] but still”
Others
FL Studio
It’s actually not realtime. It takes some time to process tracks first (hence maybe it’s the best out of the three).
It uses Demucs 4, but maybe not the ft model and/or with low parameters applied or/and it's their own model.
"Nothing spectacular, but not bad."
"- FL Studio bleeds beats, just like Demucs 4 FT
- FL Studio sounds worse than Demucs 4 FT
- Ripple clearly wins"
pymss-ara
https://github.com/pymss-project/pymss-ara
It also doesn't work in real time, but it allows separations inside your DAW premade in a separate process (currently Windows and CUDA supported). The forte is, most current Roformers are supported.
demucs-rs
https://github.com/nikhilunni/demucs-rs
It has VST3 CLAP variant working on MacOS using Metal acceleration. Or the regular CLI variant with Vulkan backend on Windows/Linux (faster than DirectML in UVR)
djay Pro 5.x
“very good realtime stems with low CPU” Allegedly “faster and better than Demucs, similar” although “They are not realtime, they are buffered and cached.” but it’s very fast anyway. It uses AudioShake. It can be better for instrumentals than UVR at times.
Neutone VST
It has a Demucs quantized model to use in realtime in a DAW
(it uses light “retrained, smaller version” version of Demucs_mmi)
https://neutone.space/models/1a36cd599cd0c44ec7ccb63e77fe8efc/
It doesn't use GPU, and it's configured to be fast with very low parameters, also the model is not the best on its own. It doesn't give decent results, so it's better to stick to other real-time alternatives. It won’t work correctly on low-end CPU (e.g. C2Q), breaking audio in the middle and giving inconsistent audio stream with some random breaks.
Peel Stems
https://products.zplane.de/products/peelstems/
VST for real time source separation (probably the same models like in MPC stems)
https://www.youtube.com/watch?v=0Js5bWQWY7M
- Service rebranded to
Fadr.com from SongtoStems.com
It is just Demucs 4 HT, but paid.
"My assumption, Fadr uses Gain Normalize [for instrumentals] was right [...].
Demucs 4 HT seems to get a cleaner result. The rest = practically identical." And someone even said that vocals in VirtualDJ with Stems 2.0 had less artifacts on vocals.
Apple Music Sing
“I heard a few snippets, and what stood out is, whether intentional or not, the vocals remained in the background just enough to actually hear them.
Now that could be great for Karaoke, so u have a kind of lead to go on.” but as for just instrumentals, it’s bad.
Voxless
VST “uses AI to separate vocals and instrumental in real time. Now it is designed to be used in a DAW, but you can also run it in soundsource [on Mac, or probably SAVIHost (VST2/3) or Equalizer APO (VST3) or JBridge on Windows] so you can use it on your system audio live. It has low latency and doesn't use CPU a lot. The software has a very simple interface, just two knobs to increase/ decrease the instrumental or vocals or a mute/solo button for each. As for the quality it sounds like the first ever days of audio separation with AI like Demucs v1 or Spleeter in 2019 - 2020 but a little worse somehow, since it is very low latency not CPU heavy, but it does the job. Voxless has a trial of 7 days if you wanna check it but the license costs 100$ which I think is quite a lot for a software that separates vocals with barely first gen quality.” midol
Ozone 11 Master Rebalance
“I select vocals and have them dialed down to max using an EQ inside of it (may sound complicated, so you gotta watch a tutorial to see how ozone works). However, the results were far from voxless quality. It leaves so much bleed and whenever vocals are quite loud you can barely hear anything from the way it's fighting it, so it sounds like a complete mess. Both from the master rebalance and the main AI interface” - midol
(x) BL-Rebalance
“The most important thing which is the separation quality, is horrible unfortunately, dialing the vocals all the way down to -120db the max, barely picks up vocals to cancel, the song sounds like it's just playing normally with vocals being suppressed in a very horrible way, it's muddy, and it leaves a lot of bleed, also again, when vocals are quite loud, you barely hear anything else because it's fighting hard.” midol
algoriddim djay
App for Windows, Android, Mac.
Judging by strings in stemseparation.dll, they seem to use “bytesep” which is a package name of this repository: bytedance/music_source_separation.
____________________________________________________________________
Music to MIDI transcribers/converters
Multitrack stem MIDI transcription:
https://github.com/magenta/mt3
Good results for piano:
https://github.com/magenta/magenta/tree/main/magenta/models/onsets_frames_transcription
“On high quality piano recordings its almost flawless”
If you have notes:
musescore
transkun transcriber (now also on MVSEP)
“it's the most accurate piano transcription algorithm ever trained and is unequalled in accuracy and absolute indifference to literally *any audio quality*
as long as the piano being transcribed is at A440 it'll spit out a 95 percent accurate transcription from virtually any recording no matter how absolute garbage it is”
Piano2Notes
(notes and midi output, paid, 30 seconds for free, very good results)
“Tried Basic-Pitch and It is way worse than MT3 as It produces MIDI tracks without an identifier.”
“not 100% accurate but it's certainly the easiest to use and maybe most versatile”
https://github.com/muscriptor/muscriptor
“For drum transcription this is better than ADTOF, since it can also classify the different toms and not just one like ADTOF. Not only the tabs are accurate, they are also quantized optimally.” - Vinctekan
Allegedly the best for now, and the best model is MuScriptor Large
“I've tried to separate the stems into piece-by-piece, but MuScriptor is doing the best when it works with full instrumentals”
AnthemScore
(paid)
Harmonic mixing (find the song key)
“Since mixed in key change from 10 to 11 the software has several failures especially when overwriting the file name and an error that base 84 error, and you are left without the analysis of the file thing. Which is essential when doing remixes and having a clarity of the tone and bpm. Someone knows of an alternative that does not make a mistake”
https://www.reddit.com/r/DJs/comments/n5byah/key_detection_comparison_mixed_in_key_10_vs_85/
__________
Older separation services
Audioshake
Paid, $16 per wav stem, 2 or 5 stems (6? (guitar and piano) or 4 stems for preview (Indie creators)
Better piano model than GSEP.
"gsep piano model is very clean but sometimes fails in bigger mix, when there are a lot of instruments"
And also guitar stem
Instead of Audioshake you can use:
- myxt.com (also paid, 3 stem model, prob. 16kHz cutoff which Audioshake normally doesn't have. No other stem. Results, maybe slightly better than Demucs)
- Algoriddim djay pro
- Neural Mix Pro (part of Algoriddim, also uses Audioshake), but it’s only for MacOS
- LANDR Stems (cheaper, also uses Audioshake; plugin, probably doesn’t work locally, free access won’t give you access to stems; “LANDR Stems is only included in Studio Standard + Studio Pro” it’s not included in trial; SDR: 1 | 2)
- https://twoshot.app/model/289
Audioshake is suspected that it is just MDX with expanded dataset, but there’s no evidence at the moment. Comparing to UVR/MDX-UVR NET 1 model, vocal stem is 9.793 vs 9.702 in free MDX-UVR, so they’re close as for vocals.
Their researcher said they were training UMXHQ model at this period of time of 2020 Demixing Challenge.
Free Demucs 3 has a much better SDR for drums and bass than Audioshake, however the SDR for vocals and others is worse.
It accepts only non-copyrighted music for separations, but you can slow it down to circumvent it (some music like K-Pop BTS is not detected) but changing speed to 110% yields better results, even vs reversing the track.
Upload limit is one minute, so theoretically you can cut and merge chunks, but AS will fade out each chunk, so you need to find specific overhead to begin every next chunk with, to merge chunks seamlessly (I don’t remember if it solves the problem of AS watermark, though).
Then, you can download preview (chunk) for free using similar method like described in allflac section (Chrome Dev Tools -> Network -> Set filter to amazon) but result file is unfortunately only 128kbps mp3.
They are now limiting how many audio files you can upload to preview, but that can easily be mitigated by just using a temporary email provider or adding “+1” or “+2” or “.” to your gmail address, so you will still receive your email e.g. y.o.u.r.m.a.i.l.@gmail.com is the same for Google as yourmail@gmail.com.
You can also ping Smudge, Baul Plart or Bas Curtiz in #request-separation to isolate some song to make this all easily just for you (edit. 09.02.2023 - at least the Bas’ tool stopped working, so the rest like AS Tool might be dead too - at least in terms of API access, not sure).
Lalal.ai
7 stem
Acoustic and electric guitar models, piano, bass, drums and vocal with instrumental (for 2 stem UVR/MDX should do the job better)
Online service with 10 minutes/50MB per file limitation per free user.
Now they have some voc/inst models sounding like some ensemble of public Roformers, but still not as good, but close. Some specific models are worth trying out, e.g. lead guitars - the model got better by the time or piano model.
Older notes:
“I love Demucs 3, although for some specific songs (with a lot of percussion and loops) I still find lalal better.
Demucs is great at keeping punchy drums, for example hip-hop, rap, house etc songs”
“lalal is[n’t] worth it anymore, most of their models like strings or synths are crap and don't work at all” ~becruily
How to… abuse it. Doesn't always work for everyone, and sometimes you'll receive only 19 seconds snippets.
Go to the signin/register screen and use a temp email from https://tempail.com/
When you are in, make sure you use the settings with a P icon, P meaning Pheonix, which seems to be some hybrid mvsep lalal shit they made
I'd recommend making the processesing level normal, although you can play around with the settings to see what sounds better
They will later process it and since lalal has shorter queues, you get them faster. It took me like 10 seconds to get a preview for a song and 20 seconds for full which is wild.
You will get a sample and if you like it, you can submit it and get your stems!"
You can also use dots in Gmail addresses, instead of +1 (and more) at the end, which is unsupported in lalal. You'll receive your email with dots in its username anyway, and it will be treated as a separate email by their system.
Their app uploads input files to separate on external servers.
DeMIX Pro V3
Paid, 6 stem model, trial
Official site:
https://www.audiosourcere.com/demix-pro-audio-separation-software/
paid 33$/month, or x10 for year, or x2,5 permanent license, 7 days trial available
https://www.audiosourcere.com/demix-pro-audio-separation-software/
Vocal, Lead Vocal, Drum, Bass & Electric Guitar
https://www.demixer.com/ has the same models implemented, though they don’t currently even describe that guitar model is available, but when you log in, it’s there. Guitar might be a bit worse than RipX (not confirmed)
“audioshake [had] the best guitar model [at some point] (its combined [paid only]), second place is deemix pro (electric guitar)”
"Demix launched a new v4 beta, and it can now process songs locally + new piano and strings models
the piano model is not bad at all, it sounds a bit thin/weak, but it detects almost all notes
hadn't found good songs to test the strings model yet, but it might be good too"
Hit'n'Mix RipX DeepAudio
Moises.ai
Not really a good models before introduing BS-Roformer ones, no previews for premium features.
You can use apk when it allows previewing for free without downloading, but isling found some workaround googling for “moises premium free apk”.
Some information here might be outdated.
“also has a guitar and a b.v. model, and a new strings model, but it's not that good, in my opinion it is not worth buying a premium account.
4-STEM model is something like demucs v2 or demixer.
B.V. model is worse than the old UVR b.v..
GUITAR model is not really good, it's probably MDX, it has a weird noise, and it tries to take the "guitar" where is not at all. It takes acoustic and electric guitar together.
PIANO model is just splitter, maybe better at some songs.
STRINGS model is interesting, It's good for songs with orchestra, but still not that clean
Their service is very interesting, and the appearance of their site is clear and simple, but the models have better competitors.” thx, sahlofolina.
Byte Dance
available on https://mvsep.com/
“This algorithm took second place in the vocals category on Leaderboard A in the Sony Music Demixing Challenge. It's trained only on the MUSDB18HQ data and has potential in the future if more training data is added.
Quality metrics are available here (SDR evaluated by his authorship non-aircrowd method):
Demos for Byte Dance: https://mvsep.com/demo.php?algo=16 “
(8.08 SDR aicrowd for vocal)
MDX-UVR SDR vocal models (kimberley_jensen a.k.a. KimberleyJSN) were evaluated by the same dataset as ByteDance above (aircrowd):
https://discord.com/channels/708579735583588363/887455924845944873/910677893489770536
and presumably the same goes to GSEP and their very first vocal model (10 SDR) since their chart showed the same ByteDance SDR score like in aircrowd.
___UVR settings for ensemble (section deprecated, see the section above)__
Ensemble can provide different results from one current main model, but not especially better in all cases, so it’s also a matter of taste and conscious evaluation.
- Aggressiveness shouldn’t be set to more than 0.1
(also check 0.01)
- high_end_process: bypass (official recommendation) or mirroring 2 (in some cases)
- In most cases, you shouldn’t use more than 4 models to not decrease the quality (developer recommendation)
Don't use postprocessing in HV Colab for ensemble (doesn't work).
Other recommended models for ensemble:
HP2-4BAND-3090_4band_arch-500m_1.pth,
HP2-4BAND-3090_4band_arch-500m_2.pth
(+new 3 band?)
as they currently the best (15.08.21) but feel free to experiment with more (I also used old MGM beta 1 and 2 with two above,
some people used also vocal models as well, and later there was also HP2-MAIN-MSB2-3BAND-3090_arch-500m model released, which gives good results solo).
(old) ___Good UVR accapella models______
In general, it’s better to use MDX-UVR models for clean acappellas, but for UVR, these are going to be your best bet:
- Vocal_HP_4BAND_3090 - This model with come out with less instrumental bleed.
- Vocal_HP_4BAND_3090_AGG - This is a more aggressive version of the vocal model above.
“If you wanna removes the vocals but keeping the backing vocals, you can use the latest BV model”
HP-KAROKEE-MSB2-3BAND-3090.pth
(HV)
For clean vocal, you can also use ensemble with following models:
https://cdn.discordapp.com/attachments/767947630403387393/897512785536241735/unknown.png
(REUim2005)
__How to remove artefacts from an inverted acapella?_____
This section is old, and “cleaning inverts” in current models section can provide more up-to date solutions.
0) Currently, GSEP is said to be the best in cleaning inverts. But at least for vocal you can use some MDX model like Kim, or even better MDX23 from MVSEP beta.
- by charm
(rather outdated) Use Vocal_HP_4BAND_3090_arch-124m.pth at 0.5 aggressiveness, tta enabled
then use any model u like with Vocal_HP_4BAND_3090_arch-124m.pth instrumental results to filter out any vocals that weren't detected as vocals with Vocal_HP_4BAND_3090_arch-124m.pth model
combine the two results
then use model ensemble with whatever models u like (i used HP2 4BAND 1 and 2)
drag both vocal hp 4band+another model and ensemble results into audacity
use the amplify effect on both tracks and set it to -6.03
render
then use StackedMGM_MM_v4_1band_arch-default.pth
tbh vocal models even at 0 aggressiveness really help inverts
Or 0.3
I mostly use acapellas for mashups and remixes, so the little bit of bleed i get at 0.0 aggressiveness is fine
drums-4BAND-3090_4band.pth
0.5 optionally (less metallic sound)
2)
Utagoe (English version with guide and error messages translated by Anjok) - if the invert isn't good, then try utagoe, but it’s not the best (although rather better then Align Inputs in UVR).
Settings for Utagoe by HV:
“if your tracks don't invert perfectly” (even when aligned)
“if it's perfectly inverting”:
Some other settings:
“It has a weird issue sometimes tho, even when everything is perfectly aligned and inverts perfectly, utagoe misses some places, and it won’t insert for a second or so”
by Mixmasher00
“There is no actually settings depending on songs, but that is what I use, which is the default one.
Going higher than 1.3 [of extractable level] imo won't do good at cleaning. Additional tip too, if you want to do just an "invert" and "keep the original vocal volume" just choose "by waveform”.
I have been using Utagoe for inversions recently because it keeps the original volume of the vocals, and then I ran it on UVR or MDX. If the chunks are soft, I prefer using UVR but if there are chunks [that are] heavy like drums, I'd use MDX.
Also, I find it better to clean an invert via UVR or MDX than Utagoe because it's better and cleaner without destroying the vocals“
- “When using Utagoe, or UVR5, for aligning inputs, and inverting them, I get this really strange cracking noise, that not even doing a vocal separation with AI later can get rid of. Anyone know what could possibly be causing this?
So, I actually found a solution to this, for anyone running into a similar issue.
With inversions like this, you're already gonna have to use AI to get rid of the left over noise, since it's not gonna be a perfect inversion. So, the solution isn't to get a perfect one, it's simply to get rid of the noise that the AI cannot recognize, right?
With this particular inversion, I originally was using really compressed MP3s from around 2007 for the instrumentals, because the lossless versions of the instrumentals were lost media, up until a few months ago.
I thought it was odd, because I don't remember this noise being an issue with the MP3s, and that's when it hit me, MP3s cut off the noise, with compression, and added just a small bit more of that noise you get with imperfect inversions.
So I converted the lossless instrumentals to an MP3 with Foobar, and it was better, but still had that damned drum crackling! So I kept trying. I used OPUS, OGG, different bitrates of MP3, even AAC.
I have found that OPUS is the best at removing the drum overlap, I cannot hear any in fact, with OPUS.
So, my final guide is,
If you are getting cracking/crackling/overlap on drum hits in your inversions, then:
- Convert the instrumental to OPUS (128 Kbps) with Foobar2000.
- Use a software like Audacity to amplify it to a peak amplitude zero DB (since apparently OPUS auto declips to floating levels?)
- Export it as a WAV at the original sample rate (since OPUS only supports 48 kHz, I actually tried resampling the instrumental, and original to 48 kHz before converting to OPUS, but found that results in a WORST output.)
- Do your inversion (hopefully in Utagoe)
- Use whichever vocal model you like the output for best for cleanup.” - sausum
PS. Once, I ran into a similar issue. And I fixed it, actually similarly. I think I was trying to invert with an mp3 VBR, and the other file was lossless, so I converted it to the same codec and bitrate/V preset.
Yes, it wasn't perfect, but better.
I wonder if simply applying a cutoff at 20kHz wouldn't be a better solution. That's what Opus more or less does, plus upsampling to 48kHz.
- Despite the fact that separation is in 32 bit float, align inputs option in UVR uses something lower internally, hence the clipping may occur.
- As a better alternative to Utagoe and UVR’s Align feature, you can use paid Auto Align Post 2 (maybe even cheaper MAutoAlign).