What are BigShifts?
It's based on Demucs' shift trick, but for Demucs it is limited to 0.5 second shifting max (with a randomly chosen value).
Each BigShifts here shifts the audio by 1 second, no more random shifting.
F.e. bigshifts=2, it will do 1 pass with 0 shifting, and a second pass with 1 second shifting, then merge the results
bigshifts=3 means 1 pass with 0 shifting + 1 pass with 1 sec shift + 1 pass with 2 sec shift, etc...
Overlap is doing almost the same thing but at audio chunks level, instead of full audio, and the way overlap is implemented (in MVSEP-MDX23), f.e. with overlap=0.99, the first audio chunk will have 1 pass, 2nd audio chunk will have 2 passes, etc... until the 99th audio chunk and following ones will have 99 passes. With BigShifts, the whole audio is processed with the same number of passes.
So BigShifts shifts the audio forward one second each time.
Overlap computing is different between MDXv2 models and the other ones in the fork:
For MDXv2 models (like VOC-FT), it uses the new code from UVR and goes from 0 to 0.99.
For MDXv3 (InstVoc) & VitLarge models [introduced in v2.3] it uses code from ZFTurbo (based on MDXv3 code from KUIELab, https://arxiv.org/abs/2306.09382) and it goes from 1 to whatever.
I'm using low overlap values in the fork because it's kinda redundant with the BigShifts experimental feature I've added and which is based on Demucs' "shift trick" (described here https://arxiv.org/pdf/1911.13254.pdf, chapter 4.4). But instead of doing shifts between 0 and 0.5 sec like Demucs by adding silence before input, BigShifts are much larger (and related to input length). Having larger time shifting gives more amplitude in possible results.
Instead of adding silence before input to shift it, which would be a waste of time & resources as BigShifts can be above 30s or 1 min of shifting, instead, it changes the shifted part position in audio input (like move the 1st minutes of audio at the end of the file before processing and restores it after processing).
Then like Demucs original trick all shifted & restored results are merged together and averaged.
From my tests, it can influence results from -2 SDR to +2 SDR for each shifted result, depending on input and BigShifts value. It's not linear!
Using BigShifts=1 (disabled) and high overlap value probably gives more stable results, on the other end, but maybe not always as high and/or fast as what BigShifts can give.
Weights have been indeed evaluated on MVSep's multisong dataset. I haven't tried every possible setting, but default values should be not far away from optimal settings, if not optimal [already].
Q: Wasn't the BigShifts trick in the MDX23 Colab relying on a slowed-down and sped-up separation ensembling?
I think increasing the parameter too much, rather tends to increase bleeding.
A: It's unrelated to bigshifts, but it was doing that for MDX2 models with a cutoff around 16-17khz (to get fullband results from them) but since it's using only fullband models, I've removed that part (in v2.2 iirc)
There are a few other "tricks" used in the [MDX23 Colab] fork:
The phase inversion denoise trick (was already in original code from ZFTurbo, also used in UVR):
Some archs (MDXv2 mostly, so VOC-FT here) are adding noise to output signals. So to attenuate it, we process the input 2 times, including one time with phase polarity inverted before processing, and restored after processing. So, only the model noise is phase cancelled when the 2 passes are mixed together. (It doesn't cancel 100%, but it's attenuated). This is also applied to Demucs processing (since original code).
MDXv3 & VitLarge don't seem to add noise (or at insignificant volume) so this trick is not used with these models.
Segment_size (dim_t) original model value has doubled since v2.3 of the fork.
Some benchmarks done by Bas Curtiz showed that it gives a little bit better results (here with VOC-FT, there's the same benchmark with the InstVocHQ model here).
Multiband ensembling:
I'm using a 2-band ensemble, with different ensemble in frequencies below 10 kHz and above. This is a workaround to get fullband final results even when not fullband models are part of the ensemble (like VOC-FT). Without it, the instrumental stem, obtained by vocals’ phase inversion against the input audio, would have small permanent vocals bleeding above VOC-FT's cutoff, as phase cancellation would be biased there.
It was a really more essential feature in previous versions when most of the models were not fullband.
VitLarge is not used too in high freq band, but it's a more personal taste (so in the end there's only InstVoc model results above the crossover region)
In fact, alternatively you could separate your instrumental with Demucs single models used by the Colab (demucs_ft, demucs 6s, demucs, mmi) and use SCNet XL and BS-Roformer models from here.
As, along with demucs_ft, they have the best overall SDR for 4 stems separation (actually MDX23C model1 can give interesting results too compared to demucs_ft).
And then perform manual weighted ensemble in DAW by setting volume of the stems manually to your liking after importing and aligning lossless stems.
Because rarely ensembling of more than 4 stems gives good results, IG you could get rid of some demucs models with lower SDR for it (I think the mmi has the lowest SDR, and then 6s).
If it's too much of a hassle, you could change the volume of the stems from a specific model by the same volume.
Guide how to use MDX23 Colab v 2.5
(reworked Infisrael text)
0. If you plan to use your GDrive for input files, go there now, and create folder called “input” and upload your files there. Create also output folder (not sure if the Colab creates both already). That way you may decrease the time till timeout when the Colab is initialized (esp. for people with slower connection).
Now open the Colab
1. Click the “play” button on the Installation cell and wait until it's finished (should show a green checkmark on the side)
2. Click the “play” button on the GDrive cell.
It will ask you for permission for this notebook to access your Google Drive files, you can either accept or deny it (it is recommended to accept it if you want to use Google Drive as i/o for your files).
After you've done installing it, go to the Separation section below.
Default settings are already balanced in terms of SDR, and too resource-intensive.
3. Click on the play button to start the Separation, **make sure** you uploaded the audio file in the `folder_path`.
After it's done, it will output the stems in the `output_folder`.
Also note, "`filename_instrum`" is the inversion of the separated vocals stems against the original audio.
"`filename_instrum2`" when “Separation_mode:” is set to 4 stems (slower) is the sum of the Drums + Bass + Other stems that are obtained by processing "`instrum`" with multiple Demucs models.
So "`instrum`" is the most untouched and "`instrum2`" can have fewer vocals residues or sound a bit muddier.
Experimenting on settings you can set BigShifts to 5 or 7, although it may not give a noticeable difference vs default 3, while increasing separation time severely, but some people use 20 or even 30.
Comparisons of MDX23 (probably v. 2.0) vs single demucs_ft model by A5
The Beatles - She Loves You - 2009 Remaster (24-bit - 44.1kHz)
So I tried out the MDX23 Colab with She Loves You, which is easily the most ratty sounding of all the Beatles recordings, as it is pure mono and the current master was derived from a clean vinyl copy of the single circa 1980. So if it can handle that, it can handle anything. And well, MDX23 is very nice, certainly on par with htdemucs_ft, and maybe even better. I'm surprised. You can hear the air around the drums. Something that is relatively rare with demucs. And the bass is solid, some bleed but the tone and the air, the plucking etc is all there. Plus, the vocals are nicer, less drift into the 'other' stem.
John Lennon - Now and Then (Demo) - Source unknown (16-bit - 44.1kHz)
OK, another test, this time on a John Lennon demo, Now and Then. The vocals are solid, MDX23 at 0.95 overlap is catching vocals that were previously in htdemucs_ft being lost to the piano. So, yeah, it's pretty good. MDX23 is now my favored model. In fact, upon listening to the vocals, it's picking up, from a demo, from a poor recording, on a compact cassette, lip smacks, breathing and other little non-singing quirks. It's like literally going back and having John record in multitrack.
Queen - Innuendo - CD Edition TOCP-6480 (16-bit 44.1kHz)
Every single model fell down with Freddie Mercury's vocals, not anymore. (...) I've heard true vocal stems from his vocals and the MDX23 separation sounds essentially like that. We're now approaching the 'transparent' era of audio extraction.
NOTE: [voc_ft not tested] for Innuendo, will be tested by 07/07/2023
Colab instruction by Infisrael for old versions
Install it, click on the play button and wait until it's finished (should show a green checkmark in the side).
It will ask you for permission for this notebook to access your Google Drive files, you can either accept or deny it (it is recommended to accept it if you want to use Google Drive as i/o for your files).
After you've done installing it, go to the configuration, it's below the 'Separation' tab.
https://i.imgur.com/qD9jsYG.png (dead)
(Recommended settings)
Input "`overlap_large`" & "`overlap_slow`" with what you desire, at the highest (1.0), it will process slower but will give you a better quality. The default values for large are (0.6), and for small (0.5) [with 0.8 still being balanced in terms of speed and quality].
Input "`folder_path`" with the folder destination where you have uploaded the audio file you'd like to separate
Input "`output_folder`" with the folder you'd like the stems to be separated
Change your desired path after `/content/drive/MyDrive/`, so for example:
> `folder_path: /content/drive/MyDrive/input`
> `output_folder: /content/drive/MyDrive/output`
You can also make use of "`chunk_size`" and put it in a higher value by a little, but if you experience memory issues, lower it, default value for it is 500000.
Afterwards, click on the play button to start the separation, **make sure** you uploaded the audio file in the `folder_path` you provided.
After it's done, it will output the stems in the `output_folder`.
Also note, "`filename_instrum`" is the inversion of the separated vocals stems against the original audio.
"`filename_instrum2`" is the sum of the Drums + Bass + Other stems that are obtained by processing "`instrum`" with multiple Demucs models.
So "`instrum`" is the most untouched and "`instrum2`" can have fewer vocals residues.
Installing MDX23 Colab locally
NVIDIA 12GB VRAM GPU recommended
"I think it's possible to use Colab notebook .ipynb files locally with anaconda and jupyter, but I've never tried.”
“I didn't get it to work yet but a simple tkinter gui should be easy to throw together I reckon”
Now you can use ready MDX23 WebUI fork by RedsAnalysis.
[Alternatively]
“You can git clone the repo, install requirements and use the inference.py script, but the command line can be really long to type manually (on Colab it's managed with the GUI):
python inference.py \
--input_audio "{file_path}" \
--large_gpu \
--BSRoformer_model {BSRoformer_model} \
--weight_BSRoformer {weight_BSRoformer} \
--weight_Kim_MelRoformer {weight_Kim_MelRoformer} \
--weight_InstVoc {weight_InstVoc} \
--weight_InstHQ4 {weight_InstHQ4} \
--weight_VOCFT {weight_VOCFT} \
--weight_VitLarge {weight_VitLarge} \
--overlap_demucs {overlap_demucs} \
--overlap_VOCFT {overlap_VOCFT} \
--overlap_InstHQ4 {overlap_InstHQ4} \
--output_format {output_format} \
--BigShifts {BigShifts} \
--output_folder "{output_folder}" \
--input_gain {input_gain} \
{filter_vocals} \
{restore_gain} \
{vocals_only} \
{use_VitLarge_} \
{use_VOCFT_} \
{use_InstHQ4_} \
{use_InstVoc_} \
{use_BSRoformer_} \
{use_Kim_MelRoformer_}
Q: How do you use the example {useVitLarge}
like the other stuff ik how to use
A: These last arguments are boolean based, there are generated before the command line and depending on the option selected in the GUI with:
use_InstVoc_ = '--use_InstVoc' #forced use
use_BSRoformer_ = '--use_BSRoformer' #forced use
use_Kim_MelRoformer_ = '--use_Kim_MelRoformer' #forced use
use_VOCFT_ = '--use_VOCFT' if use_VOCFT is True else ''
use_VitLarge_ = '--use_VitLarge' if use_VitLarge is True else ''
use_InstHQ4_ = '--use_InstHQ4' if use_InstHQ4 is True else ''
restore_gain = '--restore_gain' if restore_gain_after_separation is True else ''
vocals_only = '--vocals_only' if Separation_mode == 'Vocals/Instrumental' else ''
filter_vocals = '--filter_vocals' if filter_vocals_below_50hz is True else ''
Q: So you don't need to use them?
Only using the ones with the two -- before right
A: For example, if you want to activate vocals filtering below 50hz, you add "--filter_vocals" to the command line
Q: How do you do this
A: (click)
Q: oh yeah I just have to change the default number then right
It works #general
A: If you have multiple GPUs and the CUDA one is not labelled device "0", maybe that can be the cause too, it's hardcoded for Colab, but you can change it in first lines of inference.py file gpu_use = "0"
If your GPU is not detected in Anaconda, use Python (can be 3.12). If it's the same:
https://pytorch.org/get-started/locally/#start-locally
Where it says "run this command" I basically uninstalled the modules it had in there
so I did pip uninstall torch torchvision torchaudio
then ran that command to install it
and it fucking fixed it (knock)
1.0 original code used kim vocal 1 (later 2), kim inst and (at least for 4 stems) Demucs models.