Mateus Contini's methods

Page 96 of 228 · Open the original GDoc ↗

#1 (old)

-“TIP! For busy songs: I was testing some ensembles trying to get Instrumental Stems with less volume variation (muddy), preserving guitar solos, pads the most and I had great results doing the following, for anyone interested:

Ensemble (Demucs + 5_HP-Karaoke with Max for Instrumental stem) - The result will be the Instruments + Backing Vocals and this preserves most of the guitar solos, pads and things that MDX struggles.

Instrumental Stem Output > Demucs to remove the Backing Vocals from the track - This pass will remove the rest of the Vocals. In some cases will be some minor leftovers that you can clean later with other methods.

I find the results better than Demucs alone/ MDX models or other ensembles for what I'm looking for. I'm not evaluating noise, but fuller instrumental Stems, trying to preserve most of it and also the cost (time) to do it.

Since I'm not interested, for this case, in doing manual work song by song and just use these stems to sing over it, I find the results great.” - Mateus Contini

Q: Do you mean that you process Demucs 2 times? Once for ensemble with VR then the result was processed using Demucs again?

A: You can add other models with the ensemble, like Demucs, VR_5-Karaoke and HQ3 for an extra, before processing again with Demucs.

Also, this method is very good for leave good backing vocals into the instrumentals (only the ensemble result). I find extracting bv from the Vocal Stem to be less effective, giving you less material (comparing if you would join the bv with instrumentals later)

M.Contini Method #2 (newer)

Well, I tried to improve the results of the method I posted, so here it is, for **anyone interested in get fuller Instrumentals**, with a bit of bleed in some songs, wielding great results overall.

I'm doing this in the UVR-gui. The idea behind it is to scoop the vocals little by little, so the instrumentals is preserved the most. The proccess requires 3 extractions. Here are the Ensembles:

1. pass Ensemble: 5_HP-Karaoke-UVR + Inst HQ3 + htdemucs - Min/ Max

- If the song doesn't have BV, this will already give you good Instrumental Stem results. If you have Vocals bleeding into the Instr, continue to pass 2, but sometimes jumping straight to pass3 will produce better results.

- If the song have BV, this you keep a fuller **Instrumental Stem with BV** in it. If you want to keep the BV, but there is some Main Vocals bleeding through the Instr, continue to pass 2.

2. pass Ensemble: Kim Vocal 2 + Inst HQ3 + MDX Karaoke 2 - Min/Max

- This pass will try to preserve the BV in the Instrumental Stem while removing Main Vocal bleed. You can stop here if you want the **Instrumental Stem with BV**

3. pass Ensemble: Kim Inst + Inst HQ3 + htdemucs - Min/Max

- This pass will try to remove BV from the instrumental Stem and other Main Vocal Bleed while keep the Instrumental fuller.

The idea behind it, is to have less volume variation where the vocals are extracted, leaving the Instrumental Stem less muddy. Since the extraction of the vocals is done little by little using the Min/Max, the Models will not be so aggressive. This is a great starting point if you want to improve further in a DAW or just sing over it. The Con is that, sometimes, the track will have tiny bleeds. If you try this method, please post the results here.

- -try this ensemble: 9_HP (10 agression) + HQ3 (chunks on) + demucs_ft, Min/Max

- it preserves most of the instruments.

M. Contini method #4 (new)

Another Ensemble suggestion for good instrumentals with minimized bleeding vocals and a bit of noise in some cases:

Ensemble: 9_HP + HQ3 + Demucs_6s (secondary model 50%: full_292) - Algorithm [min/max]

Configs:

9_HP Window[512], Agress[10], TTA[on], Post[off], High-End [off])

HQ3 Chunks[on] [auto], Denoise[on], Spectral[off]

Demucs_6s Chunks[on] [auto], Split[off], Combine[off], Spectral[off], Mixer[off], Secondary Model - Vocals/Instr [MDX-Inst_full_292] [50%]

Why Demucs_6s and not _ft - I compare them in some songs and 6s have less vocal bleed in the instrumental track.

Description:

The idea is to take the good bits of the models using only one from each Group (VR, MDX and Demucs). The secondary model on Demucs is to minimize some vocal bleeding with sustained notes that was happening in some songs.

Comparing the results from multiple models, I find that Chunks enabled on MDX and Demucs removes some bleeding vocals from the Instrumental track and gives better results overall. This ensemble in my machine completes in about 5 min per song (GTX 1070 8GB, 16GB RAM, Ryzen 1600x). [chunks have been replaced by newer method in newer UVR GUI versions]

____________

- “The best combo is the HQ instrument models ensemble average/average including HQ3/Main/Main Inst/Kim1/2/Kim Inst/demucs3 (mdx_extra)/htdemucs_ft/hdtdemucs6s” (MohammedMehdiTBER)

"Wow, I tried out the ensemble with all those models you said, and it actually sounds pretty good. There's a definitely more vocal bleed but in a saturated/detailed distortion type of way. I can't tell which one I like better, the ensemble sounds more full and has more detailed frequencies, but the vocal bleed is a lot more obvious. The HQ_3 by itself has almost no vocal bleed but sounds more thin and watery."

- Kim instr + mdx net instr3 + HQ2 + HQ3 + voc ft max/max

The result is so amazing… Now can hear more detail on instrumental result where before I cannot hear a bit of music parts. (Henry)

- "I am very much enjoying making an ensemble of HQ3 and MDX23C_D1581, then inverting the vocals into the instrumental and running that through hq3 with 0.5 overlap" (Rosé)

__________________________________

Ensembles for specific genres

Evaluation based on public models available at 23.04.23 and multisong dataset on MVSEP. The list might be outdated, as it doesn’t take all the current models into account.

SDR sorted by genre

By Bas Curtiz

"If we remove **Kim vocal 2**, so only those that are available right now will be taken into account:

- Ensemble Rating 1 scores highest on average overall

[Probably this one:

Kim vocal 2 + Kim FT other + Inst Main + 406 + 427 + htdemucs_ft | Avg/Avg

At least it was the best for the given date.

But now we have ensembles which score better.]

- Kim vocal 1 is best for Rock

- Kim vocal 1 & Ensemble Rating 1 are best for RnB/Latin/Soul/Funk

- MDX'23 Best Model is best for Pop

- Main 427 & MDX'23 Best Model are best for Other

- Main 427 & MDX'23 Best Model are best for Blues/Country

- Main 427 & Ensemble Rating 1 are best for Jazz

- Main 427 & Ensemble Rating 1 are best for Acoustic genres

- Ensemble Rating 1 is best for Beats

- Ensemble Rating 1 is best for Hip Hop

- Ensemble Rating 1 is best for House

Sheet where **Kim vocal 2 **is removed:

https://docs.google.com/spreadsheets/d/1ceXA7XKmECwnsQvs7a0S81XZOUokIXUN8ndsUDcYRcc/edit?usp=sharing"

Further single MDX-UVR models descriptions

E.g. used for ensembles above, but if a model has a cutoff, using ensemble with models/AIs without cutoff like Demucs 2-4 will fill the gap above. But it's still a good alternative for people without decent Nvidia GPUs or are force to use Colab.

UVR-MDX models naming scheme

All models called "main" are vocal models.

All models called "inst" and "inst main" are instrumentals.

NET-X [9.X/9.XXX in Colab] are vocal models

Kim vocal 1/2 (self-explanatory)

Inst main is 496

Kim other ft is Kim inst

Model labelled as just ‘main’ is vocal, and was reported to have the same checksums as 427 and 423, but it doesn't seem to be true as 427 and main have different SDR (427 has better SDR than main, so apparently main is 423 [CRC32: E3C998A6]).

- MDX HQ_1/2 models - excellent, vivid snares, no cutoff (22kHz) high quality, rarely worse results than narrowband inst1-3 models, HQ_2 might have slightly less loud snares, but can have fewer problems with removing some vocals from instrumentals

- MDX-UVR Inst 3 model (464) - 17.7 cutoff (the same cutoff as for Inst 1, 2 inst main, but maybe not applicable for vocals after inversion in Colab), it was the third-best single model in our SDR chart at the time, available in Colab update and UVR5 GUI with VIP models package - now available for free.

- Forth-best single model for instrumentals back then was inst main (496, MDX 2.1), then inst 1 and inst2.

- There was some confusion about MDX 2.1 model (iirc on x-minus) being vocal 438 (even 411), but it’s currently inst main.

- Full band MDX-Net models without cutoff (better SDR than Demucs 4 ft)

As for SDR, the epochs score is following: 292<403<386<(inst 1)<338<382<309<337

<450 (first final, HQ_1)<498 (HQ_2)<(inst3)<(Kim inst)<HQ_3<HQ_4

Epochs 292, 403 and 450 and newer are also in Colab (and in UVR5, older when VIP code is redeemed)

- (currently the best, maybe not single model, but custom ensemble, as for vocals) MDX23 in MVSEP beta,

and in UVR5 - Kim vocal model -

It's a further trained MDX-UVR vocal model from their last epoch (probably UVR-MDX-NET Main). It's based on a higher n_fft scale which uses more resources.

Not always gives that good results for instrumental as SDR may suggest, and also more people shares that opinion [both Colab and UVR users, so i’ts not due to no cutoff in Colab]).

In UVR5 generally for the best vocal result use vocal models, and for the best instrumental result use instrumental models or eventually 4 stem Demucs 4 ft.

"[Kim_Vocal_1] is an older model (November), than Kim uploaded at 2022-12-04 to" https://mvsep.com/quality_checker/leaderboard.php?sort=insrum&ensemble=0

(steps below no longer necessary, the model is added to GUI and these are the same models)

You can download her (so-called “old”) model from here (it still gets better results for vocals than inst 3 and main): https://drive.google.com/drive/folders/1exdP1CkpYHUuKsaz-gApS-0O1EtB0S82?usp=sharing

When you copy/paste the model in `C:\Users\YOURUSERNAME\AppData\Local\Programs\Ultimate Vocal Remover\models\MDX_Net_Models` It asks you to configure, hit Yes.

Then change `n_fft to 7680`."

For instrumentals, it gets worse results, frequently with more bleeding, and UVR manually applies cutoff above training frequency to instrumentals after inversion, to avoid some noise and possibly bleeding. Colab version of Kim model doesn’t have that cutoff, so instrumentals as a result of inversion have max 22kHz frequency (but UVR applies it to prevent some noise).

- (generally outperformed by models above) MDX-UVR 9.7 vocal model a.k.a. UVR-MDX-NET 1 (instrumental is done by inversion, older model) - available in Google Colab/mvsep (here 24 bit for instrumentals)/UVR5 GUI.

Compared to 9.682 NET 2 model, it might have better results on vocals, where 9.682 NET might have better results for instrumentals, but everything might still depend on a song. Generally, 9.7 model got better SDR both in Sony Demixing Challenge and on MVSEP. Generally, 438 vocal, or 464 inst_3 should give better results for instrumentals. 427 vocal model tends to give worse results for instrumentals than even this older 9.7/NET1 model.

More about MDX-UVR models -

If they don't have more vocal bleeding than GSEP, they’re better in filtering more vocal leftovers which sometimes GSEP tend to leave (scratches, additional vocal sounding sounds, also so-called “cuts” [short multiple lo-fi vocal parts] which GSEP doesn’t catch, but MDX-UVR does probably due to bigger dataset). But using single instrumental MDX-UVR models instead of ensemble will result in cut off of a training frequency (e.g. 17.7kHz or lower).

Also, MDX-UVR like GSEP may not have this weird constant "fuzz" which VR models tend to leave as vocal leftovers (but in other cases, 9.7 model can leave very audible vocal residues, so test out everything on this list, till you get the best result).

The 9.7 model (or currently newer models) is also good for cleaning inverts (e.g. when having lossy a cappella and regular song).

If you tested all the alternatives, and you stick to the MDX-UVR 9.7 for some song, and it doesn't have (too much) bleeding, to fine-tune the results you can try out two 9.6 models to check whether it's better for you than 9.7 in this specific case (they're available at least in HV Colab and UVR5 GUI).

Newer MDX-UVR 423 vocal model usually provides more audible leftovers than 9.7 model.

To further experiment with MDX-UVR results, and you’re stuck with Colab, you can enable Demucs 2 model on Colab to "ensemble" it with MDX-UVR model (although metrics say it slightly decreases SDR, I like what it does in hi-end - it was suspected at some point the SDR decreasing problems may come out from enabling chunking).

________________

- Demucs 4 (htdemucs_ft) - no cutoff, it’s 4 stem, but you can perform mixdown without vocals in Audacity for instrumental - sometimes it may give you louder snare than in GSEP, but usually muffled shakers compared to GSEP. Also, it will give you more vocal residues than GSEP and MDX-UVR 464 (Inst 3). 6 stem models gives more vocal residues than 4 stem model (ft is the best one and also outperformed mdx_extra model [better than mdx_extra_q - quantized) but in some cases that might be worth to check old mdx_extra model as well (but

- (outperformed in many cases when used at least as a single models)

VR-architecture models (Colab, CLI or UVR5 GUI) sometimes provide cleaner and less muddy results for instrumentals than single narrowband models of MDX or even GSEP, only if they do not output too much vocal bleeding (which really happens for VR models frequently - especially for heavily processed vocals in contemporary music), but bleeding also depends on specific model:

- E.g. 500m_1 (9_HP2-UVR) and MSB2 (7_HP2-UVR) models are the most aggressive in filtering vocals among VR models, but other, less aggressive VR models may provide better sounding, less spoiled instrumentals (only if it is not paid for with worse vocal bleeding [BTW. I haven’t heard the newest 2022 VR model yet (available at least in UVR5 GUI, maybe for Patreons, not sure]).

All parameters and settings corresponding to specific models you’ll find in “VR architecture models settings” section.

- VR models-only ensemble settings - if your track doesn’t have too many problems with bleeding using VR-models above, to fine-tune the results achieved with VR, and to get rid of some mud, and e.g. get better sounding drums in the mix, I generally recommend VR-architecture models ensemble with settings I described in the linked section above.

I'd say it's pretty universal, though the most time/resource-consuming method.

Also, these ensemble settings from the UVR HV Colab seem to make decent job for extracting vocals in some cases when above solutions failed (e.g. claps leftovers).

Check also demucs_6s with 9 HP UVR and gsep in min-specs mode

Also, UVR5 GUI has rewritten MDX, so it can use their Demucs-UVR models from Demucs 3 (I think mvsep doesn't provide ensembling for any MDX models):

- (generally outperformed by MDX-UVR 4xx models) Demucs-UVR models - 1 and 2 models beside "bag" are worth trying out (mainly 1) on their own if the results achieved with above methods still provide too much bleeding - better results than e.g. bare MDX-UVR 9.7 or VR models or even GSEP in some specific cases (available on MVSEP and UVR5 GUI). They're Demucs 3, 2 stem better trained models by UVR team. No cutoff - 22kHz.

_______________________________

- As for extracting -