Medley Vox (different vocalists)

Page 160 of 228 · Open the original GDoc ↗

Usually outperformed by newer models here (which are also not narrowband).
Use already separated vocals as input (e.g. by Roformers, vox_ft or MDX23C fullband a.k.a. 1648 in UVR or 1666 on MVSEP).

Local installation video tutorial by Bas Curtiz:

https://youtu.be/VbM4qp0VP8
(NVIDIA GPU acceleration supported, or perhaps CPU - might be slow)

Cyrus version of MedleyVox Colab with chunking introduced, so you don't need to do chunking manually:

https://colab.research.google.com/drive/10x8mkZmpqiu-oKAd8oBv_GSnZNKfa8r2?usp=sharing (07.02.25 fork with fairseq fix and GDrive integration)

Currently, we have a duet/unison model 238 (default in Colab),

and main/rest 138 to uncomment in Colab.

Recommended model is located in vocals 238 folder (non ISR-net one).

While:

“The ISR_net is basically just a different type of model that attempts to make audio super resolution and then separate it. I only trained it because that's what the paper's author did, but it gives worse results than just the normal fine-tuned.”

MedleyVox is also available on MVSEP, but it has more bleeding and “doesn't work as well as the Colab iteration with duets”. (Isling/Ryanz)

The "duet/unison model 238" will be used by default.

``and main/rest 138 to uncomment in Colab`` if you need it.

Then go to the first cell again. To "uncomment" means to delete the "#" from the beginning of the line before the "!wget" so the line will be used to download the model files.

Do it for both pth and json lines

(you might be asked whether to replace existing pth and json files by the alternative model you just downloaded in the place of the previous one)

``Recommended model is located in vocals 238 folder (non ISR-net one).``

That's the model used in the Colab by default. You can ignore that information. It's for users using the MV on their own machine.

The output for 238 model is 24kHz sample rate (so 12kHz model in Spek).

You might want to upscale the results using e.g. AudioSR or maybe even Lew’s vocal enhancer location further below the linked section.

The output is mono.

You might want to create a "fake stereo" as input by copying the same channel over the two, then do the same with another channel, and then create the stereo result from both channels processed separately in dual mono with MV.

The AI will create a downmix from both input channels instead of processing channels separately.

Be aware that “dual mono processing with AI can often create incoherencies in stereo image (like the voice will be recognized in some part only in left channel and not the other, as they are processed independently)” jarredou

"The demos sound quite good (separating different voices, including harmonies or background [backing] vocals)"

It's for already separated or original acapellas.

The model is trained by Cyrus. The problem is, it was trained with 12kHz cutoff… “audiosr does almost perfect job [with upscaling it] already, but the hugging page doesn’t work with full songs, it runs out of memory pretty fast”.

It was possible at some point that later stages of the training, looking like over fitting were responsible for higher frequency output.

It’s sometimes already better than BVE models, and the model has already similar to demo results on their site.

Sadly, the training code is extremely messy and broken, but a fork by Cyrus with instructions is planned, with releasing datasets including the one behind geo-lock. Datasets are huge and heavy.

Original repo (Vinctekan fixed it - the video at the top contains it)

https://github.com/jeonchangbin49/medleyvox

____

Outdated

https://colab.research.google.com/drive/1StFd0QVZcv3Kn4V-DXeppMk8Zcbr5u5s?usp=sharing (pip issues fixed 29.08.24, defunct as of 06.02.25)

(outdated instructions, current Colab explains everything)

“Run the 1st cell, upload song to folder infer_file, run the 2nd cell, get results from folder results = profit”

Further explanations how to use the Colab:
``Run the 1st cell``

So press the first "play" button then you load the Colab

``upload song to folder infer_file``

Looks like the folder for the input file has changed from infer_file to input in newer Colabs.

So, once you started the first cell, and it finished, open Colab file manager (folder icon on the left) and go to /content/MedleyVox/input\

Now paste your song there and wait till it's done.

``run the 2nd cell``

So the next play button below the first one once you scroll down a bit. Now it will start separation

``get results from folder results``

Go to file manager again and find /content/MedleyVox/results

right-click on the result file and download it. Wait till it's done.

``Currently, we have a duet/unison model 238 (default in Colab)``

So you don't have to change anything in the Colab to separate using it.

Old info

https://media.discordapp.net/attachments/900904142669754399/1050444866464784384/Screenshot_81.jpg (dead)

Colab old

https://colab.research.google.com/drive/17G3BPOPBPcwQdXwFiJGo0pKrz-kZ4SdU

Older Colab

https://colab.research.google.com/drive/1EHJFBSDd5QJH1FQV7z0pbDRvz8yXQvhk

(The same one, but here you need to change the .ckpt, .json and .pth files there from Cyrus [more details in the video above].)

____________________________________________________________________