Anjok’s interview on YT

 

TL;DW: UVR’s documentary + training, archs and demudder explained

 

Anjok is the developer of Ultimate Vocal Remover 5 (UVR5 GUI).

He intended UVR to be a Swiss army tool - to contain everything you need for separation, and also contain models made by the community (e.g. dereverb/denoise/deecho).

 

History of UVR

 

Anjok in times where Spleeter was still a thing, found a VR arch made by Japanese developer, tsurumeso, and received better results than Spleeter. He started to make his own model on laptop 1060 6GB on 100 or 150 pairs with the absolute minium parameters, and it turned out to be a better model than tsurumeso's one. Later he transitioned to faster GPU [probably before 3090 yet].

Anjok wanted to create a GUI for VR, and found BoskanDilan on Fiver and simply contracted him, paying him to build the foundations of what UVR is today. BoskanDilan turned out to be a very good and talented coder.

They put the work on GitHub, and Aufr33 contacted Anjok with ideas on how the VR models can be improved etc.

 

Then BoskanDilan left in mid 2021 for personal reasons. Then the GUI work was taken by Anjok who was mentored by BoskanDilan to improve on understanding the coding. Anjok started working on UVR exclusively, spending 10 hours a day for UVR in 2022.

He decided to make a simple installer in one package, as he received lots of issues on GutHub from people not knowing how to install it. He also re-coded the UVR to make the code easier to maintain. Then Bas Curtiz helped Anjok on design aspects of UVR, e.g. designed the new logo, and gave some advice, and good amount of feedback from UVR user perspective. The early 2022 phase of UVR development took a lot of advice from early users of UVR.

In May 2022 there was a first installer released to make UVR more accessible without e.g. installing Python or other dependencies and specialized programming knowledge to set up a proper environment.

 

Anjok was still in charge of introducing other archs than VR into UVR, being simply the only person behind the process, while normally bigger teams work on projects of that scale, when ideally  e.g. different archs would be coded into UVR by different developers. It was a stressful period of time, because Anjok intended to make the software which is also free of bugs, and still not fully rely on the community in terms of bug reporting.

 

Then the Mac version came out and M1/M2/M3/M4 support for faster GPU acceleration. Anjok found out a part of the code in the Demucs repo, which made it easier to port UVR to Macs, and it is used by every model. The music community is pretty Mac-centered, and he devoted a considerable amount of time to make it work reliably on Macs too.

 

In the new UVR version there's a planned demudder to be introduced (described later; already done), and possibly translations.

Anjok currently trains a new model coming in several weeks (done - HQ_4).

It's intended to be a little smaller in order to be not so resource intensive, but also better than the best current MDX-Net model.

Update 01.03.24

“I'm going to allow HQ4 to continue training beyond 1500+ epochs as an experiment (it's currently at 1200), and interestingly, the SDR has been steadily increasing. It has significantly surpassed HQ3 in terms of SDR and listening tests, and it also outperformed MDXC23 in listening tests, though not in SDR (yet!). The most recent evaluation on the multi-dataset showed a score of 15.85, using the default settings. Clearly, there's a limit to how much further training can enhance performance, but up to this point, improvements are still being observed. This model has been in training since October! I'm chipping away at the next GUI update as well, and the demudder will be in it.”

The model was released, with already HQ_5 scheduled in the following month/s (done).

 

 

The archs in UVR and their technicalities summarized

 

VR

 

VR uses audio spectrograms and converts them to FFT spectrograms.

VR uses only magnitude spectrograms, not phase.

Phase represents timing where the data is, while magnitude represents the intensity of each frequency.

Phase is much harder to predict.

Actually VR uses the original phase from the mixture and saves it during the process "and it just does the magnitude".

That's the reason why VR tends to have more artefacts in it. The smearing in instrumentals of VR is because the phase from the mixture is still in there.

 

Aufr33 later introduced 4 bands support for UVR.

Let's say for the first of three bands between 0-700Hz there will be different resolution, for all other frequency ranges there will be different. E.g. knowing that vocals are in a specific frequency range, you can optimize it further.

That feature made UVR and VR arch much better.

 

Later they introduced -

Ensembling

So a way to use multiple models to potentially get better results.

The three ways of ensembling:

avg - gets the average of vocals/instrumentals

max - is maximum result of each stem, e.g. in a vocal you'll get the heaviest weighted vocal from each model, and the same goes for instrumental, giving a bit cleaner results, but more artefacts

min

 

MDX-Net

 

Uses full spectrogram with phase and magnitude

The tradeoff is muddier results, but natural, cleaner sound.

 

Training

 

Anjok separated on nearly every genre you can think of, and stated that the hardest genre for separation is metal and vocal-centered mixes. Also, if the instrumental has a lot of noise, e.g. distorted guitars, the instrumental will come out muddier.

MDX-Net was the arch, addressing lots of VR issues in its core.

Tracks from the 70-80s can separate well. 50-60s will be harder, e.g. recorded in mono. The early stereo era gets a little better.

 

A good model needs to be as good as the dataset for a model.

There was lots of work scrapping it from the internet.

Aufr33 was the mastermind behind the Karaoke model and its dataset.

Demucs model wasn't as successful, as probably was more meant for more stems, and MDX-Net gave better results for 2 stems.

 

[Training details covered in this interview can be found at the top of Traning models guide section of the doc]

 

The biggest issue in terms of archs and the source of muddiness, is phase. Currently, in audio separation there's not a great way to calculate phase in a model like the phase spectrogram as it's not as obvious as the magnitude spectrogram.

 

You take the vocal out of a heavy rock track, but the process is not perfect, so it will take some part of the instrumental with it. Even if you don't hear instrumental in vocals, there's still instrumental data in there in the phase of that vocal track.

At the end of the day, source separation is prediction. It's predicting where it thinks it is, but there will always be some imperfections, e.g. whenever you hear muddier sound in a track which has more noise like metal tracks.

 

Anjok emphasizes the (current) lack of correlation between SDR and the fact that a bigger SDR metric doesn’t necessarily mean better. He tried some top of the SDR chart results before, and wasn’t quite happy about them around 2024 or before.

 

Because phase is a big part of the issue, now the new upcoming -