Bleedness and fullness leaderboard
Python evaluation script by jarredou (prob. mirror), Torch version (with Bas Curtiz),used on Quality Checker
Librosa version added to ZFTurbo training repo.
More detailed and reliable method of evaluation on multisong dataset than the popular SDR metric.
(old) Bas Curtiz’ evaluation chart with some Roformers tested with that method:
https://docs.google.com/spreadsheets/d/1pPEJpu4tZjTkjPh_F5YjtIyHq8v0SxLnBydfUBUNlbI/edit?usp=sharing (shortened version - it’s outdated - all metrics are rewritten in the models sections above)
For some newer models not on the list, you could search here for the model name, and bleedless/fullness metrics for new models are now provided in the evaluation description when you click on the result, but plenty of model evaluations have names not corresponding to final model names and were shared along with models on our Discord and later pasted to this document above.
Also, sorting by specific metric on MVSEP was added in June 2025, so you can track the exact evaluation by provided metrics in this document that way, or by searching Discord, but it can be difficult, as links to some older models’ evaluations were not indexed by the metrics bot, so once the evaluation was posted, the bot wasn’t showing metrics from the beginning, and not all models were evaluated along with the model release.
Explanations on the metric
Spectrogram difference showcase diagram
“Blue is what is missing from a separated stem (compared to a clean source).
Red is bleed in a separated stem.
White is perfect
(dB scale on the right seems wrong, I haven't checked, but it's not really important to see what is going on).
Same formula [can] be used for a metric, which would theoretically measure bleedness and fullness of the evaluated models
I think that for a metric, it's better to then separate negative values of diff array on one side, and keep positive values on other side, and average/scale each of them separately, so we get 2 scores, 1 for bleedness and 1 for fullness.
It has to be experimented further (and with better stft, it's only working on single chunk currently)
(Not sure that so high n_fft/mel_bins values are really needed, it was just nicer on the plot with that)”
“bleedless/fullness metrics are stft magnitude-only based and as they are discarding the phase data, they have some kind of blind spots.” - jarredou
Random noise added to results can increase fullness metric:
https://mvsep.com/quality_checker/entry/7709
https://mvsep.com/quality_checker/entry/7708
“l1_freq, the simplest way to explain it - it’s a mix between fullness and bleedless but without the noise issue (in a sense it’s the real fullness/bleedless metric) (...)
there’s no universal metric still sadly, we have to rely on a combination of them (and our ears)” - becruily
Q: How can you really spot when someone trains on the valid set anyway?
A: When it scores very well on valid tracks but is shit on anything else - becruily
Reservations and alternatives
“-l1_freq = bleedless (higher is cleaner)
-aura_mrstft = fullness (higher is fuller)
they maybe don’t have the issues fullness and bleedless have but I haven’t played to check that” - becruily
“aura_mrstft - is more perceptually relevant than SDR for musical content imo” - gilliaan
“Fullness and bleedless are not very reliable.
In my opinion it’s l1_freq and aura_mrstft both measure fullness and bleedless at once better (l1_freq has the edge because it also can show if model has phase issues like some fullness models)” - becruily
Summary
(If it's still confusing, how I understand it, and correct me if I made a mistake somewhere in this conclusion)
L1_freq recognizes phase.
Fullness metric can be cheated by adding noise which can be fixed by replacing phase.
But then, the fullness metric won't even see it and won't change.
L1_freq was described as "real fullness" - so if a model has phase issues and noise, it won't score high as the fullness metric, but it can still show fullness, because in fact, it's a mix of fullness of bleedless - it's more perceptual metric and single one for both.
Why aura_mrstft = fullness, is because it's not a STFT metric. It's a time domain metric. And the latter is blind to phase. So it's closer to fullness metric in that regard. More on it explains unwa more technically later below.
Read for discussion
“[The] problem with bleedless/fullness metric is that you can easily increase them by multiplying stem on constant.
Multiply predictions by 0.97 - it increases fullness and reduces bleedless
Multiply predictions by 1.03 - it greatly increases bleedless and reduces fullness” - ZFTurbo (metrics/discussion)
Q: I don't understand what fullness models do differently that result in this bleeding that is typical of fullness models
A: I would say that fullness models are leaving a larger ring of fire*/shell of the untargeted stem (link)
(I guess that with a different n_fft resolution or number of bands, that bleed noise would sound also differently with Roformers iirc) - jarredou
Other metrics:
Log WMSE - good “at least for drums or anything rich in low frequency content” - jarredou
"It is a relatively new time-domain metric over SDR and SI-SDR that is not overly sensitive to low frequencies like SDR and can accurately evaluate silent intervals.
In addition, time-domain metrics can be evaluated for both amplitude and phase." - Unwa
Metrics ignore phase, so probably phase fixer won't affect fullness/bleedless metric.
More by unwa:
“https://github.com/ZFTurbo/Music-Source-Separation-Training/blob/main/utils/metrics.py
First, metrics can be categorized as either time-domain or time-frequency domain.
SDR, SiSDR, and log_wmse are time-domain metrics.
l1_freq, aura_stft, and aura_mrstft are all STFT-based metrics and belong to the time-frequency domain.
Time-domain metrics assess how accurately the waveform matches the original sound, while time-frequency metrics compare how closely the spectrogram resembles the original sound's spectrogram.
Time-domain metrics are biased toward low frequencies and are less affected by high frequencies.
Time-frequency domain metrics can evaluate both low and high frequencies equally when based on a linear-scale spectrogram.
When based on Mel spectrograms, they exhibit a bias closer to human hearing.
However, the weakness of time-frequency metrics is their lack of phase consideration, since all metrics here are based on amplitude or power spectrograms.
The log_wmse metric is unique in that it allows weighting tailored to human hearing.
Although it is a time-domain metric, it is special in that it also takes the frequency domain into account.
The difference between aura_stft and aura_mrstft is that stft compares spectrograms of a single resolution, while mrstft compares mel spectrograms of multiple resolutions.”
For evaluating specific instrument stems, interesting read:
https://arxiv.org/abs/2507.06917v2
_____________________________
Top metrics of publicly available Roformers for instrumentals available for download
(as for 18.06.25)
Instrumental models sorted by instrumental fullness metric:
INSTV6N (41.68)>INSTV5N (40.47)>inst_Fv4Noise (40.40)>Inst V1e (38.87)>Inst Fv3 (38.71)>INSTV7N (36.83).
While V1e+ (37.89) might be already muddy in some cases