SDR leaderboard & explanation

Tested on multisong dataset
https://mvsep.com/quality_checker/leaderboard2.php?&sort=instrum

(some models/AIs/methods are not public, or only on MVSEP, all others you will find in UVR's and/or download center if you can't find some models, some only after using VIP code, or somewhere in this doc if it’s public)

Older, “synth” dataset more of older models, a bit less reliable, no longer updated leaderboard by the results of new models

https://mvsep.com/quality_checker/leaderboard.php?sort=insrum

Diagram by ZFTurbo of influence of chunk_sizes on SDR on various archs

The biggest SDR doesn't automatically have to mean that your result will be the best for your song, and your use case (inst/voc/stem). Read the list of all the best models and methods at the top, and experiment.
Apart from bleedless/fullness metric, models with bigger SDR than others might pick up instruments better (e.g. less wind instruments recognized as voice).

Also, “The way I see high SDR is it indicates the lower frequencies will be more accurate to the original stem, and be more free of distortion or noise. And I also see it sometimes indicates better quality of fundamental frequencies (closer to the original gain/phase, more consistent separation), but I don’t know much beyond that lol” - stephanie

- For specific songs, different ensemble configurations can give better results than for others.

- "Since the SDR [on MVSEP’s] synth dataset is flawed from the get-go due to the dataset being used isn't really music, but sample-based, don't get your hopes up too much.".

But it generally reflects in greater extent differences between models, e.g. used in Demixing Challenge 2021, so it's not totally bad and multisong dataset might be even better (and still not perfect) - just be aware that different settings can give you better results for your particular song rather than average best combination of models on the SDR chart.
- Bas Curtiz conducted some tests with commercial music as evaluation dataset, and it turned out that only models already close in SDR switched ranks, and most models kept the same. So in conclusion, multisong dataset can be considered as still reliable (although bleedless and fullness metric is more suitable for our tasks now - more below).

About SDR evaluation on MVSEP and how important factor is that to the final result  

It still depends on the specific song, what bag of models/ensemble or what specific models will come out the best in specific scenarios. Suggesting by SDR of at least multisong dataset can be misleading. For example, the metric doesn’t really reflect the differences between e.g. HQ_3 and MDX23C fullband model in case of bleeding in instrumentals occurring in lots of contemporary songs. Although, the bleeding issue doesn’t always occur, and then, HQ_3 results can be more muffled, so in this case, SDR metric would be more accurate to human listening scenario where MDX23C models gets better metric, so it can be misleading, because SDR can vary very much from song to song.
“The thing is that SDR evaluates at the same time how "full" the stem separation is and how much bleed there is in the separated stem. You can't know, only based on SDR score, which of "fullness" or "bleedless" is impacting the score the more” - jarredou

Also, according to some SDR evaluations conducted by Bas Curtiz, it turned out that permanent bleeding don’t have more impact on SDR than occasional bursts of bleeding here and there.

Still, in some scenarios SDR metric of multisong dataset on MVSEP can be a safe approach, giving you some reassurance that the result in a strict test scenario will be at least decent in some respects, although you can (or even should when some instruments are missing) still experiment trying to get a better result, but it doesn't have to be reflected in SDR.
To sum up, SDR evaluation is only kind of averaging toward a specific dataset of songs, and it’s unpredictable based on just SDR how certain model will behave on specific song, plus its algorithm is limited vs human ears too. For example, if you could measure SDR for a specific song by its official, perfectly inverting instrumental, then it may not get the best result by the settings of the best ensemble combination measured by SDR for the time being. Suggesting by SDR means there’s just higher chance to hit a good result in a certain spectrum of sonic changes - it’s a good starting point to experiment further.

Based on 9.7 NET 1 models, MVSEP synth dataset usually gives ~0.7 higher scores than on Demixing Challange 2021 leaderboard. Also, it favours Bas Curtiz FT model more than multisong dataset due to some characteristic features ZFTurbo pointed out.

“A calculation by a computer isn't a human ear”.

- Another way to at least sonically evaluate a model/ensemble, is to test it on a set of AI killing tracks which tend to have specific issues after separation with most if not all models, and to see how better or worse it got. Childish Gambino – Algorhythm is a good starting point to chase differences in vocal bleeding in instrumentals among various models, due to specific effects applied to vocals.

How does SDR even work in Python

def sdr(reference, estimate):

    delta = 1e-7  # avoid numerical errors

    num = np.sum(np.square(reference), axis=(1, 2))

    den = np.sum(np.square(reference - estimate), axis=(1, 2))

    num += delta

    den += delta

    return 10 * np.log10(num / den)

Q: Is there a way to compare SDR between an official instrumental and the filtered instrumental

A: Bas has shared an .exe script to do that easily ⁠(uvr-general⁠)

What you could do, but only if u have the original vocal or instrumental, is to check on SDR with this:

usage:

sdrcalc.exe "c:\your-input-folder" "c:\your-output-folder"

make sure they have the exact same extension + filename

“Here is an idea for multisong leaderboard V2, with the songs edited to have the loudness of real music. In this paper, they show that lots of models SDR value decrease when evaluated on real music https://arxiv.org/pdf/2208.14355”

After the community used SDR on synth, and later multisong dataset extensively, later

jarredou invented a new method of automated evaluation of models: