Separating people in recording
Guide and script for WhisperX by Pashahlis/ai_characters
“A script on the AI hub discord for automatically separating specific voices from an audio file, like separating a main character's voice from an anime episode.
I massively updated this script now, and I am also posting it here now, since this discord is literally about that kinda stuff.
Script to automatically isolate specific voices from audio files
(e.g. isolating the main character's voice from an anime episode where many different characters are speaking).
After literal hours of work directing ChatGPT, fixing errors, etc, there is now a heavily updated and upgraded script available:
I encountered some transcription errors (musical notes, missing speaker or start and end times) that would result in the entire script failing to work. So the updated script now skips such audio. That is not a problem, however, as for a 22-min file it skipped only 16s of audio and the errored audio is just music or silence anyway.
It now also automatically merges all your audio files into one if you provide multiple, so that the speaker diarization remains consistent. This increases diarization time by quite a lot, but is necessary. The merged file will be temporarily saved as a .flac file, as .wav files have a maximum file size of 4gb. The resulting speaker files at the end of the script are created as .wav again, though, as it is unlikely they will reach 4gb in size.
I also added helpful messages that tell you at which state of the script it currently is at and which audio files it is processing at the start with the total length of audio being processed.
I also made sure that it saves the speaker files in the original stereo or mono and 16 bit or 32 bit format.
At the end of the script execution, it also lists all the speakers that were identified in order of and with the audio length for each speaker. It also lists the total amount of audio length that had to be skipped due to processing errors, as well as the total time it took to execute the script.
Last but not least, I ran this script on a vast.ai rented Ubuntu based VM with a 4090 GPU and it worked. I did this to test Linux as well as because I was processing over 4h of audio, so I wanted this to be fast. Keep in mind that if you are running this script on your home PC with a bad GPU and are processing a lot of audio, it can take quite a while to complete.
Script is attached.
https://drive.google.com/file/d/13iY2knyABBU-MOaMN5_zNAoDHFMZY6SD/view?usp=sharing
example console output and example speaker output:
https://discord.com/channels/708579735583588363/708579735583588366/1132503652033114192
Usage instructions:
install whisperx and its additional dependencies such as FFmpeg as per the instructions on the GitHub page https://github.com/m-bain/whisperX
Additionally, install pydub (and any other dependencies you might be missing if the script gives an error message indicating you are missing a dependency)
install ffmpeg-python, make sure to use the following command instead of pip install if you're running this in a conda environment, otherwise it won't work: conda install -c conda-forge ffmpeg-python
edit the script to include your huggingface token and path to the folder containing the audio files you want to process
run the script simply by python your_filename_here.py
Results are quite good for what it is, but you'll definitely need to do some additional editing in audacity and ultimate vocal remover or whatever afterwards to cut out music, noise, and other speakers that were wrongfully included. It definitely works best with speakers that appear a lot in the audio file, like main characters. It does a very good job at separating those.
I won't provide tech support beyond this, as I am no programmer and did this all by just directing ChatGPT.”
Or check alternatives