Earlier quoted context omitted.
Sure, that demonstrates the issue. Problem is with using train_test_split(X, yy, test_size=0.2..) - this assumes independent samples, which is violated for this dataset (because some come from same source audio files). The easiest (and completely acceptable) is to use one fold as the validation data, one fold for the test set, and the remaining folds as training. This problem is unfortunately quite common even in aca…
Hey jononor — we've updated the post to split the training and test sets based on the folds. Good catch and thanks again for reporting this. Some of the experiments in the project will still have the old code, but the blog post will reflect this new train/test split.
Nice. Did you update the reported results also? I think they will change quite a bit