> I have directories on my filesystem(s), files in those directories
Your library.
> powerful applications to navigate them (Total Commander / mc / Krusader).
Not powerful enough. Filemanagers are optimized for hierachical views and don't integrate well with filetypes. Namely, they usually don't extract data from files, like metadata, covers or previews. Though, filemanagers have started to do that to some degree.
But they still lack the "alternate view" on your library. Those Walled Garden-Apps that work with librarys are more like a relational database, offering flexible selections and content-optimized views that filemanagers are normally lacking.
The major difference is really just if you want a static and inflexible organization, or something dynamic. And usually with heterogeneous datas, the dynamic approach is better for consuming beacuse it allows you to break through the hierachy.
> (eg. where do I find all those books when I want to copy them?).
> I hate Calibre for the exact same reason.
Just select your books and export them. Calibre can export to a device, but also just a local directory. Calibre in that regard is one of the better gardens. Very flexible and powerful. Just the UI sucks.