I'm not advocating for the average aspiring game developer to create a game project
actually from scratch—even though, as you say, rendering, collision detection, etc. is in theory "easy" because it's such well-trodden territory. it's definitely useful to go down that route as well, but, still you get plenty of benefit from doing things in a more low-level way than you're used to, coming from something like Unity.
rarely if ever are video games created by someone having an idea, then typing some things into the computer to make that idea manifest itself in the form of an executable, and that's the whole process. the special thing about game development is that you flesh out the design of the game by actually working on it. you play around with things, explore the conceptual space of what you've created, and see what direction to take things in next. if you've only ever made games with Unity, then you only know how to think about problems within the conceptual schema of the way Unity does things (or worse: the way you ended up learning to use Unity so as to make Unity's annoyances maximally get out of the way of making your game).
when you sit down to make a game "from scratch (using libraries)", you're forced to completely rethink just about every aspect of game development that you take for granted. you don't have GameObjects and Components and Prefabs and Scenes—you have nothing, and you have to figure out how to make it into something. sure, you could just pull in some ECS library and try to continue living in that world, but there's so much benefit to be gained from making a genuine effort to not use such crutches—to figure out how to do things in a way that produces code that is reasonably efficient (not micro-optimized—just broadly "ok", efficiency-wise, is good enough for small projects running on modern computers).
you start to make observations like, woah: you don't need garbage collection/RAII/etc. at all, because, most of what you're doing that would require garbage collection/RAII/etc. is stuff that happens each frame, so you can just use a bump/arena temporary allocator that resets at the end of the frame, and, bam, that's 98% of what you were using GC/RAII for in the first place. the rest is either stuff you want to keep around for the entire duration of the executable's run, or stuff that's like per-level or per-map, that you unload and swap out when you transition between levels/maps. when you see things put into these terms, video game memory management doesn't seem all that scary, does it?
but if you've never tried to make a game "from scratch (with libraries)" on your own before, you might never encounter this. you might forever be tethered to the idea that video game logic can basically only be programmed using some kind of extremely high-level organizing principle/abstraction, like Unity's Scenes/Prefabs/GameObjects/Components, or Godot's Nodes. even if you continue using Unity or Godot (or whatever), you one day might want your game to do something complicated, and the only tool you have is a Node-hammer, so everything looks like a Node-nail, so you never consider the fundamental reality that really what you want is probably some combination of structs, arrays, and pointers, in order to make computers execute the vision of the game design idea you had in your head. and learning to be able to think about things in this way might even empower you to have the freedom to not just implement the idea you had in your head, but even do something crazier and more complex and cool, because you thought of how you could do it in the process of implementing the other idea you had!