For learning the basics about how an operating system works, I recommend Remzi and Andrea Arpaci-Dusseau's "Three Easy Pieces", which is freely available at
http://pages.cs.wisc.edu/~remzi/OSTEP/. Both Remzi and Andrea Arpaci-Dusseau are great writers and are prolific researchers in the areas of operating systems and file systems, and their work is top-notch.
For learning the basics about how the kernels of Unix-like systems are designed, I recommend the following books:
- The Design of the UNIX Operating System (1986) by Maurice Bach. You could also compare this book to the source code of Version 7 Unix, which is freely available under a BSD-style license. Many of the basic design principles of Unix-like operating systems have not changed dramatically over the years, and so studying the design and implementation of older, smaller implementations of Unix-like operating systems can aid with figuring out the code base of Linux, which is huge.
- UNIX Internals: The New Frontier (1996) by Uresh Vahalia. This discusses some more modern features that weren't part of AT&T Unix.
- The Design and Implementation of the FreeBSD Operating System (2014) by Marshall Kirk McKusick et al. I've met "Kirk" in person a few times at USENIX FAST conferences; he's not only a great writer, but he is also very friendly and approachable. This is a well-written text describing FreeBSD 11, including modern features such as ZFS.
There are some good Linux kernel books (I'm partial to "Linux Kernel Development" by Robert Love, which helped me out nine years ago when I was writing a device-mapper device), but what I've found during my experience working with the Linux kernel is that the Linux kernel changes much faster than the books describing it get updated. Nevertheless, the best thing to do is to get a solid understanding of how Unix works, then get a basic understanding of the Linux kernel, then start diving into the source code, which is ultimately the definitive documentation of the Linux kernel.
When I was working on that device-mapper device, I found that perusing the Linux kernel mailing list was excellent for learning about how relevant portions of the kernel changed over time.
Good luck and have fun! I haven't done much kernel-level work lately, but I miss working at that level!