
See how the core components of the Windows operating system work behind the scenes--guided by a team of internationally renowned internals experts. Fully updated for Windows Server(R) 2008 and Windows Vista(R), this classic guide delivers key architectural insights on system design, debugging, performance, and support--along with hands-on experiments to experience Windows internal behavior firsthand. Delve inside Windows architecture and internals: Understand how the core system and management mechanisms work--from the object manager to services to the registry Explore internal system data structures using tools like the kernel debugger Grasp the scheduler's priority and CPU placement algorithms Go inside the Windows security model to see how it authorizes access to data Understand how Windows manages physical and virtual memory Tour the Windows networking stack from top to bottom--including APIs, protocol drivers, and network adapter drivers Troubleshoot file-system access problems and system boot problems Learn how to analyze crashes
The Wikipedia article on NT architecture is decent enough (https://en.wikipedia.org/wiki/Architecture_of_Windows_NT), and some quick searching brought up this slide-deck from 2008 that's pretty good too: http://www.cs.fsu.edu/~zwang/files/cop4610/Fall2016/windows....
I'm most impressed w/ the kernel object manager and the pluggable user-mode "personalities". I've always dreamed of an NT "distribution" that didn't have Win32 but, instead, shipped with the Interix (aka Service for Unix) POSIX personality.
for plain books:
Windows Internals by Solomon and Russinovich
AI: A modern Approach by Russell and Norvig
http://www.amazon.com/Windows-Internals-Edition-Developer-Re...
http://www.amazon.com/Windows-Internals-Edition-Developer-Re...
Additionally the books for user mode programming from Jeffrey Richter and Charles Petzold tend to be quite good and full of valuable information as well.
It's not for writing applications, but rather understanding how Windows works (The Cache Manager section for example)
http://www.amazon.com/Windows®-Internals-Including-Windows-P...
If you're in school, Microsoft also has an academic program that allows universities (almost) full access to the source code of various Windows kernels. Its called the Windows Research Kernel program and might help you along if you can get ahold of it because it comes with a ton of documentation: the original NT design papers, proposed class materials and a full copy of the aforementioned Microsoft Windows Internals book in PDF form.
For Mac, the online kernel guides are pretty good. The Mac OS X Internals book isn't so great for driver or kernel development directly, but is a good reference for learning about how things work. Having said that, the book is a bit dated at this point and could use an update for the later kernels.