
The information inside of this book is vital for the lives of people of color. This book only serves as the first step in our life-long journey of self-REdiscovery.Knowledge of self is priceless.
Fun fact: his original concept needed 9 pins and therefore was going be forced to have a 14 pin package. A late epiphany got it down to the 8 pin version we know today.
probably the best place to start for this, in a top-down sequence, is the art of electronics by horowitz and hill. it explains how transistors and diodes act in §1.6, §2, and §3, initially explaining transistors with the simplified 'current amplifier' model (which already goes beyond the 'transistor switch' model you're thinking of), then quantitatively with the ebers–moll model; they're focused on how to use this information to design working circuits from discrete components you can put together on a circuit board. camenzind's designing analog chips (available free online) goes into how to use this information to design actual chips (not only does nand2tetris get into things like metastability and noise margin, the authors seem to be confused about what a chip even is, thinking you can make a chip by assembling other chips)
but the ebers–moll model is still not solid-state physics knowledge. so far the best overview i've found of that is madou's 'fundamentals of microfabrication and nanotechnology' which has a couple of chapters about solid-state physics, going into the historical development of quantum mechanics. but it's not really a quantum mechanics textbook; it's just an overview that shows where quantum-mechanical knowledge fits into understanding solid-state physics
'the feynman lectures on physics' is the best quantum mechanics textbook i've found so far, but because my knowledge of quantum mechanics is even more minimal, please don't trust my recommendation on this
hope this helps. good luck in your learning voyage!
For digital circuits, Mead and Conway's Introduction to VLSI Systems is a good source of information. It's available online: https://ai.eecs.umich.edu/people/conway/VLSI/VLSIText/VLSITe... (the link seems to be down right now, though.)
For analog circuits, the book "The Art of Analog Layout" is good if you can find a cheap used copy. The designer of the 555 timer wrote an interesting book "Designing Analog Chips" which I found very interesting. It is available online: http://www.designinganalogchips.com/
There's also an online tutorial on IC reverse engineering: http://siliconzoo.org/tutorial.html
At first I didn't see this, but Ken linked Camenzind's book in the article footnotes. It's highly regarded and free online.
Shirriff's schematics are a lot nicer-looking than Camenzind's. I don't know what software Camenzind was using to draw his schematics, but despite being well organized and laid out, they are superficially rebarbative, almost as bad as KiCad's. Square dots, bright red transistors contrasting with bright green wires, barbed arrows on transistors, collectors intersecting with emitters at the base (or, randomly, not), active-low overbars that are the wrong length, resistors full of 90-degree angles and only four corners, no line width variation, P-channel MOSFETs with the arrow overlapping a wire-connection dot, MOSFET gates too far from the rest of the MOSFET, text overlapping wires, u for μ, etc. The effect is as if he did every schematic in Comic Sans.