
One thing a friend of mine said, which I think has been very good advice, is to get several books on a single topic. Eventually every author will lose you and you'll get stuck; having alternate discussions will help you get through it. This is easy to do and inexpensive if you pick up some Dover math books, but I've been making heavy use of the local academic library. The math books you want to read are not in high demand at the library! You can get three or four and see if any of them are good enough to warrant a purchase later on, because you probably won't get through them in a month or whatever.
I find that for many topics there is a really good text. Calculus by Spivak is a great example, it straddles the line between calculus-in-college and analysis. Every topic seems to have a few really good books like this one, and there are often books that will take a totally different approach, like H Jerome Keisler's nonstandard calculus book using infinitesimals.
I used to see it as a real problem that I was learning math outside class, but more and more I see it as a benefit, because you can pick up the stuff you want at the resolution you want and benefit from the best books rather than whatever the publishers are bribing professors to use. Going at your own pace, you're not going to go through as much stuff as quickly, but you will actually _really_ learn it. I've spent the last three weeks or so thinking about the construction of the real numbers... in a classroom setting, you would be forced to get through this quickly to get on with the rest of the curriculum, even if you aren't interested in the rest of it.
I think both our roads are eventually going to lead us to differential geometry, and the only thing I know about that is that there appears to be a very good book on Amazon (Tapp, Differential Geometry of Curves and Surfaces), and that you may want to avoid older books that use the older notation for it. I have heard great things about the book Gravitation, but I'm totally afraid of it, not ready to go there yet. Also check out Physics from Symmetry, that book looks amazing to me but I haven't read it yet, just flipped through the contents, but it might be exactly what you're after, since it discusses the math right before applying it to specific areas of physics.
I'd recommend it if you're looking for some math reading.
For a more classical introduction to differential geometry requiring only multivariate calculus and some real analysis/point set topology, Do Carmo's "Differential Geometry of Curves and Surfaces" is a great textbook.
I put together a summary (key definitions/theorems) from an undergraduate course following Do Carmo at [2].
[1]: http://www.amazon.com/Differential-Geometry-Curves-Surfaces-...
[2]: http://ajtulloch.github.io/PDFs/MATH3968LectureNotes.pdf
Also see page 41 of Differential Geometry of Curves and Surfaces by Manfredo Do Carmo