
If you're looking for a more formal approach, ie the infinitesimal analogue to the usual real analysis, it's called nonstandard analysis and you could probably start with the original, eponymous book written by the creator, Abraham Robinson, for which I unfortunately don't have a link.
If this stuff interests you btw I would also check out Knuth's book on surreal numbers[3], which I believe, in some sense, are the fullest possible extension of what we think of as numbers? But it's been a while since I read into those.
[0] https://www.bravernewmath.com/ [1] https://intellectualmathematics.com/calculus/ [2] https://people.math.wisc.edu/~hkeisler/keislercalc-06-03-26.... [3] https://people.math.harvard.edu/~knill/teaching/mathe320_201...
In teaching calculus we still use both the prime/Newton notation (y', y", and so on) and the fraction/Leibniz notation ("dy/dx", etc.) for derivatives. They're both useful for writing and for visualizing how things work, and both are used outside math (e.g. in physics and engineering). "dy/dx" was originally thought of as the quotient of two infinitesimal quantities --- arbitrarily small but (not necessarily) zero.
Limits became accepted as the foundation for calculus, and infinitesimals were regarded as heuristic or non-rigorous. But if you're careful, "you can get the right answers", and Abraham Robinson's discovery of nonstandard analysis (around 1950 - 1960) showed that infinitesimals could be put on a rigorous basis. Besides Robinson's "Non-standard Analysis" and Jerome Keisler's (now free) calculus book someone else mentioned (and the supplement which provides a lot of the theory), people who are interested in this can check out "Infinitesimal Calculus" by James Henle and Eugene Kleinberg (https://store.doverpublications.com/0486428869.html - a short introduction) or "Applied Nonstandard Analysis" by Martin Davis (https://store.doverpublications.com/0486442292.html - it starts with the logic and set theory and like Keisler's supplement is fairly theoretical).
Why aren't infinitesimals used in teaching calculus? After all, we can now make them rigorous, and the notation fits well with differential forms, which are used in differential geometry. Keisler's book (which I think came out in the 1970s) was an otherwise traditional calc book which made the attempt, but there is just too much inertia behind the limits approach. Some people were bothered also by the fact that infinitesimals are definitely non-constructive; the Bulletin of the American Math Society oddly had Errett Bishop, a noted constructivist, review Keisler's book when it came out, and the review was not very positive. (I don't know why they picked him as a reviewer.)
But there's still a problem - you're defining the derivative as a limit of difference quotients, but it sure looks like a fraction. What to do? Calc books often fudge this in the following way. They define the derivative f'(x) (note the prime notation) as the limit of [f(x + h) - f(x)]/h as h goes to 0. Then "dx" is regarded as synonomous with "Δx", and both are considered to be simply increments/changes in x --- no infinitesimals. (So, e.g., dx could be 0.3.) Then you define dy to be f'(x) dx., so of course dy/dx = f'(x). In this interpretation, "dy" and "dx" are increments in y and x as you move along the tangent line to the curve. Rather than refer to these as "differentials", calc books will often put this in a section called "linear approximation", which usefully discusses using the tangent line to a curve to approximate the curve.
I think it's good to bring up this history in calc courses - I always tried to.
Math is rigorous and proceeds according to logic (that's the aspiration, anyway), but the way math develops and is taught is a social thing.
https://people.math.wisc.edu/~keisler/foundations.html
It's a bit more than an instructor's manual. It covers the construction of the hyperreals in two ways: One (in section 1E) is relatively short, and is like the compactness theorem approach Abraham Robinson originally used in his book ("Non-standard Analysis", Princeton U. Press).
The second construction uses ultraproducts (in Chapter 1*). Keisler gives a quick introduction to the logic needed to understand the ultraproduct construction, beginning with formal languages and sentential logic. So the prerequisites are there, and explained concisely but clearly. The ultraproduct construction comes up elsewhere; for applications to other areas of math, see Martin Davis's book "Applied Nonstandard Analysis" (Dover Publications). (There's a review of the books by Keisler, Davis, and Stroyan and Luxemburg: https://www.ams.org/journals/bull/1978-84-01/S0002-9904-1978...).
The rest of Keisler's monograph contains mostly nonstandard proofs of many of the results in the textbook. I like it a lot, and anyone who's interested in this stuff should grab the PDF.
Another book that is short and moderately elementary is "Infinitesimal Calculus" by James Henle and Eugene Kleinberg, also published by Dover [which also publishes a paper version of Keisler's text, and is generally a great publishing company].
When Keisler's book came out, someone decided that it would be reviewed for the Bulletin of the AMS by Errett Bishop, who (as I recall) was a noted constructivist. You can read the review here: https://www.ams.org/journals/bull/1977-83-02/S0002-9904-1977... Having a constructivist review a calc textbook that uses nonstandard analysis was probably not a great idea. The review ends with: "Now we have a calculus text that can be used to confirm their experience of mathematics as an esoteric and meaningless exercise in technique." I believe Keisler wrote a reply in a subsequent issue of BAMS, but I can't find it.
For young students, a great introductory textbook is Calculus Made Easy. It is around 100 years old, and develops all the material using infinitesimals. Which is essentially modern non-standard analysis, minus rigor. It is also the way Newton and Leibniz thought about calculus, and the way most physicists intuitively think about problems.
For a more mature audience, I like Infinitesimal Calculus by Henle & Kleinberg.