I would suggest a look at Excellence Without a Soul by Harry Lewis (sometime dean of Harvard College) before you conclude that it was only those comments that lost Summers his job.
There is an interesting chapter in _Excellence Without a Soul_ by Harry Lewis, suggesting that garde inflation is less a problem that widely believed. You could look it up.
In Excellence Without a Soul, Harry Lewis, sometime dean of Harvard College, quotes a head of house, scholarly and not at all athletic as saying that athletes add a great deal to the school: They are the only people who know how to lose.