Google Testing Blog

I'm irrationally passionate about testing. Shameless plug: (rendered version at https://gist.github.com/androidfred/501d276c7dc26a5db09e893bb7c9f3ee)

# Test your tests

## Summary: test interfaces, not implementations

Tests that pass when something is actually broken, and fail when things actually do work, are worse than useless- they are positively harmful.

## The requirement

Let's say we've been tasked with returning `400` when `GET` `/users/<userId>` is called with a negative `userId`.

## The test

The requirement can be turned into a test that hits the endpoint with a negative `userId` and checks that a `400` is returned:

```java
@Test
public void getUser_InvalidUserId_400() {
expect().statusCode(400).when().get("/users/-1");
}
```

## Implementation

A ubiquitous style of implementation may look something like this: (ignore whether you like the style of implementation or not, it's just an example)

```java
public class UserResource {

@Inject
private UserService userService;

@GET
@Path("/users/{userId}")
public Response getUser(@PathParam("userId") final Long userId) {
try {
return Response.ok(userService.findById(userId)).build();
} catch (final UserException e) {
return Response.status(e.getErrorCode()).build();
}
}
}
```

```java
public UserService {

@Inject
private UserDao userDao;

public User findById(final Long userId) throws UserException {
…

What I meant by unsustainable is that I feel the retaining talent problem is less related to money. My guess is smart people are leaving mostly because of the environment and not because of pay and benefits. When those smart people leave, theoretically the quality of the product suffers and overtime the competitor with better people wins. If MS profits go down over time, and investing in things like Azure and Bing and Entertainment (all which loose money), I don't see how they can continue to throw money at the talent problem. But man if I could get paid more AND work in a better environment (one where I had an impact) I would go for it in a heart beat. So in that sense money DOES matter. I agree with the rest of what you said though about it being difficult to measure.

This is what I think of when the subject is talented people leaving MS: http://www.techflash.com/seattle/2008/12/Key_data_center_architect_leaves_Microsoft_for_Amazon36057114.html

http://googletesting.blogspot.com/2009/06/james-whittaker-joins-google.html

I think Amazon and Google are better because of it.

I was spoiled as my internship January last year with Google was my first introduction to the world of corporate IT. Google as a culture really encourages engineers to follow best practices - "Testing on the Toilet"[1] was always a highlight.

As you say, the emphasis on unit tests is amazing and something not found in a lot of professional companies. When you tell someone you haven't committed your code yet because you're writing unit tests they don't tell you to hurry - it's not considered something you can rush.

Additionally some of the research that has come out of Google Research [2] is stunning and I suggest if people are interested they can keep tabs on new publications via their feed [3]. A lot of their tools and products really are pushing the boundaries of the field they're in, both in complexity and scale, so the "reports from the front line" offer a unique insight.

[1] - http://googletesting.blogspot.com/2008/08/tott-100-and-counting.html
[2] - http://research.google.com/pubs/papers.html
[3] - http://research.google.com/pubs/atom.xml

I think we must be misunderstanding each other. Developers are human. (At least, I think we are.) We respond to the incentives and structure of the environment in which we're developing. A culture of "throw crap over the wall to QA" naturally creates an incentive for quickly writing up features, without balancing that with robustness, because finding the problems "is QA's fault". A culture of "the devs are responsible for the quality of their code" produces better code. The way to treat the devs well is to have the QA folks go off and create or bring in easy-to-use automated testing and CI frameworks, to make it easy for the devs to do the right thing.

This is not a new observation - it's been happening over the last 10 years. see, for example, http://blogs.forrester.com/mike_gualtieri/11-02-17-want_better_quality_fire_your_qa_team
or http://product.hubspot.com/blog/the-classic-qa-team-is-obsolete
and, while it's from 2007, http://googletesting.blogspot.com/2007/03/difference-between-qa-qc-and-test.html

I think there is an aspect to this article that won't be understood unless you're really part of the Python community. For a while now, one of Python's selling points to users from other languages has been "you don't need to write all that code" (most likely directed at Java), more specifically you can simply use a function rather than having to define a class just to define static methods etc. Over time, this has grown into the mantra of "You don't need a class just use a function/module". A lot of people seem to follow this viewpoint blindly, so much so that they consider it to be "Pythonic"; the way you "should" write python.

I think Armin's post is somewhat in line with the following post from the google testing blog.
http://googletesting.blogspot.com/2008/12/static-methods-are-death-to-testability.html

I work at IMVU as well, know the author personally, and can tell you that yes, he unit tests too. In fact, like all of us, writes all code test-first.

For test strategy in general, you need a whole seminar. There's a good blog on testing over at http://googletesting.blogspot.com/ <- go there!

As for code:test ratio, I'm not sure what metric you want, but using just php files, there are ~1k active test files and ~7k php files total. So if you take into account that 1k of that 7k is the test, and some is test-setup and obsolete files we no longer use, it's maybe about 5:1 code:test. However, a lot of the code is third party OS software packages. In-house code is about 1.5:1 test:code, in my experience. These, by the way, are very rough estimates.