There is Such a Thing as Too Much Testing

Have been watching the RubyConf 2009 video. And I liked the matrix which is based on 3 dimensions to determine level of testing. The questions you have to ask to determine how much testing you need should include:
  • Did your tests stop/find bugs?

    • How often?
    • How soon?
  • Was there good design?

    • Did the tests encourage good design?
    • How easy was it to write new features?
  • What's the level of developer buy-in for testing?
  • What's the level of buy-in for managers?
  • What level of assurance do you need? Are you really sure? (Lose a tweet – not a big deal, lose a life – thing you're dead too).
The lessons are:
  • Start small (do not do Selenium at least!).
  • Do not get new resources just to run your tests. They MUST be fast enough.

    • Paralllelise.
    • Use fast machine or cloud.
    • Do not hit external resources (including DB if possible).
    • Profile and examine slow tests.
    • remove redundant tests.
  • Distinguish between model and integration test clearly. You often assert in model test instead of integration ones.