Tests vs. Specs

Why?!

I never understood my coworker’s apprehension to tests. He would opt to write endless amount of documentation. This is a former Googler! His team there had tests, but they ran too long, some tests would randomly fail, and well, sounded like nobody knew how to fix it.

I am on the opposite end of the spectrum: I love tests and will spend effort to put in tests whenever possible. I would only document when the code is ‘magical’ (i.e. include links to explain why something is done this way or is ‘magical’.)

So, it’s confusing to talk with my coworker about problems with work code. I see some problems his code base encounters which can be solved with tests. Especially anything related to ‘bugs’ or ‘surprises’ with code when ‘x’ happens. He just would not believe me. On the code base I work on, it is thoroughly tested, we have very few surprises and bugs.

Inspiration

When I saw this talk by Miško Hevery from Google explaining the “Psychology of Testing”, a few things clicked for me:

• ‘tests’ vs. ‘specs’ definition:
  • tests: software tests 1:1 function code to test ratio. If there is a function x, there is a function x_test to test it.
  • specs: executable specs 1:n function code to executable specs ratio. If there is a function x, there are numerous other functions (i.e. function x_input_params, function x_nil_params, etc.) to test it.
• Specs are small and simple
  • As part of the definition above, specs are small to test out one part of the function (i.e. the input params, nil params, etc.)
  • This makes specs very easy to understand and to modify, even for a new programmer to the project!
• Production code can be rebuilt from specs
  • If the project needs to be rebuilt only using specs, it is quite possible since each spec encapsulates a small part of the system.
  • Tests encapsulates the same or more complexity as the code it is testing.
• Specs are written before production code is written by the same person!
  • Whereas tests are written after production code is written (which may or may not be the same person!)
  • I find writing specs before writing code to be like writing an outline before writing an article. It gives me direction.

This is all great. I’m totally on-board with Miško on all these points. He’s preaching to the choir! That makes me wonder, why is Miško giving such a talk, at Google no less! The current center of the software world!

Light bulb! 💡

I realized, it’s been awhile since I have coded in a ‘corporate’ environment and realize bigger companies have dedicated Quality Assurance teams just for software testing. This changes the software development dynamic:

• Who is writing the tests or specs
  • In a start up, there’s less than a handful of people. whereas an enterprise will easily have a handful of people just dedicated to one task, like testing.
  • In a start up, tests will be written by the developers. there’s just no resources available to have tests written. (if tests are written at all!)
  • In an enterprise, there can be a dedicated department where their sole purpose is writing tests for developer’s code.
• Tools for writing specs and tests
  • Software tooling that startups use are the type that are open sourced. Where many other contributors have put good work into them and provided them for everyone to use in the most general way possible.
  • Enterprise tooling is a different beast, where tools are closed sourced and highly specialized for the task it is designed for.
• Style
  • Developer specs are more of an outline, give direction to tests, which turn into executable specs afterwards.
  • Specs can be refactored as tests are added and a resource for understanding the code.
  • QA tests are to really stress the code and make sure the developer is doing their job.

From this, I realize the word ‘test’ can be a very loaded term in the software world. Two software developers can be talking about ‘tests’ and they will mean very different things. Enterprise tests are very different than startup tests. This explains why there’s such a divide between developers on this topic, even in the same company! This probably explains why my coworker has such a bad experience with tests. Our definitions differ, even in the same working context!

Realization 😎

So, from here on, I will be more careful in my terminology. I will use ‘test’ to mean enterprise style tests (it’s only fair, I’m sure they were there first!) and ‘specs’ to mean executable specs. I want to make sure the other party understands exactly what I mean by the terms: ‘test’ and ‘spec’… which I will start using today.