Red Green Repeat Adventures of a Spec Driven Junkie

Inside Out Testing

Testing from the inside out is analogous to the idea I had for testing outside in. For an existing software system, tests are built from the inside of the code base to the outside.

Difference with Outside In?

While Outside In testing takes the perspective of an end-user for creating its tests. For outside in testing:

  • Tools to create the tests would be similar to what the end-user uses.
  • Tests are very high level and whole groups of functions are tested at once.

On the contrary, inside out testing takes the perspective of the programmer for a software system. How does the programmer see the system. Inside out testing uses:

  • Tools are only what a programmer would use to verify system correctness and never used by the end-user.
  • Tests are very low level and tests individual functions with specified inputs for expected outputs.

Difference with Executable Specs?

Executable specs are small tests which are made by programmers in the course of creating a system. If taking a similar approach, executable specs can be used to create inside out tests.

Why do this?

Many good reasons to create tests from the inside out:

  • Create tests now so the code base is easier to manage later.
  • Have the best documentation possible, especially when taking executable specs approach.
  • Keep code base manageable by having automated tests to verify system operation.
  • Allow refactoring of functions inside the system, so large functions can be broken down into smaller ones when sufficient test coverage is there.
  • Be able to deploy a version of the system knowing which parts of the system are not working by running the automated tests.

In general, the larger and longer running the system is, the greater the need for tests, inside or out. If the large long running system is to be built upon, tests from the inside are a definite must! It is the best way I know to prevent feature regressions, where users ask: “this feature used to work, but does not now. Why?” I hate those kinds of questions.

How to test from the inside out?

Practical Object Oriented Design in Ruby

Start reading Sandi Metz’s book Practical Object Oriented Design in Ruby. This is a fantastic book that gives a great overview of how to think about systems, refactoring them to be testable, with examples. I really enjoyed this book and it shapes almost every piece of code I write now in any language. “Ruby” is in the title, but its lessons are highly transferable to any programming language.

Get a testing framework

A testing framework will go a long way in organizing tests to be run in the system. It makes life a lot easier to have a testing framework.

Some testing frameworks for different languages:

  • Rspec (Ruby): my personal favorite.
  • Minitest (Ruby): another alternative for Ruby.
  • Jasmine (JavaScript): my goto whenever I have to work on JavaScript.
  • Mocha (JavaScript): a more advanced one for JavaScript.
  • PyUnit (Python): a unit test system for Python.

Get comfortable using one by programming an algorithm starting with tests. I find this a great way to know the capabilities of a testing framework and understanding how functions and tests are connected with the framework.

Find the ‘happy path’

Now that the testing framework has been hooked up and functions can be tested, where to start?

Look for the main ‘happy path’ of the system. This is the ‘ideal path’ a user would take through the system.

It probably starts at a login function, where a successful login moves onto another area. Start testing the login function with valid credentials and make sure it can be successfully tested repeatedly.

From there, expand into the application more. What should happen after a user login? What is the main feature of the application? What functions do these depend on? What are the most common parameters for these functions to pass. What result should those functions return with the common parameters?

Keep digging deeper into the application more and more until all the functions on the ‘happy path’ are fully tested.

Find ‘edge cases’

After the happy path as been mapped out, start investigating any side paths or ‘edge cases’ which would be part of the main path. The path a user would encounter using the system, but not so common or would find in error.

From there, continue mapping out functions and creating tests for them.

Things to do to make testing from the inside easier

Functions should always return a value

A function is easier to test when there can be an expected value to match against. For example, a login function:

1
2
3

login(good_username, good_password) # => true
login(bad_username, bad_password) # => false
login(good_username, bad_password) # => false

It’s very hard to tell internally whether the login function failed because the username was bad, or the password was bad. (For security concerns, there should not be so much revealed, but internally, to make things easier to manage).

Maybe a better way to have the function would be:

1
2
3

login(good_username, good_password) # => :successful_login
login(bad_username, bad_password) # => :no_such_user
login(good_username, bad_password) # => :bad_password

And on the user facing side, the error message for :no_such_user and :bad_password would be: “Sorry, bad username or password”.

Simple to evaluate return types

Testing a function that returns a type which is easy to evaluate makes testing easier. Simple types like: Boolean, a number, string, arrays are simple to evaluate. Did the function successfully pass or fail? Use a Boolean. Does a function do calculations with its inputs? Return its output (and false if an edge case is reached.) Simple return types make testing easier.

Hard to test types: File handles (especially when the function manipulates the file contents.) Lambda - a call to another function.

Investigate how the test framework evaluates values. Some may handle array equivalence, so evaluating array equality can be easy… Or hard if there is no support.

Many small functions \» few large functions

This is particularly true when testing from the inside. Small functions generally mean: more focused functions, which handle less. Testing these will be easier as its range of inputs and outputs will be limited.

Large functions mean larger range of input or output, which makes testing a bit more trickier. The good thing about getting tests around large functions is refactoring them will be easier when tests are in place, making large functions small.

Advantages of testing inside out

Inside out testing is meant to give programmers a way to test out parts of the system being built. The tools used to test from the inside are all programmer oriented and are very different from what the user of a system would use.

With automated tests in place, deployments can be done quickly as the state of the system can be known at any time.

If the executable spec approach is used for creating tests on the inside, it is creating the best documentation possible. Functions are tested in small specs. Failing specs are immediately apparent when specs fail. It’s a real joy to work with a system that has executable specs.

With tests on the inside, system robustness can be improved. Either by running the tests after each new feature to ensure there are no feature regressions or using the tests as a way to refactor the system into more manageable parts.

Disadvantages of testing inside out

Programmer oriented. Tests created won’t have an immediate benefit to the end user until there is sufficient test coverage. If there is a desire for immediate end user benefits for tests, it’s much better to test from the outside in.

Limited to system under test. Inside out testing tooling are designed to only test the system under test and no more. Any other external services to be tested become a bit tricky and can be brittle.

Having a testing framework really helps a lot as it provides a lot of support to test functions.

Conclusion

Testing inside out has been presented. It’s a great strategy to introduce a system without tests from the programmers perspective. Tests from the inside provide a way to make a system more robust and cover more edge cases simply than an outside in testing strategy would.