Red Green Repeat Adventures of a Spec Driven Junkie

Testing from the Outside In

What is testing from the outside in?

Testing from the outside in is testing functional groups of a system at once (i.e. can a user login with the right password?) instead of the the finer programming details (i.e. does the login method return the right value with the correct user and password settings?)

How is this different than…

…inside out testing?

Outside in testing is creating tests from the end user’s perspective instead of the programmer’s perspective. The tests created from the outside will essentially be the same as what the system’s end user would see and do.

…specs?

These are totally different than specs. The tests created when testing from the outside in will be tied very closely to the sytsem being tested and be very long, since whole groups of functions are being tested at once.

Why would testing from the outside be useful?

Testing from the outside in is very useful when:

  • For systems where the internals are not accessible anymore (i.e. a legacy system) or not practical to make tests from the inside.
  • Getting tests into systems without really digging into the sytsem but get the benefits of the test (i.e. refactor system but making sure the interface is consistent.)
  • Automating whole workflows from a high level (i.e. user logs in, buys item, pays with na credit card.)

In systems with specs, it’s not that useful, because there will be a lot of overlap. Systems with internal tests won’t either. Outside in tests for these cases will server as another system check.

This is one technique I have used when incorporating tests into existing system that has zero tests and I want to get the most bang for the (owner’s) buck.

How to test from the outside?

Have a tool that can be programmically controlled and can interface with the system to be tested.

  • For a web app: Selenium is perfect because it can control different web browsers in a variety of programming languages.
  • For a JSON API, make a JSON client which can make and accept API calls. Curl is a UNIX tool which does this, another: Postman which is a great Chrome extension to make REST requests.

Where to start?

Make sure operations to be tested can be done manually. There is no point in trying to script a test if the operation cannot be done by hand.

Understand how to program the tool which interfaces the system to be tested. Selenium is a great tool which can programmically control a web browser, simulating the system’s end user, but there are many options and its documentation is tricky to search for since there are a variety of language interfaces.

Break down the steps of the operations to very small steps. So small that it seems ridiculous. Many operations take a lot more steps and are more complex than I realized.

Example: steps for a user to log in on a web page from scratch:

  1. Open browser
  2. Select browser URL element
  3. Enter "#{login_page_URL}" into browser URL element
  4. Command browser to load browser URL
  5. Wait for page to load
  6. Select username text field
  7. Enter "#{username}" into the username field
  8. Select password text field
  9. Enter "#{password}" into the password field
  10. Select login button
  11. Click the login button
  12. Wait for browser URL to change successful login page

This might be a bit more detailed needed, but programmically controlling a tool to login might take this many steps! Do not be surprised with how many steps it takes, a lot of these steps are taken for granted or just internalized by the user, I definitely did until I had to use Selenium.

Start scripting slowly. Take each step from the break down and start scripting them. I always feel it is more work to script out a process I can do by hand because there are so many small steps involved and there are so many areas be an error or items unaccounted for. In the long run, having an automatic script saves a lot of effort since items can be tested independent of a person.

Script, run, verify

The process will be to:

  1. write a little bit of the script
  2. run script on system
  3. verify script is doing what I expect
  4. repeat 1

I always want to make sure my script is matching up with the manual operation I am automating. Using Selenium for this is great because I can see what is going on exactly.

This process will take a long time since I build iteratively. I also refactor once I reach certain milestones and look for common methods to group together. I advise against refactoring early on. Scripting like this has very few reusable parts on the first pass. A test is being created, not a tight spec, it’s very procedural.

What to do when script is done?

Congratulations! A script is done. With the one path done, I usually like to move on and start scripting out the ‘negative’ paths, like when a user uses a bad password. Password reset using a recovery email, etc. Start building out scripts for a user to use the system. Process flows which are essential to the user of the system.

When there is enough coverage, setup the tool to run in a minimal mode. Selenium can run the browser in ‘headless’ mode which does not display the browser when it is running. A great way to multi-task.

Including print messages at key points in the script helps when running in minimal mode as well as finding a place to start debugging when things go wrong later.

Advantages of testing outside in

Full end-user automation

Having a tool which perfectly replicates what an end user would do on a system is awesome for debugging. If there are any issues that come up the procedure can be tested right away and repetitively.

System and test independence

The script and the system can use very different technologies. If I’m testing out a JSON API, I can change out the server completely. Create my JSON client in Ruby while the server is NodeJS.

Multiple system interaction

The script can interact with multiple different systems it can interface with, allowing greater range of tests. One example is email password recovery. When a user requests their password to be reset, the email should arrive in their email system of choice and when clicking the specified link, a certain page be losed with certain options.

Disadvantages of testing outside in

Testing speed

Even though the script takes longer to write, running the test will always be slower than testing from the inside. (i.e. using Selenium, a whole web browser has to start, just to initiate the smallest test on the system.

Longer scripts

The script written will be very procedural and depending on the operation, will require multiple interdependent steps (i.e. single page apps require time to load because of browser rendering)

Tight coupling with system

If the system being tested changes and the script has to update with those system changes, updating the script might take longer due to the nature of the script being so procedural. Refactoring script will help making updates easier.

Conclusion

Outside in testing has been presented. It’s a great way to get a system under test which its internals are not easily accessible but automation for end user actions are desired. It is important to have the right tool to test from the outside in, a tool that can interface and be controlled programmically are essential. For web apps, Selenium is the best tool.

Break down procedures to be tested into the finest bit possible and start scripting from there. I like to iterate slowly, making sure my script matches the actions I would do manually.

It can feel like a slow process initially, but after the script is done, I love the feeling of having a script which can perform long complex actions while I focus on other parts of the problem.

Stay (automated) testing, my friend.