Coffee Space


Listen:

Testing Products

At some point I want to design something to test fixed systems that have well defined inputs and outputs. I mean, for a system where the transactions are simple, how hard can it be to test both functional and non-functional use cases?

I think it's worth mentioning that the most important feature for a hardware based product would be to automate that hardware, after all I'm interested in seeing how the software reacts (as I'm biased towards software anyway). Hardware is somebody else's domain and by all accounts much more reliable.

I thought about crunching a large formulae that describes all the inputs and outputs of the system where anomalies are looked at as potential weaknesses of the system. The problem with this is that it could get overly large at some point. It also doesn't explain when state machines get messed up, etc where programmers rely on states not being possible for example. This also puts out the idea of just testing individual functions.

This leads my thinking towards not looking at the system as a function at all, but more as a Turing machine. A well defined system should in theory not have states that either are infinite or run for too long anyway as user feedback is often essential.

Solution

I would therefore test each function individually as part of a tree and look for significant RAM changes (i.e. changes in state). Anything that leaves the product in a different state to that of what it started should be tested again in the next tree level. Duplications in state should be condensed and tested as one, although a recording should be made that the duplication was made if the solution has to be implemented by restricting the inputs to the function.

As the tree expands, loops are expected and are closed so that the most shallow tree (or first discovered tree) is preferred. These are then explored to see how different inputs could in turn affect these states further.

Defining incorrect Function

As part of the automation I would like to see it pick out scenarios which are incorrect without knowing how the system works. To help with this, functions must be broken down as much as possible to reduce complexity. This is both good practice and easier to test.

The way in which it reports these I think is significant, so I'll split it up into two types of warning:

Warning

  • One to many mapping depending on state: The same input in different states gives different outputs. This is potentially a problem but okay if this is the design (for example there is a moving average in the function).

Dangerous

  • State doesn't reoccur in deep search: After a maximum search depth (or time constraint) one of the previously tested states cannot not be reproduced (no loop-back in the tree). There is also no sign of random variables being used in the function.
  • Random variable used in function: Functions using random variables can never be guaranteed to always behave in any particular way. Potentially there are too many states to test.

Conclusion

It's highly possible with access to the code, enough time and resource without knowing what it is the individual functions are supposed to do, removing the human aspect. The more complex the system, the harder the code is to test but the safer it is once tested. Most parts of this can in fact be automated.