Strides Interactive

Software Quality - Part 12Filed Under: Weekly Tuesday Dose of goodness

Hi all,

This week’s post comes a little later than usual because I’m not satisfied with the quality of my draft. After some serious thoughts and revamps, I’ve managed to rewrite the article to better serve this week’s topic - Dangers of using stubs.

By the end of this post, we’d have covered most, if not all of the basics in Software Quality.

We’ll then move on (continuing on the series) to other parts of software quality processes which can help you in your daily development or higher level processes.

So what’s so dangerous about stubs? Let’s find out…

Introduction

When it comes to stubs, it can come from at least 2 areas.

1. Unit Testing
2. Runtime Testing with stubs

We must also remember that stubs are temporary measure to fulfill a prerequisite and should never be considered as a means to an end. This being said, there’re also some not-so-obvious mechanisms hidden when instrumented stubs are used.

In most cases, there’ll always be a safety barrier when it comes to stubs.

Safe Stubs

We’ve mentioned this type of stubs in our previous article. What are they used for? As mentioned in my previous article, safe stubs are usually either generated automatically or pre-generated to intercept certain potentially dangerous function calls.

What it does basically is to return a failure value, be it a return code or NULL pointer, it’s main purpose for doing that is to protect the caller from accidentally or unintentionally making those calls which leads to disastrous results.

In a good stub framework, many of these dangerous functions will be stubbed out automatically as safe stubs. Here’s the following to name a few:

1. f_open()
2. f_write()
3. f_prinf()

* Arguments are omitted because I’m lazy…
** The functions have a’ f_’ instead of ‘f’ because of a problem. Most host applications use the ‘f’ variant while some embedded systems uses the ‘f_’ variant.

Just take f_open() for example, if the following code is executed in original definition (ie, without being stubbed out), what will happen?

FILE* fp = f_open ("abc.txt", "w") ;

This piece of code may be a file open operation, but in reality, it actually overwrites the existing file (if any) without any warning and set it to 0 bytes.

Imagine 100 test case, data driven, each with a different filename, some sensitive, executing without safe stubs. True, some files may be protected by the OS, but ask yourself, what are the chances that one of these files isn’t?

It may not be an OS file. It could be something important to you or to your company.

Now, this is a danger that occurs only if safe stubs aren’t used. Evidently, we’ve seen safe stubs in action automatically so what’s the problem?

Circumvention

This word has been mentioned several times last week and as explained, this is a useful way to control the stub. A good stubbing framework gives the user the ability to do at least the following below:

1. Controlled return value
2. Bridging to original definition by conditions
3. Exception testing

All these 3 have their own dangers. However, before going deep into it, we must first understand that, circumvention is usually done by choice. Thus this increases the risks of introducing errors into the stub itself.

Out of these 3, the 2nd point is the most dangerous of all. Why?

Reality vs Stub Limbo vs Actual

Instrumented stubs or hand-made stubs all have a common characteristic. That is, they both sit in between the reality and the actual execution.

Let’s first talk about reality and actual execution, what’s the difference?

You can understand reality as the program execution during the run while actual refers to the original definitions of other functions.

The stub itself serve as a limbo. Which means to say, it can alter reality by preventing the program flow from ever reaching the actual execution.

Since it’s in a limbo, users can control the end state of the stubs, leading to a few possible dangerous scenarios

1. Erroneous stub returns causing incorrect or unstable program flows
2. Erroneous stub bridge, causing the wrong data to be passed to the original definitions
3. Erroneous exception triggers, causing the tests to fail incorrectly

Let’s talk about them one by one.

1 - Erroneous stub returns causing incorrect or unstable program flows

Stubbing requires full knowledge on the unit itself. Without sufficient or full knowledge, the stub writer will not have a clear picture on what not to return.

Of course, one can defend that this is meant to test for the unit’s robustness. However, we must be careful with such tests. Even with techniques like defensive programming (which I’ll talk about soon), there’s a limit.

Simply put, we don’t test things that can never happen. If you force things to work unnaturally, even if you managed to write up a test suite, the stubs will end up triggering all the incorrect behavior.

Naturally, one may put the blame on the application design. But - that’s where the limit comes in. Going in with erroneous stubs is a big step towards test failures.

2 - Erroneous stub bridge, causing the wrong data to be passed to the original definitions

This is caused by incorrect test case identification or a lack of communications between the test case writer and the stubs programmer.

The case in point is when functions like f_open() has a certain requirements to really be opened and most of the time be a safe stub. This means, that, one, by circumvention, I go around my original safe stub. Two, I channel my stub flow according to the test case ID assigned by my test case writer.

If this flow is channeled correctly, then things will go well. But even if it’s well-channeled, we’ll still need to be concerned on the data that’s being passed into the stubs. Should the data have something that the stubs cannot handle, ie, new test cases with a new set of data, then there’s danger of file corruption via circumvention implemented in a stub.

This is in fact the most damaging possibility when it comes to stubs.

3 - Erroneous exception triggers, causing the tests to fail incorrectly

Lastly, we have incorrect exception triggers.

This is very simple. A function or method with try-catch blocks will usually have 1 or more catch(s) to handle different types of exception. In C++, we’re also discouraged from catching the mother of all exceptions - std::exception.

While this is so, there’ll be cases where we’re forced to catch this mother exception because we simply don’t know what will happen when we call that particular method.

By right, this is bullshit. You should know which method you’re calling and know which exception it can possibly throw and when and how it’ll throw.

Well - yes. You’ll know if you have the documentations. You’ll know if you have access to the source codes.

However, you won’t know if your API is using another API. If such things are not properly documented, then here it goes.

So back to stubs - usually in your codes, you’ll stub out the function that can throw exception and circumvent it accordingly. Here’s an example:

try
{
    DoThis();
}
catch( std::bad_alloc& exp )
{
}
catch( someException& exp )
{
}

In this example, the stub programmer is only supposed to throw 2 types of exceptions:

1. std::bad_alloc
2. someException

However, if the stub throws std::exception then we have a problem. The code did not cater for such an exception to be thrown and therefore has to throw up further.

In terms of unit testing, this unit has failed to handle the exception on its own. Throwing back an exception means that the test unit has failed.

Thus, throwing the wrong exception can lead to incorrect results too. Though, in all, this consequence isn’t anywhere as damaging as others.

Conclusion

Finally, we’ve come to an end to stubs and mock objects. I’ve mostly covered stubs and less on mock objects due to what I’ve experienced.

All I can say is, stubs can make you, but it certainly can break you big time as well.

Stubs are usually needed when it comes to testing static libraries, where its default build process doesn’t include any linking. This causes an effect known as declaration without definition to be sustained without any errors.

The errors that appear later will be known as linker errors - unresolved external link error. Symbol XXX is used in class YYY.

I also hope that the dangers of stubs is more or less (more of less I feel) described in this article. Hopefully it can help someone. Remember, these points are not exhaustive. I’m in no ways representative of any standards that tells you that this is the truth and the only truth.

In my future articles, I’ll talk about other areas in software quality. Hopefully that’ll help you in your development career no matter where you are, which language you code in or whatever.

From here on, I’ll need to take a break to settle into my new digs. I’ll probably return in early April to continue with my Software Quality series.

Thanks for reading my posts! I truly appreciate it!

Thus, have a great weekend ahead. Hope you’ll have a great month of March and I’ll see you again in April!

Signing off,
Jeremy

Series Navigation

«Software Quality - Part 11Software Quality - Part 13 (Backtracking time?)»