Finally! I have some time to post something new here. As time goes by, the challenges I face gets tougher by the day. Remember some time ago I talked about memory leaks that occurs due to design flaws and abuse despite having a good high-level memory management pointer container?

Well, my talk has just caught up with me once more. It seems that the simple case of allocator race leak is no longer that simple. Usually, we’re always on the illusion that the memory we allocate is the same memory that is supposedly deallocated immediately (alike the allocator when sourcing out for RAM).

But no. This is indeed not the case.

Read on to find out why…

Introduction

First of all, this post should be regarded more like a speculation rather than a fact sheet of how things work. The reason is because some of the contents requires proper understanding of the host OS which I admit that I only have limited knowledge.

Secondly, different OS have different means of handling memory. The allocator we’re talking about in this article is also based on the de-facto compiler. What I’m saying is, for Windows, we’ll be referring to a Visual Studio compiler and CRT. For Linux, one can assume GCC and its CRT to be the one.

High-level memory management vs Low-level memory management

When we talk about high level and low level, it’s usually based on the level of control you have on your memory. One can argue that he/she have absolute control over his/her memory. Granted. Only if the number of allocations are minimal and each allocation is large.

A low-level memory management is like managing which block of memory is to be allocated for the requested size. Managing memory to such details are usually for platform-level languages such as Java and her JVM. Have you ever wondered why the JVM takes up a contiguous amount of memory and this amount is usually in large amounts, such as 256MB onwards?

A high-level memory management is like cSmartPtr<T> where its management is dictated by technical design of the object. It doesn’t care how the memory is allocated or when the memory is deallocated from the OS. It only knows when to delete an object when the time comes for it to be destroyed.

Deleting an object doesn’t necessarily guarantee a direct refund of memory. This is especially so for debug CRTs as compared to release CRTs.

This article’s topic isn’t on the difference between debug and release CRTs. But one notable difference is that, debug CRTs usually take up more memory like a glutton compared to release CRTs.

Memory Paging

In Windows, from my limited knowledge, memory paging is used. The exact size of each page is unknown to me, but my guess is around 4k bytes per page.

My second guess in this article is that - the allocator cannot get a memory refund unless all the allocations on the page is deleted. Thus, if there’re 100x 4k pages and that these pages only have 1x 64 bytes of allocation, then for 64k worth of memory objects, 400K of RAM is used up instead. This is indeed memory fragmentation - fragmentation beyond what high level management can do (unless the user creates an array and manages it manually…)

Thus, until there’s a customized solution for game-based applications, who has high volume (rate of construction/destruction per second) of object construction and destruction, the amount of memory taken up will increase gradually no matter how leak-free your application might be.

Possible Solutions

There’re several solutions which are of a similar nature that can help reduce or eliminate the problem completely. I’ll list them in order of complexity (I hope I’m right)…

1. Use limited managed arrays

Although arrays are known to be evil, sometimes if properly employed, can be a powerful ally. I mentioned the use of managed - that’d mean arrays represented by a proper data structure such as std :: vector<T> or cArray<T>.

The crux of this solution is to design the application in such a way that certain objects have a limitation in numbers. Such as projectiles limits, particle (dust, sparks, fire, etc) limits, and other object limits. Different games require different limits, some prohibit limitations due to the way the game is meant to be played. Either way, someone has to make the decision.

The disadvantage of employing this solution is limitation and increases processing load. It’s difficult to calculate which of the array slots will be available during runtime and thus the algorithm will usually place the game object in the next available slot and goes back to index 0 to start searching for one when the index has come to an end.

The search for an available slot is timed at O ( n ) though in my personal opinion, it should be somewhat lesser.

On top of that, the execution update loop will need to iterate all slots to see if they’re NULL or not and then process those that are not NULL.

2. Use a large memory buffer and manage manually

This is a significantly more complex solution which requires a high-level allocator to search, allocate, release and maintain a heap. Thus one requires a heap data structure to properly manage the user-defined memory.

The crux of this approach is to reduce the number of default allocation calls. This in turn will gradually reduce the amount of memory page fragmentation and thus the memory leak itself.

The disadvantages of this approach is that the user have to manually manage the memory buffer. The amount of effort for coding this management system; which may not be reusable or transferable easily without massive recoding, makes it highly undesirable.

3. Implement your own garbage collector / memory allocator

The 4 major C++ operators new, new [], delete and delete [] must be overridden. This applies to all objects, thus templates shouldn’t be in the design considerations. That’s the first step.

Next, the internal allocation mechanism must allocate an page aligned memory buffer that is large enough by estimation to accommodate the user’s requests. On top of that, the system must also expand as necessary.

The way it sounds, it certainly sounds like a garbage collector isn’t it? Well, it certainly is - with the exception that your low level mechanism doesn’t include memory housekeeping. This should be done by a smart pointer container such as cSmartPtr<T>.

The crux is to ensure that memory management is done from bottom-up and top-down as well. Since the memory is pre-allocated, the number of default allocations are minimized. In fact, a well-designed application can even eliminate such memory leaks altogether.

This is certainly the best solution since it doesn’t require any major recoding in order for it to be reused.

Disadvantages? Well, the system is notoriously hard to implement and may take a long time to validate itself. Kudos to those who wrote the Java language. You guys really know your stuffs. Too bad, many of your subscribers are too stuck up to see what you’ve done for them.

Conclusion

This episode that I’ve encountered taught me an important lesson. That is - never to take memory allocation for granted. Understanding the OS is just but a first step to understanding how its memory allocations are done.

A good game developer will at least understand this and know the limitations and thus will not design something that the system cannot handle.

At this stage, I believe this is the end of the spectrum for memory leaks. This topic has been quite a hot topic on stridesdev.org. I hope that this article can raise the awareness of memory allocations among the community. Don’t take it for granted.

Do feel free to correct me should you feel that my guesses are wrong. I’ll be more than willing to acknowledge my mistakes and post them here.

Signing off,
Jeremy

I’ve gone into a retreat within the deepest recessions of creativity. Thus I will not be able to make posts for the next few months unfortunately.

However, during this period, I’ll try to eek out some time for a post if I can help it.

Something’s big is coming up.

All shall be revealed by the end of June 2011.

Regards,
Jeremy

This week, we’re going to talk about OoI and OoD.

Don’t get me wrong, they’re not meant to be read literally.

OoI stands for Order of Initialization

OoD stands for Order of Destruction

The former applies to all programming language with object (or memory buffer) constructs while the latter only apply to C/C++ only. Or does it?

So what’s so problematic about them that leads to software quality issues?

Read on…

Introduction

Programmers usually have the tendency to take certain things for granted. One of the many things is the order of initialization(construction) and destruction.

Naturally some might argue that it’s not truth. Fine. However, it’s hard to say that programmers don’t take other people’s order of constructs seriously.

When this happens, the effects are obvious. Not only objects may not be initialized properly, it may even cause memory leaks in languages that have garbage collectors. This is especially so if one’s codes have a A->B and B->A relationship. This is possible in C++ using forward declarations and is also possible in other languages just as Java or C#.

Problems

How does it happen? A has B - B is the full implementation, child class. A is the class holding B. Thus it can be said that A is composed of B and other objects.

Now, let’s look at B. If B requires a parameter, ie, the parent of A. In this case, in Java or C# it can be the base object class. So if the initialization is like: (In Java)

public class A extends ParentObject {
    private B bObj;

    //constructor
    public A() {
        this.bObj = new B(this);
    }
};

public class B {
    private ParentObject pObj;
    //constructor
    public B(ParentObject obj) {
        this.pObj = obj;
    }
};

At  this point, you’re technically cross-referencing both objects. I’m not 100% sure on how Java garbage collector works but if it’s just a simple reference counting pointer, then there’ll a memory leak still.

Why? Because the instance of A will be lost in its composed object, bObj. Even if A is set to null, it won’t be able to garbage collect since B is holding another reference to A. Remember, if A extends ParentObject, then A IS ParentObject! There’s no difference in terms of base memory address.

In this scenario, unless the garbage collector has additional information to decide that bObj is useless as well, otherwise it cannot, by the designs of reference counting pointer mechanisms, release the instance of A. Thus causing a memory leak.

Problems In C++

Order of destruction problems are compounded in C++ by the fact that you can arbitrarily delete objects at any time. Thus even with reference counting pointers, programmers cannot escape order of destruction problems.

The main reason is the order itself. A lot of programmers like to delegate the order of destruction to the garbage collector, in C++, it’ll be the reference counting pointer container to handle.

The over-reliance on this mechanism easily leads to crashes that doesn’t make a lot of sense. That’s because we’re taught to understand reference counting pointer that there’s only valid or null pointers. Nothing else.

Messing up the order of destruction will overturn this teaching.

Reason? Well, the problems generated by the order of destruction isn’t about null pointer dereferencing (assuming you’ve done your null checks properly). The biggest problem is generated by dangling pointers where certain modules are released already without waiting for the reference counting pointer.

How is this possible?

The reference counting pointer can only cover as far as it could based on allocator and deallocator.  It doesn’t cover things like when a DLL is detached, for example, how can you ensure that every reference counting container is set to NULL?

You can’t.

Thus the only way to prevent this without using the full version of Strides Interactive’s cSmartPtr<T>, is to ensure that the order of destruction is done properly. This require proper design adherence and design declaration in the early stages of the project. I’ll not go into details for these 2 terms just yet.

So, how is it so that Strides’ smart pointer can handle what normal reference counting pointer can’t?

Well, it allows you to terminate an object and ensures that everybody else returns a NULL when checked against.

Order of Initialization Problems

So we’ve covered order of destruction problems. How about order of initialization? Surely there isn’t any wrong with allocating objects right? Does memory limitations play a part?

Well - there’s nothing wrong allocating objects and memory limitations do play a part but in terms of the order, it doesn’t play a major role unless the case is very specific.

We all know that all applications require a specific sequence of initialization. This sequence actually has an underlying order which is categorized accordingly to certain application stages.

Here’s an example of an order of initialization of a typical game:

1) Initialize Windows API to create a form
2) Initialize all devices (renderer, audio, input, etc)
3) Initialize game engine
4) Initialize script engine
5) Initialize game

Usually the first 4 steps are done for you if you use a 3rd party engine. Thus leaving the order of initializing the game to the programmer.

So what happens in a game order of initialization? Let’s look at this example:

1) Load the textures
2) Load the meshes
3) Create the particle samples
4) Create the particle emitters
5) Load data
6) Create entities
7) Create user interface to expose entity properties

If you notice, there’s a certain dependency in the order of initialization. If this dependency is broken, it’ll usually lead to broken implementations and functionality. Thus, it’s important to understand this order and respect the dependencies.

The keyword is respect. That means, to take it really seriously, consult with the technical design and ask the right questions.

Self-awareness

It’s also important to understand your own module and be aware of what’s required for your codes to function properly. With that, you’ll better understand where and when your codes can be initialized.

Sometimes, it’s not a matter with the codes, but rather, a problem with the order of scripting.

“How can a particle be created without a texture?” . These type of questions should be asked. If there’s no answer, then there can only be  the following possibilities:

1) Your module is not applicable to the project at all
2) Your module is completed way too early for any integration to be possible
3) The technical design of the module isn’t clear enough
4) Project management isn’t good enough to optimize your time and skills

Conclusion

Thus, as I sign off, I’d like to stress the importance of the Order of Initialization and the Order of Destruction. You’ll need to seriously respect these 2 nature of things in order for your project to be better off.

Why do I say it’s nature of things rather than nature of programming? That’s because, we’re sub-consciously doing it already. We cannot expect things to happen just like that. There’s a natural progression and a sequence of events that will lead to it.

Normally, we’ll even plan that sequence and hope that things will go our way. And most of the time, it doesn’t go our way, leading to exceptions.

So if we want to go toilet, we’ll first need to walk to it right? Sounds simple? Yes.

Taken for granted? Yes, unfortunately. It may not be just this case, but many other cases. For example, when you order food, do you expect it to come in a timely fashion? Yes you do. But over time, we take it for granted.

When it comes to programming, there’ll be no one to cover us taking for granted and such problems may not be easily traceable as it seems to be. However, once you’ve traced it - there’ll be at least 2 outcomes.

1) You get confused by the design of the flow and head back to the docs for an answer
2) You get demoralized because this is a technically very simple (and possibly stupid) mistake.

Thus, please - have respect for nature and you should be just fine.

Signing off for now. Have a great week ahead!

Regards,
Jeremy

Dear all,

It has been quite some time since I’ve posted. And I must say, I’ve been thinking through about the next set of software quality parts and I do not feel very comfortable posting them here. The primary reason is due to the fact that those articles are highly subjective and debatable. It’s unlike the first 12 parts where all of the components described are actually in practice.

Thus for the next few weeks, since my 2nd set of software quality isn’t ready for release, I’ll try my best to share my other experiences too.

This week, we’ll talk about a small change that can lead to a very irritating development cycle. So what’s that?

Read on…

Introduction

Many a times when we’re coding using 3rd party libraries, sometimes these assholes actually have a time limit as to when you can or cannot use it.

Thus once this time limit is reached, your application grinds to a halt.

So what’s next? Well…. either you get the full version or you try to set the date back to “give” yourself more time. Well-protected shareware libraries will tend to guard against time back tracking to prevent such scenarios from happening.

However, others may be kind enough to allow you to do that.

So you’ve managed to change the time. The project builds fine and it runs alongside the 3rd party as though as it’s still under evaluation. What’s the problem then?

Problem

The bloody problem occurs when you back track time. In IDE (Integrated Development Environment) such as Visual Studio, it’ll be kind to check your object files as well as your executable to see if they need to be rebuild. This is also known as the “up-to-date” checker.

So what happens when you try to build again? The IDE will rebuild virtually, your entire project. Don’t think it’s trivial. In a standard project size, around 300,000 lines of codes, this can be a nightmare scenario.

You’ll start to wonder what on Earth have your IDE gone wrong and probably the first thing you’ll do is to restart your machine.

Guess what? It doesn’t do a thing!

Thus you’ll have to endure your project rebuilding every time even when you didn’t make ANY changes at all!

Worse of all, you may have forgotten that you’ve changed your date or worst of all, you don’t even know you changed your date.

This problem is very apparently in the older Windows XP where if you open the calendar in an impromptu manner, ie, a singe click on your date time and then you select a date some time before, this becomes the set date.

Conclusion

This is a very short article by the way. Hopefully it’ll ease the pain of having to read so much. I’ll probably do the same over the next few weeks for my articles too.

Anyway, the conclusion is very simple. You’ll have to manage any time changes if you ever must change your system date. If not, this should be one of the things you should look out for if your solution keeps rebuilding itself even when there isn’t any changes introduced.

Everybody have a great week ahead! Signing off for now!

Regards,
Jeremy

Hi all,

This week’s post comes a little later than usual because I’m not satisfied with the quality of my draft. After some serious thoughts and revamps, I’ve managed to rewrite the article to better serve this week’s topic - Dangers of using stubs.

By the end of this post, we’d have covered most, if not all of the basics in Software Quality.

We’ll then move on (continuing on the series) to other parts of software quality processes which can help you in your daily development or higher level processes.

So what’s so dangerous about stubs? Let’s find out…

Introduction

When it comes to stubs, it can come from at least 2 areas.

1. Unit Testing
2. Runtime Testing with stubs

We must also remember that stubs are temporary measure to fulfill a prerequisite and should never be considered as a means to an end. This being said, there’re also some not-so-obvious mechanisms hidden when instrumented stubs are used.

In most cases, there’ll always be a safety barrier when it comes to stubs.

Safe Stubs

We’ve mentioned this type of stubs in our previous article. What are they used for? As mentioned in my previous article, safe stubs are usually either generated automatically or pre-generated to intercept certain potentially dangerous function calls.

What it does basically is to return a failure value, be it a return code or NULL pointer, it’s main purpose for doing that is to protect the caller from accidentally or unintentionally making those calls which leads to disastrous results.

In a good stub framework, many of these dangerous functions will be stubbed out automatically as safe stubs. Here’s the following to name a few:

1. f_open()
2. f_write()
3. f_prinf()

* Arguments are omitted because I’m lazy…
** The functions have a’ f_’ instead of ‘f’ because of a problem. Most host applications use the ‘f’ variant while some embedded systems uses the ‘f_’ variant.

Just take f_open() for example, if the following code is executed in original definition (ie, without being stubbed out), what will happen?

FILE* fp = f_open ("abc.txt", "w") ;

This piece of code may be a file open operation, but in reality, it actually overwrites the existing file (if any) without any warning and set it to 0 bytes.

Imagine 100 test case, data driven, each with a different filename, some sensitive, executing without safe stubs. True, some files may be protected by the OS, but ask yourself, what are the chances that one of these files isn’t?

It may not be an OS file. It could be something important to you or to your company.

Now, this is a danger that occurs only if safe stubs aren’t used. Evidently, we’ve seen safe stubs in action automatically so what’s the problem?

Circumvention

This word has been mentioned several times last week and as explained, this is a useful way to control the stub. A good stubbing framework gives the user the ability to do at least the following below:

1. Controlled return value
2. Bridging to original definition by conditions
3. Exception testing

All these 3 have their own dangers. However, before going deep into it, we must first understand that, circumvention is usually done by choice. Thus this increases the risks of introducing errors into the stub itself.

Out of these 3, the 2nd point is the most dangerous of all. Why?

Reality vs Stub Limbo vs Actual

Instrumented stubs or hand-made stubs all have a common characteristic. That is, they both sit in between the reality and the actual execution.

Let’s first talk about reality and actual execution, what’s the difference?

You can understand reality as the program execution during the run while actual refers to the original definitions of other functions.

The stub itself serve as a limbo. Which means to say, it can alter reality by preventing the program flow from ever reaching the actual execution.

Since it’s in a limbo, users can control the end state of the stubs, leading to a few possible dangerous scenarios

1. Erroneous stub returns causing incorrect or unstable program flows
2. Erroneous stub bridge, causing the wrong data to be passed to the original definitions
3. Erroneous exception triggers, causing the tests to fail incorrectly

Let’s talk about them one by one.

1 - Erroneous stub returns causing incorrect or unstable program flows

Stubbing requires full knowledge on the unit itself. Without sufficient or full knowledge, the stub writer will not have a clear picture on what not to return.

Of course, one can defend that this is meant to test for the unit’s robustness. However, we must be careful with such tests. Even with techniques like defensive programming (which I’ll talk about soon), there’s a limit.

Simply put, we don’t test things that can never happen. If you force things to work unnaturally, even if you managed to write up a test suite, the stubs will end up triggering all the incorrect behavior.

Naturally, one may put the blame on the application design. But - that’s where the limit comes in. Going in with erroneous stubs is a big step towards test failures.

2 - Erroneous stub bridge, causing the wrong data to be passed to the original definitions

This is caused by incorrect test case identification or a lack of communications between the test case writer and the stubs programmer.

The case in point is when functions like f_open() has a certain requirements to really be opened and most of the time be a safe stub. This means, that, one, by circumvention, I go around my original safe stub. Two, I channel my stub flow according to the test case ID assigned by my test case writer.

If this flow is channeled correctly, then things will go well. But even if it’s well-channeled, we’ll still need to be concerned on the data that’s being passed into the stubs. Should the data have something that the stubs cannot handle, ie, new test cases with a new set of data, then there’s danger of file corruption via circumvention implemented in a stub.

This is in fact the most damaging possibility when it comes to stubs.

3 - Erroneous exception triggers, causing the tests to fail incorrectly

Lastly, we have incorrect exception triggers.

This is very simple. A function or method with try-catch blocks will usually have 1 or more catch(s) to handle different types of exception. In C++, we’re also discouraged from catching the mother of all exceptions - std::exception.

While this is so, there’ll be cases where we’re forced to catch this mother exception because we simply don’t know what will happen when we call that particular method.

By right, this is bullshit. You should know which method you’re calling and know which exception it can possibly throw and when and how it’ll throw.

Well - yes. You’ll know if you have the documentations. You’ll know if you have access to the source codes.

However, you won’t know if your API is using another API. If such things are not properly documented, then here it goes.

So back to stubs - usually in your codes, you’ll stub out the function that can throw exception and circumvent it accordingly. Here’s an example:

try
{
    DoThis();
}
catch( std::bad_alloc& exp )
{
}
catch( someException& exp )
{
}

In this example, the stub programmer is only supposed to throw 2 types of exceptions:

1. std::bad_alloc
2. someException

However, if the stub throws std::exception then we have a problem. The code did not cater for such an exception to be thrown and therefore has to throw up further.

In terms of unit testing, this unit has failed to handle the exception on its own. Throwing back an exception means that the test unit has failed.

Thus, throwing the wrong exception can lead to incorrect results too. Though, in all, this consequence isn’t anywhere as damaging as others.

Conclusion

Finally, we’ve come to an end to stubs and mock objects. I’ve mostly covered stubs and less on mock objects due to what I’ve experienced.

All I can say is, stubs can make you, but it certainly can break you big time as well.

Stubs are usually needed when it comes to testing static libraries, where its default build process doesn’t include any linking. This causes an effect known as declaration without definition to be sustained without any errors.

The errors that appear later will be known as linker errors - unresolved external link error. Symbol XXX is used in class YYY.

I also hope that the dangers of stubs is more or less (more of less I feel) described in this article. Hopefully it can help someone. Remember, these points are not exhaustive. I’m in no ways representative of any standards that tells you that this is the truth and the only truth.

In my future articles, I’ll talk about other areas in software quality. Hopefully that’ll help you in your development career no matter where you are, which language you code in or whatever.

From here on, I’ll need to take a break to settle into my new digs. I’ll probably return in early April to continue with my Software Quality series.

Thanks for reading my posts! I truly appreciate it!

Thus, have a great weekend ahead. Hope you’ll have a great month of March and I’ll see you again in April!

Signing off,
Jeremy

There has been some weird issues going on now that I can’t upload Software Quality part 12.

Please hang on while I try to resolve the issues here. Will share this article asap. Sorry for the delay.

Regards,
Jeremy

Hi all,

This week we’ll talk about stubs in greater detail. We’ve seen how stubs can be a useful helper and there’re many types of stubs as well.

We’ll also need to cover some hidden dangers of stubs and mock objects as well.

So what’s so dangerous about stubs and mock objects anyway? Let’s find out.

Introduction

In this article we’ll talk about several aspects of stubs and how we can make full use of them to our full advantage.

First of all, as explained in my previous posts, stubs and mock objects are temporary assets which help bridge functionalities or mock state representation of a certain module.

We’ll talk about how much efforts should we invest in such temporary assets, what types of stubs and mock objects do we have and the dangers involved in using such assets.

Efforts vs Effects

Looks familiar isn’t it? This is the name of one of the post that was published a long long time ago.

When it comes to manual processes, stubs and mock objects stand in a single line. That’s because, if I can implement a mock object, I might as well do the stubs inside the mock object so that I can save time for others.

That’s a pretty standard and likely clever way of doing things. The only problem is - the code has to know of the existence of your mock objct.

Thus the amount of efforts come not just from coding the mock object and integrating it but also from maintaining the mock object integrations.

For example, you’ll have to account for how many integrations to this mock object is done so that you’ll know where to remove them when the project is finally ready for SIT or UAT. Very likely SIT (System Integrated Testing).

When you actually remove the mock objects from integration, you’ll begin to feel that it was such an inefficient use of time. Objects that are coded with care are now poised for removal from project, possible complete deletion once the project finishes.

And the effects? Well - it served it purpose more than less.

One more thing we’ll have to take in mind is that if we’re doing C++, then extra care must be taken when coding this object so that it doesn’t cause memory leaks or stub-triggered memory problems.

It’s like - nursing a pet from a young till its teen years and then slaughter it for no apparent reason or usage. That hurts.

Stubbing/Mock Object Framework vs Manual Efforts

Naturally, a stubbing or mock object framework can prove to be very useful in terms of helping the stub/mock object programmer to create such assets with ease. However, for such a framework to be viable, it must at least have the following qualities:

1) Ease of use
2) Visibility of what’s stubbed, what’s not
3) Ease of management
4) Auto-integration
5) Does not TAINT the original source codes

Let’s talk about all the 5 qualities.

Ease of Use

Naturally, this is one of the expectations when it comes to stubbing frameworks. However, the notion of “ease” can be subjective. For one, “ease” could be a code snippet generator, while for others, “ease” could be as complicated as a stub management system.

Visibility

Anyone who does stubs will be concerned as to which method is being stubbed out or which objects are passed in as mock objects. This is to be able to know the exact ingredients used for a particular white-box test case so that the end results will be correct still.

Ease of Management

Naturally, once the locations of all the stubs and mock objects are known, the next step will be the ease of accessing or removing them in a list. Thus removing the need for stub programmers to write/remove/search for declarations and definitions manually.

Auto-integration and taint-free source codes

This is one of the hardest things to do in stubs. Auto-integration means that quite a lot has to be done. There’re several approaches here but in my opinion, the best auto-integration will be a seamless one, ie, integration without any changes to the original source codes.

This is only possible with instrumentation.

Stub Types

Now, let’s put mock objects down and go into stubs itself. As we know, stubs by definition are nothing but methods returning a dummy value.

However, even so, there’re several types of stubs. Namely:

1) Original
2) User
3) Safe
4) Virtual

So what are they?

Original - Not stubs. It’s really just original source codes
User - User-defined stubs, the ones that you’ll write
Safe - Functions that are deemed to be dangerous by the stubbing framework will automatically stub them out for you
Virtual - It’s to indicate that this stub is based on a virtual method call

A good stubbing framework will allow several types of tests to be done. They’re namely:

1) Backdoor testing via circumvention
2) Specific return value via circumvention
3) Controlling who gets the stub value, who gets the real value
4) Exception testing

The word ‘circumvention’ appeared twice. So what exactly is circumvention? For the benefit of those whose command of the English language may not as good as others, circumvention essentially means going around.

So what are we going around? Well, the question is answered by why are we doing stubs in the first place. The answer? White Box Test Cases!

Let’s take a look at the 4 types of testing.

Backdoor Testing

I’ve mentioned these type of testing many times in my previous articles. The reason for this type of testing to exist is because of the inherent limitations of white box testing. As we know, white box testing primarily involves methods that allows arguments to be passed in.

Thus, methods that pull values using the Pull Approach will be nearly impossible to test.

The solution to this problem is backdoor testing and the ingredient to this solution is stubs.

Specific Value via Circumvention

This method is closely tied to backdoor testing with the exception that the former is done because there’s no other choice. The former is done because the original codes cannot reproduce the values required for the test.

Such scenarios are common among pull approaches that pull values from the hardware. A good scenario will be the commercial jetliner autopilot system. This system makes use of readings from the instruments to make certain decisions, ie, to ascend or descend.

Thus, if there’s a test to test such a condition, it’ll be impossible to just simply power the hardware on and hope that the plane can be simulated at 15000 feet. This is provided that the hardware even comes with a simulator or simulation device.

Remember here, we’re testing codes, not integration here - thus if real hardware could be avoided, they should be avoided.

Thus, circumvention comes in handy.

Stub Return Value Control

One of the most important aspects of stubs is to be able to return a different return value based on a certain predicate. In terms of white box testing, the most direct predicate will be the test case name. If registered correctly, the test case name can be used to check if the test runner is running a particular test case.

Once this condition is fulfilled, the stub will then be able to return the desired value.

It can also reconnect to the original source codes and return the original value instead if desired.

Exception Testing

One of the hardest aspects in unit testing is to trigger an exception. As we know, exceptions are thrown for unexpected behavior. Therefore, there’re 2 ways to generate this unexpected behavior.

1) Create the scenario to generate this behavior, ie, data reading from thumb drive and you pull it out to trigger an IO Exception.
2) Use stubs

Naturally, the use of stubs is a more enticing proposition than to physically test the code via manual means. This usage is naturally evolved from specific return values and stub control.

Based on test case predicates, the stub can actually check and throw exceptions at certain test cases to simulate an exception scenario where the catch handling code comes into play.

On top of that - it allows you to see how the exception is being handled and how much coverage is derived from it.

Conclusion

Again, it has been a lot of information this week. I’m afraid I’d have to leave the dangers of using stubs behind till next week.

One thing I’d like to note. The stubs we’ve been talking so far all belong to the same stub approach known as Method-Specific Stubs.

There’s another stub approach known as Test-Specific Stubs.

The differences are -
1) Method specific stubs allows pipe-blockage of all the method calls to the method it stubs out
2) Test specific stubs only work for the test case it’s tied to

We’ll however, not talk abt test specific stubs.

That’s it for this week. Please have a great weekend ahead!

Signing off,
Jeremy

Dear all,

Time to move on to the next topic in Software Quality! Today, we’ll talk about stubs and clear up the myth between stubs and mock objects.

Stubs are commonly used a temporary gateway to tackle missing implementations. The question is how is this gateway being implemented. Many a times, people might as well just do an entire class to mock the entire functionality. Well… so is that a mock object or stub?

What are their differences?

Read on more to find out…

Introduction

Remember in my last article, I mentioned about Neo in the Matrix, being stuck in a train station, sort of like a limbo to nowhere. The only person that can access that place is the train master.

This is by right the proper way to do stubs and mock objects.

Most of the time, our codes have to be modified in order to accomodate the stub or the mock object that implements the stub. That code modification is considered to be undesirable since it increases the need to keep track of the changes so that it doesn’t mess up with the final build.

Nobody would want to have a mock object integrated into the final version of the build.

However, in order to achieve modification-free stubbing, you’ll need to instrument your codes. Thus there’ll be a need to use a certain tools out in the market that can help you with this. There’s certain known framework out there that can already do it flawlessly, I’m not at liberty to reveal them though; you’ll have to find out those tools yourself. All I know is that there’re existing solutions, some free and some paid.

Prerequisites

Even stubs and mock objects have their own prerequisites. I’ll mention them here so that you won’t need to fall into the same trap.

1) Knowing the difference between static libraries and executables
2) Understanding declarations and definitions
3) Knowing the difference between transient and persistent
4) Knowing the limitations of white box testing
5) Pull-approach design

Static Libraries and Executable

Static Libraries and Executables are handled differently during their respective builds. The biggest difference is the final part where static libraries are archived, executables are linked.

Dynamic linked libraries and shared objects are considered executables.

The difference between the archive and linking process is that, the linker searches all the definitions required and reports a linker error with errors like undefined symbol used in certain codes.

Why are we so concerned about this? The reason is very simple - if you’re working on an Application or Shared Object/DLL project, you’ll know what are the symbols that could possibly be missing simply by building the project.

However, if you’re working on a Static Library project, there’s a chance that there might be some functions or classes that has declarations, but no definitions.

Static Libraries can avoid undefined symbol problems because the symbols search is deferred till it’s linked onto an application that uses it. Thus the linker errors won’t appear only until you link and static libraries don’t link!

Declarations and Definitions

This is really going back to the basics. But I’ll make it short. When you declare a method, it’s usually in a header file and it has no body unless it’s inlined. The method body is usually in a cpp file and that body is known as the method definition.

This definition must always be around during linking so that the linker can resolve the missing symbols.

Transient and Persistent

This are really 2 IT words that ring about across the industry. But I’ll make it very simple here as well.

Transient - No state, no variables and it comes and goes, such as a function.
Persistent - Has state, has variables and have a lifespan and represents a certain behavior, such as an object.

Limitations of White Box testing / Pull-approach design

As mentioned in my previous posts, white box testing is poised to test methods that pass in arguments. But it doesn’t seem to cater well to methods that call other methods to get values.

This is known as the pull approach. Instead of working on the values passed into itself, it works more independently by drawing values from other areas. For example, reading from value from a piece of hardware.

Thus it’s important to know that there’ll always be some people that uses the pull-approach methodology. This will also affect the reason as to why we’re using stubs as well.

Differences between Mock Objects and Stubs

Let’s talk about the former - mock objects.

Mock Objects and Mock Instances

First of all, it’s an object that has state and some degree of behavior.

A cousin of mock objects is mock instance. A mock instance is based on a fully implemented object type.

A mock object may not necessarily be fully implemented.

Stubs

Stubs by itself, in definition, is simply just a method that returns a dummy value.

A stub may not necessarily be part of a mock object. It can also be part of a half-implemented class as a temporary measure to satisfy technical build requirements.

In any case, it serves more of a temporary bypass and more importantly, it doesn’t by default contain any state. That being said, it simply implies that it doesn’t represent an entire object.

However, we can make changes to it. Such as creating a static instance of a complete object and then return it from a stub.

The whole setup is still a stub-return, but it now can hold state.

That’s probably the reason why there’s so much confusion between a stub and a mock object.

Moreover, a mock object doesn’t require a stub function/method to return an instance of itself. It simply exists as a dummy implemented or half-implemented form and definitely can hold states.

Conclusion

In conclusion, we’ve spoken on the prerequisites of understanding stubs and mock objects, we’ve also discussed on their differences and their usage.

In the next article, I’ll talk about the different types of stubs and when or how each type of stubs should or should not be used.

One of the key concerns or consideration regarding the dummy world or stub world, is the amount of modifications on the actual source codes required for stubbing and mock object to take place.

We’ll also talk about the effects of stubbing on coverage as well.

Till then, have a great week ahead!

Signing off,
Jeremy

Dear all,

As promised, today we should be able to wrap up coverage completely and call it a day. Coverage has been a big topic as we all know and has so far taken up quite a few posts for the last few weeks.

Today we’re going to discuss the last coverage metric - Modified Condition Decision Coverage (MC/DC).

What’s so important about this coverage? Who should pay extra attention?

Let’s find out!

Introduction

Modified Condition / Decision Coverage is the hardest coverage type to understand and then to master. There’re some prerequisites in order to draw out its benefits fully.

One of the important prerequisites to have is truth table and understanding of binary. The next most important prerequisite to have is understanding the order of evaluation. I’ll explain in further detail as we go along.

MC/DC is used to test all possible permutations in a single condition block within a single method or function.

To get 100% coverage for a single condition, you’ll need to fulfill all of its conditions via a “truth table”. I’ll explain later on how to create this truth table and then eliminating the impossible truths via order of evaluation.

I’m not too sure about its history but I’ve heard that this metric came up after a serious incident occurred to a fighter jet, reportedly a F-16 variant. The cause seems to be due to the instruments blacking out for no apparent reason.

Is 100% MC/DC coverage possible?

By using unit testing in a single thread, it’s possible to achieve 100% MC/DC coverage. Unit testing falters when it comes across multi-threaded applications. The most common flaw it has is inconsistent results from one run to another due to the lack of deterministic execution time for threads.

Even so, attaining 100% MC/DC is an arduous process. On top of that - no tools can automatically generate meaningful test cases that will cover MC/DC coverage. Any coverage is almost certainly coincidental. It’ll be good enough to have tools that tell you what are the parts of the condition truth table that is being covered so that you’ll know what to test next.

Basically, the tool knows what’s being tested or not, but it doesn’t understand the significance of the predicates. That means, it won’t be able to understand your sub-condition and determine which value or which stub to use to trigger that predicate.

On top of that, the road to achieving MC/DC also means another thing; the codes must be stable enough and shouldn’t be in full development by then. The reason is simple -if a programmer changes the condition one day and patches another condition another day, the testing team will go crazy.

There’s a lot of work involved in getting all the permutations drawn up. Since there’re no known tools out there that can display the possible permutations via a truth table, it’s usually up to the tester to draft out a test plan for MC/DC.

By the time the test cases are generated, the tester would have understood the code well enough. That’s how much work it requires.

MC/DC Truth Table

What’s a MC/DC truth table? Let’s take a look at this simple example here:

if( a < 2 && b > 3 )
{
}

First of all, you’ll need to calculate the worst case scenario for your tests. The simplest and least accurate way to get a rough worst-case scenario guide is to take 2 power of N conditions.

In this case, there’re 2 conditions. Therefore it’s 2 power of 2 - giving you 4 test cases minimum. So take a look at this truth table:

Looks like the case right?

Well - no. Here’s where the 2nd prerequisite applies - Order of Evaluation.

As we know, C/C++ doesn’t blindly evaluate every single sub-condition. It depends very heavily on the comparison operator used and reacts accordingly. This is done mostly for safety and bits of optimization as well.

Thus if we apply order of evaluation to this case, the final truth table will be :

So what does “X” mean?

Based on the order of evaluation, if the first condition is false and the comparison operator is &&, then there’s no point to continue on. Thus, the outcome of the 2nd sub-condition doesn’t matter if the first one is false.

Please note - this whole exercise is only for a single condition in a single method belonging to a single class or namespace.

If there’re 10 conditions inside a method, you’ll have to repeat it 10 times. On top of that, you’ll have to get through the conditions above in order to allow the conditions below be hit.

Coverage In Conclusion

Finally we’ve come to the end of coverage. For the last few posts, we’ve been talking about different coverage types all day long. Some may wonder - is there a way to combine all the coverage together and generate a single report.

Let me tell you - it’s not a good idea to do it that way. The reason is very simple - not all coverage types combine well. Generating a single report may be good enough for a stamp of approval but it could also at the same time, be very confusing as well.

In such a case, it’ll be better to generate a report based on module or even based on granularity as low as a class itself.

This allows every component to be tested independently before being put together for another round of testing. On top of that, the tests and test results are well-documented and is easy to maintain.

I hope that I’ve managed to bring out the importance of having code coverage, whether it’s recorded via unit testing or recorded via runtime testing, having a coverage report is better than having none. More importantly, this coverage report should tell you where you’ve managed to test and which parts still need more tests to penetrate.

In all, coverage can be seen as a statistic, a visibility matrix or an indication of a possible technical, code  design or business design flaw.

Thus we’ve concluded all topics for coverage. If there’s any questions, feel free to leave a comment.

Conclusion

In conclusion as a whole, we’ll move towards the backdoor of unit testing. Also known as stubs. However, there’s a myth surrounding stubs. Are stubs and mock objects the same?

Do they share state or store states?

Well.. I’ll answer that next week in Software Quality part 10. Meanwhile, a thought for you to ponder on. Remember in Matrix Evolution, there’s a train station where Neo was stuck in? Sort of like a limbo but it’s a clear physical area rather than a subconscious construct. Try to imagine stubs as the train station.

Have a great week ahead!

Signing off,
Jeremy

Hi all,

This week, I’ll continue to describe the last 2 types of coverage which falls under condition-based category. Just a recap, last week we’ve covered line-based and snippet-based categories of coverage.

Why didn’t I finish off the post by covering the last 2 coverage types? First of all, they’re significantly more important than the rest of the coverage types. Secondly, it’s not as straightforward as any of the coverage types.

To be specific, today I’ll just cover Simple Condition Coverage only due to the amount of explanations and materials required for the details. I’ll cover MC/DC next week. So, please bear with me till then.

So why are they so special? Read on….

Introduction

Condition-based coverage types are the ones that can tell you the most without revealing too much of itself. We’re going to talk about the following coverage types today.

* Simple Condition Coverage

The more popular coverage type is probably Modified Condition / Decision Coverage (MC/DC), especially in the aerospace industry. There’re standards out there such as DO-178B that mandates 100% compulsory MC/DC coverage for all unit tests. This is to minimize, if not, eliminate all possible software problems which may lead to cases like instrument blackout which can be fatal on board a commercial jet liner.

Ironically, it’s also the hardest coverage type to master and understand completely.

We’ll discuss the technique towards understanding MCDC next week.

Simple Condition Coverage

Simple condition means as it is literally. However, not all conditions are as simple as it looks. For example:

...
if( a > 0 && b > 3 && c < 2 && d == 5)
{
}
else
{
}
...

It looks like a complex condition right? Well yes. On top of that, there seems to be 4 different cases right? This by right leads to a case of 2^4 which is 16 test cases right?

No.

Remember, this is simple condition coverage. A simple condition is derived from the final binary outcome of the condition block. That means for this statement:

if( a > 0 && b > 3 && c < 2 && d == 5)

Can only either have an absolute positive or negative path based on the evaluation of all the sub-conditions along with the order of evaluation. The final outcome is known as the simple condition predicate.

In order to have 100% Simple Condition coverage for the previous line of code, both positive and negative blocks must be executed at least once.

Thus the user may be required to write at least 2 test cases or more, depending on the complexity of the sub-conditions, in order to achieve 100% simple condition coverage.

Speciality of Simple Condition Coverage

In tools that demarcate Simple Condition Coverage, the good ones will only mark a condition with both simple condition outcomes executed as covered. It’ll also mark the condition as not covered if the test cases fail to reach the condition in any scenario.

There’s however, a 3rd state. This is the only coverage type that has a 3rd state color. Let’s assume that if covered is green, not covered is red. This 3rd state will either be yellow or teal.

What is this 3rd state?

It means that after executing all the test cases, the condition will only go into a positive or negative branch only. For example, if I have 100 test cases that ran, and after the tests are done, the condition has only so far executed either the positive or the negative branch.

This is a warning sign to a few possible problems. I’ll list a few of them.

1) Possible unreachable codes due to one-sided condition(s)
2) Possible short-circuited logic
3) Possible code design flaw
4) Possible business design flaw
5) Possible lack of test case robustness

We’ll just talk about the first 2 items listed.

Possible Unreachable Codes due to one-sided condition(s)

These type of codes may exists due to volatile changes made to other parts of the codes which inadvertently led to a one-sided effect.

The simplest example of all would be:

if(true)
{
}
else
{
... //this branch will never be executed
}

Now, let’s think again. Could there have been some parts of the code which, under certain circumstances, return a constant value? Take this piece of code for example:

if( CheckIsSystemOn() && CheckIsSystemUnlocked() && CheckMemoryStatus() )
{
}
else
(
//hard to tell whether if this block will be executed right?
)

It’s hard to tell yes unless we check all the 3 function calls. If all 3 function calls return a constant value, regardless of true, false, zero or non-zero, it’ll cause a branch to be blocked off always.

Usually, we’re advised to put an else block as a “guarantee” or condition-block or condition-sealer to avoid logic problems later on. However, when else block is used in an overly conservative manner, there might be a case as described above. It’ll really look stupid to tell people that you’ve done so much as a contingency plan only to realise that it’ll never be executed.

Possible short-circuited logic

This is only available in programming languages that allow short-circuited logic to occur. C/C++ is definitely one of them.

Short-circuited logic occurs when a non-comparison operator is used instead. For example:

if( a & b )

instead of

if ( a && b )

Short-circuited logic are common and is used for certain specific purposes in many applications despite strong advice to avoid it. Alike pointers, short-circuits can only be used when you’ve attained a certain level of mastery in the language of your choice. Otherwise, you’ll end up busting your own codes with logic outcome that you’ll never even thought will happen.

In C++, a very common form of short-circuited logic comes from the comparison of pointers to see if they’re null or not.

Instead of checking for null like:

int* a = NULL; //from somewhere
if( a == NULL ) {}

we’ll do something like:

int* a = NULL; //from somewhere
if(!a) {}

If in any case, should the code above really be true, then you’ll have a partial simple condition coverage scenario since pointer a will always be null. The opposite occurs if you didn’t assign anything to pointer a. In this case, the pointer will be a non-zero random address. Thus if(!a) will always be false.

That’s probably all for Simple Condition Coverage. It should be quite straightforward to understand other than the special cases.

Conclusion

That’s another rather heavy load of information to digest. I’ll need your mind to be clear for MC/DC next week. After that, I’ll wrap up the topic of coverage and move on to stubs as promised.

Simple condition coverage is actually very easy to understand. The final outcome of every condition will either yield a true or false. For a complete coverage, both cases must happen during the tests. Otherwise, gaps will appear in results, including the middle case where the condition is either always true or always false.

The effect of unused codes is quite minimal but is also subjected to the project size. The bigger the team, the bigger the project, the lesser the discipline, more unused codes will creep in due to all kinds of reasons and excuses.

When this happens, your initial heap consumption will be so high that you’ll be wondering what went wrong. Thus Simple Condition Coverage can help remove unused codes as well.

That’s about it for today! I’ll write up the next post earlier so that I can post it early by 10am SGT instead of 10pm SGT.

Signing off,
Jeremy