Strides Interactive

Testing tools - a bane or a blessing?Filed Under: Weekly Tuesday Dose of goodness

Hi all,

This week I’ll talk about some of the feedback and experience I’ve gotten as a senior technical consultant.

I’ve been working with testing tools for at least 3 years already and have understood their value to management and usefulness to developers.

However, I’ve also received feedback such as, testing tools become a bane to their self-improvement or even their jobs.

Let’s find out if this is true.

Introduction

It seems that my articles nowadays have introduction and stuffs like that right? Yeah, I’d prefer my articles to be more organized nowadays. Otherwise I might have a problem of not knowing when to stop! ;p

Testing tools has been a part and parcel of software development. Like it or not, most of these tools do provide value to their respective users and in most cases, if proper tools are used, the effect could be extraordinary.

This contrast cannot be appreciated by developers who’ve used tools right from the start at the beginning of their programming careers. The reason is simple, if you haven’t seen the ugliest, then you’ll never know what’s truly ugly. Vice-versa for beauty.

Therefore, is it a bane or blessing? Let’s go into the tools themselves.

Testing tools - in categories

We all know that software development isn’t just about pure coding. It’s also about a myriad of other requirements which require other tools to test. But all these requirements are sometimes mixed with technical requirements or they’re inadvertently linked together due to the nature of the business requirements.

Testing tools come in various categories though. The good one may come as a solution, encompassing one or more categories of testing. But anyway, let’s go through a few of them:

Static Analysis

The most basic tests of all is to conduct a static analysis simply by reviewing your source as it is. However, this category of test has always been misunderstood or abused by marketing personnel. How so?

Yes, Static analysis does look into your codes. But, the real thing look into your preprocessed codes, not just your text codes. Preprocessed codes simply mean the final outlook of your codes before it’s being compiled into its object code.

In C++, that’d mean replacing all your #defines and macros as well as #if and #endif to pick the selected code branch for the final version of the build.

In Java, well.. if I’m not wrong, it also means to replace your little string concatenations into monstrous lines of new Strings(), ie, String str = “abc” + “bde” + “abaa”;  Processed codes? String str = new String( “abc” ) + new String(”bde”) + new String(”abaa”);

It’s very different from what many had thought it was - a text checker. Come mon, coding standards can be picked up easily by IDEs in most cases. What we’re talking about here are usually things that cannot be detected by the compiler. Although this point is disputable, my point is that Static Analysis isn’t as simple as many think it is.

Data Flow Analysis

So if static analysis gave you a lot of false positives or aka as maligning violations, then here comes Data Flow Analysis.

It’s still static, it’s still checking on your preprocessed source codes without execution. It uses keywords such as if, else, switch, case, do, while, for and so on in order to discern data flow. It then makes an assumption on the positive and negative or case outcomes to simulate flow.

Once in, it’ll detect certain possibilities of a bug, such as division by zero, null pointer dereferencing and so on. One of the biggest advantages it has is the ability to find API-based resource leaks.

One good example would be java.sql.Connection leakages.

Unit Testing - White Box Test Cases

A tool should also be able to automatically generate test cases for stability testing.

Let me list out a list of unit testing techniques:

1) Stability testing - to test how stable your codes are, how robust they are and how they handle exceptional data
2) Russian Roulette testing - It’s a roulette form of testing. The number of test cases are kept small, but the data that are passed in changes rapidly.
3) Use Case testing - Testing source codes by passing in meaningful values (positive, negative and exception)
4) Data Driven testing - Encompasses all the above with the exception of being able to use a data source such as a CSV file, Excel spreadsheet or Database to provide the test data.
5) Coverage testing - Tests how much codes are covered by the design and how much more are there left to test. It also reveals possible short-circuited logic and unused private/protected methods.
6) Stubbing - A gateway between your source codes and the actual original function call. Used primarily to substitute implementations that are yet to be completed, it’s also used as a backdoor approach to test codes that not easily tested by normal test cases.
7) Regression - By asserting the state of the entire class before and after each call, one can easily ascertain the workability of the class and its reliability with ease. This however requires user data and this data should be contained within a range (defined by the design). A regression error in codes would almost certainly mean a bug and possibly a cascading fan effect that is occurring across the project.

With that, let us begin to debunk some of the myths out there:

Myth #1 : With all these tools to help, does it mean that a developer or tester is out of job?

Well… the answer is of course NO!

I’ve heard a few telling me that testing tools will eventually replace their jobs and things like that. Let me tell you now, that’s complete BULLSHIT!

If a testing tool can replace their jobs, they deserved to be fired!

Remember, a testing tool can only help and not replace an intelligent being such as a programmer. The tool cannot derive logical values for you. It can only derive heuristic, boundary and null values for your test codes at best.

The reason is simple, no tools can understand your design! If it could, then the tools will sell by itself and I’ll be the first to be out of job!

Myth #2 : With these tools, my programmers will now depend on them and become lazy.

Again, let me debunk this myth by stating a very obvious tool that ALL programmers out there use and still become lazy. What’s that tool? The COMPILER (and LINKER for C/C++)!!

Don’t tell me none of us depend on the compiler to parse our codes and ensure that there’re no syntax errors in our codes? That’s again, complete BULLSHIT!

We depend on our compilers to tell us where we went wrong in our codes, both in errors and warnings. Tools offer to report violations in a higher perspective, in terms of language facilities, usage and standards. In terms of best practices and top tens (such as OWASP ten 10 security vulnerabilities).

We already depend on our compilers to lessen our syntax errors. What shame does we have if we depend on the good tools that help lessen our bad practices altogether?

How did we learn to lessen our syntax errors? Via compiler errors!

How did we learn to improve our coding standards? Via good static analysis tools!

So beat it! We’re already lazy, let’s stop being slaves to our own codes and stupid mistakes!

Myth #3 : With these tools, I’ll have more work to deal with than I did previously

Well, this is a common case of a shallow and short-sighted developer whose aim is solely to reduce the amount of work assigned to him/her.

Unfortunately, the more work is true. But true on the surface.

If a piece of code has passed User Acceptance Tests, does it mean that it won’t fail during Production?

No.

Is there any visibility or assurance? No.

How can one promise that it won’t fail? Oh… I put my “reputation” at stake.

Let me tell you, this “reputation” stinks! Its stench is far worse than any land fills you’ll ever find! This is as good as - “I cross my fingers and hope that it works anyway”.

And what happens in the end? More work. People stay back late nights to debug, trace back and eat more suppers as a result. Ruining health, ruining lives and families one by one.

Why?

Because there were no tools to help me find a Connection leakage! I have to sit at my PC for like 10 days, 4 hours to complete a validation of a trace path! Then I suddenly realised that I forgot to do a finally block or catch the right exception. The viola! I’m a genius! I resolved the bug!

No, if I’m that person, then I’m an IDOIT!

A good testing tool can and WILL easily find out this problem within 30 minutes based on the same scale I’ve mentioned. Data Flow Analysis would have dug out Connection leakages.

So, yes initially, there’ll be a few more clicks, a few more detours and a few more files to read. But that’s just a fraction compared to the number of files you have to go through during debug.

Therefore, look far, don’t be stupid. A tool can only help you. It doesn’t give you more work. If a click and give you 4 more hours of time with your family, which one will you choose? It’s a no-brainer!

So please, don’t be stupid.

Myth #4 : With these tools, I’ll have lesser opportunities to level up my programming skills

Again, this is complete BULLSHIT. A good programmer is never measured by the kind of bugs he/she has resolved, or the number of lines of codes that he/she has written.

A good programmer is one that make conscientious efforts to ensure that his work has a certain level of quality and visibility. His/Her work is transparent to all and can be justified against the design documentations.

A good tool that comes with good documentations will serve as a guide to a good programmer. But at the same time, it’ll serve only as a hindrance to a lousy programmer who will find excuses to suppress violations that are actually valid. Or worse, make one the following claims:

1) It’ll NEVER happen!
2) It’s not used.
3) It works now why bother to change?
4) I’ve been a programmer for 10 years, so back off!

Conclusion

In conclusion, the purpose of having a testing tool is not to replace anybody’s job. Instead, it’s there to provide visibility to everyone.

Management can also take the opportunity to assert governance over their developers whom previously had no control over.

A testing tool can be a bane if it falls into a hands of the wrong kind of developers or testers. It can also be a bane if testing is done last minute. The amount work the tester and developer has to do can be astronomical.

A testing tool is usually a blessing because it reveals potential bugs that can hide from everyone.

It’s not a matter of whether it’s there or not.

It’s a matter of knowing whether it’s there or not.

With this, I’ll sign off here. I’ll try to get my Jeremy’s 101 ways of development series started soon on Weekly Tuesdays.

Thanks for your continued support!

Signing off,
Jeremy