100 Coverage Proves That the Program Can Not Fail
This comprehensive tutorial explains what is Code Coverage in Software Testing, its types, benefits, and drawbacks:
The ultimate goal of any software development company is to develop software that is of good quality. To achieve this goal, the software must be thoroughly tested.
Testing is thus an integral part of developing a software application. Hence it is essential, that the software developed is reviewed by the developer (which is done during the unit testing phase) and is then passed onto the QC team to be thoroughly tested to ensure that it has minimal or no bugs.
The software is unit tested before being released to the actual test team for testing. As this testing involves testing at the code level, it is done by the developer. This is done to ensure that each part of the code being tested works as expected.
Here, small chunks of code that have been developed are tested in isolation to ensure their correctness. But, the question that often looms in the mind of a developer is how much of unit testing should be done and the answer to this lies in the Code Coverage.
This tutorial will give you a deep knowledge of what Code Coverage is and why we need it. You would get to know how it differs from Test Coverage.
We shall also take a look at the tools and methodologies that are used for Code Coverage and towards the end of this tutorial, we would see the benefits along with its drawbacks. Some of the myths associated with Code Coverage would also be covered here.
What You Will Learn:
- What Is Code Coverage
- Why We Need Code Coverage
- Code Coverage Vs Test Coverage
- Methodologies
- Statement Coverage
- Function Coverage
- Condition Coverage
- Branch Coverage
- Tools For Code Coverage
- Benefits
- Drawbacks
- Myths Vs Facts
- FAQ's
- Methodologies
- Conclusion
- Recommended Reading
What Is Code Coverage
This is an important unit testing metric. It comes handy in knowing the effectiveness of unit tests. It is a measure that indicates what percentage of source code would get executed while testing.
In simple terms, the extent to which the source code of a software program or an application will get executed during testing is what is termed as Code Coverage.
If the tests execute the entire piece of code including all branches, conditions, or loops, then we would say that there is complete coverage of all the possible scenarios and thus the Code Coverage is 100%. To understand this even better, let's take up an example.
Given below is a simple code that is used to add two numbers and display the result depending on the value of the result.
Input a, b Let c = a + b If c < 10, print c Else, print 'Sorry'
The above program takes in two inputs i.e. 'a' & 'b'. The sum of both is stored in variable c. If the value of c is less than 10, then the value of 'c' is printed else 'Sorry' is printed.
Now, if we have some tests to validate the above program with the values of a & b such that the sum is always less than 10, then the else part of the code never gets executed. In such a scenario, we would say that the coverage is not complete.
This was just a small example to clarify the meaning of Code Coverage. As we explore more, you will gain better clarity on it.
Why We Need Code Coverage
[image source]
Various reasons make Code Coverage essential and some of those are listed below:
- It helps to ascertain that the software has lesser bugs when compared to the software that does not have a good Code Coverage.
- By aiding in improving the code quality, it indirectly helps in delivering a better 'quality' software.
- It is a measure that can be used to know the test effectiveness (effectiveness of the unit tests that are written to test the code).
- Helps to identify those parts of the source code that would go untested.
- It helps to determine if the current testing (unit testing) is sufficient or not and if some more tests are needed in place as well.
Code Coverage Vs Test Coverage
To understand the difference between Code Coverage and Test Coverage, let's first understand the meaning of Test Coverage.
Test Coverage
It is a measure of how many parts of the expected testing have been covered during testing a software. By 'expected testing' we mean the complete set of test cases that are written to be executed to test a given software.
Suppose, to test a software a set of 500 total test cases have been written. Now, as a part of the testing activity, only 300 test cases were executed. Let's assume this to be due to lack of time. In this case, the below would be the test coverage.
Test Coverage = (Executed Test cases/Total Test cases) * 100
= (300/500) * 100
= 60 %
Let's compare this with what Code Coverage is.
Code Coverage
It is a measure that shows the extent to which a source code of an application gets executed during testing the code. It thus shows the degree to which a source code would get tested.
Suppose to test an application with 500 lines of code, only 400 lines of the code get executed by tests. Let's assume that this is due to a certain loop/condition not getting executed. In this case, the below would be the code coverage.
Code Coverage = (number of lines of code executed/ total number of lines of code)* 100
= (400/500) * 100
= 80 %
Enlisted below are the differences between Code Coverage and Test Coverage:
Methodologies
Here, we shall discuss the various methods that are/can be used to measure the Code Coverage.
To understand these methodologies, let's take a look at the below code snippet:
Add (int a, int b) { If (b > a) { b = b - a Print b } If (a > b) { b = a – b Print b } Else Print '0' } Statement Coverage
This methodology is a measure that tells if all possible executable statements of code in source code have been executed at least once. It is a method to ensure that each line of the source code is covered at least once by the tests.
This might sound simple but caution needs to be exercised while measuring the Statement Coverage. The reason being, in a source code there could be a certain condition that might not get executed depending on the input values.
This would mean that all the lines of code would not be covered in testing. Thus, we may have to use different input value sets to cover all such conditions in the source code.
For example, in the above source code if input values are taken as 2 & 3 then, the 'Else' part of the code would not get executed. However, if the input values are of type 3 & 2 then the 'If' part of the code would not get executed.
This means that with either set of values of our Statement Coverage would not be 100%. In such a case, we may have to execute the tests with all three [(2, 3), (3, 2), (0, 0)] set of values to ensure 100% Statement Coverage.
Function Coverage
As the name suggests, this methodology measures the extent to which the functions present in source code are covered during testing. All functions that are in the source code get tested during test execution. Again, it must be ensured that we test these functions for varying values so that the function gets tested thoroughly.
In a source code there may be multiple functions and depending on the input values used a function may or may not be called. Thus the purpose of Function Coverage is to ensure that we have each function called for.
For example, in the source code above if our tests call the 'Add' function even once, then we would call this as a complete Function Coverage.
Condition Coverage
In a source code wherever we have a condition, the result would be a Boolean value of either true or false. Condition Coverage aims at establishing if the tests cover both the values i.e. true, false.
In the source code, when each occurring condition is evaluated for both true and false states, then the Condition Coverage for the code is said to be complete.
For example, in the above code if value sets (2, 3) and (4, 2) are used then Condition Coverage would be 100%. When data set (2, 3) is used then (b > a) evaluates to true and (a > b) evaluates to false. Similarly, when data set (4, 2) is used then (b > a) evaluates to false and (a > b) evaluates to true.
Thus both the conditions have both the values i.e true and false covered. Hence the Condition Coverage would be 100%.
Branch Coverage
This methodology aims at ensuring that every branch appearing in each conditional structure gets executed in source code. For instance, in the above code, all the 'If' statements and any accompanying 'Else' statement should all be covered by the test for a 100% Branch Coverage.
For example, in the above code if value sets (2, 3), (4, 2), (1, 1) are used then Branch Coverage would be 100%. When data set (2, 3) is used then (b > a) and the first 'If' branch gets executed. Similarly, when data set (4, 2) is used then (a > b) evaluates to true and the second 'If' branch gets executed.
Then with the data set (1, 1) the 'Else' branch evaluates to true and gets executed. Thereby, ensuring 100% Branch Coverage.
Branch Coverage Vs Condition Coverage
Branch Coverage is often confused with Condition Coverage, however, the two are different.
Let's understand this with a simple example.
If (a >0) & (b >0) Then Print "Hello" Else Print "Bye"
Let us write down the data set needed for complete Branch Coverage:
(1, 1) – In this case, 'a' and 'b' both are true, so the If condition gets executed.
(1, 0) – In this case, 'a' is true and 'b' would be false, so the Else part of the code is executed.
As we know the purpose of Branch Coverage is to get every branch executed at least once and this purpose is achieved.
Condition Coverage:
(1, 0) – In this case, 'a' is true and 'b' would be false.
(0, 1) – In this case, 'a' is false and 'b' would be true.
The purpose of Condition Coverage is to get each of true and false for every condition executed and this purpose is achieved here.
Did you notice that the else part does not get executed in Condition coverage? This is where Condition Coverage differs from Branch Coverage.
Tools For Code Coverage
To measure the Code Coverage of any Software, there are several tools available in the market.
Enlisted below are some of the tools for your reference:
- Parasoft JTest
- Testwell CTC++
- Cobertura
- JaCoCo
- CodeCover
- BullseyeCoverage
- EMMA
- OpenCover
- NCover
- Squish COCO
- CoverageMeter
- GCT
- TCAT C/C++
- Gretel
- JCov
Recommended Reading=> Code Coverage Tools
The above link will include the following information on these tools:
- Key Features
- License Type
- Official URL
- Pros and Cons
- Latest Version
Benefits
As seen above, it is a very useful test metrics for the below reasons:
- It helps to identify those areas in a source code that would remain untested/uncovered by the tests.
- It comes handy in identifying used/dead code thereby improving the code quality.
- The effectiveness of the unit tests can be known with the help of Code Coverage.
- Software having better quality can be delivered using these metrics.
Drawbacks
- Trying to aim for a 100% code coverage sometimes causes a lack of robustness of the tests resulting in missing out on capturing the defect prone scenarios.
- Unlike the common perception, it cannot guarantee if the designed software caters to all the requirements.
Myths Vs Facts
FAQ's
Q #1) What is an acceptable Code Coverage?
Answer: Achieving 100% code coverage should not be the goal while unit testing software code. But, why not? To understand the reason you may have to dive a little deeper to understand the underlying meaning.
When we target a 100% coverage, then it more often happens that all the focus in designing the tests go into ensuring if each statement, loop, branch, or condition gets tested. So we land up putting too many efforts which might not be worth considering the time spent.
Moreover, focusing on a high coverage also results in missing out on the important scenarios that are likely to have the defects because all that we are aiming is to ensure that each line of code is tested.
Focusing on a high code coverage is not that important always and it can neither be a fixed number to target to test different codes. However, in general, a coverage of 75% – 80% should be an ideal number.
While testing our code, the main focus should lie on ensuring to cover the critical and likely error-prone scenarios. If these are missed out then despite having a 100% Code Coverage our tests would simply have poor test effectiveness.
Q #2) How do I check my Code Coverage?
Answer: To test the percentage of Code Coverage that you might have achieved by the tests designed for testing the code we have several tools in the market. Depending on the programming language one is using we have various tools.
Some of them are listed below:
- Java – Cobertura, JaCoCo
- Javascript – Blanket.js, Istanbul
- Python – Coverage.py
- Ruby – SimpleCov
Using these tools we can get a complete coverage report of our tests that help us in knowing which part of the code would get executed and which would get missed out by our tests.
Q #3) Is Code Coverage a good metric?
Answer: In real-life scenarios, this is useful to a certain extent and in some particular ways.
Looking first at its limitations, we very well know that having a 100% coverage does not guarantee that the Code is bug-free, nor does it guarantee that all the requirements have been covered in the code i.e. despite a 100% Code Coverage we are very likely to have bugs in the code, the reason being that coverage does not ensure all scenarios have been tested.
Moreover, if requirements have been skipped while writing the code, then there is no mapping of requirements with the code that is taken care of as a part of the Code Coverage.
Having said that, we cannot deny that when we use Code Coverage as metrics, it gives us an idea if we have covered the basic requirements of testing each line of our code. This coverage percentage gives us an idea of how many parts of our code is getting executed with our unit tests.
We come to know how much of our code would be unexecuted. This in turn helps us to decide how much more unit tests are needed and for which parts of the code.
We can thus conclude, that having poor coverage does give us an idea of the ineffectiveness of the unit tests. At the same time, ensuring a 100% coverage is not a guarantee to a defect-free code. Thus one needs to have a balanced approach where we do not overemphasize the importance of targeting a high Code Coverage percentage.
Q #4) How can I improve my Code Coverage?
Answer: The Code Coverage Report that is provided by coverage tools such as JaCoCo, Istanbul, etc. shows the areas that are covered by the tests and also those that would go untested.
By knowing the untested parts of the code, tests can be written either manually or using any automation tool to cover the areas that would otherwise go untested and thereby increase the Code Coverage.
An important thing to note here is that while we may write hundreds of lines of code to test a function in code but still the coverage might be very less. The reason is that getting too deep to test a part of the huge code will not help in increasing Code Coverage.
Thus, if the target is to increase the coverage then care needs to be taken to cover all the functions, conditions, and loops instead of diving deep in a single function and writing large tests for that single function.
Conclusion
A high-quality software product is what is required in today's fast-paced internet world.
Ensuring good quality software is not only the responsibility of a QA Engineer but it is also the responsibility of the developer. Code Coverage is thus of great use when it comes to delivering a quality product to the QA team by the developer(s).
This tutorial explained all about Code Coverage and its uses. We also dug a little deeper into understanding the difference between Code Coverage and Test Coverage. Besides this, we got an understanding of the methodologies used along with a variety of Code Coverage tools.
Benefits and Drawbacks were briefed here. Finally, we busted some of the myths and FAQ's associated with Code Coverage
Happy Reading!!
100 Coverage Proves That the Program Can Not Fail
Source: https://www.softwaretestinghelp.com/code-coverage-tutorial/