Writing robust test cases
Writing test cases is an integral part of software development. Software developers need to write test cases for making sure that the functionality of their code has appropriate test coverage and that no further code changes to their feature do not break the existing functionality. Also since most of the test cases run in continous integration (CI) systems, the test cases need to be robust and non-brittle. In this blog post, we will go over some examples from an open source project Openswitch.net to see how some of the test strategies are not robust and brittle enough to fail or worse allow a bug to sneak into the main branch despite the test case being written for catching the same breakage.
- Some of the test cases are written to check if some test string is present in a standard output. For example for a given route 123.0.0.1/32 which is resolved the nexthop 22.22.22.2 is verified in following manner in some of python test code in OpenSwitch.net:
out = dut01.DeviceInteract(command="
retBuffer = out.get('buffer')
LogOutput('info', retBuffer)
assert '123.0.0.1/32' in retBuffer, \
" VRF static routes not configured - failed"
assert 'via 22.22.22.2, [1/0], static' in retBuffer, \
" VRF static routes not configured - failed"
LogOutput('info', "VRF static routes configured - passed")
1. Test validations need to be done on required output only.
a. At times we try checking for a string in an output which is not required to be checked at all. This creates bug and defeats the purpose of writing test cases.
Why it fails? Consider two routes in “show ip route” as follow:-
switch# sh ip route
Displaying ipv4 routes selected for forwarding
'[x/y]' denotes [distance/metric]
123.0.0.1/32, 4 unicast next-hops
via 11.11.11.1, [1/0], static
143.0.0.1/32, 1 unicast next-hops
via 22.22.22.2, [1/0], static
Verification logic for 123.0.0.1/32
retBuffer = out.get('buffer')
LogOutput('info', retBuffer)
assert '123.0.0.1/32' in retBuffer, \
" VRF static routes not configured - failed"
assert 'via 22.22.22.2, [1/0], static' in retBuffer, \
" VRF static routes not configured - failed"
LogOutput('info', "VRF static routes configured - passed")
Verification logic for 143.0.0.1/32
retBuffer = out.get('buffer')
LogOutput('info', retBuffer)
assert '143.0.0.1/32' in retBuffer, \
" VRF static routes not configured - failed"
assert 'via 11.11.11.1, [1/0], static' in retBuffer, \
" VRF static routes not configured - failed"
LogOutput('info', "VRF static routes configured - passed")
Both test cases will pass.
2. We should only use show command outputs for verification. We shouldn’t use output of CLI config commands for verification.
a. Using config command output makes the test verification very brittle. Show commands are standard so they do not change frequently
b. Use appclt output only when needed. For example for PD features we have “fp show” for dumping the hardware configured values.
assert retCode == 0, "Able to configure L2 VLAN500"
This test case depends on the config command output. This test case will fail when someone changes or removes the config command output.
We need to check the show command outputs instead.
3. Use python regular expressions instead for string comparisons wherever possible. Generic regular expressions are less likely to result in a test case failure than strict string based comparisons.
Consider:-
a. assert 'via 11.11.11.1, [1/0], static' in retBuffer,
b. match = re.match(r’(.*)11.11.11.1(.*)[
Verification in step ‘a’ is more likely to fail than verification in ‘b’ if the show command output changes.
4. Construct libraries for show commands whenever possible. Libraries for show commands could return a dictionary/list for the show command output. Show command output is easier to model using a dictionary in most cases.
a. We have a few libraries for “show ip route”/”show rib” and “show lldp neighbor”
b. Python dictionary based comparisons are generic and much easier to do.
c. This allows future test cases to be written faster.
Comments
Post a Comment