Kaner: Exploratory testing better than scripted testing |
 |
By Jennette Mullaney, Associate Editor
05 Aug 2008 | SearchSoftwareQuality.com |
|
|
TORONTO -- Exploratory testing is superior to scripted testing, resulting in better tests and better testers, noted tester Cem Kaner told attendees at the Conference of the Association for Software Testing (CAST). Kaner, a software engineering professor at the Florida Institute of Technology, advocated that testers use checklists rather than scripts in his keynote speech, "The Value of Checklists and the Danger of Scripts: What Legal Training Suggests for Testers."
|
|
|
|
|
Following scripted instruction is a great way of leaving you within the confines of the script.
Cem Kaner
Software engineering professor, Florida Institute of Technology
|
|
|
|
|
|
|
|
|
|
Kaner is famous for his groundbreaking work in software testing, but many in IT may be unaware that Kaner holds a doctorate in psychology and a law degree. Both of these backgrounds informed Kaner's checklist work.
As a psychology student, Kaner observed that brain-damaged rats followed a "script" to avoid a shock, whereas normal rats followed a "checklist." The normal rats were able to adapt their behavior to the situation and complete a few steps to find a safe area. The brain-damaged rats were unable to adapt and followed the exact same behavior regardless of their circumstances. As such, they were incapable of simply running to safety. From this experiment, Kaner concluded that "following scripts is the 'best practice' available for brain-damaged rats."
Kaner also sought to dispel many of the misconceptions he sees surrounding exploratory testing and scripted testing. Exploratory testing is not limited to manual testing, and explorers "can use any tool they want," he said. Similarly, exploratory testers may create whatever documents they wish. Exploratory testing is not limited to black box testing nor is it limited to test execution, Kaner explained.
The harmful aspects of scripted testing
The myths cloaking scripted testing are numerous and insidious. "Myths support the notion that we need complex scripts," Kaner told the audience. One of these myths is that complex scripts promote learning. "Following scripted instruction is a great way of leaving you within the confines of the script," he said. Junior testers who are given complex scripts as "training wheels" are doomed to remain junior testers because they are not cognitively engaged with the testing process, Kaner argued.
In addition to being poor teachers, scripted tests do not properly test entry conditions, said Kaner. He listed a large number of factors that might affect a test that a script would not specify, including programs running, versions of those programs, exact amount of free memory available, and even the temperature of the processor being used.
The creation of bias may be scripted testing's worst offense; it subverts the purpose of testing, Kaner said. Testers following scripts will look for certain results and ignore others.
"People constantly change what they see based on expectations," he said, adding that scripted testing "sets up expectations." This leads to confirmation bias, whereby testers interpret results based on what they expect or want. Even worse, scripted testing can encourage "inattentional" blindness, so that testers do not notice failures even as they are observing them.
Testers are "looking for events in a certain time, and something else happens...and it never gets to [their] consciousness," explained Kaner. (For a quick and dramatic example of inattentional blindness, Kaner referred attendees to this short video.)
Checklists as cognitive aid
As a prosecutor in California, Kaner used checklists to greatly increase reckless driving convictions. "We had to prove that this was beyond really bad driving," he said. Kaner had to examine what makes the driving so bad. He utilized checklists to determine recklessness, including points such as ice on the road, number of people present, traffic levels and so forth. Kaner posited that checklists could be a wonderful resource for testers.
Checklists are "not just about the questions you're going to ask. They're much more about 'why do you want to know this?'" he said.
Testers have the ability to prioritize checklist items -- an option not available in scripts. Kaner stressed how checklists keep users cognitively engaged with their work.
"Checklists demand that you think about whether you need to bother with this and why or why not," he said.
Checklists and exploratory testing
Exploratory testing requires cognitive engagement from the tester throughout the entire test process, Kaner told the audience. This is in stark contrast to scripted testing, in which cognitive work is limited to test design, he explained.
In exploratory testing and checklist-driven work, cognitive effort is connected "to real-time activity where you actually apply it, in front of a judge, a client," said Kaner. Exploratory testers use checklists in the manner they choose. The checklist is a tool; one the tester controls.
|
