Everything I know about Testing I learned from the Scientific Method Paul Coyne @paulcoyne73 Clydesdale Bank plc
Me! • Paul Coyne – Test Architect – Working in testing since 1989 • Mainly in finance & utilities • Currently at Clydesdale Bank – Failed zoologist!
Life after zoology • Postgrad in Systems Analysis & Design – – – –
SSADM Programming Business Analysis and, of course, Testing • Sorry, did I say “testing”? • That’s obviously a mistake
How I learned to Test • The situation was… – I was the new wildly enthusiastic programmer on the block – I was going to revolutionise COBOL programming (oh yeah!)
• My first assignment was… – The Test Team? – You’re kidding, right? – What did I do wrong???
How I learned how to Test
So what is the Scientific Method? • Science is not……. – – – – –
A body of knowledge Expert opinion or Textbooks Tools and lab techniques Revealed wisdom Unchangeable
The Scientific Method is… • • • •
A way of thinking A way of investigating A means to ask (the right) questions A way to learn from the answers to those questions – Even if the answer is not what you expected! – In all cases the simplest explanations are preferred • Until they are discounted
Observe, Imagine, Query • Start with an observation… • …leads to an hypothesis – It should be consistent enough with what is known to be considered reasonable – But propose something new that is worth finding out about
A Good Hypothesis • Hypotheses must be able to be proven to be false and should make specific, testable predictions – Otherwise they’re just speculation! • “There’s an invisible dragon in my garage which only I can see”
• One Black Swan is all it takes, despite a virtual infinity of white ones.
Science is Active Observation‐ • Science is based on real‐world observations – It is empirical
• It relies on those observations being repeatable – Transparency of methods is paramount
• Depends on accurate and timely recording of results and publication of ALL outcomes – No hiding ‘unfortunate’ results
So it’s a method, then? • More like a set of methods • A way of thinking and doing • “A systematic and logical approach to discovering how things work” • Rooted in rigorous observation under controlled circumstances – Is any of this sounding familiar?
And what about experiments? • Well, we have a hypothesis – which makes testable predictions – which we know we can (in theory) prove wrong – So… that’s what we set out to do
Good Experiments are… • Controlled – changing only one thing at a time (in so far as is possible)
• Repeatable – Clear methods, well stated, outcomes recorded
• Based on a model of the experimental subject – Of necessity less detailed than reality
• Blinded (single, double)
Correlation is not Causation
www.tylervigen.com
Experimental failures • Failure in an experiment is not the failure to get the expected result • It’s a methodological failure.. – To control for bias or variables – To report accurately and dispassionately – With conclusions not supported by the evidence
• The press rarely get it right either!
“Hold on – what’s this got to do with me?” • Good software testing has a lot in common with the Scientific Method – – – –
Testable predictions Control of variables Repeatability / Reporting of Methods Discovery of new information
Good Testers are into SM • • • • • • •
Rigour in thought processes Scepticism & critical thinking Seeking the “negative” outcome Clear methods, stated in advance Concise fact‐based reporting * provided you have been rigorous in Ability to repeat your methods and honest in your reporting All results are valid*
Classic Mistakes in Testing Classic Mistakes in Science too.. • • • • • • •
“Owning” quality Trying to prove it works Poor planning – non‐methodical Inadequate controls Trusting in documentation Mistaking quantity for quality Ambiguous reporting
Lessons for Testing ‐ 1 1. 2.
3. 4. 5.
Do not seek to confirm, but to disprove. You’re looking for bugs Passes and Fails are not symmetrical. A test that has passed is weak evidence. A test that has failed is much stronger evidence. Design your tests to eliminate uncontrolled variables or to limit their effects. Observe carefully, record dispassionately and be aware of your own biases and preconceptions. Keep good records. Understand that correlation and causation are not the same – consider what other plausible mechanisms might explain what you observed.
Lessons for Testing ‐ 2 6.
Have clear goals. Understand what is valuable in your context and pursue that. 7. Follow your nose, but don’t lose your way. Someone has to pay for your time! 8. All prior knowledge is provisional until disproven. However all observed facts are true, provided your methods are robust. 9. Be audacious in what you do. But be humble in your own knowledge and interpretation. Let the facts speak. 10. Challenge authority (diplomatically). Use other people’s skills wisely, but don’t rely on their truths.
Testers are Scientists • Software Development is NOT Engineering – Otherwise it would work better!
• Software Testing is Experimentation – Testers are methodical and systematic – Testers control for variability in the system – Testers record and report all results without fear or favour
