MDD and its impact on testing... A nanotech case study
Bryan Bakker
EuroSTAR virtual conference May 2012
Contents
Intro ASD MDD and Testing Experiences
© Sioux 2012 | Confidential | 2
About Bryan Bakker
Test Expert Certifications: ISTQB, TMap, Prince2 Member of ISTQB Expert Level on Test Automation Tutor of several test related courses Domains: medical systems, professional security systems, semi-industry, electron microscopy Specialties: test automation, integration testing, design for testability, reliability testing
© Sioux 2012 | Confidential | 3
About Sioux
UTRECHT
MOSCOW
EINDHOVEN HERENTALS
NEDERWEERT
© Sioux 2012 | Confidential | 4
About FEI
World leader in electron microscopes Light microscope: 1000x 200nm (limited by the wavelength of light) Electron microscope: 4Mx 0.05 nm Nm = a billionth of a meter (10-9 meter)
© Sioux 2012 | Confidential | 5
About FEI
Breast cancer cell. Magnification 5.000x
Salmonella bacteria. Magnification 80.000x
Atomic structur of Ge (Germanium). Distance is 0.5 nm
2 nm
0. 5 nm
© Sioux 2012 | Confidential | 6
ASD (Analytical Software Design)
Model Driven Development technology Component Based Development Models are verified mathematically at design time Extensive model checker: Deadlock Live-lock Starvation Race conditions, etc
© Sioux 2012 | Confidential | 7
ASD Model ASD Model: - Precise - Complete - Correct
© Sioux 2012 | Confidential | 8
ASD Model ASD Model: - Precise - Complete - Correct
Design errors
Generate formal model
Formal model and verification
© Sioux 2012 | Confidential | 9
ASD Model ASD Model: - Precise - Complete - Correct
Design errors Generate defect free source code from verified model
Generate formal model
Formal model and verification
Guaranteed equivalence
Source code: - Java - MISCRA C - C++ - C#
© Sioux 2012 | Confidential | 10
What is covered? Interface model: Specification of interfaces of components “What” Design model: Implementation of interfaces Internal implementation “How” Also the outside interfaces are specified Interfaces to not-modelled components © Sioux 2012 | Confidential | 11
ASD Typically used to model behaviour described in: State machines Sequence diagrams Algorithms What is modelled? All normal functionality (good weather) All exceptional functionality (bad weather) All illegal behaviour
© Sioux 2012 | Confidential | 12
MDD and Testing Wish, need Release
User Requirements
Acceptance Test
System Requirements
System Test
Architecture
Integration Test
Formal methods Design
Component Test
Implementation © Sioux 2012 | Confidential | 13
MDD and Testing Wish, need Release
Formal methods typically:
User Requirements
Acceptance Test
System Requirements
System Test
Architecture
Applied at Design level Only in Software For part of the software
Integration Test
Formal methods Design
Component Test
Implementation
© Sioux 2012 | Confidential | 14
MDD and Testing Wish, need
Part of the component tests are not needed anymore (Software Engineer) Test activities of Test Engineer: From integration level and up Are still needed But: MDD has impact on quality of deliverables
Release
User Requirements
Acceptance Test
System Requirements
System Test
Architecture
Design
Integration Test
Formal methods Component Test
Implementation
© Sioux 2012 | Confidential | 15
Testing experiences
Testing is still necessary Integration (with other SW, with HW) System testing Model incorrect Rubbish in, rubbish out New insights, so model will change Functionality must still be verified (and errors are found) Programming errors or integration issues: In handwritten code On boundary between generated and handwritten code (hard to analyse) Not in generated code! © Sioux 2012 | Confidential | 16
Testing experiences Reliability of generated code: very high! But crashes can still occur... (but not in generated code) Reliability tests HW failures/wear detected much earlier Integration and system test not hampered by reliability issues big improvement, less cycles
© Sioux 2012 | Confidential | 17
Likelihood
Product Risk Analysis 15 Comp1
9 Comp2
Comp3
3 3
9
15 Impact
© Sioux 2012 | Confidential | 18
Likelihood
Product Risk Analysis 15Could test
Must test Comp1
III
I
9
IV
Comp2
3 “Won´t test” 3
Comp3
9
II
Should test 15 Impact © Sioux 2012 | Confidential | 19
Likelihood
Product Risk Analysis 15 Comp1
9 Comp2
Comp3
3 3
9
15 Impact
© Sioux 2012 | Confidential | 20
15
ASD
Likelihood
Product Risk Analysis
9 Comp1 Comp2
Comp3
3 3
9
15 Impact
© Sioux 2012 | Confidential | 21
Conclusions Component tests (white box) could be skipped Internal working is correct Although functionality can be incorrect Integration and system testing a lot smoother Less interface issues Less reliability issues Reliability testing reliability measurement Integration and system test more efficient (less cycles) Architecture and design takes more effort More re-designs with ASD are planned © Sioux 2012 | Confidential | 22
Questions
© Sioux 2012 | Confidential | 23
www.sioux.eu +31 (0)40 26 77 100
[email protected] © Sioux 2012 | Confidential | 24