www.elvior.ee
Optimizing Testing of Embedded Software EuroSTAR 2004 Cologne, Germany Presented by Andrus Lehtmets Written by Andres Kull and Andrus Lehtmets Elvior Ltd, Tallinn
[email protected]
www.elvior.ee
Agenda
• • • • • • •
Our background Scope - large distributed embedded software project Classical problem areas in embedded software projects Inefficient approach of embedded SW testing Improving testing efficiency Requirements for test bed tools MessageMagic experiences
[email protected]
www.elvior.ee
Elvior introduction • Founded: 1992 • Specialized in embedded software development – – – –
GSM/UMTS BTS SW development for different base station families Access nodes workstation testing Mobile phone applications development Elevator control and monitoring systems development
• Two products:
– MessageMagic – test tool for embedded SW – MLP-Magic test bed tool for Mobile Location Protocol (MLP) applications
• Staff – 10 software engineers • Supporter member of Open Base Station Architecture Initiative (OBSAI)
[email protected]
www.elvior.ee
Large embedded distributed software project (1) • System consists of an significant number of interconnected HW nodes. • Software is distributed over the different HW nodes. Example: UMTS base station
Antenna filter units Wideband amplifiers Transmitters/receivers
- Signal processors - Summing units
[email protected]
- Operational and maintenance - telecom signaling
www.elvior.ee
Large embedded distributed software project (2)
• There are lot of developers involved in developing different HW/SW components concurrently. • Projects last long (> 2 years) • The projects are hard to manage – Evolving requirements and priorities – Project staff is not permanent. Human resources are entering and exiting the project and moving between different tasks inside the project – Lot of subcontractors and partners are involved. – Several different versions of the product may be under the development concurrently – The most permanent thing is the constant moving!!!
[email protected]
www.elvior.ee
What is typical for embedded software?
1. 2. 3. 4. 5. 6.
Embedded into physical environment Concurrency System decomposition matches the concurrency division Infinite execution Event driven state machines Asynchronous messages used for communication between SW components
[email protected]
www.elvior.ee
Abstract model of embedded software
SW component 2
SW component 1
Computational node
Events SW component 3 HW
[email protected]
SW component 4
www.elvior.ee
Embedded distributed software Computational node 1 SW component 2
Computational node 2 SW component 1
Events
SW component 4
SW component 3 HW
HW
Intercommunicating SW components are locating on the same or on the different computational node.
[email protected]
www.elvior.ee
Embedded SW components as state machines
SW component’s environment Req_1 Req_2 Resp_1
Ind_1
SW component
[email protected]
State A Req_1 s Resp_1
Req_2
Alarm State B Req_2
State C Alarm s Ind_1 State D
Req_1 s Resp_1
www.elvior.ee
Classical problem areas in the large SW projects • Business environment – Evolving requirements – Deadline pressures
• Human and resources
• Projects don’t meat deadlines • There are lot of cancelled projects • Lot of money wasted
– Lack of resources – Too many tasks for each developer – Lot of inexperienced people
• Specifications
– Usually too weak, specifications are not up-to-date – Lot of specifications in the head of developers
• Implementation
– Effort is hard to estimate beforehand – Estimations tend to be overly optimistic
• Integration
– Lack of communication between different teams – Different development tools and environments used by different teams
• Testing – – – –
Often has very weak processes Testing issues are not not taken into account in early project planning. Testing is treated on demand basis. Testing steps are combined together or dropped out to save time
[email protected]
www.elvior.ee
What can be done? • •
Each software development project’s phase may include a lot of inefficiencies. Project efficiency can be improved by improving the efficiency of each project phase individually.
There is a big potential to improve the whole project’s efficiency by optimizing test processes
[email protected]
www.elvior.ee
Optimizing test processes
1. Avoiding classical mistakes • •
A lot can be done just by avoiding classical mistakes done often in embedded software testing. Most of the classical mistakes can be avoided by understanding that saving a few dollars on testing costs a lot for the whole project.
2. Taking into use efficient test tools
[email protected]
www.elvior.ee
Avoid classical mistakes 1.
Test your software sooner
2.
Avoid too thorough and bureaucratic testing processes
3.
Measure test coverage
4.
Do not sacrifice test steps to catch up the schedule
5.
Test SW components individually before integrating them
6.
Don’t start testing in hardware too early
Follow good well known testing practices !
[email protected]
www.elvior.ee
Inefficient approach (1) Why good testing practices are often not followed?
• When testing an individual software component you need test bed code that provides the environment for the component. • Test bed development is an additional activity that may take the same amount of time as the product code itself. • Test bed code will usually be thrown away after the tests are performed.
[email protected]
www.elvior.ee
Inefficient approach (2)
1st untested component
[email protected]
2nd untested component
www.elvior.ee
Drawbacks (1)
component
B
component
A
component
C
1.
Hard to locate problems from integrated untested components
2.
Weak test coverage
3.
Neighbor components are ready in different time
[email protected]
www.elvior.ee
Drawbacks (2) Project plan A B C A
implementation implementation implementation + B + C joint unit testing
Actual A B C A
implementation implementation implementation + B + C joint unit testing
[email protected]
Planned
Actual
www.elvior.ee
How to improve test process? • Test bed development should be fast and easy for developers
– developers will become more willing to test their components properly before integration and – it will be easier for project managers to schedule more test steps into the project plans.
• User-friendly test tools for building test beds quickly should be used extensively in all test phases. • There is a need for the test tool that helps developers to create test beds quickly.
Test bed tools are the solution for embedded distributed software.
[email protected]
www.elvior.ee
Improved testing process Project plan A A B B C C
implementation unit test implementation unit test implementation unit test
Actual A A B B C C
implementation unit test implementation unit test implementation unit test
[email protected]
Planned Actual
www.elvior.ee
What do developers need for unit testing? (1) 1. SW platform for embedded component Æ RTOS emulator, etc. 2. SW component’s environment should be simulated. 3. Input events from the environment should be generated to trigger the component from state to state. 4. Output events generated by testable component should be examined. 5. Manual input events generation should be possible. Testable SW has to be bombed with different message data values to cover all branches. 6. Test scenarios should be possible to make tests repeatable.
[email protected]
SW component’s environment
Req_1
Alarm
Req_2 Resp_1
Ind_1
SW component
Req_1 s Resp_1
State A Req_2
State B Req_2
State C Alarm s Ind_1 State D
Req_1 s Resp_1
www.elvior.ee
What do developers need for unit testing? (2) SW component’s environment
SW platform for embedded component. Req_1
For all major RTOSs there are available soft-simulators that allow to run the RTOS components on the workstation environment. Code generating CASE tools (Rhapsody for example) have execution frameworks for workstation.
Alarm
Req_2 Resp_1
Ind_1
SW component
Req_1 s Resp_1
State A Req_2
State B Req_2
State C Alarm s Ind_1 State D
[email protected]
Req_1 s Resp_1
www.elvior.ee
What do developers need for unit testing? (3) SW component’s environment
SW component’s environment should be simulated. Req_1
There are two ways:
Resp_1
•To write software stubs for the environment components, and hard-code the test cases. •To use general test bed tools like MessageMagicTM from Elvior
Alarm
Req_2 Ind_1
SW component
Req_1 s Resp_1
State A Req_2
State B Req_2
State C Alarm s Ind_1 State D
[email protected]
Req_1 s Resp_1
www.elvior.ee
What do developers need for unit testing? (4) Inputs messages should be generated and output messages should be examined. Input messages with different parameters should be possible to try.
SW component’s environment
Req_1
Alarm
Req_2 Resp_1
Ind_1
SW component
• Easy to create and trace binary messages • Support for different encoding/decoding of binary messages (e.g.. ASN.1, TLV)
Req_1 s Resp_1
State A Req_2
State B Req_2
State C Alarm s Ind_1 State D
[email protected]
Req_1 s Resp_1
www.elvior.ee
What do developers need for unit testing? (5) Test scenarios to repeat test cases SW component’s
SW component’s environment
Req_1
environment
Alarm
Req_2
SW component
Resp_1
Req_1
Ind_1
SW component
Resp_1 Req_2 Req_1 Resp_1 Req_2
Req_1 s Resp_1
State A Req_2
State B Req_2
State C Alarm s Ind_1 State D
[email protected]
Req_1 s Resp_1
www.elvior.ee
MessageMagic experiences (1)
Test bed tool for embedded systems
[email protected]
www.elvior.ee
MessageMagic experiences (2)
• Embedded software testing in workstation environment – no
need for expensive laboratory equipment • Easy and fast development of test beds • Platform independent • User-friendly representation of binary messages • ASN.1 and TLV encoding • Easy scripting mechanism
[email protected]
www.elvior.ee
MessageMagic experiences (3)
•
low level tests on workstation
•
integration and unit tests on workstation
[email protected]
www.elvior.ee
Conclusion • Embedded software projects are usually large and complex
• Projects efficiency can be improved by improving the efficiency of each project phase individually • Improving efficiency in testing will save resources (money) significantly and reduce cost of whole project • To improve testing efficiency - Follow good testing practices - Take into use testing tools, such as MessageMagicTM from Elvior
[email protected]
www.elvior.ee
Questions ?
Thank you!
[email protected]
