Building a robust Embedded Linux platform FrOSCon 2012 Thilo Fromm

Before we begin Buzzword Bingo Motivation

Buzzword Bingo! Building a robust embedded Linux platform

Buzzword Bingo!

OpenE

mbed ded

?

Buzzword Bingo!

? r ile

p m OpeCnoE s m s b edded o r C

?

Buzzword Bingo!

? r ile

? d m e t s

p m y OpeCnoE S s m s b edded o r C

?

Buzzword Bingo!

? ? r d e l m i e r a e w t p f o t S ys OpeCnoEm S s e l m c y s c e b f i L o dded Cr ? ? t n e m e g a Man

Buzzword Bingo!

? ? r d e l m i e r a e w t p f o t S s y OpeCnnotEm e l S i f s e e l m r c y s c a e b f i p L s o dded TranCr ms ? ? ? t n e e m e t g a s Man sy

Buzzword Bingo!

? ? r d e l m i e r a e w t p f o t S s y AUO e FSpsee?CnnotEm l S i f e l m r c y s c a e b f i p L s o dded TranCr ms ? ? ? t n e e m e t g a s Man sy

Buzzword Bingo! ? t c e j o r p D ? T ? M x u Lin r d e l m i e r a e w t p f o t S s m y AUO FSpsee?CnnotEfile S e l m r c y s c a e b f i p L s o dded TranCr ms ? ? ? t n e e m e t g a s Man sy

Buzzword Bingo! ? t c e j o r p D ? T ? M x u UBLIin ? r d e l m i e r a e w t p f o t S s m y AUO e o FSpsee?C l S i f t n U B n IF E S ? e l m r c y s c a e b f i p L s o dded TranCr ms ? ? ? t n e e m e t g a s Man sy

Buzzword Bingo! ? t c e j o r p D ? T ? M x u UBLIin ? r d e l m i e r a e w t p f o t S s m y AUO e o FSpsee?C l S i f t n U B n IF E S ? e l m r c y s c a e N b f i p A L s N o D radseedb?lock? TranCr ms ? Ed ? t n e e m e t g a s Man sy

Buzzword Bingo!

KTHX!

Motivation Building a robust embedded Linux platform

Motivation

Arnout Vandecappelle “Safe upgrade of embedded systems” FOSDEM 2012 http://www.mind.be/content/Presentation_Safe-Upgrade.pdf http://www.mind.be/content/Presentation_Safe-Upgrade.odp

Building a robust embedded Linux platform

Agenda Platform

-vv

Challenges Concepts

Platform

-vv

Building a robust embedded Linux platform

What's a platform, anyway?

Car analogy

What's a platform, anyway?

What's a platform, anyway?

What's a platform, anyway?

What's a platform, anyway?

What's a platform, anyway?

What's a platform, anyway? Solid Base Comprehensive set of Customizations Multiple, different Use Cases

Agenda (revisited)

Platform

-vv

Challenges Concepts

Challenges Building a robust embedded Linux platform

Embedded Challenges “Server” thinking Failure vectors Platform building

Challenges - “Server thinking”

Photo by C.G.P. Grey http://www.flickr.com/photos/52890443@N02/4892006947/sizes/l/in/photostream/

Challenges - “Server thinking”

Photo by mikebaird, http://www.flickr.com/photos/mikebaird/2917873686/sizes/l/in/photostream/

Challenges - “Server thinking”

Challenges - “Server thinking” Accessibility (physical/network) Error/Sanity Detection Fallback Modes Atomic Operations

Challenges – Failure Vectors Power Fail Storage Corruption (“bit rot”) Transmission errors

Challenges – Failure Vectors Compatibility issues (config!) Fringe Situations Real Software Bugs

Challenges – Platform building SDK, Build System, Debug Tools? Software Lifecycle Management? Extensive Application Catalog? Easily Extensible?

Modern State of the Art Software Package Manager Fast, parallel, event-driven INIT

Robust Un-Brickable Safe Package Mgmt / Image Upgrades System Configuration Fallbacks

Agenda (revisited)

Platform

-vv

Challenges Concepts

Concepts (1 – 3i) x (1 + 3i)

http://xkcd.com/849/

Core Concepts System/Hardware Requirements Hardening OpenEmbedded

Setting Requirements Building a robust embedded Linux platform

Setting HW Requirements Performance wise Un-Brickability: internal+external boot Internal Storage technology => integrate these REALLY well!

Requirement: System Performance RAM Interfaces Minimum CPU req.

Requirement: HW Dual Boot Internal default + External Rescue/Dev Un-Brickable during development Fast in-field recovery

Boot

HW Dual Boot

Y Ext. Present? N

Rescue Re-Flash From external

Boot internal

Dev or Rescue? Dev

Boot external

This. IS. HARDWAAARE !!!

Requirement: Internal Storage Single Point of Optimization Robust code + workflows Common System FS layout

Requirement: Internal Storage

Setting Requirements Can increase degrees of freedom by minimizing Risk by providing reliable system features enabling for base platform concepts

Hardening Building a robust embedded Linux platform

Hardening Kernel & Root FS System Upgrade Package Management

Hardening: Kernel, Root FS Redundancy Sanity Fallbacks

Hardening: Kernel, Root FS

Hardening: Kernel, Root FS Boot

Last boot N OK? Y

Boot current

Boot fall-back

Hardening: Kernel, Root FS

Safety Timer

Mark Healthy

Root FS: Snapshots + Fall-backs Read-Only Root FS tmpfs for volatile data Transparent Overlay FS for non-volatile

Transparent Overlay FS r/o root fs /bin /boot /etc /home /lib /lost+found /media /mnt /opt /proc /root /run /sbin /tmp /usr /var

Transparent Overlay FS r/o root fs /bin /boot /etc /home /lib /lost+found /media /mnt /opt /proc /root /run /sbin /tmp /usr /var

r/w transparent overlay

Transparent Overlay FS r/o root fs /bin /boot /etc/motd /home /lib /lost+found /media /mnt /opt /proc /root /run /sbin /tmp /usr /var

r/w transparent overlay read()

Transparent Overlay FS r/o root fs

r/w transparent overlay

/bin /boot /etc/motd /etc/motd /home /lib /lost+found /media /mnt /opt /proc /root /run /sbin /tmp /usr /var

write()

Transparent Overlay FS r/o root fs

r/w transparent overlay

/bin /boot /etc/motd /etc/motd /home /lib /lost+found /media /mnt /opt /proc /root /run /sbin /tmp /usr /var

read()

Transparent Overlay FS r/o root fs

r/w transparent overlay

/bin /boot /etc/motd /etc/motd /home /lib /lost+found /media /mnt /opt /proc /root /run /sbin /tmp /usr /var

unlink()

Transparent Overlay FS r/o root fs

r/w transparent overlay

/bin /boot /etc/motd /etc/motd /home /lib /lost+found /media /mnt /opt /proc /root /run /sbin /tmp /usr /var

open() => -ENOENT

Transparent Overlay FS r/o root fs /bin /boot /etc /home /lib /lost+found

r/w transparent overlay

Root FS: Snapshots + Fall-backs

Root FS: Snapshots + Fall-backs Add new R/W overlay on top Stack multiple R/O overlays All this atomically

Achievement unlocked Snapshots + Roll-Backs!

Photo by stevent, CC BY-NC-SA 2.0, http://www.flickr.com/photos/stevent/3213246203/ Photo by pmarkham, CC BY-SA 2.0, http://www.flickr.com/photos/pmarkham/2795321710

Hardening: System upgrade & Package management Building a robust embedded Linux platform

Hardening: System Upgrade Transmission Sanity Atomic Switch “System upgrade” system state?

Hardening: System Upgrade Checksum everything Cache downloads (don't DL+Write) Verify, then Write Switch atomically

NAND Flash Organized in Pages of Erase Blocks Writes per page are limited “Writes” flip bits from 1 to 0 Erase: set all bits back to 1

Switch

Read Latest valid info Erase oldest or invalid entry

Write new Partition info, Epoch + 1 Booted =“1” Healthy=“1”

Image write

Switch Safety Timer

Flip “booted” to zero

Boot Partition Booted =”0” Healthy=”1”

Boot

Flip “healthy” to zero Booted =”0” Healthy=”0”

System

Hardening: Package Management Remember System Upgrade? Checksum, Cache downloads Verify, then Write Switch atomically

Hardening: Package Management Remember System Upgrade? Checksum, Cache downloads Verify, then Write Switch atomically

Hardening: Package Management Remember Transparent Overlays? Storage back-end is a directory How does it store its data?

Hardening: Package Management r/o root fs

r/w transparent overlay

/bin /boot /etc/motd /etc/motd /home /lib /lost+found /media /mnt /opt /proc /root /run /sbin /tmp /usr /var

write()

Hardening: Package Management

r/o root fs /bin /boot /etc/motd /home /lib /lost+found /media /mnt /opt /proc /root /run /sbin /tmp /usr /var

r/w transparent overlay /etc/motd

otd

found

Hardening: Package Management /overlays/overlay-data/some-tag/

r/w transparent overlay /etc/motd

Hardening: Package Management /overlays/overlay-data/some-tag/etc/motd

Hardening: Package Management Files, directories 1:1 in back-end Plus .mgmt files (white-outs etc.) write on AUFS, track in back-end Does it work the other way?

Hardening: Package Management /overlays/overlay-data/new-package/[write here]

otd

found

Hardening: Package Management /overlays/overlay-data/new-package/[package data]

r/o transparent overlay /[package data]

Hardening: Package Management

r/o root fs /bin /boot /etc/motd /home /lib /lost+found /media /mnt /opt /proc /root /run /sbin /tmp /usr /var

r/o transparent overlay /[package data]

Hardening: Package Management r/o root fs /bin /boot /etc/motd /home /lib /lost+found /media /mnt /opt /proc /root /run /sbin /tmp /usr /var

r/w transparent overlay

Hardening: Package Management r/o root fs

r/o transparent overlay

/bin /boot [package data] /etc/motd /home /lib /lost+found /media /mnt /opt /proc /root /run /sbin /tmp /usr /var

remount

r/w transparent overlay

Hardening: Package Management r/o root fs

r/o transparent overlay

/bin /boot [package data] /etc/motd /home /lib /lost+found /media /mnt /opt /proc /root /run /sbin /tmp /usr /var

r/w transparent overlay

Hardening Redundant Kernel and Root FS Atomic Switches + Fallbacks AUFS: R/O root, Snapshots, atomic package install

Putting it all together Building a robust embedded Linux platform

Putting it all together OpenEmbedded Building a robust embedded Linux platform

OE benefits SDK: compiler, build tools, sandboxing Thousands of apps, libs Built-in Package Management Big lively community, sub-projects

OE benefits Pluggable “Layer” concept Core, BSPs, Distributions, Tools Easy to extend

OE benefits: Many, many apps+libs Often very recent versions State of the art Well maintained

OE benefits: Package management Throughout: image is a set of pkgs Versioning, deps, conflicts => upgrade paths => Application lifecycle management

OE benefits: Angstrom Distro Layer Many features State of the art (e.g. systemd) Easily optimized

OE benefits: BSP layers Kernels, boot loaders, tools All set up and patched Some officially supported (as in: Manufacturer supported)

OE Layer example

OE Layer example

Why add custom layers?

Custom Distro Overrides System-wide optimizations Independence from distro politics Full systemd / upstart integration Custom system states

Event-driven INIT Parallelize start-up (i/o wait!) Start only what you need custom system states => upstart, systemd

Custom System states? Like “init” state, but more expressive Upstart: empty job w/ dependencies Systemd: .target systemctl start sys-upgrade.target

Custom Distro Overrides Packages for Platform Features … like transparent overlay management … or boot config tools

Custom BSP Overrides Take advantage of HW Requirements Integrate your storage layout... … and boot config features

Custom Layers Be pluggable Be re-usable Provide a foundation to build on

OpenEmbedded SDK, Tooling, App Library Pluggable, re-usable, extensible Great framework for building Platforms => integrate your concepts here

Phew. Building a robust embedded Linux platform

Wanna play with this stuff? Give HidaV a try. Go to http://dfe.github.com Building a robust embedded Linux platform

? Building a robust embedded Linux platform

KTHX! Building a robust embedded Linux platform This work is licensed under Creative Commons Attribution-ShareAlike 3.0