There WILL be bugs! !

http://www.aleax.it/bayp14_twbb.pdf!

©2014 Google -- [email protected]

1

"Will there be bugs"? YES!!!

with probability above 0.9999...

Murphy's Law fully applies (& then some:-)

2 2

"When will there be bugs"? when you're pushing the envelope

new or new-to-you algorithms, fields, libraries, frameworks, ...

...there are new traps waiting for you

when you're doing humdrum, routine development or maintenance...

...what you've done 1,000 times before...

...your attention level's likely not 100%!

...and, in all cases in-between:-)

3 3

The 1 exception in my life

my very first program: 1 run, bug-free

1974: Fortran, conditional probabilities of suits in bridge, mainframe

3 HW majors, bridge enthusiasts (key!), no programming courses (NP), humble (!)

punched cards,"big brain" mystique (helped!)

"invented" code-reviewing, pair-prog x 1.5

had PCs been easily around (a bit later)...

...bugs would have abounded (later did!-)

...never ever happened again in my life! 4 4

Fast Forward 40 years... downside: we only got 1 run so couldn't possibly have run tests

not that we'd ever heard of testing!-)

nowadays, tests must be at the heart of bug avoidance, discovery, and fixing

...as they should have been in the '70s!-)

Knuth, 1977: "Beware of bugs in the above code; I have only proved it correct, not tested it"...!-)

but code reviews & pairing still help! 5 5

Where do bugs like to hide? Well, anywhere, actually!-)

6 6

Common bug-hiding places "advanced" stuff, or where you're "clever"

key fix: *simplify* (+, unit-test!)

where you did not fully understand the problem, architecture, platform

ditto +: acceptance tests, code reviews

boilerplate/duplicate code

Don't Repeat Yourself (DRY)

rarely executed code (error handling, ...)

key fix: unit-test w/mocking for errors

previously-buggy code ("regressions")

a special case of TDD 7 7

Simplify, simplify, simplify Kernighan: "Debugging is twice as hard as writing a program in the first place. If you're as clever as you can be when you write it, how will you ever debug it?"

break up long, complicated expressions (intermediate results -> local variables)

beware complex decision chains

and nested loops / recursion, esp. with many conditional break/continue stmts

regular expressions MUST be tested a LOT ( also see: regex101.com ) 8 8

Don't Repeat Yourself (DRY) Hunt and Thomas (1999): "Every piece of knowledge must have a single, unambiguous, authoritative representation in a system".

AKA Once and Only Once (focus on code)

copy-and-paste coding is a main antipattern breaking DRY / OaOO

duplicate code is the worst code smell

everything's changing all the time: with duplicates, you'll miss some needed change

abstract what varies, merge what doesn't

9 9

Before DRY, it's WET...: if foo(bar, baz):! return bar! else:! return baz! !

...! !

if foo(zip, zap):! return zip! else:! return zap + 1 10 10

...let's DRY it up! def picker(a, b, c=None):! if foo(a, b):! return a! else:! return b if c is None else b + c! ...! return picker(foo, bar)! ...! return picker(zip, zap, 1)!

11 11

Rarely executed code handling weird errors, corner cases

unittest.mock (3.3+; backports aplenty)

with mock.patch(...) as x:!

to locally replace ... with a Mock obj x

set return_value, side_effect, ...

after, may assert about x's calls &c

BEWARE:

all attributes and methods auto-appear

-> high danger of typoes!

use auto_spec to reduce the danger 12 12

A mock.patch example def foo():! try: never.fails()! except OopsItDid as e:! logging.warm('Failed: %s', e)! raise! ...! with mock.patch('never.fails'‡) as nf:! nf.side_effect = OopsItDid! with self.assertRaises(OopsItDid):! module_i_am_testing.foo()! ‡: add autospec=True &c 13 13

Why do bugs happen? Your brain tricks you!

Perception: See what you expect to see

Attention lapses

Overconfidence

Confirmation bias

You've written it: it's YOUR code, it's YOUR baby, you're really proud of it

bugs love to hide there, as `you won't see them` -- you're "too close" to see clearly

14 14

Remedies for your “tricksy brain” egoless programming

more eyeballs

open source

pair programming

code reviews

testing

`lint` and similar static-analysis tools

15 15

Egoless Programming Jerry Weinberg, "The Psychology of Computer Programming", 1971

1.Understand and accept that you will make mistakes; find them early!

2. You are not your code; reviews are to find problems, and find them they will -don't take it personally!

...

10. Critique code instead of people – be kind to the coder, not to the code 16 16

Eyeballs: open source Eric Raymond's "Linus Law" (1997): "given enough eyeballs, all bugs are shallow"

"Given a large enough beta-tester and co-developer base, almost every problem will be characterized quickly and the fix will be obvious to someone."

Glass "Facts and Fallacies about Software Engineering" (2003): "it's a fallacy"

"the rate at which additional bugs are uncovered does not scale linearly with the number of reviewers"

`goto fail`, `heartbleed`; see Mike Bland's http://martinfowler.com/articles/testingculture.html 17 17

Eyeballs: pair programming the "driver" codes, focuses on "tactical" aspects of completing the current task

the "observer" reviews-as-it-goes, ponders "strategic" issues, plays "safety net"

frequent role switches; talking through it

many empirical studies and meta-research confirm: higher quality, faster time, but larger effort, compared with `solo` progr.

effort estimation here is biased-high

e.g: P.P produces a higher `bus number` w/o extra study/analysis effort! 18 18

Eyeballs: code reviews traditional "high ceremony" `Fagan Inspections` -- multiple meetings, printed code, thorough line-by-line critique, ...

high cost, very slow, high efficacy

may "fail to see the forest for the trees", "bikeshedding" also a risk

lightweight, informal walkthroughs and critiques - much faster, also effective

especially with the right tool!

complementary, not alternative, to pair programming and testing 19 19

Testing

Jacob Kaplan-Moss: "code without tests is broken by design":-)

informal "just run it and see if it breaks"

quite ineffective, esp. by original author

automated light-weight "unit tests"

must run fast so you can and will keep re-testing even on minor changes

great at avoiding unit-specific issues

automated middle-weight "integration tests"

complementary, not alternative, to u-t's

focus on interaction between subsystems

formal heavy-weight "acceptance tests"/QA 20 20

Testing vs Eyeballs Testing: objectively shows bugs' presence

can never conclusively show absence:-)

automated -> re-run all the time

Eyeballs: complementary, not alternative

pick up issues testing can never reveal

lack of clarity, bad naming, complexity

not automated -> consume human time

A third leg for the anti-bug "stool": lint

automated, fast

uniform style -> more productive eyeballs 21 21

Anti-bug tools most important is source code control (svn, hg, git, ...) -- trace what changed and when

beware subtle merges, cherrypicks, ...

next, a testing framework (unittest &c)

a bug tracker (ideally integrating w/SCC)

a code review tool (ditto)

22 22

“What about debuggers?” only mild importance for fighting bugs

if the code's so complicated that you have to follow step by step, simplify it

can help as "learning device"

unfamiliar code, library, framework

23 23

Beware tool-itis!

a geek's natural attitude: "if I have the right tools, all bugs will flee in terror"

they won't flee: bugs are courageous!-)

tools, at best, help: they don't solve bugs

tools can in fact hurt, by distracting you

hours spent in front of a cool debugger

...yak-shaving (tracing just-fine code)...

MUCH more important than tools' details:

attitude, skill, care, focus on team&user

and esp: good practice (light process) 24 24

And yet, tools are cool:-)

...but, excessive power can hurt the unwise

debugger to interactively examine values: log them instead! (w/logging.debug)

then code a script to sanity-checks logs

big difference: it's automated!

SCC allows fancy integrates, merges, cherrypicks, and generally funky graphs

very high risk of introducing bugs!

keep SCC graphs clean and simple!

key idea: good enough IS good enough 25 25

Testing frameworks stdlib unittest: a good starting point

+ many extensions: nose & plugins, &c

coverage's important (figleaf's ok too:-)

use doctest only for examples in docs!

it's designed for that & does it well

automated test runners / CI

nosy, nosier, nosyd, PyZen; buildbot

mocks, fakes, stubs, spies, dummies, ...

unittest.mock, but use with care!-)

specialized: web, fuzzing, GUI, acceptance... 26 26

Web testing either: simulate a browser

simple, fast -- but, no Javascript, CSS &c!

best for unit-tests

or: automate a real browser

most powerful, realistic, but, slower

best for integration & acceptance tests

specific web frameworks may further help

from google.appengine.ext import testbed!

from django import test!

...

27 27

Good enough... pick one from each group

the standard-er, the better

28 28

Example: web testing either: simulate a browser

simple, fast -- but, no Javascript, CSS &c!

best for unit-tests

or: automate a real browser

most powerful, realistic, but, slower

best for integration & acceptance tests

specific web frameworks may further help

from google.appengine.ext import testbed!

from django import test!

...

29 29

Linting &c Logilab's pylint

very powerful, configurable

unused/unassigned variables

too-long modules/functions

style violations in naming, wspace, ...

pyflakes, pep8: limited, but fast

Clone Digger

finds some "clones" (duplicate code)

... 30 30

When to use a bug tracker Always!-)

"bug tracker" (BT) is a misnomer...:

track both bugs AND features

Integration w/SCC: any changeset must identify which BT entry it regards

ideally only 1: keep CSs small!

Integration w/code review tool: code reviewer can follow the BT entry to find out WHY the CS under review exists

BT entry points back to CSs fixing it 31 31

The worst kind of bug

race conditions (& their evil cousins: deadlock, starvation, ...)

alas, test don't help much (!); code reviews may if super-duper-hyper careful/thorough

prevention is by far the best cure here

avoid shared-RW-memory concurrency

message passing, shared-RO-mem OK

if you really can't, rigorously sequence lock acquisitions, a la Dijkstra

maybe a dedicated global-memorychanging thread w/Queue of changes 32 32

Test-Driven Development Again: Jerry Weinberg, "The Psychology of Computer Programming", 1971

`In program testing, the programmer who gets early "success" with his program is likely to stop testing too soon. One way to guard against this mistake is to prepare the tests in advance of testing and, if possible in advance of coding.`

write the tests; see them fail; fix code to pass each test; check they succeed; refactor and check they still succeed. 33 33

TDD: worth using? (1) (personal opinion here, but, based on substantial experience...)

for adding features...: hmmm...

what exactly do you unit-test?

proper test targets for most features, "user stories", are more suitable for acceptance/integration than unit tests

Behavior driven development (BDD): TDD variant based exactly on user stories

ideally written by users/PMs/&c

great if you get such users/PMs/&c:-) 34 34

TDD: worth using? (2) for fixing bugs...: absolutely YES!

you know exactly what to unit-test for:

the very bug you're fixing!

first, you write the new tests

...and make sure all the new tests fail

"reproduces the bug" in a known state

then, you fix the bug

...and make sure all the tests pass

the new tests stay in the test-suite

insurance against future regressions! 35 35

TDD: worth using? (3) for refactoring...: just *NOT APPLICABLE*!

you can NEVER safely refactor code not already well covered by good tests

ensuring such coverage is not "the start of the refactoring",

it's a *PRE-REQUISITE* of it!

Michael Feathers, "Working Effectively with Legacy Code", 2002 -- http:// www.objectmentor.com/resources/articles/ WorkingEffectivelyWithLegacyCode.pdf 36 36

Q&A http://www.aleax.it/bayp14_twbb.pdf!

?

! 37 37