History and Practice of Digital Sound Synthesis Julius Smith CCRMA, Stanford University AES-2006 Heyser Lecture October 6, 2006

Julius Smith

AES-2006 Heyser Lecture – 1 / 84

Overview Early Digital Synthesis Spectral Modeling Physical Modeling Summary

Overview

Julius Smith

AES-2006 Heyser Lecture – 2 / 84

Outline Overview

• Outline • CCRMA Perspective Early Digital Synthesis Spectral Modeling Physical Modeling Summary

Digital sound synthesis approaches in approximate historical order:

• • • • • • •

Wavetable (one period) Subtractive Additive Frequency Modulation (FM) Sampling Spectral Modeling Physical Modeling

Some connections with audio coding will be noted

Emphasis:

• Sound examples • Block diagrams • Historical notes Julius Smith

AES-2006 Heyser Lecture – 3 / 84

CCRMA Perspective Overview

• Outline • CCRMA Perspective Early Digital Synthesis Spectral Modeling Physical Modeling Summary

The Knoll, Stanford University

Julius Smith

AES-2006 Heyser Lecture – 4 / 84

Overview Early Digital Synthesis Spectral Modeling Physical Modeling Summary

Early Digital Sound Synthesis

Julius Smith

AES-2006 Heyser Lecture – 5 / 84

Wavetable Synthesis in Music I-V (1957-1969) Overview Early Digital Synthesis

• Music V • KL Music • “Daisy” • Additive Analysis • Additive Synthesis • FM Synthesis • FM Formula • FM Patch • FM Spectra • FM Examples • FM Voice • Sampling Synthesis • Modern Example Spectral Modeling Physical Modeling Summary

Julius Smith

AES-2006 Heyser Lecture – 6 / 84

Music V Scripting Language (“Note Cards”) Overview Early Digital Synthesis

• Music V • KL Music • “Daisy” • Additive Analysis • Additive Synthesis • FM Synthesis • FM Formula • FM Patch • FM Spectra • FM Examples • FM Voice • Sampling Synthesis • Modern Example Spectral Modeling Physical Modeling Summary

• Essentially Supported in MPEG-4 Structured Audio Orchestra • Julius Smith

Language (SAOL) (Music V → csound → SAOL) “Encoding sounds” as “instruments” is hard, in general

AES-2006 Heyser Lecture – 7 / 84

Kelly-Lochbaum Vocal Tract Model Overview

…

Early Digital Synthesis

• Music V • KL Music • “Daisy” • Additive Analysis • Additive Synthesis • FM Synthesis • FM Formula • FM Patch • FM Spectra • FM Examples • FM Voice • Sampling Synthesis • Modern Example Spectral Modeling Physical Modeling

e(n)

Glottal Pulse Train or Noise e(n)

1 + k1 z− 1 2 k1

(Unused Allpass Output)

− k1

…

R1 z− 1 2

1 − k1

y(n)

1 + kM z− 1 2 kM

…

− kM

Speech Output y(n)

RM z− 1 2

1 − kM

Kelly-Lochbaum Vocal Tract Model (Piecewise Cylindrical)

Summary

John L. Kelly and Carol Lochbaum (1962)

Julius Smith

AES-2006 Heyser Lecture – 8 / 84

Sound Example Overview

“Bicycle Built for Two”: (WAV) (MP3)

Early Digital Synthesis

• Music V • KL Music • “Daisy” • Additive Analysis • Additive Synthesis • FM Synthesis • FM Formula • FM Patch • FM Spectra • FM Examples • FM Voice • Sampling Synthesis • Modern Example Spectral Modeling

• • • • • •

Vocal part by Kelly and Lochbaum (1961) Musical accompaniment by Max Mathews Computed on an IBM 704 Based on Russian speech-vowel data from Gunnar Fant’s book Probably the first digital physical-modeling synthesis sound example by any method Inspired Arthur C. Clarke to adapt it for “2001: A Space Odyssey” — the computer’s “first song”

Physical Modeling Summary

Julius Smith

AES-2006 Heyser Lecture – 9 / 84

Classic Additive-Synthesis Analysis (Heterodyne Comb) Overview Early Digital Synthesis

• Music V • KL Music • “Daisy” • Additive Analysis • Additive Synthesis • FM Synthesis • FM Formula • FM Patch • FM Spectra • FM Examples • FM Voice • Sampling Synthesis • Modern Example Spectral Modeling Physical Modeling Summary

John Grey 1975 — CCRMA Tech. Reports 1 & 2 (CCRMA “STANM” reports — available online) Julius Smith

AES-2006 Heyser Lecture – 10 / 84

Classic Additive-Synthesis (Sinusoidal Oscillator Envelopes) Overview Early Digital Synthesis

• Music V • KL Music • “Daisy” • Additive Analysis • Additive Synthesis • FM Synthesis • FM Formula • FM Patch • FM Spectra • FM Examples • FM Voice • Sampling Synthesis • Modern Example Spectral Modeling Physical Modeling Summary

John Grey 1975 — CCRMA Tech. Reports 1 & 2 (CCRMA “STANM” reports — available online) Julius Smith

AES-2006 Heyser Lecture – 11 / 84

Classic Additive Synthesis Diagram Overview

A1 (t) f1 (t) A2 (t) f2 (t) A3 (t) f3 (t) A4 (t) f4 (t)

Early Digital Synthesis

• Music V • KL Music • “Daisy” • Additive Analysis • Additive Synthesis • FM Synthesis • FM Formula • FM Patch • FM Spectra • FM Examples • FM Voice • Sampling Synthesis • Modern Example Spectral Modeling

noise

Σ

FIR Physical Modeling Summary

y(t) =

4 X i=1

Julius Smith

Ai (t) sin

Z

0

t

 ωi (t)dt + φi (0) AES-2006 Heyser Lecture – 12 / 84

Classic Additive-Synthesis Examples Overview Early Digital Synthesis

• Music V • KL Music • “Daisy” • Additive Analysis • Additive Synthesis • FM Synthesis • FM Formula • FM Patch • FM Spectra • FM Examples • FM Voice • Sampling Synthesis • Modern Example Spectral Modeling Physical Modeling

• • • • • • • • •

Bb Clarinet Eb Clarinet Oboe Bassoon Tenor Saxophone Trumpet English Horn French Horn Flute

• All of the above • Independently synthesized set

Summary

(Synthesized from original John Grey data)

Julius Smith

AES-2006 Heyser Lecture – 13 / 84

Frequency Modulation (FM) Synthesis Overview

FM synthesis is normally used as a spectral modeling technique.

Early Digital Synthesis

• Music V • KL Music • “Daisy” • Additive Analysis • Additive Synthesis • FM Synthesis • FM Formula • FM Patch • FM Spectra • FM Examples • FM Voice • Sampling Synthesis • Modern Example Spectral Modeling Physical Modeling Summary

Julius Smith

• Discovered and developed (1970s) by John M. Chowning • •

(CCRMA Founding Director) Key paper: JAES 1973 (vol. 21, no. 7) Commercialized by Yamaha Corporation:

◦ DX-7 synthesizer (1983) ◦ OPL chipset (SoundBlaster PC sound card) ◦ Cell phone ring tones • On the physical modeling front, synthesis of vibrating-string waveforms using finite differences started around this time: Hiller & Ruiz, JAES 1971 (vol. 19, no. 6)

AES-2006 Heyser Lecture – 14 / 84

FM Formula Overview

x(t) = Ac sin[ωc t + φc + Am sin(ωm t + φm )]

Early Digital Synthesis

• Music V • KL Music • “Daisy” • Additive Analysis • Additive Synthesis • FM Synthesis • FM Formula • FM Patch • FM Spectra • FM Examples • FM Voice • Sampling Synthesis • Modern Example

where

(Ac , ωc , φc ) specify the carrier sinusoid (Am , ωm , φm ) specify the modulator sinusoid Can also be called phase modulation

Spectral Modeling Physical Modeling Summary

Julius Smith

AES-2006 Heyser Lecture – 15 / 84

Simple FM “Brass” Patch (1970–) Overview Early Digital Synthesis

• Music V • KL Music • “Daisy” • Additive Analysis • Additive Synthesis • FM Synthesis • FM Formula • FM Patch • FM Spectra • FM Examples • FM Voice • Sampling Synthesis • Modern Example

Jean-Claude Risset observation (1964–1969): Brass bandwidth ∝ amplitude

fm = f0 g A

F

fc = f0

Spectral Modeling Physical Modeling

A

F

Summary

Out

Julius Smith

AES-2006 Heyser Lecture – 16 / 84

FM Harmonic Amplitudes (Bessel Function of First Kind) Overview

Harmonic number k , FM index β :

Early Digital Synthesis

Spectral Modeling

1

0.5

k

J (β)

• Music V • KL Music • “Daisy” • Additive Analysis • Additive Synthesis • FM Synthesis • FM Formula • FM Patch • FM Spectra • FM Examples • FM Voice • Sampling Synthesis • Modern Example

0 0

Physical Modeling Summary

2 −0.5 0

4 5

6

10 15

8

20 25 30

10

Order k

Argument β

Julius Smith

AES-2006 Heyser Lecture – 17 / 84

Frequency Modulation (FM) Examples Overview

All examples by John Chowning unless otherwise noted:

Early Digital Synthesis

• Music V • KL Music • “Daisy” • Additive Analysis • Additive Synthesis • FM Synthesis • FM Formula • FM Patch • FM Spectra • FM Examples • FM Voice • Sampling Synthesis • Modern Example Spectral Modeling Physical Modeling Summary

• FM brass synthesis ◦ Low Brass example ◦ Dexter Morril’s FM Trumpet • FM singing voice (1978) Each formant synthesized using an FM operator pair (two sinusoidal oscillators)

◦ Chorus ◦ Voices ◦ Basso Profundo • Other early FM synthesis ◦ Clicks and Drums ◦ Big Bell ◦ String Canon

Julius Smith

AES-2006 Heyser Lecture – 18 / 84

FM Voice

Early Digital Synthesis

• Music V • KL Music • “Daisy” • Additive Analysis • Additive Synthesis • FM Synthesis • FM Formula • FM Patch • FM Spectra • FM Examples • FM Voice • Sampling Synthesis • Modern Example

FM voice synthesis can be viewed as compressed modeling of spectral formants

Carrier 2

Carrier 1

Carrier 3

Magnitude

Overview

Spectral Modeling Physical Modeling Summary

0

Frequency

f

Modulation Frequency (all three)

Julius Smith

AES-2006 Heyser Lecture – 19 / 84

Sampling Synthesis History Overview

• 1979 - Fairlight Computer Music Instrument - 8-bit

Early Digital Synthesis

• Music V • KL Music • “Daisy” • Additive Analysis • Additive Synthesis • FM Synthesis • FM Formula • FM Patch • FM Spectra • FM Examples • FM Voice • Sampling Synthesis • Modern Example Spectral Modeling

◦ ◦ ◦ ◦ ◦

First commercial sampler Eight voices, 8 bits, 64 KB (4 sec) RAM, 16 kHz (mono) Editing, looping, mixing One could draw waveforms and additive-synthesis amplitude envelopes (for each harmonic) with a light pen $25,000–$36,000!

• 1981 - E-mu Systems Emulator ◦ First “affordable” sampler ($10,000) ◦ Eight voices, 128 K RAM, 8-bit, 80 lb.

Physical Modeling Summary

• 1986 - Ensoniq Mirage ◦ Breakthrough price-point ($1695) ◦ Eight voices, 144 K RAM, 8-bit

Julius Smith

AES-2006 Heyser Lecture – 20 / 84

Modern Sampled Piano Overview

Example:1

Early Digital Synthesis

• Music V • KL Music • “Daisy” • Additive Analysis • Additive Synthesis • FM Synthesis • FM Formula • FM Patch • FM Spectra • FM Examples • FM Voice • Sampling Synthesis • Modern Example

40 Gigabytes on ten DVDs (three sampled pianos) Every key sampled 4–10 “velocity layers” Separate recordings with soft pedal down Separate “release” recordings, for multiple striking velocities

• • • • •

Spectral Modeling Physical Modeling Summary

1

Julius Smith

Synthogy Ivory, $349 (Electronic Musician, October 2006) AES-2006 Heyser Lecture – 21 / 84

Fundamental Problem with Sampling Synthesis Overview

Piano timbre is determined by

Early Digital Synthesis

• Music V • KL Music • “Daisy” • Additive Analysis • Additive Synthesis • FM Synthesis • FM Formula • FM Patch • FM Spectra • FM Examples • FM Voice • Sampling Synthesis • Modern Example

• key number (1 byte) • key velocity (2 bytes more than enough) • pedal state (1 bit [or byte] per pedal) Piano control is relatively low-dimensional:

• Less than six bytes of information per note played • No continuous controls (typically) • Ratio of total sampled data to one note of control data

≈ one billion

Spectral Modeling Physical Modeling Summary

Julius Smith

AES-2006 Heyser Lecture – 22 / 84

Now consider bowed strings Overview

Control parameters:

Early Digital Synthesis

• Music V • KL Music • “Daisy” • Additive Analysis • Additive Synthesis • FM Synthesis • FM Formula • FM Patch • FM Spectra • FM Examples • FM Voice • Sampling Synthesis • Modern Example

• • • • • • • • •

Left-hand finger position(s) Left-hand vibrato Bow velocity Bow force Bow position Bow angle Shoulder damping Instrument orientation Player motion (within a room)

Spectral Modeling Physical Modeling Summary

Julius Smith

AES-2006 Heyser Lecture – 23 / 84

Difficulty of sampling bowed strings Overview Early Digital Synthesis

• Music V • KL Music • “Daisy” • Additive Analysis • Additive Synthesis • FM Synthesis • FM Formula • FM Patch • FM Spectra • FM Examples • FM Voice • Sampling Synthesis • Modern Example

• Bowed-string control is infinite-dimensional in principle • Many time-varying functions — “gestures” • • •

(we counted more than 10) Complete sampling of bowed strings on the level of pianos has apparently never been done Rule-driven navigation of the most useful recorded playing regimes has worked well (e.g., Synful Orchestra) Model-based approaches greatly reduce data requirements:

◦ Spectral models (inspired by sound perception) ◦ Physical models (model the sound source)

Spectral Modeling Physical Modeling Summary

Julius Smith

AES-2006 Heyser Lecture – 24 / 84

Overview Early Digital Synthesis Spectral Modeling Physical Modeling Summary

Spectral Modeling Synthesis (Historical Summary)

Julius Smith

AES-2006 Heyser Lecture – 25 / 84

Classic Vocoder Analysis & Resynthesis (Dudley 1939) magnitude, or magnitude and phase extraction

Overview Early Digital Synthesis Spectral Modeling

• Vocoder • Vocoder Examples • Sinusoidal Modeling • Spectral Trajectories • Sines + Noise • S+N Examples • S+N FX • S+N XSynth • Sines + Transients • S + N + Transients • S+N+T TSM • S+N+T Freq Map • S+N+T Windows • HF Noise Modeling • HF Noise Band • S+N+T Examples • SM Summary • Spec Future Physical Modeling

A

x0 (t)

x ˆ0 (t)

f

x(t)

A

x1 (t) f

A

Data Compression, Transmission, Storage, Manipulation, Noise reduction, ...

xN −1

x ˆ1 (t)

x ˆ(t)

x ˆN −1

f

Analysis

Processing

Synthesis

Summary

Julius Smith

AES-2006 Heyser Lecture – 26 / 84

Phase Vocoder Channel Model Analysis Model

Overview

Synthesis Model

Early Digital Synthesis

• Vocoder • Vocoder Examples • Sinusoidal Modeling • Spectral Trajectories • Sines + Noise • S+N Examples • S+N FX • S+N XSynth • Sines + Transients • S + N + Transients • S+N+T TSM • S+N+T Freq Map • S+N+T Windows • HF Noise Modeling • HF Noise Band • S+N+T Examples • SM Summary • Spec Future Physical Modeling Summary

Julius Smith

ak

Channel Filter Response

Spectral Modeling

ak(t)

A F Sine Osc

∆ωk 0

ωk

ωk+∆ωk(t)

ω Out

• Early “channel vocoder” implementations (hardware) only • •

•

measured amplitude ak (t) (Dudley 1939) The “phase vocoder” (Flanagan and Golden 1966) added phase tracking in each channel Portnoff (1976) developed the FFT phase vocoder, which replaced the heterodyne comb in computer-music additive-synthesis analysis (James A. Moorer) Inverse FFT synthesis (Rodet and Depalle 1992) gave faster sinusoidal oscillator banks AES-2006 Heyser Lecture – 27 / 84

Amplitude and Frequency Envelopes Overview

ak (t)

Early Digital Synthesis Spectral Modeling

• Vocoder • Vocoder Examples • Sinusoidal Modeling • Spectral Trajectories • Sines + Noise • S+N Examples • S+N FX • S+N XSynth • Sines + Transients • S + N + Transients • S+N+T TSM • S+N+T Freq Map • S+N+T Windows • HF Noise Modeling • HF Noise Band • S+N+T Examples • SM Summary • Spec Future

∆ωk (t) = φ˙k (t)

t

0

t

Physical Modeling Summary

Julius Smith

AES-2006 Heyser Lecture – 28 / 84

Channel Vocoder Sound Examples Overview

• Original

Early Digital Synthesis Spectral Modeling

• Vocoder • Vocoder Examples • Sinusoidal Modeling • Spectral Trajectories • Sines + Noise • S+N Examples • S+N FX • S+N XSynth • Sines + Transients • S + N + Transients • S+N+T TSM • S+N+T Freq Map • S+N+T Windows • HF Noise Modeling • HF Noise Band • S+N+T Examples • SM Summary • Spec Future

• 10 channels, sine carriers • 10 channels, narrowband-noise carriers • 26 channels, sine carriers • 26 channels, narrowband-noise carriers • 26 channels, narrowband-noise carriers, channels reversed • Phase Vocoder: Identity system in absence of modifications • The FFT Phase Vocoder next transitioned to the Short-Time Fourier Transform (STFT) (Allen and Rabiner 1977)

Physical Modeling Summary

Julius Smith

AES-2006 Heyser Lecture – 29 / 84

Tracking Spectral Peaks in the Short-Time Fourier Transform Overview

atan

Early Digital Synthesis Spectral Modeling

• Vocoder • Vocoder Examples • Sinusoidal Modeling • Spectral Trajectories • Sines + Noise • S+N Examples • S+N FX • S+N XSynth • Sines + Transients • S + N + Transients • S+N+T TSM • S+N+T Freq Map • S+N+T Windows • HF Noise Modeling • HF Noise Band • S+N+T Examples • SM Summary • Spec Future

Phases

s(t)

FFT

Frequencies

dB mag

Peak tracking

Quadratic Peak Interpolation

window w(n) Amplitudes

• STFT peak tracking at CCRMA: mid-1980s (PARSHL program) • Motivated by vocoder analysis of piano tones • Influences: STFT (Allen and Rabiner 1977), •

ADEC (1977), MAPLE (1979) Independently developed for speech coding by McAulay and Quatieri at Lincoln Labs

Physical Modeling Summary

Julius Smith

AES-2006 Heyser Lecture – 30 / 84

Example Spectral Trajectories Overview

f

Early Digital Synthesis Spectral Modeling

• Vocoder • Vocoder Examples • Sinusoidal Modeling • Spectral Trajectories • Sines + Noise • S+N Examples • S+N FX • S+N XSynth • Sines + Transients • S + N + Transients • S+N+T TSM • S+N+T Freq Map • S+N+T Windows • HF Noise Modeling • HF Noise Band • S+N+T Examples • SM Summary • Spec Future

t

Physical Modeling Summary

Julius Smith

AES-2006 Heyser Lecture – 31 / 84

Sines + Noise Synthesis (1989) PSfrag Overview

A1 (t) f1 (t) A2 (t) f2 (t) A3 (t) f3 (t) A4 (t) f4 (t)

Early Digital Synthesis Spectral Modeling

• Vocoder • Vocoder Examples • Sinusoidal Modeling • Spectral Trajectories • Sines + Noise • S+N Examples • S+N FX • S+N XSynth • Sines + Transients • S + N + Transients • S+N+T TSM • S+N+T Freq Map • S+N+T Windows • HF Noise Modeling • HF Noise Band • S+N+T Examples • SM Summary • Spec Future Physical Modeling Summary

Julius Smith

white noise

u(t)

y(t) =

P

filter(t)

ht (τ ) 4 P

i=1

Ai (t) cos

hR t

i 0 ωi (t)dt + φi (0) + (ht ∗ u)(t) AES-2006 Heyser Lecture – 32 / 84

Sines + Noise Sound Examples Overview

Xavier Serra 1989 thesis demos (Sines + Noise signal modeling)

Early Digital Synthesis Spectral Modeling

• Vocoder • Vocoder Examples • Sinusoidal Modeling • Spectral Trajectories • Sines + Noise • S+N Examples • S+N FX • S+N XSynth • Sines + Transients • S + N + Transients • S+N+T TSM • S+N+T Freq Map • S+N+T Windows • HF Noise Modeling • HF Noise Band • S+N+T Examples • SM Summary • Spec Future

• Piano ◦ ◦ ◦ ◦

Original Sinusoids alone Residual after sinusoids removed Sines + noise model

• Voice ◦ ◦ ◦ ◦

Original Sinusoids Residual Synthesis

Physical Modeling Summary

Julius Smith

AES-2006 Heyser Lecture – 33 / 84

Musical Effects with Sines+Noise Models (Serra 1989) Overview

• Piano Effects

Early Digital Synthesis Spectral Modeling

• Vocoder • Vocoder Examples • Sinusoidal Modeling • Spectral Trajectories • Sines + Noise • S+N Examples • S+N FX • S+N XSynth • Sines + Transients • S + N + Transients • S+N+T TSM • S+N+T Freq Map • S+N+T Windows • HF Noise Modeling • HF Noise Band • S+N+T Examples • SM Summary • Spec Future

◦ Pitch downshift one octave ◦ Pitch flattened ◦ Varying partial stretching • Voice Effects ◦ Frequency-scale by 0.6 ◦ Frequency-scale by 0.4 and stretch partials ◦ Variable time-scaling, deterministic to stochastic

Physical Modeling Summary

Julius Smith

AES-2006 Heyser Lecture – 34 / 84

Cross-Synthesis with Sines+Noise Models (Serra 1989) Overview

• Voice “modulator”

Early Digital Synthesis Spectral Modeling

• Vocoder • Vocoder Examples • Sinusoidal Modeling • Spectral Trajectories • Sines + Noise • S+N Examples • S+N FX • S+N XSynth • Sines + Transients • S + N + Transients • S+N+T TSM • S+N+T Freq Map • S+N+T Windows • HF Noise Modeling • HF Noise Band • S+N+T Examples • SM Summary • Spec Future

• Creaking ship’s mast “carrier” • Voice-modulated creaking mast • Same with modified spectral envelopes

Physical Modeling Summary

Julius Smith

AES-2006 Heyser Lecture – 35 / 84

Sines + Transients Sound Examples (Serra 1989) Overview Early Digital Synthesis

In this technique, the sinusoidal sum is phase-matched at the cross-over point only (with no cross-fade).

Spectral Modeling

• Vocoder • Vocoder Examples • Sinusoidal Modeling • Spectral Trajectories • Sines + Noise • S+N Examples • S+N FX • S+N XSynth • Sines + Transients • S + N + Transients • S+N+T TSM • S+N+T Freq Map • S+N+T Windows • HF Noise Modeling • HF Noise Band • S+N+T Examples • SM Summary • Spec Future

• Marimba ◦ Original ◦ Sinusoidal model ◦ Original attack, followed by sinusoidal model • Piano ◦ Original ◦ Sinusoidal model ◦ Original attack, followed by sinusoidal model

Physical Modeling Summary

Julius Smith

AES-2006 Heyser Lecture – 36 / 84

Multiresolution Sines + Noise + Transients (Levine 1998) Overview

Why Model Transients Separately?

Early Digital Synthesis Spectral Modeling

• Vocoder • Vocoder Examples • Sinusoidal Modeling • Spectral Trajectories • Sines + Noise • S+N Examples • S+N FX • S+N XSynth • Sines + Transients • S + N + Transients • S+N+T TSM • S+N+T Freq Map • S+N+T Windows • HF Noise Modeling • HF Noise Band • S+N+T Examples • SM Summary • Spec Future

• • • • • •

Sinusoids efficiently model spectral peaks over time Filtered noise efficiently models spectral residual vs. t Neither is good for abrupt transients in the waveform Phase-matched oscillators are expensive More efficient to switch to a transient model during transients Need sinusoidal phase matching at the switching times

Transient models:

• • • •

Original waveform slice (1988) Wavelet expansion (Ali 1996) MPEG-2 AAC (with short window) (Levine 1998) Frequency-domain LPC (time-domain amplitude envelope) (Verma 2000)

Physical Modeling Summary

Julius Smith

AES-2006 Heyser Lecture – 37 / 84

Time Scale Modification of Sines + Noise + Transients Models

original signal

Spectral Modeling

sines + noise

sines + noise

sines + noise

transients

Physical Modeling

time-scaled signal

sines + noise

transients

• Vocoder • Vocoder Examples • Sinusoidal Modeling • Spectral Trajectories • Sines + Noise • S+N Examples • S+N FX • S+N XSynth • Sines + Transients • S + N + Transients • S+N+T TSM • S+N+T Freq Map • S+N+T Windows • HF Noise Modeling • HF Noise Band • S+N+T Examples • SM Summary • Spec Future

sines + noise

transients

Early Digital Synthesis

transients

Overview

sines + noise

time

Time-Scale Modification (TSM) becomes well defined:

• • • •

Transients are translated in time Sinusoidal envelopes are scaled in time Noise-filter envelopes also scaled in time Dual of TSM is frequency scaling

Summary

Julius Smith

AES-2006 Heyser Lecture – 38 / 84

Sines + Noise + Transients Time-Frequency Map 16

Overview 14

Early Digital Synthesis

Physical Modeling

frequency [kHz]

• Vocoder • Vocoder Examples • Sinusoidal Modeling • Spectral Trajectories • Sines + Noise • S+N Examples • S+N FX • S+N XSynth • Sines + Transients • S + N + Transients • S+N+T TSM • S+N+T Freq Map • S+N+T Windows • HF Noise Modeling • HF Noise Band • S+N+T Examples • SM Summary • Spec Future

12 10 8 6 4 2 0

0

50

100

150

200

250

0

50

100

150

200

250

1

amplitude

Spectral Modeling

0.5 0 −0.5 −1

time [milliseconds]

(Levine 1998)

Summary

Julius Smith

AES-2006 Heyser Lecture – 39 / 84

Overview Early Digital Synthesis

transient

Corresponding Analysis Windows

low octave

amplitude

• Vocoder • Vocoder Examples • Sinusoidal Modeling • Spectral Trajectories • Sines + Noise • S+N Examples • S+N FX • S+N XSynth • Sines + Transients • S + N + Transients • S+N+T TSM • S+N+T Freq Map • S+N+T Windows • HF Noise Modeling • HF Noise Band • S+N+T Examples • SM Summary • Spec Future

middle octave high octave

Spectral Modeling

Physical Modeling

0 Summary

Julius Smith

50

100

150

200

250

time [milliseconds]

AES-2006 Heyser Lecture – 40 / 84

Quasi-Constant-Q (Wavelet) Time-Frequency Map 16

Overview

14

Early Digital Synthesis

frequency [kHz]

• Vocoder • Vocoder Examples • Sinusoidal Modeling • Spectral Trajectories • Sines + Noise • S+N Examples • S+N FX • S+N XSynth • Sines + Transients • S + N + Transients • S+N+T TSM • S+N+T Freq Map • S+N+T Windows • HF Noise Modeling • HF Noise Band • S+N+T Examples • SM Summary • Spec Future

12 10 8 6 4 2 0

4 0 x 10

50

100

150

200

250

0

50

100

150

200

250

2

amplitude

Spectral Modeling

1 0 −1 −2

time [milliseconds]

Physical Modeling Summary

Julius Smith

AES-2006 Heyser Lecture – 41 / 84

Bark-Band Noise Modeling at High Frequencies (Levine 1998) Overview

85

Early Digital Synthesis

80 Spectral Modeling

Physical Modeling Summary

Julius Smith

75 70 65

magnitude [dB]

• Vocoder • Vocoder Examples • Sinusoidal Modeling • Spectral Trajectories • Sines + Noise • S+N Examples • S+N FX • S+N XSynth • Sines + Transients • S + N + Transients • S+N+T TSM • S+N+T Freq Map • S+N+T Windows • HF Noise Modeling • HF Noise Band • S+N+T Examples • SM Summary • Spec Future

60 55 50 45 40 35 30

0

500

1000

1500

2000

2500

3000

3500

4000

4500

frequency [Hz]

AES-2006 Heyser Lecture – 42 / 84

Amplitude Envelope for One Noise Band Overview

• Vocoder • Vocoder Examples • Sinusoidal Modeling • Spectral Trajectories • Sines + Noise • S+N Examples • S+N FX • S+N XSynth • Sines + Transients • S + N + Transients • S+N+T TSM • S+N+T Freq Map • S+N+T Windows • HF Noise Modeling • HF Noise Band • S+N+T Examples • SM Summary • Spec Future

Original Mag. [dB]

Spectral Modeling

70

60

50

40

50

100

150

200

250

300

350

0

50

100

150

200

250

300

350

75 70 65 60 55 50 45

Physical Modeling

0

80

LSA Mag. [dB]

Early Digital Synthesis

80

time [milliseconds] Summary

Julius Smith

For more information, see Scott Levine’s thesis.2 2 AES-2006 Heyser Lecture – 43 / 84 http://ccrma.stanford.edu/~scottl/thesis.html

Sines + Noise + Transients Sound Examples Overview Early Digital Synthesis

Scott Levine Thesis Demos (Sines + Noise + Transients at 32 kbps) (http://ccrma.stanford.edu/~scottl/thesis.html)

Spectral Modeling

• Vocoder • Vocoder Examples • Sinusoidal Modeling • Spectral Trajectories • Sines + Noise • S+N Examples • S+N FX • S+N XSynth • Sines + Transients • S + N + Transients • S+N+T TSM • S+N+T Freq Map • S+N+T Windows • HF Noise Modeling • HF Noise Band • S+N+T Examples • SM Summary • Spec Future

Mozart’s Le Nozze di Figaro

• Original • Compressed using MPEG-AAC at 32 kbps • Compressed using sines+transients+noise at 32 kbps • • • •

Multiresolution sinusoids alone Residual Bark-band noise Transform-coded transients (AAC) Bark-band noise above 5 kHz

Physical Modeling Summary

Julius Smith

AES-2006 Heyser Lecture – 44 / 84

Rock Example Overview Early Digital Synthesis

Scott Levine Thesis Demos (Sines + Noise + Transients at 32 kbps) (http://ccrma.stanford.edu/~scottl/thesis.html)

Spectral Modeling

• Vocoder • Vocoder Examples • Sinusoidal Modeling • Spectral Trajectories • Sines + Noise • S+N Examples • S+N FX • S+N XSynth • Sines + Transients • S + N + Transients • S+N+T TSM • S+N+T Freq Map • S+N+T Windows • HF Noise Modeling • HF Noise Band • S+N+T Examples • SM Summary • Spec Future

“It Takes Two” by Rob Base & DJ E-Z Rock

• Original • MPEG-AAC at 32 kbps • Sines+transients+noise at 32 kbps • • • •

Multiresolution sinusoids Residual Bark-band noise Transform-coded transients (AAC) Bark-band noise above 5 kHz

Physical Modeling Summary

Julius Smith

AES-2006 Heyser Lecture – 45 / 84

Time Scale Modification using Sines + Noise + Transients Overview Early Digital Synthesis

Scott Levine Thesis Demos (Sines + Noise + Transients at 32 kbps) (http://ccrma.stanford.edu/~scottl/thesis.html)

Spectral Modeling

• Vocoder • Vocoder Examples • Sinusoidal Modeling • Spectral Trajectories • Sines + Noise • S+N Examples • S+N FX • S+N XSynth • Sines + Transients • S + N + Transients • S+N+T TSM • S+N+T Freq Map • S+N+T Windows • HF Noise Modeling • HF Noise Band • S+N+T Examples • SM Summary • Spec Future

Time-Scale Modification (pitch unchanged)

• S+N+T time-scale factors [2.0, 1.6, 1.2, 1.0, 0.8, 0.6, 0.5]

S+N+T Pitch Shifting (timing unchanged)

• Pitch-scale factors [0.89, 0.94, 1.00, 1.06, 1.12]

Physical Modeling Summary

Julius Smith

AES-2006 Heyser Lecture – 46 / 84

Spectral Modeling History Highlights

• • • • • • • • • • •

Fourier’s theory (1822) Teleharmonium (1906) Hammond organ (1930s) Channel Vocoder (1939) Phase Vocoder (1966) “Additive Synthesis” (1969) FFT Phase Vocoder (1976) Sinusoidal Modeling (1977,1979,1985) Sines+Noise (1989) Sines+Transients (1989) Sines+Noise+Transients (1998)

Julius Smith

Perceptual audio coding:

• • • • • • •

Princen-Bradley filterbank (1986) K. Brandenburg thesis (1989) Auditory masking usage Dolby AC2 Musicam ASPEC MPEG-I,II,IV (incl. S+N+T “parametric sounds”)

AES-2006 Heyser Lecture – 47 / 84

Future Prospects Overview

Observations:

Early Digital Synthesis Spectral Modeling

• Vocoder • Vocoder Examples • Sinusoidal Modeling • Spectral Trajectories • Sines + Noise • S+N Examples • S+N FX • S+N XSynth • Sines + Transients • S + N + Transients • S+N+T TSM • S+N+T Freq Map • S+N+T Windows • HF Noise Modeling • HF Noise Band • S+N+T Examples • SM Summary • Spec Future

• Sinusoidal modeling of sound is “Unreasonably Effective” • Basic “auditory masking” discards ≈ 90% information • Interesting neuroscience observation: “... most neurons in the primary auditory cortex A1 are silent most of the time ...”

•

(from “Sparse Time-Frequency Representations”, Gardner and Magnesco, PNAS:103(16), April 2006) What is a true and correct “psychospectral model” for sound?

◦ The cochlea of the ear is a real-time spectrum analyzer ◦ How is the “ear’s spectrogram” represented at higher levels?

Physical Modeling Summary

Julius Smith

AES-2006 Heyser Lecture – 48 / 84

Overview Early Digital Synthesis Spectral Modeling Physical Modeling Summary

Physical Modeling Synthesis (Historical Summary)

Julius Smith

AES-2006 Heyser Lecture – 49 / 84

Kelly-Lochbaum Vocal Tract Model Overview

…

Early Digital Synthesis Spectral Modeling

e(n)

y(n)

Physical Modeling

• KL Music • Digital Waveguide • Signal Scattering • Plucked String • Struck String • Karplus Strong • EKS Algorithm • Clarinet • Wind Examples • Bowed Strings • Distortion Guitar • Acoustic Strings • Sound Examples • Linearized Violin • Commuted Piano • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

Glottal Pulse Train or Noise e(n)

1 + k1 z− 1 2 k1

(Unused Allpass Output)

− k1

R1 z− 1 2

1 − k1

…

1 + kM z− 1 2 kM

…

− kM

Speech Output y(n)

RM z− 1 2

1 − kM

Kelly-Lochbaum Vocal Tract Model (Piecewise Cylindrical)

John L. Kelly and Carol Lochbaum (1962)

Summary

Julius Smith

AES-2006 Heyser Lecture – 50 / 84

Digital Waveguide Models (1985) Overview

Lossless digital waveguide

∆

= bidirectional delay line

Early Digital Synthesis

at some wave impedance R

Spectral Modeling Physical Modeling

• KL Music • Digital Waveguide • Signal Scattering • Plucked String • Struck String • Karplus Strong • EKS Algorithm • Clarinet • Wind Examples • Bowed Strings • Distortion Guitar • Acoustic Strings • Sound Examples • Linearized Violin • Commuted Piano • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

z−N

R z−N Useful for efficient models of

• • • •

strings bores plane waves conical waves

Summary

Julius Smith

AES-2006 Heyser Lecture – 51 / 84

Signal Scattering Overview

Signal scattering is caused by a change in wave impedance R:

Early Digital Synthesis

R − R1 k1 = 2 R2 + R1

Spectral Modeling Physical Modeling

• KL Music • Digital Waveguide • Signal Scattering • Plucked String • Struck String • Karplus Strong • EKS Algorithm • Clarinet • Wind Examples • Bowed Strings • Distortion Guitar • Acoustic Strings • Sound Examples • Linearized Violin • Commuted Piano • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

1 + k1 z−N R1

z−N k1

− k1

z−N

R2 z−N

1 − k1

If the wave impedance changes every spatial sample, the Kelly-Lochbaum vocal-tract model results.

Summary

Julius Smith

AES-2006 Heyser Lecture – 52 / 84

Ideal Plucked String (Displacement Waves) Overview Early Digital Synthesis Spectral Modeling

+

+

y (n)

y (n-N/2)

Physical Modeling

• KL Music • Digital Waveguide • Signal Scattering • Plucked String • Struck String • Karplus Strong • EKS Algorithm • Clarinet • Wind Examples • Bowed Strings • Distortion Guitar • Acoustic Strings • Sound Examples • Linearized Violin • Commuted Piano • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

“Bridge”

-1

y-(n) (x = 0)

(x = Pluck Position)

-1

“Nut” y-(n+N/2)

(x = L)

• Load each delay line with half of initial string displacement • Sum of upper and lower delay lines = string displacement

Summary

Julius Smith

AES-2006 Heyser Lecture – 53 / 84

Ideal Struck String (Velocity Waves) Overview Early Digital Synthesis Spectral Modeling

c

v+(n)

v+(n-N/2)

Physical Modeling

• KL Music • Digital Waveguide • Signal Scattering • Plucked String • Struck String • Karplus Strong • EKS Algorithm • Clarinet • Wind Examples • Bowed Strings • Distortion Guitar • Acoustic Strings • Sound Examples • Linearized Violin • Commuted Piano • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

“Bridge”

-1

v-(n) (x = 0)

(x = Hammer Position)

-1

c

“Nut” v-(n+N/2)

(x = L)

Hammer strike = momentum transfer = velocity step:

mh vh (0−) = (mh + ms )vs (0+)

Summary

Julius Smith

AES-2006 Heyser Lecture – 54 / 84

Karplus-Strong (KS) Algorithm (1983) Overview Early Digital Synthesis Spectral Modeling

+

Output y (n)

+

y (n-N)

N samples delay

Physical Modeling

• KL Music • Digital Waveguide • Signal Scattering • Plucked String • Struck String • Karplus Strong • EKS Algorithm • Clarinet • Wind Examples • Bowed Strings • Distortion Guitar • Acoustic Strings • Sound Examples • Linearized Violin • Commuted Piano • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

1/2 1/2

z -1

• Discovered (1978) as “self-modifying wavetable synthesis” • Wavetable is preferably initialized with random numbers

Summary

Julius Smith

AES-2006 Heyser Lecture – 55 / 84

Karplus-Strong Sound Examples Overview

• “Vintage” 8-bit sound examples:

Early Digital Synthesis Spectral Modeling Physical Modeling

• KL Music • Digital Waveguide • Signal Scattering • Plucked String • Struck String • Karplus Strong • EKS Algorithm • Clarinet • Wind Examples • Bowed Strings • Distortion Guitar • Acoustic Strings • Sound Examples • Linearized Violin • Commuted Piano • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

• Original Plucked String: (WAV) (MP3) • Drum: (WAV) (MP3) • Stretched Drum: (WAV) (MP3)

Summary

Julius Smith

AES-2006 Heyser Lecture – 56 / 84

EKS Algorithm (Jaffe-Smith 1983) Overview Early Digital Synthesis

Hp (z)

z −N

Hβ (z)

HL (z)

Spectral Modeling

Hρ (z)

Physical Modeling

• KL Music • Digital Waveguide • Signal Scattering • Plucked String • Struck String • Karplus Strong • EKS Algorithm • Clarinet • Wind Examples • Bowed Strings • Distortion Guitar • Acoustic Strings • Sound Examples • Linearized Violin • Commuted Piano • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

Hs (z)

Hd (z)

= pitch period (2× string length) in samples 1−p = pick-direction lowpass filter Hp (z) = −1 1 − pz N

Hβ (z) = 1 − z −βN = pick-position comb filter, β ∈ (0, 1) Hd (z) = string-damping filter (one/two poles/zeros typical) Hs (z) = string-stiffness allpass filter (several poles and zeros) Hρ (z) = HL (z) =

ρ(N ) − z −1 = first-order string-tuning allpass filter 1 − ρ(N ) z −1 1 − RL = dynamic-level lowpass filter 1 − RL z −1

Summary

Julius Smith

AES-2006 Heyser Lecture – 57 / 84

STK EKS Sound Examples Overview Early Digital Synthesis Spectral Modeling

• Synthesis Tool Kit (STK) by Perry Cook, Gary Scavone, and others — distributed by CCRMA: Google search: STK ToolKit

Physical Modeling

• KL Music • Digital Waveguide • Signal Scattering • Plucked String • Struck String • Karplus Strong • EKS Algorithm • Clarinet • Wind Examples • Bowed Strings • Distortion Guitar • Acoustic Strings • Sound Examples • Linearized Violin • Commuted Piano • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

STK Plucked String: (WAV) (MP3)

• Plucked String 1: (WAV) (MP3) • Plucked String 2: (WAV) (MP3) • Plucked String 3: (WAV) (MP3)

Summary

Julius Smith

AES-2006 Heyser Lecture – 58 / 84

EKS Sound Example (1988) Overview

Bach A-Minor Concerto—Orchestra Part: (WAV) (MP3)

Early Digital Synthesis Spectral Modeling Physical Modeling

• KL Music • Digital Waveguide • Signal Scattering • Plucked String • Struck String • Karplus Strong • EKS Algorithm • Clarinet • Wind Examples • Bowed Strings • Distortion Guitar • Acoustic Strings • Sound Examples • Linearized Violin • Commuted Piano • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

• Executed in real time on one Motorola DSP56001 (20 MHz clock, 128K SRAM)

• Developed for the NeXT Computer introduction at Davies Symphony Hall, San Francisco, 1988

• Solo violin part was played live by Dan Kobialka of the San Francisco Symphony

Summary

Julius Smith

AES-2006 Heyser Lecture – 59 / 84

Digital Waveguide Single Reed, Cylindrical Bore Model (1986) Overview Early Digital Synthesis Spectral Modeling Physical Modeling

• KL Music • Digital Waveguide • Signal Scattering • Plucked String • Struck String • Karplus Strong • EKS Algorithm • Clarinet • Wind Examples • Bowed Strings • Distortion Guitar • Acoustic Strings • Sound Examples • Linearized Violin • Commuted Piano • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

Mouth Pressure pm ( n ) 2

p−b ( n )

hm

-

Reed Table ρˆ

*

Reflection Filter h+∆

-

Embouchure Offset

Output Filter

Reed to Bell Delay

Bell to Reed Delay

p+b ( n )

Reed

Bore

Bell

Digital waveguide clarinet

• Control variable = mouth half-pressure • Total reed cost = two subtractions, one multiply, and one table lookup per sample

Summary

Julius Smith

AES-2006 Heyser Lecture – 60 / 84

Digital Waveguide Wind Instrument Sound Examples Overview

• STK Clarinet: (WAV) (MP3)

Early Digital Synthesis

Google search: STK clarinet

Spectral Modeling

◦ Synthesis Tool Kit (STK) by Perry Cook, Gary Scavone, and

Physical Modeling

• KL Music • Digital Waveguide • Signal Scattering • Plucked String • Struck String • Karplus Strong • EKS Algorithm • Clarinet • Wind Examples • Bowed Strings • Distortion Guitar • Acoustic Strings • Sound Examples • Linearized Violin • Commuted Piano • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

others — distributed by CCRMA: Google search: STK ToolKit

• Staccato Systems Slide Flute •

(based on STK flute, ca. 1995): (WAV) (MP3) Yamaha VL1 “Virtual Lead” synthesizer demos (1994):

• Shakuhachi: (WAV) (MP3) • Oboe and Bassoon: (WAV) (MP3) • Tenor Saxophone: (WAV) (MP3)

Summary

Julius Smith

AES-2006 Heyser Lecture – 61 / 84

Digital Waveguide Bowed Strings (1986) Bow Force Bow Velocity

Overview Early Digital Synthesis

Nut to Bow Delay vb

Physical Modeling

-1

Nut

-

v+∆

Bow to Nut Delay

String

Body Filter

Bow to Bridge Delay

Spectral Modeling

• KL Music • Digital Waveguide • Signal Scattering • Plucked String • Struck String • Karplus Strong • EKS Algorithm • Clarinet • Wind Examples • Bowed Strings • Distortion Guitar • Acoustic Strings • Sound Examples • Linearized Violin • Commuted Piano • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

− vs, r

+ vs, l

ρˆ Bow Table

+ vs, r − vs, l

Bow

Reflection Filter

*

Bridge to Bow Delay

String

BridgeBody

Air

• Reflection filter summarizes all losses per period •

(due to bridge, bow, finger, etc.) Bow-string junction = memoryless lookup table (or segmented polynomial)

Summary

Julius Smith

AES-2006 Heyser Lecture – 62 / 84

“Electric Cello” Sound Examples (Peder Larson) Overview

• Staccato Notes: (WAV) (MP3)

Early Digital Synthesis Spectral Modeling

•

(short strokes of high bow pressure, as from a bouncing bow) Bach’s First Suite for Unaccompanied Cello: (WAV) (MP3)

Physical Modeling

• KL Music • Digital Waveguide • Signal Scattering • Plucked String • Struck String • Karplus Strong • EKS Algorithm • Clarinet • Wind Examples • Bowed Strings • Distortion Guitar • Acoustic Strings • Sound Examples • Linearized Violin • Commuted Piano • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding? Summary

Julius Smith

AES-2006 Heyser Lecture – 63 / 84

Soft Clipper Overview Early Digital Synthesis

f (x) =

Spectral Modeling

x−

  

Physical Modeling

x ≤ −1 x3 3 ,

2 3,

−1 ≤ x ≤ 1 x≥1 3

x=−1:0.01:1; plot([−(2/3)*ones(1,100), x−x. /3, (2/3)*ones(1,100)]) 0.8

0.6

0.4

0.2

f(x(n))

• KL Music • Digital Waveguide • Signal Scattering • Plucked String • Struck String • Karplus Strong • EKS Algorithm • Clarinet • Wind Examples • Bowed Strings • Distortion Guitar • Acoustic Strings • Sound Examples • Linearized Violin • Commuted Piano • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

 2 − ,    3

0

−0.2

−0.4

−0.6

−0.8 −2

−1.5

−1

−0.5

0 x(n)

0.5

1

1.5

2

Summary

Julius Smith

AES-2006 Heyser Lecture – 64 / 84

Amplifier Distortion + Amplifier Feedback Overview

Sullivan 1990

Early Digital Synthesis Spectral Modeling Physical Modeling

• KL Music • Digital Waveguide • Signal Scattering • Plucked String • Struck String • Karplus Strong • EKS Algorithm • Clarinet • Wind Examples • Bowed Strings • Distortion Guitar • Acoustic Strings • Sound Examples • Linearized Violin • Commuted Piano • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

Pre-distortion output level String 1 Nonlinear Distortion

.. .

Output Signal Pre-distortion gain

String N

Distortion output level Amplifier Feedback Gain

Amplifier Feedback Delay

Distortion output signal often further filtered by an amplifier cabinet filter, representing speaker cabinet, driver responses, etc.

Summary

Julius Smith

AES-2006 Heyser Lecture – 65 / 84

Distortion Guitar Sound Examples Overview

(Stanford Sondius Project, ca. 1995)

Early Digital Synthesis Spectral Modeling Physical Modeling

• KL Music • Digital Waveguide • Signal Scattering • Plucked String • Struck String • Karplus Strong • EKS Algorithm • Clarinet • Wind Examples • Bowed Strings • Distortion Guitar • Acoustic Strings • Sound Examples • Linearized Violin • Commuted Piano • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

• Distortion Guitar: (WAV) (MP3) • Amplifier Feedback 1: (WAV) (MP3) • Amplifier Feedback 2: (WAV) (MP3)

Summary

Julius Smith

AES-2006 Heyser Lecture – 66 / 84

Commuted Synthesis of Acoustic Strings (1993) Overview Early Digital Synthesis

e(t) Trigger

y(t)

s(t) String

Excitation

Resonator

Output

Spectral Modeling Physical Modeling

• KL Music • Digital Waveguide • Signal Scattering • Plucked String • Struck String • Karplus Strong • EKS Algorithm • Clarinet • Wind Examples • Bowed Strings • Distortion Guitar • Acoustic Strings • Sound Examples • Linearized Violin • Commuted Piano • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

Schematic diagram of a stringed musical instrument.

Trigger

Excitation

Resonator

String

Output

Equivalent diagram in the linear, time-invariant case.

Trigger

Aggregate Excitation

a(t)

x(t) String

Output

Use of an aggregate excitation given by the convolution of original excitation with the resonator impulse response.

Summary

Julius Smith

AES-2006 Heyser Lecture – 67 / 84

Commuted Components Overview Early Digital Synthesis

Trigger

Spectral Modeling

Aggregate Excitation

a(t)

x(t) String

Output

Physical Modeling

• KL Music • Digital Waveguide • Signal Scattering • Plucked String • Struck String • Karplus Strong • EKS Algorithm • Clarinet • Wind Examples • Bowed Strings • Distortion Guitar • Acoustic Strings • Sound Examples • Linearized Violin • Commuted Piano • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

“Plucked Resonator” driving a String.

s(t)

Bridge Coupling

Guitar Body

Air Absorption

Room Response

y(t) Output

Possible components of a guitar resonator.

Summary

Julius Smith

AES-2006 Heyser Lecture – 68 / 84

Sound Examples Overview Early Digital Synthesis

Electric Guitar (Pick-Ups and/or Body-Model Added) (Stanford Sondius Project → Staccato Systems, Inc. → ADI, ca. 1995)

Spectral Modeling Physical Modeling

• KL Music • Digital Waveguide • Signal Scattering • Plucked String • Struck String • Karplus Strong • EKS Algorithm • Clarinet • Wind Examples • Bowed Strings • Distortion Guitar • Acoustic Strings • Sound Examples • Linearized Violin • Commuted Piano • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

• • • •

Example 1: (WAV) (MP3) Example 2: (WAV) (MP3) Example 3: (WAV) (MP3) Virtual “wah-wah pedal”: (WAV) (MP3)

STK Mandolin

• STK Mandolin 1: (WAV) (MP3) • STK Mandolin 2: (WAV) (MP3)

Summary

Julius Smith

AES-2006 Heyser Lecture – 69 / 84

Sound Examples Overview

More Recent Acoustic Guitar

Early Digital Synthesis Spectral Modeling Physical Modeling

• KL Music • Digital Waveguide • Signal Scattering • Plucked String • Struck String • Karplus Strong • EKS Algorithm • Clarinet • Wind Examples • Bowed Strings • Distortion Guitar • Acoustic Strings • Sound Examples • Linearized Violin • Commuted Piano • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

• Bach Prelude in E Major: (WAV) (MP3) • soundexamplewavBach silenceLoure in E Major: (WAV) (MP3) Virtual performance by Dr. Mikael Laurson, Sibelius Institute Virtual guitar by Helsinki Univ. of Tech., Acoustics Lab3

3

http://www.acoustics.hut.fi/

Summary

Julius Smith

AES-2006 Heyser Lecture – 70 / 84

Commuted Synthesis of Linearized Violin a) Overview Early Digital Synthesis Spectral Modeling Physical Modeling

• KL Music • Digital Waveguide • Signal Scattering • Plucked String • Struck String • Karplus Strong • EKS Algorithm • Clarinet • Wind Examples • Bowed Strings • Distortion Guitar • Acoustic Strings • Sound Examples • Linearized Violin • Commuted Piano • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

Amplitude(n) Frequency(n)

Impulse Train

b) Amplitude(n) Frequency(n)

Impulse Train

s(n)

e(n) String

e(n)

c) Amplitude(n) Frequency(n)

Output x(n) Resonator

Impulse-Response Train

Resonator

Output x(n)

a(n) String

Output x(n)

a(n) String

• Assumes ideal Helmholtz motion of string • Sound Examples (Stanford Sondius project, ca. 1995): ◦ Violin 1: (WAV) (MP3) ◦ Bass: (WAV) (MP3) ◦ Violin 2: (WAV) (MP3) ◦ Cello: (WAV) (MP3) ◦ Ensemble: (WAV) (MP3) ◦ Viola 1: (WAV) (MP3) ◦ Viola 2: (WAV) (MP3)

Summary

Julius Smith

AES-2006 Heyser Lecture – 71 / 84

Commuted Piano Synthesis (1995) Overview

Hammer-string interaction pulses (force):

Early Digital Synthesis Spectral Modeling

Force 0.5

Physical Modeling

• KL Music • Digital Waveguide • Signal Scattering • Plucked String • Struck String • Karplus Strong • EKS Algorithm • Clarinet • Wind Examples • Bowed Strings • Distortion Guitar • Acoustic Strings • Sound Examples • Linearized Violin • Commuted Piano • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

0.4 0.3 0.2 0.1 5

10

15

Time 20

Summary

Julius Smith

AES-2006 Heyser Lecture – 72 / 84

Synthesis of Hammer-String Interaction Pulse Overview

Impulse

Impulse Response

Early Digital Synthesis Spectral Modeling

Lowpass Filter

Physical Modeling

• KL Music • Digital Waveguide • Signal Scattering • Plucked String • Struck String • Karplus Strong • EKS Algorithm • Clarinet • Wind Examples • Bowed Strings • Distortion Guitar • Acoustic Strings • Sound Examples • Linearized Violin • Commuted Piano • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

Time

Time

• Faster collisions correspond to narrower pulses (nonlinear filter )

• For a given velocity, filter is linear time-invariant • Piano is “linearized” for each hammer velocity

Summary

Julius Smith

AES-2006 Heyser Lecture – 73 / 84

Multiple Hammer-String Interaction Pulses Overview

Superimpose several individual pulses:

Early Digital Synthesis Spectral Modeling Physical Modeling

• KL Music • Digital Waveguide • Signal Scattering • Plucked String • Struck String • Karplus Strong • EKS Algorithm • Clarinet • Wind Examples • Bowed Strings • Distortion Guitar • Acoustic Strings • Sound Examples • Linearized Violin • Commuted Piano • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

Force Impulse 1 Impulse 2 Impulse 3

δ1

LPF1

δ2

LPF2

δ3

LPF3

+

String Input

0

Time

Summary

Julius Smith

AES-2006 Heyser Lecture – 74 / 84

Multiple Hammer-String Interaction Pulses Overview

Superimpose several individual pulses:

Early Digital Synthesis Spectral Modeling Physical Modeling

• KL Music • Digital Waveguide • Signal Scattering • Plucked String • Struck String • Karplus Strong • EKS Algorithm • Clarinet • Wind Examples • Bowed Strings • Distortion Guitar • Acoustic Strings • Sound Examples • Linearized Violin • Commuted Piano • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

Force Impulse 1 Impulse 2 Impulse 3

δ1

LPF1

δ2

LPF2

δ3

LPF3

+

String Input

0

Time

As impulse amplitude grows (faster hammer strike), output pulses become taller and thinner, showing less overlap.

Summary

Julius Smith

AES-2006 Heyser Lecture – 74 / 84

Complete Piano Model Natural Ordering: Overview Early Digital Synthesis

vc

Spectral Modeling Physical Modeling

• KL Music • Digital Waveguide • Signal Scattering • Plucked String • Struck String • Karplus Strong • EKS Algorithm • Clarinet • Wind Examples • Bowed Strings • Distortion Guitar • Acoustic Strings • Sound Examples • Linearized Violin • Commuted Piano • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

Impulse δ1 Tapped δ1 GenerDelay Trigger ator Line δ2

LPF1

δ3

LPF3

LPF2

+

String

Sound Board & Enclosure Output

Commuted Ordering:

vc Sound Board & Enclosure Trigger Impulse Response

Tapped Delay Line

LPF1 LPF2

+

String

Output

LPF3

• Soundboard and enclosure are commuted • Only need a stored recording of their impulse response • An enormous digital filter is otherwise required

Summary

Julius Smith

AES-2006 Heyser Lecture – 75 / 84

Piano and Harpsichord Sound Examples Overview

(Stanford Sondius Project, ca. 1995)

Early Digital Synthesis Spectral Modeling Physical Modeling

• KL Music • Digital Waveguide • Signal Scattering • Plucked String • Struck String • Karplus Strong • EKS Algorithm • Clarinet • Wind Examples • Bowed Strings • Distortion Guitar • Acoustic Strings • Sound Examples • Linearized Violin • Commuted Piano • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

• Piano: (WAV) (MP3) • Harpsichord 1: (WAV) (MP3) • Harpsichord 2: (WAV) (MP3)

Summary

Julius Smith

AES-2006 Heyser Lecture – 76 / 84

More Recent Harpsichord Example Overview Early Digital Synthesis

• Harpsichord Soundboard Hammer-Response: (WAV) (MP3) • Musical Commuted Harpsichord Example: (WAV) (MP3)

Spectral Modeling Physical Modeling

• KL Music • Digital Waveguide • Signal Scattering • Plucked String • Struck String • Karplus Strong • EKS Algorithm • Clarinet • Wind Examples • Bowed Strings • Distortion Guitar • Acoustic Strings • Sound Examples • Linearized Violin • Commuted Piano • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

Reference: “Sound Synthesis of the Harpsichord Using a Computationally Efficient Physical Model”, ¨ aki, ¨ Henri Penttinen, Jonte Knif, Mikael Laurson, by Vesa Valim and Cumhur Erkut JASP-2004

Google search: Harpsichord Sound Synthesis

Summary

Julius Smith

AES-2006 Heyser Lecture – 77 / 84

Physical Modeling in Audio Coding Overview Early Digital Synthesis Spectral Modeling

Spectral modeling synthesis is finding application in audio coding. Can physical modeling synthesis be used as well?

• MPEG-4/SAOL already supports essentially all sound synthesis

Physical Modeling

• KL Music • Digital Waveguide • Signal Scattering • Plucked String • Struck String • Karplus Strong • EKS Algorithm • Clarinet • Wind Examples • Bowed Strings • Distortion Guitar • Acoustic Strings • Sound Examples • Linearized Violin • Commuted Piano • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

•

methods Ability to encode sounds automatically is limited

◦ Codebook-Excited Linear Prediction (CELP) is a successful ◦ ◦ ◦

source-filter model (not quite physical) There are many isolated examples of model-fitting to recorded data Good model-based denoising results have been obtained Coder problem much harder when many sources are mixed

Summary

Julius Smith

AES-2006 Heyser Lecture – 78 / 84

Best Known Model-Based Audio Coders Overview Early Digital Synthesis

A “cover band” can put together a very convincing facsimile of popular music performance

Spectral Modeling Physical Modeling

• KL Music • Digital Waveguide • Signal Scattering • Plucked String • Struck String • Karplus Strong • EKS Algorithm • Clarinet • Wind Examples • Bowed Strings • Distortion Guitar • Acoustic Strings • Sound Examples • Linearized Violin • Commuted Piano • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

JOS high-school band “Bittersweet”

Summary

Julius Smith

AES-2006 Heyser Lecture – 79 / 84

Future Physical Modeling in Audio Coding? Overview

A “Cover Band” Approach to Model-Based Audio Coding:

Early Digital Synthesis Spectral Modeling

1.

Physical Modeling

• KL Music • Digital Waveguide • Signal Scattering • Plucked String • Struck String • Karplus Strong • EKS Algorithm • Clarinet • Wind Examples • Bowed Strings • Distortion Guitar • Acoustic Strings • Sound Examples • Linearized Violin • Commuted Piano • Pulse Synthesis • Complete Piano • Sound Examples • Phy Audio Coding • Phy Audio Coding?

2. 3.

Recognize individual “audio streams” in a mix (CASA) (“I hear a trap set, electric bass, Fender Rhodes, and a strat”) For each stream, calibrate its model heuristically (“Here is what I hear the bass part doing: ...”) Fine-tune the synthetic mix to the real mix (joint “maximum likelihood estimation”)

Features of “Cover-Band Coding” (CBC):

• The “playing experience” of each “virtual performer” prevents

• •

artifacts — “musically unreasonable” parameters are made unlikely (“Bayesian priors”) An incorrect instrument must “imitate” its assigned stream New arrangements can be synthesized by deliberately choosing a new ensemble!

Summary

Julius Smith

AES-2006 Heyser Lecture – 80 / 84

Overview Early Digital Synthesis Spectral Modeling Physical Modeling Summary

Summary

Julius Smith

AES-2006 Heyser Lecture – 81 / 84

Summary Overview Early Digital Synthesis

We have reviewed a “CCRMA-centric slice” of the history of digital sound synthesis (usually starting with results from Bell Labs):

Spectral Modeling Physical Modeling Summary

Julius Smith

• • • • • • • •

Wavetable (one period) Subtractive Additive FM Sampling Spectral Modeling Physical Modeling (more in tomorrow’s 4:30 PM masterclass) Connections to audio coding

AES-2006 Heyser Lecture – 82 / 84

Overview Early Digital Synthesis Spectral Modeling Physical Modeling Summary Sound Acknowledgment

Sound Acknowledgment

Julius Smith

AES-2006 Heyser Lecture – 83 / 84

Sound Acknowledgment Overview Early Digital Synthesis

Thanks to Emu / Creative Labs for providing a superb-quality external D/A converter for this talk (an E-Mu 0404/USB)

Spectral Modeling Physical Modeling Summary Sound Acknowledgment

Julius Smith

AES-2006 Heyser Lecture – 84 / 84