Forward sequential algorithms far best basis selection

Forward sequential algorithms far best basis selection S.F.Cotter, J.Adler, B.D.Raa and K.Kreutz-Dclgado Abstract: Tlic pt.ol)lcni o r sigiial rcprcsc...
Author: Morris Thornton
2 downloads 0 Views 763KB Size
Forward sequential algorithms far best basis selection S.F.Cotter, J.Adler, B.D.Raa and K.Kreutz-Dclgado Abstract: Tlic pt.ol)lcni o r sigiial rcprcsctitation iii icrliis of' hisis vectors h i i i a lergc, overcomplete, spauiiiug dictionary has bueti the rociis o r iiiucli rcsciirch. Acliicving ii succinct, or ‘sparse', rcprcseritatinn is kriowri as tlic problem o f bcsl basis reprcscntnlioii. Mcthi)ds are considered which scck to solve this probleiii by sequentially huilrliiig tip a h a s i s set for the ~ I g ~ i i Threc iI. distinct iilgorith1~~ typcs liavc appeni-cd iii tlic liiernrurc which iirc hcrc tcriiictl biisic in;itchiiig pursuit (fjMI'), ortlcr rccursivc inntchiiig Ipiirsuit (ORMP) and tiiodilicil tuatcliitig piirsuit (MMI'). 'I'hc dgorithins iirc first tlcscribcti iiiid then their cotnpulalion is closely cx;iriiiiictl. Motlilicnhis arc inatlc to cadi riT thc procctlurcs wliich impriwc tlicir cclrn~iurarionnlcllicicncy. The coinplcxily o l each algorirlim is coiisidcrcd in two contcxtu; oiic wl~crcthe dictionary is v;iri;ildc (tinic-rlcpcriilciit} iii~tlthe other wherc tlie dictionary is lixcd (tinic-intlcpcndcnt). Expcrinicriial rcstilrs iirc prcsciiIcd wliich rlcniriiistwtc that tlic URMP metliotl is thc hcst proccdurc iti t c m s OF i l s ability Lo givc l l i c triost coinpact siynd rcprcsciitiitiori, followed by MM'I' ant1 tlicii IiMI' wli ich givcs tlic poorest rcs~ilts.I h d l y , wcigliiiig tlic pcrfnrinancc of each iilgorihn, ils coruputalion;il ciiiiqilcxity and tlic type of tlictionaiy ilvtlileblc, recotiiiiicndaliriiis arc tnarlr. as which algoritlim sIiould bc iisctl For a givcii problem.

1

Introduction

The probleiii o f sclccting a subsct of basis elcineirk h i r i il lnrgc sct of vcctms Iiw B long history aiid can lic t r x c d to tlic scerch lot opiiinal rcgrcssions in Ihc shlisiic:il 1itcl.aturc { I 211. So cslled projectioii pursuit nlgori~hiiwwcrc livsl dcvclopcrl ti) solvc this prtrl)lcrn iri 131. IIuivc~w,it was tlic &pkitimi (11' this algorittini in [4] to signal dccomposition wliicli lcd to a grcal ilcnl of inrcrcst in this problciii. In 1'41, a grcctly idgorillmi c;illcrl 'malchiny piirsuit' was tleveloped to choose vcctois h n i a largc dictionary (collcction of waveforms) to protlucc II coinpact sigiiitl rcpt'csciitation. 'l'hc iisc of ii retluiirlant tlictioiirtry allows llcxihilily in signd wprcscnlation and ilii iipproprialc clioicc O T basis will givc a compact rcprcscnlntion. Suhscqucrilly, h i - c litis becn ninch research i n achieving coinpacr. rclircscnkitions of sigritils Ihy sclccting suhscts i)f clcnictils Tram ovcr-compldc hascs [SI 61. Audio signals [7, X] mid iningcs 19-1 I]have bcen the focus of iiiost attention aiid the iiiosl coinrnoiily tiscd clictioiiaiics arc wnvclct or wavclcl packer tlictionarics. lnlcrcstingly, subsct sclcclion prnblcins arise in many diflcrcni A ~ C I I S [12], such as spcctrnl esiinintiori, fiiiiclioiial approximalion, clc. [ 13 231. Many dilicrciil algoritlims luvc bccti suggcsled L'or tlic solution of this ~~Ioblciii. Miriiinisstiou of rtirictioiials siicli

tlic I!. iioriii 124, 251 or the inorc gcncral L,, I iiorin [ 17, 261 IlilVC becn sllowll lo ~,rod~lce Sp"w sthtions. 'I'hc Iiiost commonly usctl ;tIgoritliiiis are those based 011 e forward scqucurinl scnrcli [4, 18, 19, 27.-3 I ] wI1c1.c tllc basis vcctors, wliicli will be iisctl to ctmilxictly rcprcscni thc sigiial, arc sclcctctl ~ I I Caficr the o h c r rr0111 ilie rlictioruiry of availnblc vccloi's. Thcsc iilgoritlinis we tlic tbciis of this papcr. 'I'hc rcsults of this paper AI% RII cxtctision and rcliiicincnt of soinc of nut' carliet. work i q i o r k d iii [3 I, 321. iis

2

Forward selection algorithms

TIie best basis scloclinii problctn is a s follows. [,ct U = { U ! } / - tic ii sct/tlictioriary of vectors which is Iiighly rt.dtii~rlaril,i.e. (11 E I?"' iintl ti in tlicii bc cxpandcd as

;in11

Tlie procedure teriiiiiialcs when either p = i- (lor spccilicd sparsity intlcx r) or ~l,!+l, 5 E (for slxc"iictl c). Eqiis, 1 atid 2, logclher with the clicck Tor krtoinatioii give t h c matching Inirsuit (MP) ;ilgor.illiiii (with Bo = b ) . 'k disting1iish this npproacli fioin the o1hei.s iiitrotluccd bclow wc rci'cr to this algorihiii as tlic hasic tilatching pursuit algr,rithni ( E M P). It is evidcnt Lhal tlic nlgarithiii is compul;tlir,nnlly siniplc, atid it is sliiiwn in [4] h a t it has tlic dcsirablc convcrycncc properly that thc iiorin of Ilic rcsidiial vcctrir i s iiiaiioronicnlly reduced in ciidi ircraiion. I-lowcver, tlic algorithm ha< i t s drawbacks both iii tcrnis of how Ilic rcsitlual vcctor is coriiputcd iti cqn. 2, a i d in thc ininiiticr it1 which the basis vcctor is sclcc~ed in cqii. I . Wc will elaborntc 011 thcsc dciiciciicics i i i Sectioii 2.1. As A coliscqticiicc: o f Ihcse limitatioiis, othcr algorilliim [or bnsis sdcclion have becii siiggcstctl which we discuss i ti thc Lbllowing Scctioii. 236

'l'lic algwitliin is tcrniiixitcd by using thc sntiie critcria as in tlic IjMP, i.c. whcn ci(hcr/i --.I' (lbr specified sparsity ilirlcx ~1 0 1 1161,1 5 I: (I'or spccitictl R ) . Iiqns. 5-8 coiislittite the OliMI' algoiithrn (with 6" = h, ti?"= t i / , I - I , . . . , / I ) . Note lhnt ttic residual b,, = /'th is the orthogonal prnjcctirm ol'h onto the ortliogunai complement of tlic raiigc spncc of J;,, and thcrchrc i s ~ h csiiiellcst possible error {in llic twonoi'iii sense) when b is to bc rcpi'csciiktl in the span of tlic

colulnrls ol'& Also i t is lo be iiolcd that iii cqii. 3 , or cquivnlently iii cqn. 5 , Ilic optiniis;ttion is only over ~~rcviously uiisclcchxl dictionary vcctors. Changing tlic optiinisatinii t o iricludc prcviotisly sclcclctl diciiorisry vcctors will not change the ovclall oulconic. Tlic reason fill h i s caii hc wcii by noting that adiliiig it basis clcmcnt IC/ wliich has already bccn iiscd to forin SI,-,, will not changc thc space spanncd hy tlic il;K I'm,:, -lir. Irtirij:c h i s d I ' i o c w , 1.01. 146, ,%. 5, 0uiohi.r 1999

2.3 Modified marching pursuit ( M M P ) Comliarcd lo tllc IlMII, (he ORMP iilgorithin difkrs in both llic iiiniiiier iii wliicli tlic basis vcctors are chosen end iii tlic coiiiputntioti o r thc resitliial. Uccniisc o f I:lie tiiorc cxliatislive iiatiirc of the ORMI’ vector sclcctioii proccss, thcrc is rcason to bclicvc h a t it will h c iiiorc successlid rhao B M P in lintling ii i ~ i o compact i~ rapiwciitntioii. This is supportctl by the siiiitilatioiis presciitctl i n Sec.tion 4.I’roiii R coinliiilniioiinl pcrspcctive, O R M P appcnrs at lirst glaiicc to bc inore c o i ~ p l c x . A i i w c rlctiiilctl account o f tlic coi~ipuhtio1i;ll crrniplcxily is proviclcd in Section 5.I.3. A closer cxaniin;itioii or the rcsidiid coinpiilntioii slcp i n [IMP, ns givcii in eqti. 2, rcvciils soiiie dclkieiicies of thc t3MP inclliotl for which a fix cnii bc rc;idily obt;iirictl by using thc OR MI’ rcsiduul ctrnip~itirtion appi-oach. Wiis i w t i l l s iii rlic iiindi lied inntchiiig piirsiuit (MMP) nlgorirlim. Exaiiiiniiig llic rcsirltinl coinpiitalioii stel) of thc 13MP nlgorithiii, iiotc d i e r

Tlint is, tlic scquence of o ~ i c - t l i ~ i i c ~ ~ s i oprojcctiims tiil dcfining tlic I3MI’ rcsirlual b ~ ~is‘ no!, ” it) gciicl.al, equal to itii orthogon;il projccticin ontq;;wl1cl.c

However, tlic qitatitity

.I, A

3

Computation analysis

In thc ~ ~ r c v i o iScction, is the l w i c iilgoi-ithis for sclcctiiig a subset o r tlic basis clciiicii~slinvc hccn prcsciitcd. W c iiow turn CIIW attciition to tlic coiiiputntioii itwolvcil iii cncli ;ilgoritIiiii ;irid rlcscril~ciiiotlificaticiiis which rcsult i n iiioi-c cflicicnl iiiiplciiictit;itiotis or tlic dgoritliiiis. Molivatctl lhy applicnliuiis, wc considcl. llic cr~iii~iti~~rticin;il cotiiplcxity 01’ the ;dyritliiiis in two contexts; niic wlicrt. thc dicliniiary /I is vm~:iblc (tiiiic-clel’ciidcnt), :iiid Ilic n t l i ~ r whcrc tlic tlictioiiary D i s iixcd (lime-indcpcndcii(). I(or irisl;rncc, i i i miiltiptilsc spcccli codiiig tlic dictionary \virics lirm lkiiiic to Ii.niiic [ 12, 30, 33, 341 giving r i s e tri the vari;iblc tlictioiitiry scciiwio, wliilc ii fixed tlici.ioiinry c m tic uscil in tinic-l’rcquciicy representatioiis of n signal 14, 251. ’l’lie clioicc ol‘ ii lixctl o r viirialdc tlic(ioii;iry is iniportmt hccausc i t involvcs ri barlc-t)fl’ hciwccii mciiiory iisagc niid coiillnitntioii. For illstaim, with Ilic LISC 01- a lixcrl d ict ionnry, cci~lai n c i m 11 tit a h i s ciiti hc vicwctl iis ovcrhc;id niid cnii lcail to ii IOWCI. c o m p l c x i ~ yi i i i i ~ l ~ i i i ~ I i ~:it~ tlic (ii)t~ cxpciisc or inci-ciiscrl iiicmory rccl ti ircinciils, Aviii lab I L‘ ~rsoiirccsmay ilicrcrorc dicinte wliich approxli is tiiltcii.

divisions ace cequircd in thc iteration. We iils(i includc computntioii of njko ns ail initial stcp. b'roni eqn. 12, wc nole that it is not ncccssary to ciilier explicitly ori in hl>or compute the I ~ C Wiiiiicr products ,:'I),, in each itemtion. 'l'liis rcsults in a lnrgc swiiig in cnntpolation irwe Iiave tIic inner produci lt;'lik;: w>iiI:it>Ictri LIS (scc Section 3.4). If tiicsc imcr protlacts we nut availablc then \vc iiccd 10 compute #til,,, which is coiziputntionally cquivalcnl to i o m i n g ayhl). 'Hit rccursion it1 cqii. 12 rcquircs a rurther atlditicms while foriuiiig b,,, iis in eqn. 2, icquircs 2 ~ 1ini~ltiplies.Cornputationally, ~ h clwo incthods arc hnilnr but froiii a storagc pcrspcctive, it is pi-cferablc j u s 1 to store b,] ( m locations) iiisleatI of U{'/+, ( I ) locations). The compukilioiial rcsuhs in Sectioii 3.3, whcw the dictionary i s tiinc-varying, are besed m i hriniiig /+, explicitly ;iriil tlicn rormiiig With a f i x 4 dictionaIy: a s corisidcrcd in Secrion 3.4, cqii. 12 is uscd to givc iiii cfficicnt implementntion. Thc check for termiiiniioii may requirc the c;ilcutaiitrn of llbl,j] if R I ~r: i s spccilicd but his can be easily chtirincd by not iiig that

This incans tliiil thc projcclioti oi'encli 01' tlic coluims n, irnplictl by cqn. 5, wliicli is ihc m i i i coinputational bottlciicck in OIIMP, is not reqiiircd. Thc sclcction s ~ c pcnii be rcwriitcii a s

Secondly,a rccursioii will1 tlie saine foriii R S cqn. 14 can bc LIWI to coniputc tIie iiiiiiieriitor. 'I lit! iioriii ll&)ll in the dcnoniiiinlor or this equation iniist still bc coinpiitetl for each value of 1. llowcvcr, tlicxe iiotnis can bc formcd recursively, produciiig n liirtlicr i,ediictioii iii cninpiit;itioii, using

&,,,

This rccursion coiistihitcs thc sccond iriodificnlioii to the ;ilgocitliin. Wc Ibuiid that hy cwryiiig (nit cqii. I4 and then cqri. 17, coinpiitatioii is rcduccd ovcr cxplicilly foriniiig the rcsidiinl .hi,, computing ntid Ilicii i h g eqn. 17. l'hc ~CRSOII for this i s ihal tlic cxpression

&,)

atid that both thc iiurncrntor niid tlciirmiiiatrir ol' thc linal term in this cxprcssion are avnilsble. 3.1.2 MAW: It1 tlic MMP, it is clear that wc caii iisc ir similar modification to that uscd Knr 1lic DMP (0411. 12). Iustead of uptlating 4, I , Llic iiiiier pt'otluct iiyii), is updatcd a n d h i s is done via the following mcursioii

~ i this i

rccirrsion, we

inus1 fnrin n;'q/> for cxcli

coluinn

tJ/,

anil compute q:hi,.

111tIic fixed dictioiiaiy C H S C , fiw thc HMY, hy using cqii. 12 :iiid prccooipiiling inner prt>duc.is u:Ckj>,we were aIiIc to s ~ v con coinpn(atioi1. However, in cqii. 14 prccomptitalioii oro{'q!, is iiot piwihlc since qij is not avnilablc. ThcrcTorc, based 011 thc suiiic nrgurncnts tis lit Scclinii 3.1.1, in both tliu casc o r a fixed a n d tirnc-varying dictionary, it i s dicapcr comptilalioiially and storagewisc to iinplcmciit ihc algnrithin as givcii in eqns. 9-1 1.

I$

3. 1.3 ORMP: T h c dcscripiiori ol'the O R M P wliicli was prcscntcd in Scctioii 2.2 W B S hasctl on [ I X I . Work prcscnkrl in [19, 271 altcinptcd La reducc the complexity of tlic ORMI' algorilhin. For die ovcrdetcrtnincd C B S C , i.c. m > n, it was shown that tlic co~npu~~iliotial coinplexity was rcilucctl. However, as thc autlicws stated, [heir ~iinplciiientatjons WJ'C not compirialioi~ally b e l k r than [ I S ] for thc iintlcrdc(crmined case, i.e. in ir i , which is h e CRSC of intcrcst hcrc. We tiow farniulatc tlircc motlilicatioris to the basic ORMI' iilgorithiii which substantially reetlucc i t s coinplcxity wlicrc fhc dictionary I) is uiiilcrtlctcl.iniiicd. Firsl wc recall that

so

21R

shes it1 hotli cqns. 14 niid 17 and the saving in coiiipirtation is O(aarr)multiplications per iteratiori. l'liii~dly,sincc is 110 loiigcr in tlic nunicrator, the orthogonalisation o f nll c i f thc tinchoscii c.oluinns is no longer rcquiretl. Only tlrc chosen sci ofcoiiiiutis, which i s a rnucli slnitllcl scl, inust bu orlhogoiinlisetl. This inay bc dolie usiiig the saine fortnrilatioii as was dcvelopcd for tlic MMP algoritlini (cqii. IO), and clcarly lliis ~ccprcscntsa Iiiigc saving in c n i n ~ n i t n t i ~co~~ipai-cd ~n tu h;wing tn orthogoiialisc ill1 tlic vectors at cach stcp. Tlic diITcreiice iii coinplcxity Ixlwccn this rediiccd coinplexily OICM I' tilgoritlim aiid Ilic M M P rcdiiccs essentially to ttic calculation o l ' h c tiorins lllty)!12. From cqn. 17, it is seen U I ~ II / I ~ @ IisI ~rcquirecI to correctly sclccl tlic tiw t basis clcmciit and this t1ic:iti.s that divisions are ncccssary. Tlicse divisioiis arc notcworthy os they do not arise in cithcr tlic HMI' o r MMP 3.1.4 ORMP via Choiesky decompasjrion: 71'l~c ORMI' alydhin tlcscribctl iii Section 2.2 iiscs a QR dccoinposition o f tlic biisis sct to solvc llic sequcnce or least squarcs prohlcms wliicli arise. Anothcr qiprciach io thcsc lcasl squares problems is to iisc tlic Cholcslry

dccoinposition to solvc the associatcd riormal equatii)ns [30]. As wc will c i ~ i i p i r cits coniplexity tn that o C thc algorithins alrcady dcscrilml, wc givc an outline of this algoritiitn. Recall that ti t the pth step, (p - 1) coluinns h a w been choscii ant1 tlic inatrix s$)= LS~,I, nil = [ a h , , itk,, . . . , itd,,.,,,nil i s l'oriiicd for each iiiictioscn h i s clcriicnt (I;. The iiitlex k,, is llicri clioscri i i s

This requires snlving tnany least squares problems for which Ihc corresponding normal cqtiiit iiiiis a I'C

a lowcr triatigtilnr nrnlrix.

The coniputatinnal cfiicicncy i s acliievctl by noting tlint iii the previous ilcrsiicm thc (:tiolcsky dccomposition for $,' . I S,, I liiib nlrcarfy bccii deicrniincd. I Icricc, orily llic Giial row iii llic inatrix L ( i ) has to bc cnlciilnted. 111 1301, it i s shown tlint niily thc coiiipiitiition o f t l i e last two elcinciits in tlic litial row, I::,',,. . I aiid i::,',, is rcquircrl and Iic)\lr the clciiiciits o f tlic inatrix L{i) arc used to cl'ficiciitly sclcct tlic optinid coliiinii [{A,),

8-

, 64-

2-

3.2 Computing fhe sohtion Now that tlic basis clcincnts lo be wed linve been sclcctctl, to liiitl ttic compact rcprcsentatioii, it rcmains to lincl thc c o c l i k i a i t s nssriciatctl with c a d i of' ilicsc clcincnls. The coinpiikihn irivolvcrl in doiiig lhis vnries with the nlgoritliin clioscii mid this is d e w i b c d in this Scction. A s has been st:itctl in Scctirm 2.3, thc rcsidtial a i cacti s l q of tlic U M P algciIithin docs not rcpi~cscnlllic smallest rcsiiliini obtaiiiablc, io geucrtil, wlicn the sigiiiil is rcprcseiiteil by tlic subsel 01' bnsis vcctors choscn. A lird projection using A coniugtitc gratliciit dcscctit algorithm [4] inay Ihc carried oiit i o fimi this rcsitIu;il iiivolviiig an c x ~ r acoinpu~niioiiallnnd of O((,i l ) , n r ) ii~iltiplications. With this nddiiiorial computatioii h c B M P coinplexity bccoincs coiuparnblc LO h a t OP the MMI? I-lowcvcr, bccaiisc wc mniy liiivc I-c-sclcctcrl vcctors, inorc itcrritirins tlinii arc iicccssery tiiay I i w c becii ~ i c r f ~ ~ r i i i It c t lis , also possiblc to carry o u i ii pi-ojcclion iillcr cach itcralioii but i l h i s is implcniciikrl tlicin BMP is a iiinrc coinplcx nlgoritliiii h a i i MMP. In M MI', tn snlve for tiic approxiinate soliitIoii vcclor, .U,., llic Q R decomposition ol'~Iicclioscii vcclor set Sr= [ t t k , , uk,, , ,tiL,] is ~rsctl.'I'licrclorc, .TI. = Q R aiid Ihc equalion to IIC s o ~ v c is t ~.T~.Y,.- #" - P'w1icl-c I,(*) is t~icscnrclicti for vector ant1 /P" is tIie i.cinaiijtIci. after tIic rtli stcp. ~ i i i i i ~ i u ~ y , iii tlic ciwc of both OItMI' algorithnis, thc solutirrn musl bc ohkiinctl using ii hacltstilvc.

+

..

3.3 Algorithm complexity with D variable Thc total c o i i i l h A y o r each algoritluni basctl OII our t1iacus:siun in Scciioii 3 . 1 niid i i i L l d i n g Llic cotiipiitntions rcqtiired to firirl tlic solutioii A S dct;iilcd in Section 3.2, is siimii-iariscd i n 'Ihblc 2 . Clcnrly, dcpciiding on tlic diinciisioiis of the tlictioiisry D ant1 thc sparsily required r, tlic m o u n t of' conip~itation rcquircd by each ;ilgoritlim w i l l vary. 1;or basis sclcclion problcins ili