A Physicist’s Perspective on Citation Analysis Impact and Productivity Measurements in a Changing Research Environment Washington, D.C., October 27, 2010 P. Chen (BU), S. Maslov (BNL), H. Xie (BNL)

A Physicist’s Perspective on Citation Analysis Impact and Productivity Measurements in a Changing Research Environment Washington, D.C., October 27, 2010 P. Chen (BU), S. Maslov (BNL), H. Xie (BNL)

Observations about scientific citations: amusing facts/idle gossip evolution is governed by linear preferential attachment

Google page rank analysis hidden “gems”

h-index analysis danger of reliance on h predictive value of outliers

Summary & Outlook

Phys Rev Citation Data

(as of July 07)

Phys Rev Citation Data

(as of July 07)

362,886 papers, 4,370,893 cites �# cites� = 12.04, �cite age�= 7.82

Phys Rev Citation Data

(as of July 07)

362,886 papers, 4,370,893 cites �# cites� = 12.04, �cite age�= 7.82

N.B.: Internal citations only; undercount by factor of 3-5.

(for highly cited HEP papers; SPIRES)

Phys Rev Citation Data

(as of July 07)

362,886 papers, 4,370,893 cites �# cites� = 12.04, �cite age�= 7.82

N.B.: Internal citations only; undercount by factor of 3-5.

(for highly cited HEP papers; SPIRES)

32 papers with > 1000 citations

Phys Rev Citation Data

(as of July 07)

362,886 papers, 4,370,893 cites �# cites� = 12.04, �cite age�= 7.82

N.B.: Internal citations only; undercount by factor of 3-5.

(for highly cited HEP papers; SPIRES)

32 papers with > 1000 citations 158 papers with > 500 citations

Phys Rev Citation Data

(as of July 07)

362,886 papers, 4,370,893 cites �# cites� = 12.04, �cite age�= 7.82

N.B.: Internal citations only; undercount by factor of 3-5.

(for highly cited HEP papers; SPIRES)

32 papers with > 1000 citations 158 papers with > 500 citations 426 papers with > 300 citations

Phys Rev Citation Data

(as of July 07)

362,886 papers, 4,370,893 cites �# cites� = 12.04, �cite age�= 7.82

N.B.: Internal citations only; undercount by factor of 3-5.

(for highly cited HEP papers; SPIRES)

32 papers with > 1000 citations 158 papers with > 500 citations 426 papers with > 300 citations 3693 papers with > 100 citations

Phys Rev Citation Data

(as of July 07)

362,886 papers, 4,370,893 cites �# cites� = 12.04, �cite age�= 7.82

N.B.: Internal citations only; undercount by factor of 3-5.

(for highly cited HEP papers; SPIRES)

32 papers with > 1000 citations 158 papers with > 500 citations 426 papers with > 300 citations 3693 papers with > 100 citations 12519 papers with > 50 citations

Phys Rev Citation Data

(as of July 07)

362,886 papers, 4,370,893 cites �# cites� = 12.04, �cite age�= 7.82

N.B.: Internal citations only; undercount by factor of 3-5.

(for highly cited HEP papers; SPIRES)

32 papers with > 1000 citations 158 papers with > 500 citations 426 papers with > 300 citations 3693 papers with > 100 citations 12519 papers with > 50 citations 244281 papers with < 10 citations

Phys Rev Citation Data

(as of July 07)

362,886 papers, 4,370,893 cites �# cites� = 12.04, �cite age�= 7.82

N.B.: Internal citations only; undercount by factor of 3-5.

(for highly cited HEP papers; SPIRES)

32 papers with > 1000 citations 158 papers with > 500 citations 426 papers with > 300 citations 3693 papers with > 100 citations 12519 papers with > 50 citations 244281 papers with < 10 citations 51487 papers with 1 citation

Phys Rev papers with >1000 cites (July 2003)

Phys Rev papers with >1000 cites (July 2003) cites !age" impact article title 1 PR 140 A1133 (1965) 3227 26.7 85972 Self Consistent Equations.. 2 PR 136 B864 (1965) 2460 28.7 70604 Inhomogeneous Electron Gas..

author(s) W. Kohn & L. J. Sham P. Hohenberg & W. Kohn

Phys Rev papers with >1000 cites (July 2003) cites !age" impact article title 1 PR 140 A1133 (1965) 3227 26.7 85972 Self Consistent Equations.. 2 PR 136 B864 (1965) 2460 28.7 70604 Inhomogeneous Electron Gas..

author(s) W. Kohn & L. J. Sham P. Hohenberg & W. Kohn

3 PRB 23

J. P. Perdew & A. Zunger

5048 (1981)

2079 14.4 29896 Self-Interaction Correction to..

Phys Rev papers with >1000 cites (July 2003) cites !age" impact article title 1 PR 140 A1133 (1965) 3227 26.7 85972 Self Consistent Equations.. 2 PR 136 B864 (1965) 2460 28.7 70604 Inhomogeneous Electron Gas..

author(s) W. Kohn & L. J. Sham P. Hohenberg & W. Kohn

3 PRB 23 4 PRL 45

J. P. Perdew & A. Zunger D. M. Ceperley & B. J. Alder

5048 (1981) 566 (1980)

2079 14.4 29896 Self-Interaction Correction to.. 1781 15.4 27463 Ground State of the Electron..

Phys Rev papers with >1000 cites (July 2003) cites !age" impact article title 1 PR 140 A1133 (1965) 3227 26.7 85972 Self Consistent Equations.. 2 PR 136 B864 (1965) 2460 28.7 70604 Inhomogeneous Electron Gas..

author(s) W. Kohn & L. J. Sham P. Hohenberg & W. Kohn

3 PRB 23 4 PRL 45

5048 (1981) 566 (1980)

2079 14.4 29896 Self-Interaction Correction to.. 1781 15.4 27463 Ground State of the Electron..

J. P. Perdew & A. Zunger D. M. Ceperley & B. J. Alder

5 PR 108

1175 (1957)

1364 20.2 27526 Theory of Superconductivity

Bardeen, Cooper, Schrieffer

Phys Rev papers with >1000 cites (July 2003) cites !age" impact article title 1 PR 140 A1133 (1965) 3227 26.7 85972 Self Consistent Equations.. 2 PR 136 B864 (1965) 2460 28.7 70604 Inhomogeneous Electron Gas..

author(s) W. Kohn & L. J. Sham P. Hohenberg & W. Kohn

3 PRB 23 4 PRL 45

5048 (1981) 566 (1980)

2079 14.4 29896 Self-Interaction Correction to.. 1781 15.4 27463 Ground State of the Electron..

J. P. Perdew & A. Zunger D. M. Ceperley & B. J. Alder

5 PR 108

1175 (1957)

1364 20.2 27526 Theory of Superconductivity

Bardeen, Cooper, Schrieffer

6 PRL 19

1264 (1967)

1306 15.5 20191 A Model of Leptons

S. Weinberg

Phys Rev papers with >1000 cites (July 2003) cites !age" impact article title 1 PR 140 A1133 (1965) 3227 26.7 85972 Self Consistent Equations.. 2 PR 136 B864 (1965) 2460 28.7 70604 Inhomogeneous Electron Gas..

author(s) W. Kohn & L. J. Sham P. Hohenberg & W. Kohn

3 PRB 23 4 PRL 45

5048 (1981) 566 (1980)

2079 14.4 29896 Self-Interaction Correction to.. 1781 15.4 27463 Ground State of the Electron..

J. P. Perdew & A. Zunger D. M. Ceperley & B. J. Alder

5 PR 108

1175 (1957)

1364 20.2 27526 Theory of Superconductivity

Bardeen, Cooper, Schrieffer

6 7 8 9 10 11

PRL 19 1264 (1967) 1306 15.5 20191 PRB 12 3060 (1975) 1259 18.4 23103 PR 124 1866 (1961) 1178 28.0 32949 RMP 57 287 (1985) 1055 9.2 9674 RMP 54 437 (1982) 1045 10.8 11307 PRB 13 5188 (1976) 1023 20.8 21227

A Model of Leptons Linear Methods in Band Theory Effects of Configuration.. Disordered Electronic Systems Electronic Properties of.. Special Points for Brillouin..

S. Weinberg O. K. Andersen U. Fano P. A. Lee & T. V. Ramakrishnan T. Ando, A. B. Fowler, & F. Stern H. J. Monkhorst & J. D. Pack

Phys Rev papers with >1000 cites (July 2003) cites !age" impact article title 1 PR 140 A1133 (1965) 3227 26.7 85972 Self Consistent Equations.. 2 PR 136 B864 (1965) 2460 28.7 70604 Inhomogeneous Electron Gas..

author(s) W. Kohn & L. J. Sham P. Hohenberg & W. Kohn

3 PRB 23 4 PRL 45

5048 (1981) 566 (1980)

2079 14.4 29896 Self-Interaction Correction to.. 1781 15.4 27463 Ground State of the Electron..

J. P. Perdew & A. Zunger D. M. Ceperley & B. J. Alder

5 PR 108

1175 (1957)

1364 20.2 27526 Theory of Superconductivity

Bardeen, Cooper, Schrieffer

6 7 8 9 10 11

PRL 19 1264 (1967) 1306 15.5 20191 PRB 12 3060 (1975) 1259 18.4 23103 PR 124 1866 (1961) 1178 28.0 32949 RMP 57 287 (1985) 1055 9.2 9674 RMP 54 437 (1982) 1045 10.8 11307 PRB 13 5188 (1976) 1023 20.8 21227

A Model of Leptons Linear Methods in Band Theory Effects of Configuration.. Disordered Electronic Systems Electronic Properties of.. Special Points for Brillouin..

S. Weinberg O. K. Andersen U. Fano P. A. Lee & T. V. Ramakrishnan T. Ando, A. B. Fowler, & F. Stern H. J. Monkhorst & J. D. Pack

top 10 cited Phys Rev papers (July 2007)

Phys Rev papers with >1000 cites (July 2003) cites !age" impact article title 1 PR 140 A1133 (1965) 3227 26.7 85972 Self Consistent Equations.. 2 PR 136 B864 (1965) 2460 28.7 70604 Inhomogeneous Electron Gas..

author(s) W. Kohn & L. J. Sham P. Hohenberg & W. Kohn

3 PRB 23 4 PRL 45

5048 (1981) 566 (1980)

2079 14.4 29896 Self-Interaction Correction to.. 1781 15.4 27463 Ground State of the Electron..

J. P. Perdew & A. Zunger D. M. Ceperley & B. J. Alder

5 PR 108

1175 (1957)

1364 20.2 27526 Theory of Superconductivity

Bardeen, Cooper, Schrieffer

6 7 8 9 10 11

PRL 19 1264 (1967) 1306 15.5 20191 PRB 12 3060 (1975) 1259 18.4 23103 PR 124 1866 (1961) 1178 28.0 32949 RMP 57 287 (1985) 1055 9.2 9674 RMP 54 437 (1982) 1045 10.8 11307 PRB 13 5188 (1976) 1023 20.8 21227

A Model of Leptons Linear Methods in Band Theory Effects of Configuration.. Disordered Electronic Systems Electronic Properties of.. Special Points for Brillouin..

S. Weinberg O. K. Andersen U. Fano P. A. Lee & T. V. Ramakrishnan T. Ando, A. B. Fowler, & F. Stern H. J. Monkhorst & J. D. Pack

top 10 cited Phys Rev papers (July 2007) cites !age" impact

article 1 2 3 4

PR 140 A1133 PR 136 B864 PRB 23 5048 PRL 45 566

(1965) (1965) (1981) (1980)

4930 3564 3007 2514

title Self Consistent Equations.. Inhomogeneous Electron Gas.. Self-Interaction Correction to.. Ground State of the Electron..

author(s) W. Kohn & L. J. Sham P. Hohenberg & W. Kohn J. P. Perdew & A. Zunger D. M. Ceperley & B. J. Alder

Phys Rev papers with >1000 cites (July 2003) cites !age" impact article title 1 PR 140 A1133 (1965) 3227 26.7 85972 Self Consistent Equations.. 2 PR 136 B864 (1965) 2460 28.7 70604 Inhomogeneous Electron Gas..

author(s) W. Kohn & L. J. Sham P. Hohenberg & W. Kohn

3 PRB 23 4 PRL 45

5048 (1981) 566 (1980)

2079 14.4 29896 Self-Interaction Correction to.. 1781 15.4 27463 Ground State of the Electron..

J. P. Perdew & A. Zunger D. M. Ceperley & B. J. Alder

5 PR 108

1175 (1957)

1364 20.2 27526 Theory of Superconductivity

Bardeen, Cooper, Schrieffer

6 7 8 9 10 11

PRL 19 1264 (1967) 1306 15.5 20191 PRB 12 3060 (1975) 1259 18.4 23103 PR 124 1866 (1961) 1178 28.0 32949 RMP 57 287 (1985) 1055 9.2 9674 RMP 54 437 (1982) 1045 10.8 11307 PRB 13 5188 (1976) 1023 20.8 21227

A Model of Leptons Linear Methods in Band Theory Effects of Configuration.. Disordered Electronic Systems Electronic Properties of.. Special Points for Brillouin..

S. Weinberg O. K. Andersen U. Fano P. A. Lee & T. V. Ramakrishnan T. Ando, A. B. Fowler, & F. Stern H. J. Monkhorst & J. D. Pack

top 10 cited Phys Rev papers (July 2007) cites !age" impact

article 1 2 3 4

PR 140 A1133 PR 136 B864 PRB 23 5048 PRL 45 566

5 PRL 77

6 7 8 9

PRB PRB PRB PRB

3865

title

author(s)

(1965) (1965) (1981) (1980)

4930 3564 3007 2514

Self Consistent Equations.. Inhomogeneous Electron Gas.. Self-Interaction Correction to.. Ground State of the Electron..

W. Kohn & L. J. Sham P. Hohenberg & W. Kohn J. P. Perdew & A. Zunger D. M. Ceperley & B. J. Alder

(1996)

2478

Generalized Gradient Approx...

Perdew, Burke, Ernzerhof

Special Points for Brillouin.. Efficient Iterative Schemes.... Efficient Pseudopotentials for .... Soft Self-Consistent Pseudopotentials...

H. J. Monkhorst & J. D. Pack G. Kresse & J. Furthmuller N. Troullier & J.L. Martins D. Vanderbilt

13 5188 (1976) 2277 54 11169 (1996) 1933 43 1993 (1991) 1776 41 7892 (1990) 1749

Phys Rev papers with >1000 cites (July 2003) cites !age" impact article title 1 PR 140 A1133 (1965) 3227 26.7 85972 Self Consistent Equations.. 2 PR 136 B864 (1965) 2460 28.7 70604 Inhomogeneous Electron Gas..

author(s) W. Kohn & L. J. Sham P. Hohenberg & W. Kohn

3 PRB 23 4 PRL 45

5048 (1981) 566 (1980)

2079 14.4 29896 Self-Interaction Correction to.. 1781 15.4 27463 Ground State of the Electron..

J. P. Perdew & A. Zunger D. M. Ceperley & B. J. Alder

5 PR 108

1175 (1957)

1364 20.2 27526 Theory of Superconductivity

Bardeen, Cooper, Schrieffer

6 7 8 9 10 11

PRL 19 1264 (1967) 1306 15.5 20191 PRB 12 3060 (1975) 1259 18.4 23103 PR 124 1866 (1961) 1178 28.0 32949 RMP 57 287 (1985) 1055 9.2 9674 RMP 54 437 (1982) 1045 10.8 11307 PRB 13 5188 (1976) 1023 20.8 21227

A Model of Leptons Linear Methods in Band Theory Effects of Configuration.. Disordered Electronic Systems Electronic Properties of.. Special Points for Brillouin..

S. Weinberg O. K. Andersen U. Fano P. A. Lee & T. V. Ramakrishnan T. Ando, A. B. Fowler, & F. Stern H. J. Monkhorst & J. D. Pack

top 10 cited Phys Rev papers (July 2007) cites !age" impact

article 1 2 3 4

PR 140 A1133 PR 136 B864 PRB 23 5048 PRL 45 566

5 PRL 77

6 7 8 9

PRB PRB PRB PRB

3865

author(s)

(1965) (1965) (1981) (1980)

4930 3564 3007 2514

Self Consistent Equations.. Inhomogeneous Electron Gas.. Self-Interaction Correction to.. Ground State of the Electron..

W. Kohn & L. J. Sham P. Hohenberg & W. Kohn J. P. Perdew & A. Zunger D. M. Ceperley & B. J. Alder

(1996)

2478

Generalized Gradient Approx...

Perdew, Burke, Ernzerhof

Special Points for Brillouin.. Efficient Iterative Schemes.... Efficient Pseudopotentials for .... Soft Self-Consistent Pseudopotentials...

H. J. Monkhorst & J. D. Pack G. Kresse & J. Furthmuller N. Troullier & J.L. Martins D. Vanderbilt

Theory of Superconductivity

Bardeen, Cooper, Schrieffer

13 5188 (1976) 2277 54 11169 (1996) 1933 43 1993 (1991) 1776 41 7892 (1990) 1749

10 PR 108

title

1175 (1957)

1650

Citation histories of 3 classic PR papers 200

150

KS EPR BCS

5145 internal citescites #1, 4420 internal 17057 total cites (10/16/10) (as of 04/02/08) av. citation age 59.6 (as of 7/03) 1st PR paper > 1000 cites

100

50

0 1950

1970

1990

2010

“Sleeping Beauties”

# cites > 300 /paper age > 3/4

“Sleeping Beauties”

# cites > 300 /paper age > 3/4

8 papers total, 5 on double exchange

⇒ colossal magnetoresistance

“Sleeping Beauties”

# cites > 300 /paper age > 3/4

8 papers total, 5 on double exchange

⇒ colossal magnetoresistance

100

80

de Gennes Wollan & Koehler Goodenough Anderson & Hasegawa Zener

60

40

20

0 1960

1980

2000

What is the Citation Rate? 200

Physical Review Citation Data, 1893-2003

citation rate A k

50

1980-99 1990-99

25

150

attachment rate Ak

Physics Today 58, 49 (2005)

0

1970-99

1893-1999 0

50

100

150

100

50

0

0

200

400

number of citations k

600

800

What is the Citation Rate? 200

Physical Review Citation Data, 1893-2003

citation rate A k

50

1980-99 1990-99

25

150

attachment rate Ak

Physics Today 58, 49 (2005)

0

1970-99

1893-1999 0

50

100

150

100

50

0

0

200

400

number of citations k

600

800

What is the Citation Rate? 200

Physical Review Citation Data, 1893-2003

citation rate A k

50

1980-99 1990-99

25

150

attachment rate Ak

Physics Today 58, 49 (2005)

0

1970-99

1893-1999 0

50

100

150

100

50

→ Ak ∝ k 0

0

200

400

number of citations k

600

800

Cumulative Citation Distribution −1

10

C(k) = Ae

−2

10

−a ln k−b(ln k)2

a = 0.40, b = 0.16 −3

C(k)

10

−4

10

−5

10

−6

10

1

10

k

100

1000

Google Page Rank for Citations

Brin & Page (1999)

Evolution equation for websurfers on a network: Gi = number of websurfers on node i

Google Page Rank for Citations

Brin & Page (1999)

Evolution equation for websurfers on a network: Gi = number of websurfers on node i random walk propagation

! Gj d Gi = (1 − d) + out k N j nn i j Gj i

j

�

out kj

manna from heaven

Google Page Rank for Citations

Brin & Page (1999)

Evolution equation for websurfers on a network: Gi = number of websurfers on node i random walk propagation

! Gj d Gi = (1 − d) + out k N j nn i j Gi i

j

�

out kj

manna from heaven

Google Page Rank for Citations

Brin & Page (1999)

Evolution equation for websurfers on a network: Gi = number of websurfers on node i random walk propagation

! Gj d Gi = (1 − d) + out k N j nn i j Gi i

j

�

Brin/Page: d=0.15 (bored after 6 clicks)

out kj

manna from heaven

Google Page Rank for Citations

Brin & Page (1999)

Evolution equation for websurfers on a network: Gi = number of websurfers on node i random walk propagation

! Gj d Gi = (1 − d) + out k N j nn i j Gi i

j

�

out kj

Brin/Page: d=0.15 (bored after 6 clicks)

Here: d=0.50 (don’t cite beyond 2 generations)

manna from heaven

Google Number & Citation Counts P. Chen, H. Xie, S. Maslov, and S. Redner, J. Informetrics 1, 8 (2007) 3

average google number

10

2

10

slope 1 1

10

0

10

10

0

1

10

2

10 number of citations

10

3

10

4

500

google number

400

300

200

100

0

0

500 1000 number of citations

1500

500

Cabibbo

google number

400

300

200

100

0

0

500 1000 number of citations

1500

500

Cabibbo BCS

google number

400

300

200

100

0

0

500 1000 number of citations

1500

500

Cabibbo BCS

google number

400

K/S H/K

300

200

100

0

0

500 1000 number of citations

1500

500

Cabibbo BCS

google number

400

K/S H/K

300

Onsager

Weinberg

200

100

0

0

500 1000 number of citations

1500

500

Cabibbo BCS

google number

400

K/S H/K

300

Weinberg

Onsager Slater W/S 200

F/GM

Anderson

G4

100

0

0

500 1000 number of citations

1500

500

Cabibbo BCS

google number

400

K/S H/K

300

Weinberg

Onsager Slater W/S 200

Anderson

F/GM GM/B

W/S

Hi Tc DLAGlauber S

G4

Fano

100

0

0

500 1000 number of citations

1500

The h-Index h definition: ≧ h publication with ≧ h citations

The h-Index h definition: ≧ h publication with ≧ h citations c 20000

10000

0

0

10000

2

4h

20000

The h-Index h definition: ≧ h publication with ≧ h citations

� result: h = c/4

c

J. Stat. Mech. L03005, (2010) 20000

10000

0

0

10000

2

4h

20000

Distribution of

√

c/2h

P(r) 2

1

0 0.5

1

1.5

r=

√

2

c/2h

Distribution of

√

c/2h

P(r) 2

1

who are the outliers? •extraordinary publications •timely review articles •few but good publications •junior researchers

0 0.5

1

1.5

r=

√

2

c/2h

Summary & Outlook

Summary & Outlook Citation analysis is fun! (but a Pandora’s box) Regularities and revealing irregularities Data motivate and test theories of growing citation networks Page rank analysis: helps uncover hidden “gems” h-index analysis: h is equivalent to √c but outliers, properly interpreted, are revealing

Summary & Outlook Citation analysis is fun! (but a Pandora’s box) Regularities and revealing irregularities Data motivate and test theories of growing citation networks Page rank analysis: helps uncover hidden “gems” h-index analysis: h is equivalent to √c but outliers, properly interpreted, are revealing

Future: Deeper analysis of citation data: contextual information, specialization

Larger data sources:

test universality of citation statistics