OpenSignal  comments  on  Draft  Annual  Plan  2015/16  Consultation  Document  

 

  OpenSignal1  is   creating   a   comprehensive   database   of   cell   phone   towers,   cell   phone   signal   strength   readings,   and   Wi-­‐Fi   access   points   around   the   world.   We   create   this   database   to   provide  insight  on  connectivity,  adopting  the  philosophy  that  only  a  data  driven  examination   of   the   true   performance   of   these   networks   can   lead   to   active   improvements   of   electronic   communications   infrastructure.   We   achieve   this   by   crowdsourcing   data   on   mobile   devices   through   our   Android   and   iOS   apps,   which   have   had   over   11   million   downloads   around   the   world,  and  have  approximately  50,000  active  users  in  the  UK  contributing  data.     We   are   providing   comments   on   the  Ofcom  Draft  Annual   Plan   2015/16  (hereafter  referred  to   as   ‘Annual   Plan’)   because   many   of   the   strategic   priorities   are   aligned   with   OpenSignal’s   mission  in  terms  of  improving  connectivity  for  mobile  users  through  data  analysis,  provision   of  information  to  allow  consumers  to  make  informed  choices,  and  promoting  competition  to   encourage   infrastructure   improvements   to   benefit   everyone.   Based   on   OpenSignal   as   the   market   leader   on   mobile   data   collection   and   analysis   with   respect   to   the   mobile   user   experience,  we  are  providing  comments  relating  to  how  a  crowdsourced  methodology  can   contribute  to  achieving  the  aims  laid  out  in  the  Annual  Plan.     A  recent  report  released  in  draft  form  in  early  2015  by  the  CEPT,  ECC  Report  231  on  Mobile   Coverage   Obligations2,   listed   various   methodologies   for   monitoring   mobile   coverage.   The   methodology   options   include   crowdsourcing,   (mentioning   OpenSignal   as   an   example   by   name)   although   the   document   states   that   ‘This   method   needs   further   investigation’.   The   report  also  recommends  that  ‘describing  the  different  practices  on  how  coverage  is  defined   and   assessed   among   the   CEPT   countries   could   help   administrations   in   supervision’.   OpenSignal   uses   a   crowdsourced   methodology   and   we   very   much   welcome   further   investigation,   dialogue   and   education   on   this   mechanism,   and   provide   comments   on   the   consultation   document   for   Ofcom   through   the   lens   of   offering   details   on   how   this   methodology  works,  as  recommended  by  the  CEPT.     About  OpenSignal     The   OpenSignal   data   is   collected   from   real   world   consumer   smartphones,   and   is   recorded   under  conditions  of  normal  usage.  Rather  than  approximate  the  user  experience,  we  directly   measure   it   from   the   users   of   our   smartphone   application.   Our   application   can   be   freely   downloaded   on   either   iOS   or   Android   devices   and   constantly   monitors   the   true   network   experience   that   users   are   getting   on   those   devices.   Since   our   data   is   gathered   from   consumer   smartphones,   we   are   able   to   observe   the   network   exactly   as   the   end   user   experiences   it.   This   customer-­‐centric   approach   allows   us   to   measure   the   true   end-­‐to-­‐end   experience  of  the  mobile  network.  We  do   not   run   models,  simulations  or  make  assumptions   –  our  goal  is  to  directly  measure  user  experience  through  the  eyes  of  the  users  themselves.       Although   operators   monitor   how   their   networks   perform,   there   remains   a   disconnect  

                                                                                                                1  http://opensignal.com/   2http://www.cept.org/files/1051/Tools%20and%20Services/Public%20Consultations/2014/Draft%20ECC%20Report%20231%2 0for%20PC.docx  

between  the  standard  network  KPIs  and  what  surveys  say  about  customer  experience.  We   believe  that  the  only  way  to  bridge  this  gap  is  to  measure  the  network  using  the  customer   experience  as  our  starting  point.     The   OpenSignal   website   provides   consumer-­‐focused   visualizations   and   analysis,   based   exclusively   on   data   collected   from   the   users   of   the   OpenSignal   application.   Our   website’s   features   include   coverage   maps   that   show   cellular   signal   strength   for   a   given   geographical   area,  ranking  all  of  the  networks  by  performance  in  that  location,  and  the  locations  of  all  cell   towers  within  that  particular  region.     User  experience     Under  the  strategic  purpose  of  “Promote  opportunities  to  participate”  the  annual  plan  lists   the   priority   “Promote   better   coverage   of   fixed   and   mobile   services   for   residential   and   business  consumers”.  Within  this  priority,  the  Annual  Plan  states       “We  will  develop  a  consistent  approach  to  reporting  on  coverage  and  quality  of   experience   of   mobile   services   that   reflects   real   consumer   experiences,   and   publish  operator-­‐specific  metrics  to  inform  consumer  choices.”   This   directly   aligns   with   OpenSignal’s   methodology   and   mission   of   reporting   on   and   providing  data  on  real  consumer  experience,  and  we  support  Ofcom’s  focus  in  this  area.  Our   data  tests,  run  in  our  app,  are  designed  to  measure  a  user’s  average  experience  as  closely  as   possible.   Unlike   other   testing   methodologies,   such   as   models   or   drive   testing,   we   do   not   optimize   anything   related   to   the   testing   scenario;   we   instead   keep   conditions   as   close   to   normal  usage  as  possible.  This  allows  insights  to  be  gathered  on  network  performance  that   are  not  otherwise  possible.   For  example,  we  provided  data  to  CSL,  an  operator  in  Hong  Kong,  who  wanted  to   • Gain  a  deeper  insight  into  the  experience  on  their  network  from  a  customer-­‐centric   point  of  view.     • Gain  an  understanding  of  metrics  they  were  not  currently  able  to  track  such  as  Wifi   usage  behavior  or  the  proportion  of  time  users  have  no  network  coverage.     • Benchmark   the   performance   of   their   network   against   their   competitors   whilst   reducing  the  large  capital  expenditure  on  drive  testing.       Using   our   data   in   Hong   Kong,   our   analysis   allowed   for   the   identification   of   an   issue   that   was   causing   a   large   proportion   of   their   users   to   experience   poor   latencies,   enabling   the   operator   to  roll  out  a  fix  to  their  core  network  within  10  days.  This  particular  problem  had  only  been   affecting  users  on  lower  end  devices,  which  had  not  been  included  in  the  device  subset  used   in  the  drive  and  lab  testing,  and  so  the  existing  testing  methods  had  completely  missed  the   problem.       The  data  also  provided  insight  into  certain  metrics  that  were  previously  opaque  such  as:     • Seeing   the   proportion   of   time   the   average   user   was   connected   to   LTE,   in   order   to   assess  the  success  of  their  current  LTE  network  rollout.   • Understanding  how  their  unlimited  data  plans  was  impacting  Wifi  usage  behavior.     • Seeing   the   proportion   of   their   customers   and   their   competitor’s   customers   that   were  experiencing  throttled  data  speeds.      

The  Annual  plan  references  Ofcom’s  Consumer  experience  report3,  which  states       “We  recognise  that  the  planning  tools  are  subject  to  a  margin  of  error  and  local   factors,  such  as  tall  buildings  or  trees,  can  affect  the  signal  at  different  locations.   In   addition,   the   quality   of   mobile   services   are   affected   by   factors   other   than   signal   strength,   such   as   network   capacity,   number   of   simultaneous   users   and   quality   of   handset.   In   2014   we   will   be   undertaking   work   to   measure   the   actual   consumer  experience”         OpenSignal  supports  this  goal  and  recommends  the  use  of  a  crowdsoured  methodology  as   the   only   way   to   successfully   measure   actual   consumer   experience.   The   Ofcom   report   goes   on  to  site  OpenSignal  as  an  example  tool  for  using  consumer-­‐end  devices  to  measure  mobile   coverage.     The   Annual   Plan   also   references   Ofcom’s   most   recent  Infrastructure   Report4  that  discusses   the   particular   methodologies   for   measuring   coverage   and   user   experience.   The   report   states:     “The   consumer   experience   of   mobile   services   depends   very   substantially   on   the   quality   of   mobile   coverage.   There   are   however   two   practical   issues   which   complicate   measurement   of   mobile   coverage:     The   maps   of   mobile   coverage   produced   by   operators   are   based   on   theoretical   models,   calibrated   using   measurements  of  actual  performance  that  are  broadly  accurate  overall  but  can   never  be  absolutely  accurate  in  predicting  coverage  at  a  specific  location.  We  will   continue   to   carry   out   work   in   this   area.   Consumers   use   mobile   phones   in   many   different   situations   –   indoors,   outdoors,   on   the   move,   in   cars,   as   pedestrians   along   roads   in   built-­‐up   areas   and   in   wide   open   spaces.   No   single   measure   of   coverage   can   capture   all   these   use   cases.   For   this   report,   we   have   assessed   mobile   coverage   against   a   range   of   different   measures   in   order   to   gauge   the   overall  consumer  experience.”     At  OpenSignal  we  would  argue  that  a  crowdsourced  methodology  a)  is  independent  and   empirical,  and  therefore  removes  the  first  issue  stated  of  models  that  are  produced  by  the   operators  themselves,  and  b)  does  in  fact  take  into  account  the  various  use  cases  described   above  indicative  of  user  behaviour.  Since  data  is  gathered  while  these  activities  are  going  on,   the  datasets  and  are  representative  of  the  amount  of  time  users  actually  spend  on  these   activities.  Moreover,  since  data  is  collected  both  actively  and  passively,  it  is  possible  to   detect  and  account  for  any  potential  bias  of  actively  submitted  data.     Competition  and  informed  choices  for  consumers   Strategic   Purpose   1   of   the   Annual   Plan   is   ‘promote   effective   competition   and   informed   choice’,   with   the   work   area:   “Promote   effective   choice   for   consumers   by   ensuring   that   clear   and   relevant   information   is   readily   available”.   The   Annual   Plan   goes   on   to   state   in   section   A2.15     “Consumer   information   plays   a   critical   role   in   ensuring   that   competitive  

                                                                                                                3  http://stakeholders.ofcom.org.uk/market-data-research/market-data/consumer-experiencereports/?a=0   4  http://stakeholders.ofcom.org.uk/market-data-research/market-data/infrastructure/infrastructure2014/  

communications   markets   work   for   consumers.   We   will   continue   to   publish   information   in   a   range   of   areas   to   ensure   that   consumers   can   access  clear   and   relevant  information  about  the  performance  of  different  providers.”     OpenSignal   embraces   this   approach   and   benchmarks   network   operators’   performance   against   each   other,   providing   this   information   publically   so   consumers   can   make   informed   choices.   Accurate   and   useful   information   to   aid   consumer   choice   leads   to   healthy   competition   and   encourages   companies   to   innovate,   improve   their   service   offering   and   lower   costs,   which   can   in   turn   be   passed   onto   consumers.   When   network   operators   self-­‐ report   coverage,   it   is   often   hard   to   compare   on   a   consistent   basis.   Drive   testing   methodologies   differ,   as   do   testing   devices,   making   apples-­‐to-­‐apples   comparison   across   networks  effectively  impossible.  A  crowdsourced  methodology,  independent  of  network  and   device   type   allows   for   consistent   reporting   of   performance   of   the   competition.   OpenSignal’s   crowdsourced   data   compares   network   operators’   performance   at   the   user   level.   One   way   in   which  this  data  is  presented  is  through  our  coverage  map,  available  on  our  website  and  in   the  app  itself.  This  map  tool  allows  the  user  to  compare  all  network  types  in  a  given  region,   and  filter  to  view  the  coverage  of  both  individual  operators  and  network  types  (e.g.  2G,  3G   or  4G).     Regular   analysis   of   the   performance   of   networks   within   a   country   can   highlight   opportunities  for  competition  for  infrastructure  improvements.  An  example  is  OpenSignal’s   study   on   Denmark’s   Notspots   in   20135.   The   report   found   21,098   'notspots'   -­‐   areas   where   multiple  users  reported  have  no  3G  access  on  a  particular  network,  demonstrating  a  tangible   opportunity  for  Danish  networks  to  expand  their  service  into  these  areas.     OpenSignal’s   NetworkRank   algorithm   uses   a   series   of   Quality   of   Service   (QoS)   parameters   (listed   below),   all   of   which   are   measured   empirically   from   users   of   the   OpenSignal   mobile   app.   To   calculate   the   ranking,   the   score   assigned   to   an   operator   for   a   given   parameter   is   calculated  according  to  the  best  performing  operator  for  that  parameter  in  a  given  region.  In   other  words,  it  is  a  measure  of  the  performance  of  an  operator  in  relation  to  the  local  best   operator.   Therefore   the   network   rank   is   not   based   on   an   absolute   scale   and   rankings   of   operators   from   different   regions   cannot   be   compared   directly.   However,   OpenSignal’s   coverage  maps  on  the  website  and  in  the  app  also  provide  average  values  of  network  speeds   and  latency,  which  can  be  compared  across  markets.     Network  Rank  Parameters     • Signal  Coverage:  This  is  an  average  of  the  signal  strength  reported.  For  LTE  this  is  the   RSRP,   but   for   all   other   network   types   this   is   the   RSSI   (as   defined   by   3GPP   specifications  TS  27.007  8.5).     • Download  Speed:  This  is  an  average  of  the  download  speeds  reported  by  any  data   network   tests   (see   data   network   test   section   or   more   detail)   carried   out   in   the   region.   • Upload  Speed:  This  is  an  average  of  the  upload  speeds  reported  by  any  data  network   tests  carried  out  in  the  region.   • Latency:  This  is  an  average  of  the  latency  times  reported  by  any  data  network  tests   carried  out  in  the  region.   • Data  Reliability:  This  is  based  on  a  combination  of  the  success  rates  of  the  latency,   download   and   upload   tests.   It   is   the   proportion   of   data   requests   that   are  

                                                                                                                5  http://opensignal.com/reports/denmark-notspots-jan-2013.php  

successfully  completed.     OpenSignal  has  partnered  with  consumer  groups  to  ensure  that  end-­‐users  are  informed  of   the   performance   of   the   network   options   they   are   considering.   For   example,   we   have   partnered  with  Which6,  the  UK  based  consumer  group,  to  embed  our  coverage  map  on  their   website  and  to  publish  a  number  of  reports7  examining  the  status  of  coverage  in  several  UK   cities.   We   are   also   in   discussions   with   several   other   consumer   groups   across   Europe   and   Latin   America   to   explore   the   same   type   of   partnership.  OpenSignal   has   also   partnered   will   cell  phone  retail  websites  to  allow  consumers  to  be  fully  informed  of  their  options,  such  as   mobiles.co.uk8,  which  also  has  an  embedded  coverage  map  on  their  website.       Since  Opensignal  empowers  users  to  report  on  the  quality  of  their  cell  phone  coverage,  both   actively   and   passively   through   the   application,   this   creates   an   opportunity   for   network   operators   to   compete   to   offer   better   services   to   their   customers.   This   in   turn   encourages   and  incentivizes  further  investment  in  network  infrastructure.     Switching   For  the  Priority  “Improve  the  process  of  switching  providers  for  consumers,”  Section  A1.15   of  the  Annual  plan  states     “In   order   to   take   advantage   of   competition   in   the   communications   sector,   it   is   vital  for  consumers  to  be  able  to  exercise  choice  by  switching  providers  easily.  It   remains   important   that   we   make   further   progress   in   ensuring   the   effectiveness   of  switching  procedures  for  consumers.”   At   OpenSignal   we   also   support   consumers   having   the   option   to   switch   between   mobile   networks   as   this   promotes   efficient   markets   and   infrastructure   improvements.   We   are   working   with   consumer   groups   to   provide   our   coverage   maps   for   group   customer   tariff   switching  between  networks  based  on  reverse  auctions  by  the  networks.  

Reporting  on  mobile  coverage  and  speeds   The  Annual  Plan  explains  that  Ofcom  will       “publish   mobile   broadband   speeds   and   mobile   coverage   information   to   enable   consumers   to   compare   the   mobile   coverage   that   each   MNO   offers   across   the   UK’s  nations.”       Further,  it  is  an  interim  outcome  of  the  work  plan  that  Ofcom  will  indeed,  specifically     “publish   mobile   broadband   speeds   research   in   Q1   2015/16   and   2G,   3G   and   4G   coverage  research  in  Q3  2015/16.”    

                                                                                                                6  http://www.which.co.uk/technology/phones/reviews-ns/best-mobile-phone-networks/best-mobilenetworks-overview/   7  http://www.which.co.uk/technology/phones/reviews-ns/best-mobile-phone-networks/mobile-phonenetwork-state-of-the-nation-report/   8  http://www.mobiles.co.uk/4g/coverage-checker-map.html  

At  OpenSignal  we  strongly  support  Ofcom’s  plans  to  publish  speed  and  coverage  data.  The   Annual   Plan   provides   a   link   to   a   recent   report 9  by   Ofcom   on   measuring   broadband   performance   in   the   UK,   on   3G   and   4G   services   offered   by   the   UK’s   MNOs.   While   OpenSignal   supports   making   information   of   this   nature   public,   there   are   limitations   to   the   methodologies   used   by   Ofcom   to   calculate   the   results,   which   could   be   ameliorated   with   consideration   of   crowdsourced   data   on   mobile   user   experience.   OpenSignal   published   a   response   to   this   Ofcom   report,   available   at   http://opensignal.com/blog/2014/11/14/the-­‐ ofcom-­‐mobile-­‐broadband-­‐report-­‐a-­‐response/.   An   excerpt   from   this   response   is   included   below,  and  the  full  response  is  included  as  an  annex  to  this  document.   Excerpt   of   OpenSignal’s   response   to   Ofcom’s   report   ‘Measuring   mobile   broadband   performance  in  the  UK’   “We  feel  very  strongly  that  these  kind  of  reports  are  exactly  what  OfCom  should  be  doing,   and   that   any   and   all   independent   data   made   available   to   consumers   is   a   positive   step   leading  to  a  more  efficient  market.  We  have  made  clear  in  the  past  that  we  want  OfCom  to   do  more  independent  testing,  and  make  use  of  different  datasets  (such  as  OpenSignal)  –  so   that   better   consumer   information   on   network   performance   will   force   the   operators   to   compete  more  in  terms  of  the  actual  cellular  service  they  provide.     We  are  supporters  of  all   independent   information   that   helps   to   achieve   this   end,   and   we   are   delighted   that   OfCom   chose   to   cite   our   report,   as   it   shows   they   are   open   to   innovation   and   aware   of   the   limitations   of   traditional   testing   methods.   That   being   said,   we   retain   a   number   of   reservations  about  the  methodology  used  in  the  OfCom  study.”     The  response  goes  on  to  list  specific  limitations  (available  as  an  annex  to  this  document).  In   summary,   taking   account   of   data   that   is   truly   representative   of   phone   users’   experience   can   facilitate  in  obtaining  more  useful  and  accurate  data  for  consumers.  

Quality  of  service     The  Annual  Plan  explains  in  section  2.14,       “Even   in   areas   where   fixed   and   mobile   networks   offer   widespread   coverage,   consumers  and  businesses  have  expressed  concerns  about  the  quality  of  service   that  they  receive  from  communications  providers.”       At  OpenSignal  we  agree  that  it  is  important  not  only  to  measure  and  report  on  coverage,  but   also   on   the   quality   of   the   connection   that   the   phone   user   experiences.   The   crowdsourcing   approach   that   OpenSignal   adopts   allows   for   both   of   these   measures.   The   variable   that   we   use   to   measure   coverage   is   our   ‘time   on’   metric,   which   is   the   proportion   of   time   a   phone   user   on   a   given   network   and   technology   (e.g.   3G,   LTE   etc)   has   access   to   that   network.   Quality   of   service   and   performance   data   collected   includes   variables   such   as   upload   and   download  speed,  latency,  data  success  rates  and  signal  strength.     International  collaboration     As  part  of  the  work  area  “contribute  to  and  implement  public  policy  defined  by  Parliament”  is   the  activity  “Engage  with  the  legislative  programme  of  the  new  European  Commission.”    The   Annual  Plan  explains  that  Ofcom  

                                                                                                                9  http://stakeholders.ofcom.org.uk/market-data-research/other/telecoms-research/broadbandspeeds/mobile-bb-nov14  

  “actively  participate[s]  within  European  regulatory  networks  including  the  Body   of   European   Regulators   for   Electronic   Communications   (BEREC),   the   European   Platform   of   Regulatory   Authorities   (EPRA),   the   European   Regulators   Group   for   Audiovisual   Media   Services   (ERGA)   and   the   European   Regulators   Group   for   Postal   Services   (ERGP).   We   are   also   active   within   the   Radio   Spectrum   Policy   Group   (RSPG),   which   advises   the   European   Commission   on   spectrum   policy-­‐ making.”   OpenSignal   understands   Ofcom   collaborates   with   a   number   of   international   bodies   also   striving   for   the   same   goals   of   improving   connectivity   for   consumers.   There   are   challenges   with   adopting   consistent   methodologies   to   report   on   mobile   network   coverage   across   borders.  An  additional  benefit  of  crowdsourcing  data  from  users’  phones  it  that  it  is  agnostic   of   nation   states,   allowing   for   consistent   frameworks   for   reporting   on   and   comparing   coverage  between  countries.  OpenSignal  has  engaged  with  many  multilateral  organizations   in   this   space   through   presentations,   private   meeting   and   strategy   consultations,   including   BEREC,   the   CEPT   and   the   European   Commission.   OpenSignal   presented   on   crowdsource   methodology   at   the   International   Telecommunications   Union   (ITU)   Quality   of   Service   Development  Group  in  Dubai  in  November  2014.     summary     In  Summary,  OpenSignal  supports  much  of  the  Ofcom  Annual  Plan’s  priorities.  However,  it  is   important   to   consider   the   methodology   by   which   these   actions   are   carried   out,   and   OpenSignal   believes   that   crowdsourcing   user   experience   data   is   a   powerful   tool,   and   for   the   most   part,   necessary,   for   obtaining   and   publishing   accurate   data   and   achieving   the   aims   laid   out  in  the  consultation  document.           For  more  information,  contact  Ellie  Ereira  at  [email protected]  or  +447929854557.        

ANNEX    

The  OfCom  Mobile  Broadband  Report:  a  response   on  November   14,   2014   on   the   OpenSignal   blog.   Web   version   available   at   http://opensignal.com/blog/2014/11/14/the-­‐ofcom-­‐mobile-­‐broadband-­‐report-­‐a-­‐response/   Posted  

  Yesterday  OfCom  released  a  report  [pdf]10  on  the  current  state  of  UK  mobile  networks  –  a  report   similar   to  ours   on   the   subject   from   last   week11,   which   found   that   4G   speeds   had   halved   over   the   past  year  in  the  UK.  OfCom  focussed  their  report  on  network  performance,  rather  than  coverage,   and  cited  our  report  to  show  that  independent  analysis  is  being  carried  out  on  the  comparative   coverage   provided   by   MNO’s.   Importantly,   OfCom   carried   out   their   study   by   comparing   performance   in   sites   around   the   UK,   both   indoor   and   outdoor,   where   all   four   networks   were   present   –   and   their   findings   were   different   to   ours,   with   higher   reported   speeds   on   both   3G   and   4G.   In   the   OfCom   report   EE   ranked   as   the   fastest   4G   LTE   network   while   Vodafone   were   the   fastest  4G  network  based  on  OpenSignal  user  data.     We   feel   very   strongly   that   these   kind   of   reports   are   exactly   what   OfCom   should   be   doing,   and   that   any   and   all   independent   data   made   available   to   consumers   is   a   positive   step   leading   to   a   more   efficient   market.   We   have   made   clear   in   the   past   that   we   want   OfCom   to   do   more   independent   testing,   and   make   use   of   different   datasets   (such   as   OpenSignal)   –   so   that   better   consumer   information   on   network   performance   will   force   the   operators   to   compete   more   in   terms   of   the   actual   cellular   service   they   provide.     We   are   supporters   of   all   independent   information  that  helps  to  achieve  this  end,  and  we  are  delighted  that  OfCom  chose  to  cite  our   report,  as  it  shows  they  are  open  to  innovation  and  aware  of  the  limitations  of  traditional  testing   methods.   That   being   said,   we   retain   a   number   of   reservations   about   the   methodology   used   in   the  OfCom  study.   1)   The   data   is   up   to   eight   months   old:  With  the  current  pace  of  network  upgrades,  using  data  up   to   eight   months   old   is   not   likely   to   reflect   the   current   state   of   network   performance.   OfCom’s   data   collection   ran   from   March   to   June,   while   our   OpenSignal   report   covers   the   three   months   leading   to   October   1.   This   is   especially   a   concern   when   it   comes   to   4G   measurements,   as   our   UK   report   showed   that   average   4G   LTE   speeds   across   all   networks   had   fallen   15%   from   March   to   October,   meaning   that   OfCom   may   be   overstating   the   speeds   currently   experienced   by   real   consumers.   2)   The   type   of   device   used:  OfCom   tested   using   a   Galaxy   Note   III,   a   mobile   phone   which   is   available   across   all   networks.   They   justify   this   by   saying   they   want   to   test   the   network   without   it   being  affected  by  the  user’s  device,  as  different  devices  experience  different  speeds.  While  this  is  

                                                                                                                10  http://www.ofcom.org.uk/static/research/mbb.pdf   11  http://opensignal.com/reports/2014/10/uk-networks-report/  

a  legitimate  way  to  judge  comparative  network  performance,  we  do  not  feel  that  it  is  the  best   way   to   show   the   real   user   experience   of   the   network.   Testing   using   a   consumer   device   in   this   way   sits   at   an   uncomfortable   halfway   point   between   using   specialised   testing   equipment   and   data  crowdsourced  from  real  user  devices,  and  is  unable  to  fully  capture  the  full  spread  of  user   experience.   Many   users   do   not   have   high-­‐end   devices,   so   testing   using   a   device   such   as   the   Galaxy  Note  III  is  likely  to  overstate  speeds.  Our  network  averages  represent  the  fact  that  users   have   many   different   devices,   and   our   data   is   therefore   naturally   weighted   by   device   market   share   –   as   every   user   counts   the   same   –   giving   a   more   accurate   picture   of   the   diversity   of   the   UK’s  mobile  device  usage.  We  feel  our  methodology  complements  the  OfCom  data,  as  it  would   be  too  time  consuming  to  road-­‐test  using  a  full  spread  of  devices  and  OfCom  do  not  claim  to  be   directly  measuring  the  user  experience,  despite  using  a  consumer  device  for  their  testing.   3)   The   location   of   tests:  OfCom   test   both   indoors   and   outdoors   (a   positive   development   from   the   days   when   network   testing   included   almost   no   indoor   tests)   but   they   test   50%   on   each,   which   is   not   necessarily   representative   of   typical   use.   Above   all,   by   testing   in   areas   of   good   connectivity   (where   all   four   networks   are   present)   they   are   potentially   skewing   their   results   to   the  faster  end.   4)   Testing   on   EE   ‘double   speed’:  OfCom   ran   their   tests   using   the   EE   4G   ‘double   speed’   sim   (without  fully  explaining  why,  as  they  stated  they  intended  to  test  both  the  ‘single’  and  ‘double’   speed  tariffs,  but  ended  up  only  testing  the  faster  one).  Many  EE  4G  users  are  not  on  the  faster   speed   tariff   and   so   reported   speeds   for   EE   are   only   representative   of   the   experience   available   to   users  who  are  paying  for  the  faster  data,  and  therefore  possibly  not  representative  of  those  who   are  paying  for  4G  LTE  on  EE  on  the  ‘single  speed’  network.   5)   What   OfCom   are   actually   measuring:  OfCom   are   not   measuring   the   typical   speeds   users   actually   experience   (and   this   is   by   design),   and   therefore   overstate   the   speeds   consumers   are   likely  to  get.  This  explains  why  we  report  the  mobile  networks  to  be  slower  on  both  3G     and  4G   than   OfCom,   as   our   testing   is   directly   measuring   performance   as   experienced   by   users   rather   than  modelling  it  based  on  controlled  tests.   6)   Lack   of   Coverage   data:  OfCom   rely   on   coverage   data   self-­‐reported   by   the   operators   themselves,  looking  at  the  proportion  of  premises  covered  in  the  UK.  This  metric  is  an  attempt  to   combine   raw   geographic   coverage   with   is   impact   on   users,   but   we   feel   that,   in   isolation,   it   is   some   way   divorced   from   the   actual   availability   of   networks   for   consumers.   OfCom   cite   our   ‘time   on’   metric,   which   looks   at   the   proportion   of   time   users   have   a   connection,   as   an   alternative   metric  for  coverage  and  we  feel  it  is  entirely  complementary  to  the  more  traditional  geographic   metrics   used   by   OfCom,   as   it   helps   put   the   ‘premises’   figure   into   perspective.   OfCom’s   testing   methodology  is  not  able  to  gather  accurate  data  on  coverage,  and  this  means  that  their  report   on   network   performance   cannot   be   entirely   complete,   as   it   only   records   data   from   where   all   networks   are   present.   While   differences   in   performance   are   right   to   be   noted,   and   can   be   significant,  what  is  more  significant  is  the  actual  availability  of  the  network  itself.  

The   rise   of   independent   reporting   is   vital   for   the   on-­‐going   success   of   a   competitive   mobile   market  in  the  UK,  and  we  feel  that  OfCom’s  report  is  important  for  bringing  questions  of  network   performance   to   the   forefront   of   consumers’   minds.   We   do,   however,   believe   that   additional   useful   and   up-­‐to-­‐date   information   can   be   made   available   to   consumers   through   additional   techniques,   12

such  

as  

crowdsourcing,  

and  

would  

13

encourage  

the  

OfCom  

[pdf]  and  OpenSignal  reports  to  be  read  side-­‐by-­‐side  to  paint  a  more  complete  picture  of  the   current  state  of  network  provision  in  the  United  Kingdom.    

                                                                                                                12  http://www.ofcom.org.uk/static/research/mbb.pdf   13  http://opensignal.com/reports/2014/10/uk-networks-report/