5/23/2013

Use case for implementing Operational Business Intelligence g

TDWI 2013 Munich TVP - Technology ec o ogy Department epa t e t a and d Atos tos Poland oa d 1

5/23/2013

Topics •

Why OBI is important for business

•

How to prepare business justifications

•

What are operational requirements and standard solutions

•

How to design High Level Architecture

•

What are required components

2

5/23/2013

Presentation case

▶Described solution was delivered by TVP - Technology Department and Atos Poland ▶Solution covers optimization for video content distribution on edge d servers off TVP VOD system t ▶Solution was built in the 2012 H1

3

Telewizja Polska S.A. (TVP S.A.) Polish Public Television Broadcaster

Technology Department

5/23/2013

Media Technologists General, regional, HD and speciality channels: TVP 1, TVP 2, TVP Polonia, TVP Info, Info TVP Sport, Sport TVP Historia, Historia TVP Kultura, TVP HD, TVP Parlament oraz TVP Seriale. New technologies: VOD, live streaming, SmartTV, HbbTV.

4

New European IT champion with global reach

Your Business Technologists

5/23/2013

Powering progress At Atos we strive to create the firm of the future. We believe that bringing together people, technology and b i business iis th the way fforward. d Every day we power sustainable progress for our clients and partners, the wider community and ourselves. It is our unique approach as business technologists that makes this possible. Thierry Breton, CEO Atos

5

5/23/2013

Speaker Data and Information Management Warsaw; Poland

Unlock the hidden value of your data and information Tomasz Bawor DW/BI Solution Architect tomasz bawor@atos net [email protected] @tbawor

6

Why OBI is Wh i iimportant t t for f business

7

5/23/2013

Operational Business Intelligence d fi i i definitions

5/23/2013

▶ A set of services, applications and technologies for monitoring, reporting on, analyzing and managing the business performance of an organization’s organization s daily business operations “Embedded BI” By Judy Davis and Colin White, www. B-EYE-Network.com Research 2008 paper

▶ Right time Business Intelligence “Right-Time Business Intelligence: Optimizing the Business Decision Cycle” by Judy Davis, www. B-EYE-Network.com Research paper (2006)

▶ Adaptive Business Intelligence The phrase was coined in 1999 by SolveIT Software Chairman and co-founder Dr. Zbigniew Michalewicz

▶ Decision Management Systems “Decision Management Systems: A Practical Guide to Using Business Rules and Predictive Analytics” By James Taylor Published: Sep 30, 2011

8

What do we mean by Operational B i Business IIntelligence lli ▶ Operational system and Business Intelligence combined to drive better performance for the business process ▶ Operational and BI technology stack which should be used in most optimized way

9

5/23/2013

What is the added value

PREPARE DATA

ACT

5/23/2013

MONITOR

▶ Performance improvement is the concept of measuring the output of a particular process or procedure, then modifying the process or procedure to increase the output, increase efficiency, or increase the effectiveness of the process or procedure. http://en.wikipedia.org/wiki/Performance_improvement

▶ Basically we would like to decrease resources and/or increase results. Other definitions based on simmilar pillars: BPM, Kaizen, Six Sigma

10

When OBI is most suitable

5/23/2013

▶ When there is impact on performance by: ▶ Integrate additional data ▶ Store historical data ▶ Use Advanced Analytics based on sophisticated p calculations ▶ Use High Performance DW/BI or Big Data technology

11

TVP business context

5/23/2013

▶ Video on Demand (VOD) system ▶ Video content could be broadcasted through: laptop, notebook or PC with www; Tablet, SmartPhone, SmartTV with dedicated application ▶ About 50 thousands different content per month ▶ About 6 millions requests for content per day

12

TVP business requirement

5/23/2013

Improve p Quality y of Service ((QoS)) for broadcast popular video content without stop video playback 13

TVP technology context

5/23/2013

▶ VOD system contain few servers with two layers: ▶ Fast based on SSD drives ▶ Slow based on SATA which is 20 times bigger than SSD layer

▶ Cache management (SWAP) which contains: ▶ SWAP IN – video content is loaded from SATA into SSD ▶ SWAP OUT – last played video content is removed from SSD each time when there is no space left

▶ High volume of low popular content were mixed with high popular on the same servers

14

TVP technology requirements

5/23/2013

▶Decrease number of cycles for SWAP IN and SWAP OUT for popular content ▶Split content based on the popularity through servers

15

5/23/2013

TVP solution ▶ Prepare p and store

▶ Additional data about content with information about g y hierarchy y category ▶ Calculations for content popularity in different dimensions

▶ Learn and forecast content popularity ▶ Act by delivery of whole list how to distribute content ▶ Monitor content popularity between distributions and modify distribution list if necessary

16

How tto prepare b H business i j justifications

17

5/23/2013

Start from business process

5/23/2013

▶ Each management level has its own Set Goals and Monitor

Strategic

point of view. For operational level we should proceed different than on above above. ▶ Strategic -Gather information and options

Build Plans and Notify

Tactical

▶ Tactical - Integrate information according to strategic goals

Operationalize and Act

▶ Operational - Design business process Operational

according g to tactical p plan

18

Model business process

5/23/2013

▶OBI could act on step level of business processes ▶OBI is complementary to legacy systems ▶OBI acts on hard to change decision making components of legacy systems with more manageable, more agile components

19

Characteristics of suitable d i i decisions

5/23/2013

▶ Repeatable ▶ The decision is made at defined times; the same information is used each time; defined set of actions; consistent measurement of success

▶ Nontrivial ▶ Policies and regulations; domain knowledge and p analysis y is required; q large g amounts of expertise; data; large numbers of actions; trade-offs must be made; continued updates

▶ Measurable business impact ▶ Candidates for automation 20

5/23/2013

Time to M T Market

Prioritize business processes to change h

Build roadmap starting from small changes with big g impact p 21

Solution evaluation

5/23/2013

▶Find all the technical alternatives ▶Evaluate these alternatives ▶Select the best alternative for implementation ▶OBI should be more agile, manageable, reliable and simple ▶Start hypothesis testing as soon as possible

22

TVP technical alternative evaluation l i

5/23/2013

▶ Cache management algorithms: ▶ Least Recently Used (LRU) – SWAP OUT for last used content ▶ Least Frequently Used (LFU) – SWAP OUT for least popular ▶ Adaptive Replacement Cache – combination of LRU and LFU (Patented by IBM in 2006)

▶ Content aware systems: ▶ Content Delivery Network – geographically extensive network for content distribution ▶ Transparent/Edge Caching – keeping content close to the target audience ▶ Adaptive Content Delivery – isolation of content with unpredictable characteristics, which results in increased traffic on the network

23

TVP content hypothesis testing D Does content iinsight i h add dd any value? l ?

5/23/2013

Drop Page Fields Here MATERIAL ID MATERIAL_ID

3000

4309239

Sum of REAL_REQ_NUMBER

7811575 7821552 7832365 7832416

2500

7832441 7863829 7863915

2000

7864003 7864056 7864079

1500

7864227 7892292 7892348

1000

7892369 7903760 7903804

500

7913348 7913404 7923528 2012-09-24

2012-09-23

2012-09-22

2012-09-21

2012-09-20

2012-09-19

2012-09-18

2012-09-17

2012-09-16

2012-09-15

2012-09-14

2012-09-13

2012-09-12

2012-09-11

2012-09-10

2012-09-09

2012-09-08

2012-09-07

2012-09-06

2012-09-05

2012-09-04

2012-09-03

2012-09-02

2012-09-01

2012-08-31

2012-08-30

2012-08-29

2012-08-28

0

7923593 7923614 7929935 7929984 7930008

DATE

7944502

Characteristics of the information program 24

2012-07-14 2012-07-27 2012-07-28 2012-07-29 2012-07-30 2012-07-31 2012-08-01 2012-08-02 2012-08-03 2012-08-04 2012-08-05 2012-08-06 2012-08-07 2012-08-08 2012-08-09 2012-08-10 2012-08-11 2012-08-12 2012-08-13 2012-08-14 2012-08-15 2012-08-16 2012-08-17 2012-08-18 2012-08-19 2012-08-20 2012-08-21 2012-08-22 2012-08-23 2012-08-24 2012-08-25 2012-08-26 2012-08-27 2012-08-28 2012-08-29 2012-08-30 2012-08-31 2012-09-01 2012-09-02 2012-09-03 2012-09-04 2012-09-05 2012-09-06 2012-09-07 2012-09-08 2012-09-09 2012-09-10 2012-09-11 2012-09-12 2012-09-13 2012-09-14 2012-09-15 2012-09-16 2012-09-17 2012-09-18 2012-09-19 2012-09-20 2012-09-21 2012-09-22 2012-09-23 2012-09-24 (blank)

TVP content hypothesis testing D Does content iinsight i h add dd any value? l ? 2500

DATE

Characteristics of the popular TV series 25

5/23/2013

Drop Page Fields Here MATERIAL ID MATERIAL_ID

Sum of REAL_REQ_NUMBER 3994796

3994809

3996934

3996954

2000 3996956

4071827

4071831

4080896

1500 4080900

4119634

4119646

1000 4165790

4165792

4198836

4198863

500 4198866

4239309

4362349

0 4362350

4442927

4444267

4444271

4480765

4480777

4519317

4525473

2012-07-14 2012-07-27 2012-07-28 2012-07-29 2012-07-30 2012-07-31 2012-08-01 2012-08-02 2012-08-03 2012-08-04 2012-08-05 2012-08-06 2012-08-07 2012-08-08 2012-08-09 2012-08-10 2012-08-11 2012-08-12 2012-08-13 2012-08-14 2012-08-15 2012-08-16 2012-08-17 2012-08-18 2012-08-19 2012-08-20 2012-08-21 2012-08-22 2012-08-23 2012-08-24 2012-08-25 2012-08-26 2012-08-27 2012-08-28 2012-08-29 2012-08-30 2012-08-31 2012-09-01 2012-09-02 2012-09-03 2012-09-04 2012-09-05 2012-09-06 2012-09-07 2012-09-08 2012-09-09 2012-09-10 2012-09-11 2012-09-12 2012-09-13 2012-09-14 2012-09-15 2012-09-16 2012-09-17 2012-09-18 2012-09-19 2012-09-20 2012-09-21 2012-09-22 2012-09-23 2012-09-24 (blank)

TVP content hypothesis testing D Does content iinsight i h add dd any value? l ? 500

DATE

26

5/23/2013

Drop Page Fields Here MATERIAL ID MATERIAL_ID

Sum of REAL_REQ_NUMBER 5962293

5980705

450 5999150

5999311

400 5999437

6158442

350 6158458

6158514

300 6158529

6163466

250 6163511

200 6163595

6163758

150 6163892

6291308

100 6294064

50 6294083

6294088

0 6294106

6294136

6294996

6295467

6314322

6318215

6445099

6445116

Characteristics of the TV series with long history

Build equation to compare Champion Challenger Testing

Production Proces

Output

5/23/2013

▶Champion Challenger testing when there is possible to test in

Input Background g Proces

Output p

A/B Testing

background on the whole production data

Production Proces A

Output

Production Proces B

Output p

Input

▶A/B testing when there is possible to test on the splitted production data

Statistics Testing

▶Statistics testing when above are

Production Proces Input

Output Gather Statistics

impossible 27

TVP solution assessment

5/23/2013

Chosen statistics testing g based on forecast accuracy y due to no possibility to test old and new solution in the same time

28

5/23/2013

What are operational requirements and standard solutions

29

Operational requirements

5/23/2013

▶ Prepare Data ▶ Real Time – the same state as on the operational system; Near Real Time – state with minimal latency

▶ Store Data ▶ High Performance Databases

▶ Learn ▶ High Performance Advanced Analytics

▶ Monitor ▶ Full automatization; Full accountability

▶ Act ▶ High Availability; High number of requests 30

Standard solutions

5/23/2013

▶ Prepare Data ▶ Stream/event processing; Minimal batches

▶ Store Data ▶ In memory databases

▶ Learn ▶ In database Advanced Analytics

▶ Monitor M it ▶ Messages; SLA management

▶ Act ▶ Business Rules Management; Failover; Load balancing; SOA

31

TVP operational challenges

5/23/2013

▶Variety of applications resulting in different representations in the logs ▶Internal I t l algorithms l ith resulting lti iin complex system behavior and logs representation ▶Variety of content with different metadata 32

How tto design H d i Hi High hL Levell Architecture

33

5/23/2013

5/23/2013

Single EDW Set Goals and Monitor

Self Service Dashboards

Build Plans and Notify

Self Service Ad Hoc

Operationalize and Act

Self Servise Advanced Analytics

Operational Business Intelligence

Full Service Predefined Reporting

Full Service Advanced Analytics

Business Rules Management g

Data Marts

Enterprise Data Warehouse

Big Data

ODS

Cost of creating such a low-latency BI environment may be more than the actual benefit the company receives 34

5/23/2013

Federated EDW

E h componentt used Each d

Replication n

only for its purpose with data replication p between them

35

5/23/2013

TVP before OBI ▶ Play video content:

▶ User requests video content through www ▶ DISPATCHER directs to proper server ▶ Content if available in CACHE (SSD) than broadcasted ▶ Content if not available in SSD than loaded from SATA into SSD

▶ Algorithms: Al ith ▶ DISPATCHER based on hash algorithm to distribute content ▶ SSD and SATA management based on internal application server algorithm

36

5/23/2013

TVP with OBI

▶DISPATCHER is based on OBI list • OBI list directs each video content on proper server • Additional Additi l metadata t d t about content used for calculations and learning

37

5/23/2013

What are required components

38

5/23/2013

OBI components

Each of components should be used only if it is necessary therefore whole development process will be agile and fast 39

TVP used components

5/23/2013

Data Store – stores dedicated calculations; Advanced Analytics – once per day prepares list with forecast; Business Rules Management – through day updates list after corrections from ongoing broadcast; Notification – sends alert if there is an issue; Automatization – whole process without ith t h human iinteraction t ti 40

5/23/2013

Conclusions

41

5/23/2013

Further reading

42

5/23/2013

Questions Atos IT Services sp. z o.o. ul. Postępu 18 (budynek Neptun) 02-676 02 676 Warszawa tel. +48 22 444 6500 fax. +48 22 444 6501

Tomasz Bawor Solution Architect [email protected]

For p public q questions and comments after session please use twitter @tbawor 43

5/23/2013

Thank you y Atos, the Atos logo, Atos Consulting & Technology Services, Atos Worldline, Atos Sphere, Atos Cloud, Atos Healthcare (in the UK) and Atos Worldgrid are registered trademarks of Atos SA. September 2011 © 2011 Atos. Confidential information owned by Atos, to be used by the recipient only. This document, or any part of it, may not be reproduced, copied, circulated and/or distributed nor q quoted without p prior written approval from Atos.

44