An introduction to Logger and ESM Web Services APIs Shivdev Kalambi Principal developer and manager correlation team
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Agenda Logger Web Services APIs • •
Look at the Login Service, Report Service and Search Service Look at some use cases
ESM Web Services APIs • •
Look at the Login Service, Query Viewer Service, and Report Service REST & SOAP Examples
Q&A
2
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Logger Web Services APIs
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Web Service APIs What are web services?
Wikipedia definition
External client
• Web services are typically application programming interfaces (API) or web APIs that can be accessed over a network, such as the Internet, and executed on a remote system hosting the requested services.
4
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Web Server that exposes Web Services (e.g. Logger)
Internal client
So what are we doing here? The idea is simple Fetch data from Logger and apply it to your use case
5
Do some analytics?
Write a simple client?
Show the source Address on a map
For the command line folks
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Got data? What must I know?
Prerequisites • Logger Search Syntax 100110011010101001101001101010 ? • Logger Reporting (SQL) • Knowhow of Web Services in general • Write code for simple SOAP clients
6
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Logger Web Service APIs APIs available in Logger v5.3 APIs Available • Services • LoginService • ReportService • SearchService • WSDL Location • https:///soap/services//.wsdl • https://192.168.35.9/soap/services/ReportService/ReportService.wsdl
7
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
LoginService Methods
Arguments
Return
login
username, password,timeout
cookie
logout
cookie
getVersion extendSession
Login to a logger and establishes a cookie Ends a session identified by the cookie
String cookie
Description
Returns the version of the web service Extends the session identified by the specified cookie
Example String cookie = loginService.login(“admin”, “password”, 120); 8
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
ReportService Methods
Arguments
Return
Description
getReportGroups
cookie
Group[]
Get the Report Groups (Categories)
getReportsInGroup
groupID, cookie
Report[]
Get the Reports in a Group
String
Base64 encoded Report Result (for eg. in CSV format).
String[]
Get a list of devices
runReport
getDevices
reportId, startTime, endTime, scanLimit, rowLimit, devices, deviceGroups, storageGroups, reportParams, reportFormat, cookie cookie
Example String report = reportService.runReport(report_ID, System.currentTimeMillis() – 2 * 60 * 60 * 1000, System.currentTimeMillis(), 10000, 100, null, null, null, null, “csv”, cookie); 9
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
SearchService Methods
Arguments
startSearch
queryString, startTime, endTime, cookie
Starts the search
endSearch
cookie
Ends a search session identified by the cookie
getNextTuples
count, timeout, cookie
Tuple[]
Get an array of Tuples that matched the Search
hasMoreTuples
cookie
boolean
Find out whether the search has more tuples
getHeader
cookie
String[]
The Format of the Data in the Tuples
10
Return
Description
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Sample Java Code Simple Search using the Iterator Pattern // Login String cookie = loginService.login("admin", "password", 60); // Start the Search searchService.startSearch("ERROR“, System.currentTimeMillis() - 10 * 60 * 1000, System.currentTimeMillis(), cookie); // Loop over and find resulting rows while (searchService.hasMoreTuples(cookie)) { Tuple [] tuples = searchService.getNextTuples(500, 1000, cookie); if (tuples != null) { for (Tuple tuple : tuples) { String [] arr = tuple.getData(); // Custom Processing of the data } } } // End the Search searchService.endSearch(cookie); loginService.logout(cookie); 11
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Use case 1 Command Line Utility • Run a search (or report) from the Logger Web UI – Logger | cef name | top 5 name • Simulate the same search using Web Services from a command line utility
12
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Use case 1 Command line utility
13
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Use case 1 INtegrate the command line utility with ESM ESM/Logger Integration • When a rule fires in ESM • Execute the utility to query logger for events • Send an email with events
14
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Use case 2 Plot geo locations for top 5 source IP addresses on a map Run a search CEF | cef sourceAddress | top 5 sourceAddress Run a report using this SQL SELECT arc_sourceAddress, COUNT(arc_eventId) FROM events GROUP BY arc_sourceAddress ORDER BY COUNT (arc_eventID) DESC LIMIT 5
Feed the results to a mapping tool 15
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Best practices Some points to keep in mind • Login Service – Always logout – Sessions get purged but will take a while to clean up • Search Service – Always endSearch – The Searcher will be instantly cleaned up – From v5.3, you will see performance improvements in the getNextTuples(count, timeout, cookie) – by changing the count • Report Service – Data returned from runReport(…) call is Base64 encoded – so you would need to decode it
16
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Writing clients Please note • Users are expected to write their own SOAP Clients • We provide a reference JAVA implementation and have provided some sample JAVA code to serve as an example • Documentation is available on the customer support site
17
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
ESM Web Services APIs
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
ESM Web Services APIs Introduction • Login Service • Query Viewer Service – Fetching data using REST – Fetching data using SOAP • Java Code Snippets • Report Service • Required Libraries & Interesting Observations
19
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Use cases Top 10 most common events Fetch query viewer data • Sample Query Viewer • Fetch data – Using REST – Using SOAP
20
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Use case 1 REST call to login service https://localhost:8443/www/core-service/rest/LoginService/login?login=admin&password=password Test the REST • From the browser • Invoke LoginService – Copy the authToken • Invoke QueryViewerService – Pass the QueryViewer ID
https://localhost:8443/www/manager-service/rest/QueryViewerService/getMatrixData?authToken=__&id=__
21
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Use case 1 REST call to QueryViewer service https://localhost:8443/www/manager-service/rest/QueryViewerService/getMatrixData?authToken=__&id=__ QueryViewerService using REST
QueryViewerService using SOAP
22
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Use case 2 SOAP Call to login service // Set the Base URL System.setProperty("com.arcsight.coma.client.ws.baseURL", "https://” + host + "/www/"); // localhost:8443 // Get the LoginService and login LoginServiceClientFactory factory = new LoginServiceClientFactory(); LoginService service = factory.createClient(); String authToken = service.login(null, “admin”, “password”); // This authToken is required in subsequent calls
23
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Use case 2 SOAP call to query viewer service // Get the QueryViewerService and get the data QueryViewerServiceClientFactory factory = new QueryViewerServiceClientFactory (); QueryViewerService service = factory.createClient(); MatrixData md = service.getMatrixData(authToken, "cwswTlzgBABCKipuKVcyzlg=="); // Get the Column Names List headers = md.getColumnHeaders(); int col = 0; for (String header : headers) { System.out.printf((col++%2 == 0 ? "%60s" : "%20s\n"), header); } // Get the Data List rows = md.getRows(); for (ListWrapper row : rows) { List value = row.getValue(); for (Object obj : value) { System.out.printf((col++%2 == 0 ? "%60s" : "%20s\n"), obj); } } 24
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Use case 2 I found that I needed the following static block to trust the hostname // Static Block static { HttpsURLConnection.setDefaultHostnameVerifier(new HostnameVerifier() { public boolean verify(String hostname, SSLSession session) { // Make sure that hostname is valid return true; } }); }
25
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Use Case 3 REST Call to Report Service Start the Report Generation
https://localhost:8443/www/manager-service/rest/ArchiveReportService/initDefaultArchiveReportDownloadById?authToken=_&reportId=_&reportType=Manual
Get the Download ID and download the report
https://localhost:8443/www/manager-service/fileservlet?file.command=download&file.id=DOWNID
26
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Required libraries and interesting observations Tips from an end user • Even though it’s SOAP under the covers AXIS2 libraries didn’t work – manager-ws-client-1.2.0.release.107.jar – core-ws-client-1.5.0.release.51.jar – coma-infrastructure-1.4.0.release.240.jar • For now, the SOAP APIs can only be written in Java and using these libraries • I was not able to get it to work with AXIS2 libraries in the CLASSPATH – Marshalling Errors • Don’t forget to implement a HostnameVerifier (by default it will NOT be a verified hostname) • Documentation is available on the Support Site
27
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Recap Key Takeaways • Logger Web Services APIs – Supports only SOAP – Login Service – Local Authentication – Search Service – Start Search and Iterator Pattern to go over resulting Tuples – Report Service – Run a Report and get back a Base64 encoded result • ESM Web Services APIs – REST – for simple use cases – SOAP – For now, Java clients using the provided libraries – GWT-RPC is also used by our UI team
28
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Find out more Attend these sessions
• Session ID: 1257 Title: Gain quick and relevant solutions made possible by the HP ArcSight Logger API.
Meet these folks
• Logger PMs: Roopak Patel & Suresh Venkatraman • ESM PMs: Monica Jain & Saran Selvaraj • ESM, CORRE, Logger Dev: Anurag Singla, David Wiser, & Vivek Vallachira
After the event
• Contact your sales rep • Visit the Social Pages at: http://www.facebook.com/HPSecur e Twitter: @HPSecure • Download the slides at: http://protect724.arcsight.com/
Your feedback is important to us. Please take a few minutes to complete the session survey.
29
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Thank you
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Security for the new reality © Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
