sar(1) tool, from the openSUSE package "sysstat", provides a comprehensive method for collecting performance data on a running system.
There isn't, however, a satisfactory way to display historical
sar data. It is not uncommon for users who experience a performance degradation to include a few days worth of
sar archives in their bug reports. The experts looking into the report often resort to scan these archives using text-oriented utilities such as
awk, which may fail to reveal the story hidden behind the data.
We aim at devising a method and tool to visualize large historical
Goal for this Hackweek
Take one or two sample
sar datasets and feed them into Grafana[LINK], the Perfetto trace viewer[LINK] and Performance Co-Pilot (PCP)[LINK]. The solution of choice will need to compare favorably with my two previous
sar visualization attempts made with ad-hoc scripts: [LINK-1][LINK-2]. Evaluate the outcome, write a report and select one single tool to focus future efforts on.
To the best of my knowledge, these are the methods currently available for plotting
- kSar[MAYBE-LINK-1][MAYBE-LINK-2][MAYBE-LINK-3] Ksar is a self-contained Java GUI. This tool is advertised in our openSUSE Tuning Guide, chapter 2 "System monitoring utilities", section 126.96.36.199 "Visualizing
sardata"[LINK] although we don't have a package for it. Its drawback is it can display a
sardatafile at a time: as
sarwrites one datafile per day, to visualize a week worth of archives one has to have multiple kSar windows open. The resulting charts will have different Y axis scales, which makes them difficult to compare. You can find a set of kSar screenshots illustrating this usability problem here: [LINK](requires SUSE confluence login).
- sadf -g
sadf -g your_datafile [ -- sar_options ] > output.svgsadf(1) can emit svg files that can be viewed in web browsers. The same limitation of kSar applies here: multiple days require multiple plots.
- sar2pcp[LINK] There exist a Performance Co-Pilot (PCP) plugin to import sar data, which we package in openSUSE as "pcp-import-sar2pcp". Last time I checked, sar2pcp had to be invoked with
LD_PRELOAD=/usr/lib64/libpcp_import.sobecause that shared library wasn't correctly linked in the build of the package. That can easily be fixed, but it's unclear to me how well the
sar+ PCP combination works, as I haven't yet tried it. It could very well suffer from the one-day-per-plot limitation of the previous two tools. The feature is advertised in both
sarand pcp's documentation:
- sadf -j
sadf -j your_datafile [ -- sar_options ] > output.jsonConverting sar data to json, then write an ad-hoc script to produce the charts. This is what I've done in the past, but writing special-purpose code every time takes energy away from actually analyzing the data and debugging the problem at hand. The experience was valuable though, as it showed what these charts should look like (see [LINK-1][LINK-2])
It seems a sensible choice to leverage an existing graphing tool and make adjustments so that it's tailored at
sar data (eg. create a plug-in or a new plot type within an existing framework). Candidates are Grafana (web tool), the Perfetto trace visualizer (web tool) and Performance Co-Pilot (desktop app). I'm slightly biased towards interactive charts as opposed to static images since zooming in/out and moving around the data range may help exploring the dataset. Interactivity is best achieved with web based tools, which offer the possibility of sharing access to the visualization with a URL, without the need for the recipient to install new software locally.
This project is part of:
Hack Week 22
This project is one of its kind!