Modern Storage Area Networks (SANs) are challenging to manage due to complex network topologies and the difficulties that can arise in identifying the source of performance problems and resolving them.
Good tooling for analyzing Small Computer System Interface (SCSI) devices for Linux on System z used to be hard to find. However, that’s no longer the case, as the ziomon toolset released by IBM in early 2009 improves things. It consists of:
- ziomon, a tool to collect data on SCSI device topology and performance data for a specified time
- ziorep_config, ziorep_traffic and ziorep_utilization (referred to as ziorep tools from here on), which are tools to generate reports from the data collected with ziomon.
This article examines a real-life problem scenario and uses the ziomon tools to resolve it. While doing so, we will go over the individual tools and discuss their main features and use cases. We also will look at the reports generated by the tools, and how to interpret and put them to good use.
A typical usage cycle is comprised of a data collection step with ziomon followed by multiple analysis steps using the ziorep tools.
Ideally, the ziomon package is already available in your Linux distribution. If not, a manual installation is possible, but not trivial, as there are several dependencies that must first be met, including the correct blktrace version and the right level of the zfcp kernel module. Check your distribution for the complete ziomon toolset.
Now let’s assume the Linux system administrator has received complaints about bad performance. He has ruled out the applications as possible culprits. Also, CPU and memory utilization indicate there are still resources left. So, the problem must be elsewhere. The applications use various multi-path devices, including:
- /dev/mapper/36005076303ffc1040000000000002120, abbreviated mpA
- /dev/mapper/36005076303ffc1040000000000002121, abbreviated mpB
- /dev/mapper/36005076303ffc1040000000000002122, abbreviated mpC.
Many users complain about bad performance of applications that use mpA, and there are a few complaints that mpC is slow, too. However, no users have complained about applications using mpB.
To investigate this problem and solve it, you should start by collecting data using ziomon. This can be as simple as typing in the following:
> ziomon -d 10 -o investigation /dev/mapper/36005076303ffc1040000000000002120