Using eBPF to trace disk transfer actions
Never one to hold back on providing input, Mihalis Tsoukalos explains how to keep a track of disk I/O using eBPF.
EBPF
Credit: https://ebpf.io
OUR EXPERT
Mihalis Tsoukalos
is the author of Go Systems Programming and Mastering Go, 3rd edition. You can reach him at www.mtsoukalos.eu and @mactsouk.
QUICK TIP
Seekwatcher is a tool that can generate graphs from blktrace data in order to help you reveal I/O patterns and visualise performance. You can find more information about Seekwatcher at https://github. com/tnm/ seekwatcher.
We are going to learn how to use the capabilities of eBPF to monitor disk I/O operations in order to discover and solve performance issues related to input and output operations. We’ll kick off by discussing the ways of approaching issues related to disk I/O, because there’s a wide range of tools that can help you and choosing the right tool for the job is often the most difficult task. As a result, not all issues should be resolved with the help of eBPF, especially when there are simpler tools that can do the job more quickly. Therefore, some logical steps that can be used for resolving disk I/O issues are as follows:
1. Begin with some traditional Unix tools such as iostat, vmstat, btrace or htop to get a more accurate idea of how serious the performance issue is.
2. If you cannot find a solution using a single or a combination of traditional Unix tools, begin examining the system using eBPF-related tools.
3. You might need to begin your analysis by examining the filesystem with the help of eBPF. For that you might have to wait for next month’s eBPF tutorial.
4. If it is not a filesystem issue, you should begin experimenting with the tools in this tutorial. Two key metrics to consider are high disk usage – higher than required, to be exact – and higher latency than normal.
5. If it is a system-wide issue, the previous steps might give you the answer. If it is application-specific, use the tools to identify and check specific applications. Database servers without indexes or without proper normalisation are often the cause of Disk I/O issues.
6. Keep looking until you understand the problem better and focus on a smaller area.