High disk I/O is one of the most misunderstood Linux performance problems. Administrators often log into a sluggish server, see high %wa latency, and immediately assume a disk is failing or that one application is overwhelming the storage subsystem. In reality, disk activity is often a symptom rather than the root cause.
A busy database performing scheduled checkpoints, the kernel flushing dirty pages, a RAID rebuild, excessive swapping, or even memory pressure can all produce similar symptoms. The process consuming the most I/O is not always responsible for the slowdown, and the busiest storage device does not always create the bottleneck.
The goal isn’t simply to find a process for reading or writing data. It’s to determine whether storage is genuinely limiting system performance, identify what triggered the workload, and decide whether intervention is necessary or whether the observed activity is expected.
Understand What “High Disk I/O” Actually Means
Linux exposes several storage metrics, each describing a different aspect of disk activity. Treating them as interchangeable often leads to incorrect conclusions.
A storage device can be busy because it is transferring large sequential files, servicing thousands of small random operations, or waiting for a slow underlying device to complete requests. These workloads behave very differently despite all appearing as “high disk usage.”
The following table summarizes the most useful metrics.
| Metric | What it Indicates | High Values May Mean |
|---|---|---|
| IOPS | Number of read/write operations | Many small requests |
| Throughput (MB/s) | Amount of data transferred | Large sequential transfers |
| Latency | Time required to complete requests | Slow storage or queue congestion |
%util |
Percentage of time the device is busy | Device approaching saturation |
%wa |
CPU time waiting for I/O | Applications blocked by storage or memory pressure |
One common misconception is that “high” %wa always means the disk itself is slow. It simply means CPUs are waiting for I/O operations to complete. The delay may originate from network storage, software RAID, filesystem behavior, or memory reclaim rather than the physical drive.
Confirm Storage Is Actually the Bottleneck
Before identifying a process, determine whether the storage subsystem is truly limiting performance.
Begin with a system-wide view.
vmstat 1
Running the command with a one-second interval provides a continuously updated snapshot of CPU scheduling, memory activity, and I/O. Watch the wa column, which represents CPU time spent waiting for storage operations.
Occasional spikes are normal. Sustained high values while applications remain unresponsive suggest storage deserves further investigation.
Next, examine block device statistics.
iostat -xz 1
This command reports utilization, throughput, average request size, and service latency for each storage device.
Healthy NVMe drives typically show low latency even under heavy throughput, while mechanical hard drives naturally exhibit higher response times during random workloads. Rather than focusing on utilization alone, pay attention to whether latency increases dramatically as utilization approaches saturation.
If it%util remains near 100% while request completion times continue climbing, requests are likely queuing faster than the storage device can service them.
High Utilization Doesn’t Always Require Action
Many administrators panic when they see storage utilization reach 100%.
On its own, that metric says very little.
A backup server writing sequential data overnight may legitimately keep a disk fully occupied without affecting users. Conversely, an SSD showing only moderate utilization can still cause severe application delays if request latency suddenly increases.
Context matters.
Ask questions such as:
- Are users actually experiencing slow responses?
- Is the workload what we would typically expect at this time?
- Has latency increased compared to normal?
- Did the workload begin after a configuration or software change?
Without baseline performance data, it’s difficult to distinguish normal utilization from an abnormal storage event.
Identify Which Process Is Generating I/O
Once you’ve confirmed that storage is the likely bottleneck, please identify the processes that are issuing read and write requests.
One of the most useful tools is iotop.
sudo iotop
Unlike top, which focuses primarily on CPU and memory usage, iotop displays processes actively performing disk I/O.
Look for processes consistently generating reads or writes rather than brief bursts of activity. Temporary spikes during log rotation or scheduled maintenance are often harmless.
For historical statistics rather than a live display, use:
pidstat -d 1
This view reports per-process disk activity over time, making it easier to identify workloads that appear only intermittently.
If the process disappears before you can inspect it, cumulative mode iotop can help.
sudo iotop -ao
Instead of showing only current activity, cumulative mode tracks total I/O since monitoring began, making short-lived backup jobs or maintenance tasks much easier to identify.
When Monitoring Tools Show Nothing
One of the more confusing situations occurs when storage appears busy but iotop reports very little activity.
This does not necessarily mean the monitoring tool is broken.
Linux aggressively caches filesystem writes in memory. Applications often complete write operations quickly while the kernel delays flushing modified pages to disk until later. During that writeback phase, the original application may already be idle even though storage activity continues.
Similarly, filesystem maintenance, RAID synchronization, snapshot operations, or storage drivers themselves may generate I/O outside the lifetime of the user process that initiated it.
When no obvious process appears responsible, broaden the investigation rather than assuming the storage hardware has failed.
Examine Open Files Instead of CPU Usage
Processes consuming disk bandwidth often use very little CPU.
A backup application copying files between storage devices may spend most of its time waiting for reads and writes rather than performing calculations.
Listing open files can reveal what a process is actually accessing.
lsof
Because itlsof can produce thousands of entries on busy systems, filtering by process name or directory usually provides more useful information.
For example, finding that a web server repeatedly writes enormous log files may immediately explain otherwise unexplained storage activity.
Open file inspection is particularly valuable when multiple services share the same storage pool.
Don’t Ignore Memory Pressure
High disk writes frequently originate from memory management rather than application behavior.
When physical memory becomes scarce, Linux must reclaim pages, flush dirty buffers, and potentially swap inactive memory to disk. During these periods, storage activity may increase dramatically even though no application has changed its workload.
Check memory utilization alongside disk statistics.
free -h
Also monitor virtual memory activity.
vmstat 1
Large increases in swap activity or sustained page reclaim indicate that insufficient memory—not slow storage—may be driving the observed I/O.
Adding faster storage will not eliminate performance problems caused by chronic memory exhaustion.
Containers and Virtual Machines Add Another Layer
Self-hosted environments increasingly run applications inside Docker containers, Kubernetes clusters, or virtual machines. These environments complicate I/O diagnosis because the process generating disk activity may not be visible from inside the guest operating system.
A container reporting minimal activity may reside on a host where another container is saturating the same storage device.
Likewise, a virtual machine experiencing high latency may simply be sharing storage with several other guests competing for the same physical disks.
Whenever possible, investigate I/O from the host operating system before concluding that an individual guest or container is responsible.
Container runtimes using OverlayFS also introduce additional write amplification for certain workloads. Applications generating large numbers of small file modifications may behave differently than they would on a native filesystem, particularly on slower storage devices or NAS-backed volumes.
Correlate Metrics Instead of Trusting a Single Tool
No single command explains storage performance.
Experienced administrators compare several independent observations before drawing conclusions.
For example:
| Observation | Possible Interpretation | Next Step |
|---|---|---|
High %wa, low disk utilization |
Memory reclaim or network storage latency | Check RAM usage and filesystem mounts |
| High latency with modest throughput | Random I/O saturation | Identify workload type |
| High writes with almost no active processes | Kernel writeback or filesystem maintenance | Inspect background storage activity |
| One disk saturated in a RAID set | Rebuild or degraded array | Verify RAID health before stopping applications |
Looking for agreement between multiple tools produces far more reliable diagnoses than reacting to whichever metric appears most alarming.
When the Process Isn’t the Real Cause
Finding a process with high disk activity doesn’t automatically explain why storage performance has degraded. Many workloads generate large amounts of I/O as part of normal operation, and terminating them may interrupt legitimate maintenance rather than resolve the underlying issue.
Databases are a prime example. Systems such as PostgreSQL and MariaDB periodically flush modified data to disk through checkpoints. During these intervals, write activity can increase significantly, particularly on busy servers. The database isn’t malfunctioning—it is ensuring data reaches persistent storage.
Backup software behaves similarly. Incremental backups, deduplication jobs, and snapshot creation often generate sustained sequential reads that occupy storage bandwidth for extended periods. If backups coincide with other I/O-intensive workloads, latency may increase even though every process is functioning as expected.
The objective is to determine whether the workload is unexpected, excessive, or simply poorly timed.
Background Storage Tasks Can Saturate Disks
Not every storage-intensive operation appears in application monitoring.
Many Linux systems perform maintenance tasks that run below the application layer.
Common examples include:
- Software RAID rebuilds or consistency checks (
mdadm) - Btrfs balance or scrub operations
- ZFS scrub or resilver tasks
- Filesystem journal replay after an unclean shutdown
- Log rotation compressing large files
- Antivirus or indexing services scanning the filesystem
These activities may generate substantial I/O even when user applications appear mostly idle.
If your server uses software RAID, verify array status before investigating applications.
cat /proc/mdstat
A healthy array reports all expected devices as active. If a rebuild or recovery is in progress, elevated disk activity is expected until synchronization completes.
Similarly, systems using ZFS or Btrfs should verify whether scheduled maintenance tasks are running before concluding that storage hardware or applications are at fault.
Network Storage Changes the Investigation
Storage connected over NFS, SMB, or iSCSI introduces additional variables.
Applications may appear to be waiting on local disk operations when the actual delay exists elsewhere on the network. Congested switches, overloaded NAS devices, or high network latency can all produce symptoms that resemble slow local storage.
This is particularly important in virtualization environments where multiple hypervisors share the same storage appliance. A single busy host may affect every virtual machine using that shared datastore.
When investigating network-backed storage, correlate storage metrics with network latency rather than examining block devices alone.
Recovery Should Match the Cause
Once the workload has been identified, it is advisable to refrain from terminating it immediately.
Stopping a backup process may be reasonable if it unexpectedly overlaps with production traffic. Stopping a database performing checkpoint writes or recovery operations may prolong the incident or even risk data consistency.
Instead, consider the least disruptive corrective action.
| Situation | Better Response | Why |
|---|---|---|
| Scheduled backup affecting users | Reschedule or throttle the job | Preserves backup integrity while reducing contention |
| Large log files causing constant writes | Fix excessive logging rather than deleting logs | Addresses the source instead of the symptom |
| Memory pressure causing swap activity | Reduce memory usage or add RAM | Eliminates unnecessary disk I/O |
| RAID rebuild saturating disks | Allow rebuild to complete unless hardware is failing | Interrupting rebuilds often increases recovery time |
| Container monopolizing storage | Apply I/O limits or adjust workload scheduling | Prevents one workload from starving others |
Linux also provides I/O scheduling controls, such as ionice those that can reduce the impact of non-critical workloads without stopping them entirely. This is often preferable for maintenance jobs running on production systems because it allows progress to continue while giving latency-sensitive applications higher priority.
Verify That Performance Has Actually Improved
A temporary drop in disk activity does not necessarily indicate that the problem has been solved.
After making changes, monitor the same metrics used during diagnosis.
Check whether:
- Storage latency has returned to normal.
- Application response times have improved.
%wahas decreased.- The same process immediately resumes heavy I/O.
- New kernel or storage errors appear in the logs.
Inspect recent system messages for hardware or driver-related errors.
journalctl -k -p err -b
A healthy system should show no recurring storage timeouts, controller resets, or filesystem errors related to the original incident. If errors continue despite lower I/O activity, the root cause may be deteriorating hardware rather than workload contention.
Build a Baseline Before the Next Incident
One of the biggest challenges during performance troubleshooting is determining what “normal” looks like.
A storage device consistently operating at 70% utilization during business hours may be perfectly healthy. The same utilization on a lightly loaded server could indicate an emerging problem.
Collect baseline metrics during periods of normal operation, including:
- Average read and write throughput
- Typical IOPS
- Storage latency
- Queue depth
- Peak utilization during maintenance windows
- Common background jobs and their schedules
Monitoring platforms such as Prometheus with Grafana, Netdata, or enterprise observability tools make historical comparisons much easier than relying on real-time command output during an incident.
Historical trends also reveal gradual performance degradation that would otherwise go unnoticed until users begin reporting problems.
Prevent High Disk I/O From Becoming an Outage
Many storage incidents are predictable long before they become operational emergencies.
Avoid scheduling backups, filesystem maintenance, database optimization, and large file transfers during the same maintenance window. Individually, these jobs may be harmless; together, they can saturate even fast SSD arrays.
Review application logging regularly. Excessive debug logging, rapidly growing audit files, or misconfigured services can generate continuous write activity that quietly consumes storage bandwidth and accelerates SSD wear.
On container hosts, monitor storage usage at both the host and workload level. Overlay filesystems, shared volumes, and container image layers can make I/O patterns difficult to interpret if only one layer is monitored.
Finally, monitor latency alongside utilization. High utilization alone rarely indicates a problem, but increasing latency under a familiar workload often provides the earliest warning that storage performance is beginning to degrade.
Frequently Asked Questions
Why is it %wa high even though disk utilization is low?
High %wa indicates that processes are waiting for I/O, not necessarily that a local disk is busy. Memory reclaim, swapping, network-attached storage, or filesystem delays can all increase wait time without fully utilizing the underlying device.
Why does it iotop show almost no activity while the storage LED is constantly blinking?
The kernel may be flushing cached writes to disk after the originating process has already finished. Filesystem maintenance tasks, RAID synchronization, or storage drivers can also generate I/O independently of active user processes.
Can insufficient RAM cause high disk I/O?
Yes. When memory becomes constrained, Linux writes modified pages to disk more aggressively and may begin swapping inactive memory. In these cases, adding RAM or reducing memory pressure often improves storage performance more than upgrading the disk itself.
Is replacing the SSD the best solution for persistent high disk I/O?
Not always. Faster storage helps only when the storage device is the limiting factor. If the workload is caused by excessive logging, poorly scheduled maintenance jobs, inefficient applications, or memory pressure, replacing hardware may provide little measurable improvement.
Final Thoughts
Effective disk I/O troubleshooting is less about finding the busiest process and more about understanding why the storage subsystem is busy in the first place. Applications, the Linux kernel, filesystems, RAID layers, containers, and virtualization platforms all influence how disk activity appears during an incident.
A structured investigation—confirming that storage is genuinely the bottleneck, correlating multiple performance metrics, identifying the workload responsible, and validating changes before declaring success—produces far more reliable results than reacting to a single command or utilization percentage.
Most importantly, treat high disk I/O as an observation rather than a diagnosis. Once you understand what the storage stack is doing and why, the appropriate corrective action usually becomes much clearer.
