Debug an IO throttle issue

Issue

Have you ever felt that write to a partition has become relatively slower than previously?? Ever wondered how to debug such kind of issues Then this post is for you.
Useful commands are

• lsblk
• iotop
• iostat
• dd

Check the partition type using  df -Th

Debugging

• dd command is used to emulate disk write to the device. 
• In this example we are just writing the stream of  /dev/zero of size 1G to our partition.
• A key observation was that the real time was way higher than sys and user time combined.

for i in {0..60}; do time dd if=/dev/zero of=/data/1GB.bin bs=1G count=1; done

output

real 31m27s

user 00m00s

sys 00m806s

• dd process was observed to be in uninterruptible sleep (D) state using the following command for the most part of the time as the above command was under execution.
  
• iotop (run as a root user) showed that there was no disk activity on any of the disks on the device.
  
  • Few takeaways:
  • /dev/zero produces a continuous stream of NULL (zero value) bytes.
  • if is input file
  • of is output file

Misleading Assumptions

• In case if there is any doubt that memory usage and CPU consumption are a cause for the io slow down which very unlikely. please stop all the processes running on the server and perform the above exercises

Findings

When we run in combination with sleep eg:

for i in {0..60}; do time dd if=/dev/zero of=/data/1GB.bin bs=1mb count=100; sleep 5 ; done

we see that the real time dropped significantly by 30 Seconds.
Which makes it clear that the I/O is going to sleep before writing. We suspected it to be an I/O throttle.

Solution

• If a similar issue is noticed, try to contact the vendor or Cloud Provider as soon as possible.

Commands

To see the processes which are in the waiting state.

ps aux | awk '$8 ~ /D/  { print $0 }'

and below to see for a time interval of 5 seconds

for x in `seq 1 1 10`; do ps -eo state,pid,cmd | grep "^D"; echo "----"; sleep 5; done

References

• dd_usage
• understading_dd
• Controlling_IO