-

Comments

Post a Comment

Popular posts from this blog

Setup K8s to use nvidia drivers

-
Setting up k8's to use nvidia Setting up k8's to use nvidia Prerequisites Quick Start Preparing your GPU Nodes Enabling GPU Support in Kubernetes Checks Sample yaml file References Destroy Prerequisites The list of prerequisites for running the NVIDIA device plugin is described below: NVIDIA drivers ~= 410.48 nvidia-docker version > 2.0 (see how to install and it's prerequisites) docker configured with nvidia as the default runtime . Kubernetes version >= 1.10 kubeadm installation . Post installation use flannel as the network plugin - $ sudo kubeadm init --pod-network-cidr=10.244.0.0/16 - $ kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/2140ac876ef134e0ed5af15c65e414cf26827915/Documentation/kube-flannel.yml - $ kubectl taint nodes --all node-role.kubernetes.io/master- Quick Start Preparing your GPU Nodes The following steps need to be executed on all your GPU nodes. This README assumes that the NVIDIA drivers ...
Read more

Debug an IO throttle issue

-
Issue Have you ever felt that write to a partition has become relatively slower than previously?? Ever wondered how to debug such kind of issues Then this post is for you. Useful commands are lsblk iotop iostat dd Check the partition type using  df -Th Debugging dd command is used to emulate disk write to the device.  In this example we are just writing the stream of  /dev/zero of size 1G to our partition. A key observation was that the real time was way higher than sys and user time combined. for i in {0..60}; do time dd if=/dev/zero of=/data/1GB.bin bs=1G count=1; done output real 31m27s user 00m00s sys 00m806s dd process was observed to be in uninterruptible sleep (D) state using the following command for the most part of the time as the above command was under execution. iotop (run as a root user) showed that there was no disk activity on any of the disks on the device. Few takeaways: /dev/zero produces a continuous st...
Read more