Introduction
This article concentrates on some useful commands that come in handy while analyzing memory related performance issues on Linux.The article is inspired from this video. I've listed down all the commands mentioned in that video. The video is pretty long and this article is an attempt to create a gist of items mentioned there.
Commands:-
1. vmstat:-
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 0 0 4698476 172856 1552720 0 0 3 12 82 79 2 1 98 0 0
2. dstat --top-oom:- This command helps us to find out which process would be terminated first in case of high memory consumption. In such a case, the kernel terminates the highest memory consumer. This command helps us find that out.
3. getconf PAGE_SIZE:- This returns the default page size on the machine. On my Ubuntu 14.04 64 bit, I see this as follows:-
mukund@mukund-desktop:~$ getconf PAGE_SIZE
4096
4. pmap:- This returns the address mapping of a process. Here, in the output, we can see the virtual address and the location of the actual file. The 'anon' are heap and stack memory.
mukund@mukund-desktop:~$ ps -ae | grep term
2205 ? 00:00:20 gnome-terminal
mukund@mukund-desktop:~$ pmap 2205
2205: gnome-terminal
0000000000400000 284K r-x-- gnome-terminal
0000000000646000 4K r---- gnome-terminal
0000000000647000 12K rw--- gnome-terminal
0000000000b93000 8400K rw--- [ anon ]
00007f309c000000 136K rw--- [ anon ]
00007f309c022000 65400K ----- [ anon ]
00007f30a4000000 132K rw--- [ anon ]
00007f30a4021000 65404K ----- [ anon ]
00007f30a958d000 4K ----- [ anon ]
00007f30a958e000 8192K rw--- [ anon ]
00007f30a9d8e000 35272K r---- icon-theme.cache
00007f30ac000000 136K rw--- [ anon ]
00007f30ac022000 65400K ----- [ anon ]
00007f30b0000000 324K rw--- [ anon ]
00007f30b0051000 65212K ----- [ anon ]
00007f30b4113000 4K ----- [ anon ]
00007f30b4114000 8192K rw--- [ anon ]
00007f30b4914000 356K r-x-- libibus-1.0.so.5.0.505
00007f30b496d000 2048K ----- libibus-1.0.so.5.0.505
00007f30b4b6d000 8K r---- libibus-1.0.so.5.0.505
00007f30b4b6f000 4K rw--- libibus-1.0.so.5.0.505
00007f30b4b70000 4K rw--- [ anon ]
00007f30b4b71000 24K r-x-- im-ibus.so
00007f30b4b77000 2048K ----- im-ibus.so
00007f30b4d77000 4K r---- im-ibus.so
00007f30b4d78000 4K rw--- im-ibus.so
00007f30b4d79000 132K r-x-- liblzma.so.5.0.0
00007f30b4d9a000 2044K ----- liblzma.so.5.0.0
00007f30b4f99000 4K r---- liblzma.so.5.0.0
00007f30b4f9a000 4K rw--- liblzma.so.5.0.0
00007f30b4f9b000 1392K r-x-- libxml2.so.2.9.1
00007f30b50f7000 2048K ----- libxml2.so.2.9.1
...
total 662024K
5. top:- This command can prove really useful at times. Here, 'VIRT' shows the amount of virtual memory being used and 'RES' shows the amount of memory present in RAM. There's a catch with the 'RES' memory. There are some libraries, like libC that are used by several processes. There's a single copy of this lib in physical memory, which is mapped into virtual memory of all processes. However, top counts this per process. As a result, the sum of all 'RES' memory is generally greater than the actual physical memory.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2003 mukund 20 0 2210808 1.009g 89752 S 42.9 13.0 88:17.77 firefox
1151 root 20 0 359768 68188 32756 S 5.7 0.8 18:06.65 Xorg
1887 mukund 20 0 1630664 182452 75164 S 3.0 2.2 10:08.64 compiz
1305 mukund 20 0 379444 11852 7008 S 1.0 0.1 0:40.29 ibus-daemon
1678 mukund 9 -11 513332 12108 9728 S 1.0 0.1 1:04.37 pulseaudio
2205 mukund 20 0 662024 34672 24640 S 1.0 0.4 0:21.38 gnome-terminal
Example:-
Here, we see libc mapped as follows, for PID 2003
mukund@mukund-desktop:~$ pmap 2003 | grep libc
00007f9277d1d000 1772K r-x-- libc-2.19.so
00007f9277ed8000 2044K ----- libc-2.19.so
00007f92780d7000 16K r---- libc-2.19.so
00007f92780db000 8K rw--- libc-2.19.so
For PID 2205, we see the same libc mapped as follows:-
mukund@mukund-desktop:~$ pmap 2205 | grep libc
00007f30c6580000 1772K r-x-- libc-2.19.so
00007f30c673b000 2044K ----- libc-2.19.so
00007f30c693a000 16K r---- libc-2.19.so
00007f30c693e000 8K rw--- libc-2.19.so
So, we can see that there is only one physical copy of the file, however mapped into the virtual address space of two different processes. However, top counts this as two separate 'RES' memory.
This script can be used to determine a pretty accurate picture of the actual physical memory being used.
6. cat /proc/<PID>/smaps:- This command can be used to find the clean, dirty pages per process.
7. cat /proc/meminfo: This can be used to find the total memory related information in the system. This below command can be used to find inactive and dirty pages.
mukund@mukund-desktop:~/Desktop$ cat /proc/meminfo | egrep -i "inactive|dirty"
Inactive: 725508 kB
Inactive(anon): 16384 kB
Inactive(file): 709124 kB
Dirty: 88 kB
The anonymous memory is the memory not mapped onto files, like the heap or the stack memory. This memory can only be moved to the swap space to page out.
8. min_free is the amount of memory after which the kernel(kswapd0 therad) starts seeing OOM and starts freeing memory.
mukund@mukund-desktop:~/Desktop$ ps -aux|grep swap
root 52 0.0 0.0 0 0 ? S Jul06 0:00 [kswapd0]
mukund@mukund-desktop:~/Desktop$ sudo sysctl -a | grep -i min_free
sysctl: reading key "net.ipv6.conf.all.stable_secret"
sysctl: reading key "net.ipv6.conf.default.stable_secret"
sysctl: reading key "net.ipv6.conf.eth0.stable_secret"
sysctl: reading key "net.ipv6.conf.lo.stable_secret"
vm.min_free_kbytes = 11350
9. sysq-trigger:- This is an interesting command to do special requests to the Linux kernel. The documentation can be found here.
Example:-
echo f > sysrq-trigger
This triggers the OOM killer.
10. lscpu:- This is a nice utility to get great information about your CPU.
mukund@mukund-desktop:~/Desktop$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 1
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 42
Stepping: 7
CPU MHz: 2964.828
BogoMIPS: 5587.24
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 6144K
NUMA node0 CPU(s): 0-3
11.ps -o min_flt,maj_flt <pid>: This command can be used to find the major and minor page faults of a process.
