Chase Mao's blog

Linux Monitor Commands

2023-12-14

Introduction

Monitoring a Linux system is an important part of server development and operations. Below are some commonly used commands and their use cases for system monitoring.

CPU

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
# CPU monitoring
top
htop
iotop
mpstat
pidstat
ps aux

# Sampling CPU 0 every 1 second
sar -P 0 1

Memory

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# Checking memory
sar -r 1
free -h

# Process memory usage in order
ps aux --sort=-%mem | head -n 10

# Checking memory swapping in and out
# minflt: Virtual memory has no physical memory mapping
# majflt: Physical memory is swapped out to disk
sar -B 1

IO

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
# Get block device IO status, also provides CPU usage
iostat

# List block devices
lsblk

# Get thread IO status, p switches between thread/process
iotop

# Check block device read/write statistics
blktrace -d /dev/vdb1
blkparse -i vdb1 -d vdb1.out
btt -i vdb1.out

# Writing to disk will first write data to the page cache, check the upper limit of dirty memory, when reached, IO will be throttled, check memory related conditions
cat /proc/sys/vm/dirty_ratio

Network

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
# Network connections
netstat -antup
ss

# Network card traffic
sar -n DEV 1

# Routing table
ip route show table all 

# All routing tables
vim /etc/iproute2/rt_tables
ip rule show

# Traffic control
tc qd

# TCP slow start will cause a 500k packet, need to wait for five or six RTTs, can increase the initial size of the kernel sliding window, if it is a long connection, can turn off the configuration of the sliding window automatically shrinking
# TCP sliding window maximum space, RTT, will limit the available bandwidth, can increase the sliding window maximum space, set to bandwidth*RTT, just the required cache size

Process Monitoring

1
2
3
4
# Check the execution of the process, CPU, cache hit, etc.
perf stat -d [run program]
perf record -g -e [from perf list] [run program]
perf report -g

System Call

1
2
# Check system call
strace -ttT -f run_something