The following sections showcase scripts that trace network-related functions and build a profile of network activity.
This section describes how to profile network activity.
nettop.stp provides a glimpse into how much network traffic each process is generating on a machine.
#! /usr/bin/env stap
global ifxmit, ifrecv
global ifmerged
probe netdev.transmit
{
ifxmit[pid(), dev_name, execname(), uid()] <<< length
}
probe netdev.receive
{
ifrecv[pid(), dev_name, execname(), uid()] <<< length
}
function print_activity()
{
printf("%5s %5s %-7s %7s %7s %7s %7s %-15s\n",
"PID", "UID", "DEV", "XMIT_PK", "RECV_PK",
"XMIT_KB", "RECV_KB", "COMMAND")
foreach ([pid, dev, exec, uid] in ifrecv) {
ifmerged[pid, dev, exec, uid] += @count(ifrecv[pid,dev,exec,uid]);
}
foreach ([pid, dev, exec, uid] in ifxmit) {
ifmerged[pid, dev, exec, uid] += @count(ifxmit[pid,dev,exec,uid]);
}
foreach ([pid, dev, exec, uid] in ifmerged-) {
n_xmit = @count(ifxmit[pid, dev, exec, uid])
n_recv = @count(ifrecv[pid, dev, exec, uid])
printf("%5d %5d %-7s %7d %7d %7d %7d %-15s\n",
pid, uid, dev, n_xmit, n_recv,
n_xmit ? @sum(ifxmit[pid, dev, exec, uid])/1024 : 0,
n_recv ? @sum(ifrecv[pid, dev, exec, uid])/1024 : 0,
exec)
}
print("\n")
delete ifxmit
delete ifrecv
delete ifmerged
}
probe timer.ms(5000), end, error
{
print_activity()
}
Note that function print_activity()
uses the following expressions:
n_xmit ? @sum(ifxmit[pid, dev, exec, uid])/1024 : 0
n_recv ? @sum(ifrecv[pid, dev, exec, uid])/1024 : 0
These expressions are if/else conditionals. The first statement is simply a more concise way of writing the following psuedo code:
if n_recv != 0 then
@sum(ifrecv[pid, dev, exec, uid])/1024
else
0
nettop.stp tracks which processes are generating network traffic on the system, and provides the following information about each process:
PID
— the ID of the listed process.
UID
— user ID. A user ID of 0
refers to the root user.
DEV
— which ethernet device the process used to send / receive data (e.g. eth0, eth1)
XMIT_PK
— number of packets transmitted by the process
RECV_PK
— number of packets received by the process
XMIT_KB
— amount of data sent by the process, in kilobytes
RECV_KB
— amount of data received by the service, in kilobytes
[...]
PID UID DEV XMIT_PK RECV_PK XMIT_KB RECV_KB COMMAND
0 0 eth0 0 5 0 0 swapper
11178 0 eth0 2 0 0 0 synergyc
PID UID DEV XMIT_PK RECV_PK XMIT_KB RECV_KB COMMAND
2886 4 eth0 79 0 5 0 cups-polld
11362 0 eth0 0 61 0 5 firefox
0 0 eth0 3 32 0 3 swapper
2886 4 lo 4 4 0 0 cups-polld
11178 0 eth0 3 0 0 0 synergyc
PID UID DEV XMIT_PK RECV_PK XMIT_KB RECV_KB COMMAND
0 0 eth0 0 6 0 0 swapper
2886 4 lo 2 2 0 0 cups-polld
11178 0 eth0 3 0 0 0 synergyc
3611 0 eth0 0 1 0 0 Xorg
PID UID DEV XMIT_PK RECV_PK XMIT_KB RECV_KB COMMAND
0 0 eth0 3 42 0 2 swapper
11178 0 eth0 43 1 3 0 synergyc
11362 0 eth0 0 7 0 0 firefox
3897 0 eth0 0 1 0 0 multiload-apple
[...]
Example 4.1. nettop.stp Sample Output
4.1.2. Tracing Functions Called in Network Socket Code
This section describes how to trace functions called from the kernel's net/socket.c
file. This task helps you identify, in finer detail, how each process interacts with the network at the kernel level.
[...]
0 Xorg(3611): -> sock_poll
3 Xorg(3611): <- sock_poll
0 Xorg(3611): -> sock_poll
3 Xorg(3611): <- sock_poll
0 gnome-terminal(11106): -> sock_poll
5 gnome-terminal(11106): <- sock_poll
0 scim-bridge(3883): -> sock_poll
3 scim-bridge(3883): <- sock_poll
0 scim-bridge(3883): -> sys_socketcall
4 scim-bridge(3883): -> sys_recv
8 scim-bridge(3883): -> sys_recvfrom
12 scim-bridge(3883):-> sock_from_file
16 scim-bridge(3883):<- sock_from_file
20 scim-bridge(3883):-> sock_recvmsg
24 scim-bridge(3883):<- sock_recvmsg
28 scim-bridge(3883): <- sys_recvfrom
31 scim-bridge(3883): <- sys_recv
35 scim-bridge(3883): <- sys_socketcall
[...]
4.1.3. Monitoring Incoming TCP Connections
This section illustrates how to monitor incoming TCP connections. This task is useful in identifying any unauthorized, suspicious, or otherwise unwanted network access requests in real time.
While
tcp_connections.stp is running, it will print out the following information about any incoming TCP connections accepted by the system in real time:
Current UID
CMD
- the command accepting the connection
PID
of the command
Port used by the connection
IP address from which the TCP connection originated
UID CMD PID PORT IP_SOURCE
0 sshd 3165 22 10.64.0.227
0 sshd 3165 22 10.64.0.227
4.1.4. Monitoring TCP Packets
This section illustrates how to monitor TCP packets received by the system. This is useful in analyzing network traffic generated by applications running on the system.
While
tcpdumplike.stp is running, it will print out the following information about any received TCP packets in real time:
Source and destination IP address (saddr
, daddr
, respectively)
Source and destination ports (sport
, dport
, respectively)
Packet flags
To determine the flags used by the packet,
tcpdumplike.stp uses the following functions:
urg
- urgent
ack
- acknowledgement
psh
- push
rst
- reset
syn
- synchronize
fin
- finished
The aforementioned functions return 1
or 0
to specify whether the packet uses the corresponding flag.
-----------------------------------------------------------------
Source IP Dest IP SPort DPort U A P R S F
-----------------------------------------------------------------
209.85.229.147 10.0.2.15 80 20373 0 1 1 0 0 0
92.122.126.240 10.0.2.15 80 53214 0 1 0 0 1 0
92.122.126.240 10.0.2.15 80 53214 0 1 0 0 0 0
209.85.229.118 10.0.2.15 80 63433 0 1 0 0 1 0
209.85.229.118 10.0.2.15 80 63433 0 1 0 0 0 0
209.85.229.147 10.0.2.15 80 21141 0 1 1 0 0 0
209.85.229.147 10.0.2.15 80 21141 0 1 1 0 0 0
209.85.229.147 10.0.2.15 80 21141 0 1 1 0 0 0
209.85.229.147 10.0.2.15 80 21141 0 1 1 0 0 0
209.85.229.147 10.0.2.15 80 21141 0 1 1 0 0 0
209.85.229.118 10.0.2.15 80 63433 0 1 1 0 0 0
[...]
4.1.5. Monitoring Network Packets Drops in Kernel
The network stack in Linux can discard packets for various reasons. Some Linux kernels include a tracepoint,
kernel.trace("kfree_skb")
, which easily tracks where packets are discarded.
dropwatch.stp uses
kernel.trace("kfree_skb")
to trace packet discards; the script summarizes which locations discard packets every five-second interval.
The kernel.trace("kfree_skb")
traces which places in the kernel drop network packets. The kernel.trace("kfree_skb")
has two arguments: a pointer to the buffer being freed ($skb
) and the location in kernel code the buffer is being freed ($location
).
Monitoring for dropped packets
51 packets dropped at location 0xffffffff8024cd0f
2 packets dropped at location 0xffffffff8044b472
51 packets dropped at location 0xffffffff8024cd0f
1 packets dropped at location 0xffffffff8044b472
97 packets dropped at location 0xffffffff8024cd0f
1 packets dropped at location 0xffffffff8044b472
Stopping dropped packet monitor
Example 4.5. dropwatch.stp Sample Output
To make the location of packet drops more meaningful, refer to the
/boot/System.map-`uname -r`
file. This file lists the starting addresses for each function, allowing you to map the addresses in the output of
Example 4.5, “dropwatch.stp Sample Output” to a specific function name. Given the following snippet of the
/boot/System.map-`uname -r`
file, the address 0xffffffff8024cd0f maps to the function
unix_stream_recvmsg
and the address 0xffffffff8044b472 maps to the function
arp_rcv
:
[...]
ffffffff8024c5cd T unlock_new_inode
ffffffff8024c5da t unix_stream_sendmsg
ffffffff8024c920 t unix_stream_recvmsg
ffffffff8024cea1 t udp_v4_lookup_longway
[...]
ffffffff8044addc t arp_process
ffffffff8044b360 t arp_rcv
ffffffff8044b487 t parp_redo
ffffffff8044b48c t arp_solicit
[...]