Abstract
These are some notes, and links on my research for verifying a network packet capture workflow, and determining if that was supporting nanosecond timestamps.
My conclusion is that while Sub-Second resolution is certainly possible in network packet capture analysis; and a nanosecond is equal to 1000 picoseconds or 1/1000 of a microsecond, and it should be remembered that this does not guarantee that your source packets are stamped with this accuracy!
This article examines the requirements for this accuracy, and why properly understanding this can lead to more accurate assessments of network performance of applications, whether nanosecond packet captures are taken or not. I caution that assumptions about packet accuracy may cause erroneous conclusions on collected data.
About Time Stamp Accuracy on the Netscout Packet Flow Operating Software (PFOS)
The hardware deserialization and decoding procedures for 10G fiber, 1G fiber, and 1G copper interfaces are not the same, and each type offers a differing static time delay in the input path before time stamping. Therefore, any comparison of time stamps between port types will not be accurate at the nanosecond level. […] Port stamping and time stamping never alter network through traffic. Inline tapping ports always contain an exact replica of the original packet. Only the monitor ports on the system see packets with port and/or time stamps. […] The eight-byte (64-bit) time stamp has this format:
| Bits | Description |
|---|---|
| 0-31 | Time in seconds since Epoch (00:00:00 GMT January 1, 1970) |
| 32-61 | Subsecond time in nanoseconds, in 20nsec increments |
| 62-63 | Time synchronization source: |
| 00: Uncalibrated; internal clock only | |
| 01: NTP synchronization | |
| 10: GPS (1PPS with TSIP) or 1PPS-only synchronization | |
| 11: PTP synchronization |
Measuring Latency in Linux at nanoseconds resolution on Linux
If you are capturing on the OS side, you may want to check if you are able to stamp the capture at a nanosecond resolution:
Both of these problems are solved in more recent CPUs^[confessions of a wall street programmer] : a constant TSC keeps all TSC’s synchronized across all cores in a system, and an invariant (or nonstop) TSC keeps the TSC running at a fixed rate regardless of changes in CPU frequency. To check whether your CPU supports one or both, execute the following and examine the values output in flags:
$ cat /proc/cpuinfo | grep -i tsc
flags : ... tsc rdtscp constant_tsc nonstop_tsc ...
Netscout InfiniStream
InfiniStream appliances with SFP- and XFP-based monitor interfaces support nanosecond time resolution on packet data. You can toggle nanosecond time support on or off, by entering the following command from the local console command line:
set timestamp_ns on/off
Enabling Microsecond Resolution for 10/100/1000 Adapters
You can enable microsecond resolution timestamps for 10/100/1000BASE-T adapters from the Agent Configuration Utility command line.
Microsecond resolution is disabled by default. So use the following command from the Agent Configuration utility command line interface to toggle it on or off:
set ts_resolution on/off
Note: Packet order is not guaranteed across 10/100/1000BASE-T ports.
Riverbed SteelCentral Packet Analyzer
Export and use of pcap data with SteelCentral Packet Analyzer Plus allows either millisecond or nanosecond resolution, these screenshots show where this is configured.
If you have a file that was captured elsewhere, you can confirm the resolution based on the magic number:
What’s the difference between a PCAP file with a magic number of “0x4d3cb2a1” and “0xd4c3b2a1”?
A pcap file with a magic number of 0xa1b23c4d or 0x4d3cb2a1^[stackoverflow.com] is a pcap file in which the packet time stamps are in seconds and nanoseconds. A pcap file with a magic number of 0xa1b2c3d4 or 0xd4c3b2a1 is a “regular” pcap file in which the packet time stamps are in seconds and microseconds.
Do not edit this number to change the resolution. You will only have zeros padding to the end of the millisecond resolution:
Read nanosecond pcap file using libpcap
Additionally, by changing the magic number at the header of nanosec.pcap to that of normal pcap (0x1A2B3C4D) and I got a segmentation fault error from the terminal (Ubuntu). Any expert here could advice how could I display the nanosecond^[Read nanosecond pcap file using libpcap:stackoverflow.com] part of the timestamp by using libpcap?
Once you have your data you may want to analyze it with R:
Convert data frame with epoch timestamps to time-series with milliseconds in R
Your timestamps are in milliseconds^[Convert data frame with epoch timestamps to time-series with milliseconds:stackoverflow.com] . You need to convert them to seconds to be able to use them with as “POSIXct”.
Here we see that timestamps are applied at the capture point:
Time display formats and time references
While packets are captured, each packet is timestamped. These timestamps will be saved to the capture file, so they will be available for later analysis.^[Time display formats and time references:wireshark.org]
Understand how this reflects “time”, the packet is stamped during capture.
Wireshark Time Stamps
While packets are captured, each packet is time stamped as it comes in. These time stamps will be saved to the capture file, so they also will be available for (later) analysis. […] While capturing, Wireshark gets the time stamps from the libpcap (WinPcap) library, which in turn gets them from the operating system kernel. If the capture data is loaded from a capture file, Wireshark obviously gets the data from that file.^[Wireshark Time Stamps:wireshark.org]
Linux Kernel Timestamping
1.3.1 Timestamp Generation
Some bits are requests to the stack to try to generate timestamps. Any combination of them is valid. Changes to these bits apply to newly created packets, not to packets already in the stack. As a result, it is possible to selectively request timestamps for a subset of packets (e.g., for sampling) by embedding an send() call within two setsockopt calls, one to enable timestamp generation and one to disable it. Timestamps may also be generated for reasons other than being requested by a particular socket, such as when receive timestamping is enabled system wide, as explained earlier. ^[Timestamping: www.kernel.org]
Microburst analysis for wireshark
the question is: during this 1 millisecond interval sample, based on the bits that were seen, if we were to have this many bits for 1000 consecutive milliseconds, what would be the bits per second rate. And unfortunately, wireshark doesn’t seem to calculate this for us in IO graph.
However, you can get a very detailed throughput graph when you go to statistics > tcp stream graph > throughput graph. the only problem with this is it only gives you the stats for the tcp conversation you have highlighted.^[Microburst analysis for wireshark:osqa-ask.wireshark.org]
Packet timing measurement
adding a time stamp to each packet in the plurality of packets, wherein time stamps of plurality of packets are generated with data from time stamp engines synchronized to a global clock; sending the plurality of packets with their time stamps to a tool; and performing analysis on the plurality of packets using the tool.^[Packet timing measurement:patents.google.com]
Can ‘wireshark’ truly sense response in nansecond?
Wireshark’s timestamp code uses nanosecond resolution internally and the pcapng and pcap file formats can store nanosecond timestamps. Whether or not you a) get ns timestamps when you capture and b) those timestamps are in any way accurate depends on your hardware and other factors. You probably won’t get accurate nanosecond timestamps from a standard NIC but you definitely can with some high-end NICs and purpose-built hardware such as network taps and capture appliances.^[Can ‘wireshark’ truly sense response in nansecond:stackoverflow.com]
Wireshark Time display formats and time references
While packets are captured, each packet is timestamped. These timestamps will be saved to the capture file, so they will be available for later analysis.^[Wireshark Time display formats:wireshark.org]
Nmap Packet Capture (NPcap)
Npcap is the Nmap Project’s packet sniffing library for Windows. It is based on the wonderful Winpcap / Libpcap libraries, but with improved improved speed, portability, security, and efficiency.^[Nmap Packet Capture (NPcap):wireshark.org]
Reference time from pcap header rather than keyframe
The thing with conversion to pcap format is that you have to make time adjustments based on keyframes to find wall-clock time, and if you calculate latencies based on this resolved timestamp it will introduce jitter. The most accurate way to calculate latencies it to therefore decode raw timestamps directly and only consider the delta between packets.^[Reference time from pcap header rather than keyframe:github.com]
Solution Brief Maximizing Network Visibility
As traffic enters the Gigamon Visibility Fabric nodes, the packets can be accurately time stamped with either microsecond or nanosecond accuracy clocks. Precision time stamping closer to the point of origin provides accurate time-of-transmission of the packet, thereby eliminating latencies and distortions introduced due to buffering on intermediate hardware.^[Solution Brief: www.riverbed.com]
Packet optimization for high-performance networks
As packets flow through the VSS Monitoring vBroker™and Protector™appliances, each packet is time-stamped at the time of capture to create a permanent time record —precise to the nanosecond —that is permanently embedded in the packet […] The time stamp is inserted as an 8-byte data between the payload and the cyclic redundancy check (CRC) value at the end of the packet.^[Solution Brief: www.riverbed.com]
Wireshark Documentation
7.6. Time Stamps Time stamps, their precisions and all that can be quite confusing. This section will provide you with information about what’s going on while Wireshark processes time stamps.
While packets are captured, each packet is time stamped as it comes in. These time stamps will be saved to the capture file, so they also will be available for (later) analysis.
So where do these time stamps come from? While capturing, Wireshark gets the time stamps from the libpcap (WinPcap) library, which in turn gets them from the operating system kernel. If the capture data is loaded from a capture file, Wireshark obviously gets the data from that file.
7.6.1. Wireshark internals
The internal format that Wireshark uses to keep a packet time stamp consists of the date (in days since 1.1.1970) and the time of day (in nanoseconds since midnight). You can adjust the way Wireshark displays the time stamp data in the packet list, see the “Time Display Format” item in the Section 3.7, “The “View” Menu” for details.
While reading or writing capture files, Wireshark converts the time stamp data between the capture file format and the internal format as required.
While capturing, Wireshark uses the libpcap (WinPcap) capture library which supports microsecond resolution. Unless you are working with specialized capturing hardware, this resolution should be adequate.
IP Packet Delay Variation Metric for IP Performance Metrics (IPPM)
This document refers to a metric for variation in delay of packets across Internet paths. The metric is based on the difference in the One-Way-Delay of selected packets. This difference in delay is called “IP Packet Delay Variation (ipdv)”.^[IP Packet Delay Variation Metric for IP Performance Metrics (IPPM):ietf.org]
Timestamp-resolution Problem of Traffic Analysis on High Speed Networks
Tcpdump and Wireshark are commonly used tools for analyzing network traffic between two communicating endpoints over the internet. Capture capabilities of these tools are based on the libpcap library. This library’s 1.0.x version only supports 10−6 second native timestamp resolution, however on Gigabit Ethernet and faster network speeds nanosecond resolution would be preferred.
Gigamon’s time stamp technology together with Riverbed Shark performance analytics, further reduce mean time to resolution (MTTR) by synchronizing time stamps across the network for performance troubleshooting and enable network managers to quickly determine where in the network problems are occurring. […] Gigamon’s time stamping technology for the GigaVUE H Series, including the GigaPORT-X12-TS, offers hardware-based nanosecond accurate time stamping enabling IT organizations to optimize networks for all types of traffic, including high frequency trading communications. Gigamon’s de-duplication removes duplicate packets - a result of port aggregation where the same packet is picked up at multiple places along its route - before they are unnecessarily processed or stored by tools for performance and accuracy enhancement.[Riverbed and Gigamon Partner to Enhance Network and Application Performance Management: www.riverbed.com]
SteelCentral Packet Analyzer Plus
Supports both pcap and pcap-ng files(default format) for Wireshark nanosecond granularity.^[SteelCentral Packet Analyzer Plus: www.riverbed.com]
SPAN Port Vs. TAP: The Latency Impact
The test results show that a SPAN port created more latency between the packets and additional latency when a packet was received. In conclusion, you should be aware of your SPAN port limitations and where TAPs might complement your toolbox.^[SPAN Port Vs. TAP: The Latency Impact:networkcomputing.com]
Understanding Queuing With Hierarchical Queueing Framework
The public pools of the mid-range routing series offers two possible locations to store packets: the Fast Switching buffers (F/S) and the normal pool buffers (depending on the IOS version, these buffers are used as the backup pool when the private interface pool is filled with packets). These buffers vary in size and access speed.^[Understanding Queuing With Hierarchical Queueing Framework: www.cisco.com]
Catalyst 3850: Troubleshooting Output drops
Output drops are generally a result of interface oversubscription caused by many to one or a 10gig to 1gig transfer. Interface buffers are a limited resource and can only absorb a burst up to a point after which packets will drop. Tuning the buffers can give you some cushion but it cannot guarantee a zero output drop scenario.^[Troubleshooting Output drops: www.cisco.com]
Buffer Tuning for all Cisco Routers
Buffer tuning allows you to modify the way a router allocates buffers from its available memory, and helps prevent packet drops during a temporary burst of traffic.
To determine whether your router needs to have its buffer tuned, use the show interfaces and show buffers commands.^[Buffer Tuning for all Cisco Routers: www.cisco.com]
Understanding Buffer Misses and Failures
[…] if an interface processor needs to pass a 20 byte packet to the RP, it “asks” for a Small buffer. If an interface processor needs to pass a 500 byte packet to the RP, it asks for a Middle buffer, and so forth. […] When IOS fails to get a Small buffer, it does not drop the packet. It increments the failed counter and falls through to the next level buffer, which is the Middle buffer and requests a buffer there. If it fails to get a Middle buffer, it requests the next level buffer, which is a Big buffer. This process continues until it hits the Huge buffer pool. If it fails to get a Huge buffer, then it drops the packet.^[Understanding Buffer Misses: www.cisco.com]
TCP Pacing Revisited
We conclude that TCP pacing brings signicant benefits for many applications, and though paced flows sometimes suffer in performance when competing with non-paced flows, there are enough incentives for applications to migrate from using nonpaced TCP to paced TCP.^[TCP Pacing Revisited:citeseerx.ist.psu.edu]