These are some notes, and links on my research for verifying a network packet capture workflow, and determining if that was supporting nanosecond timestamps.
My conclusion is that while Sub-Second resolution is possible in network packet capture analysis; and a nanosecond is equal to 1000 picoseconds or 1⁄1000 of a microsecond, and it should be remembered that this does not guarantee that your source packets are stamped with this accuracy. This article examines the requirements for this accuracy, and why properly understanding this can lead to more accurate assessments of network performance of applications, whether nanosecond packet captures are taken or not.
Riverbed SteelCentral Packet Analyzer
Export and use of pcap data with SteelCentral Packet Analyzer Plus allows either millisecond or nanosecond resolution, these screenshots show where this is configured.
If you have a file that was captured elsewhere, you can confirm the resolution based on the magic number:
What’s the difference between a PCAP file with a magic number of “0x4d3cb2a1” and “0xd4c3b2a1”?
A pcap file with a magic number of 0xa1b23c4d or 0x4d3cb2a11 is a pcap file in which the packet time stamps are in seconds and nanoseconds. A pcap file with a magic number of 0xa1b2c3d4 or 0xd4c3b2a1 is a “regular” pcap file in which the packet time stamps are in seconds and microseconds.
Do not edit this number to change the resolution. You will only have zeros padding to the end of the millisecond resolution:
Read nanosecond pcap file using libpcap
Additionally, by changing the magic number at the header of nanosec.pcap to that of normal pcap (0x1A2B3C4D) and I got a segmentation fault error from the terminal (Ubuntu). Any expert here could advice how could I display the nanosecond2 part of the timestamp by using libpcap?
Once you have your data you may want to analyze it with R:
Convert data frame with epoch timestamps to time-series with milliseconds in R
Your timestamps are in milliseconds3 . You need to convert them to seconds to be able to use them with as “POSIXct”.
Here we see that timestamps are applied at the capture point:
Time display formats and time references
While packets are captured, each packet is timestamped. These timestamps will be saved to the capture file, so they will be available for later analysis.4
Understand how this reflects “time”, the packet is stamped during capture.
Wireshark Time Stamps
While packets are captured, each packet is time stamped as it comes in. These time stamps will be saved to the capture file, so they also will be available for (later) analysis. […] While capturing, Wireshark gets the time stamps from the libpcap (WinPcap) library, which in turn gets them from the operating system kernel. If the capture data is loaded from a capture file, Wireshark obviously gets the data from that file.5
Linux Kernel Timestamping
1.3.1 Timestamp Generation
Some bits are requests to the stack to try to generate timestamps. Any combination of them is valid. Changes to these bits apply to newly created packets, not to packets already in the stack. As a result, it is possible to selectively request timestamps for a subset of packets (e.g., for sampling) by embedding an send() call within two setsockopt calls, one to enable timestamp generation and one to disable it. Timestamps may also be generated for reasons other than being requested by a particular socket, such as when receive timestamping is enabled system wide, as explained earlier. 6
Microburst analysis for wireshark
the question is: during this 1 millisecond interval sample, based on the bits that were seen, if we were to have this many bits for 1000 consecutive milliseconds, what would be the bits per second rate. And unfortunately, wireshark doesn’t seem to calculate this for us in IO graph.
However, you can get a very detailed throughput graph when you go to statistics > tcp stream graph > throughput graph. the only problem with this is it only gives you the stats for the tcp conversation you have highlighted.7
Packet timing measurement
adding a time stamp to each packet in the plurality of packets, wherein time stamps of plurality of packets are generated with data from time stamp engines synchronized to a global clock; sending the plurality of packets with their time stamps to a tool; and performing analysis on the plurality of packets using the tool.8
Can ‘wireshark’ truly sense response in nansecond?
Wireshark’s timestamp code uses nanosecond resolution internally and the pcapng and pcap file formats can store nanosecond timestamps. Whether or not you a) get ns timestamps when you capture and b) those timestamps are in any way accurate depends on your hardware and other factors. You probably won’t get accurate nanosecond timestamps from a standard NIC but you definitely can with some high-end NICs and purpose-built hardware such as network taps and capture appliances.9
Wireshark Time display formats and time references
While packets are captured, each packet is timestamped. These timestamps will be saved to the capture file, so they will be available for later analysis.10
Nmap Packet Capture (NPcap)
Npcap is the Nmap Project’s packet sniffing library for Windows. It is based on the wonderful Winpcap / Libpcap libraries, but with improved improved speed, portability, security, and efficiency.11
Reference time from pcap header rather than keyframe
The thing with conversion to pcap format is that you have to make time adjustments based on keyframes to find wall-clock time, and if you calculate latencies based on this resolved timestamp it will introduce jitter. The most accurate way to calculate latencies it to therefore decode raw timestamps directly and only consider the delta between packets.12
Solution Brief Maximizing Network Visibility
As traffic enters the Gigamon Visibility Fabric nodes, the packets can be accurately time stamped with either microsecond or nanosecond accuracy clocks. Precision time stamping closer to the point of origin provides accurate time-of-transmission of the packet, thereby eliminating latencies and distortions introduced due to buffering on intermediate hardware.13
Packet optimization for high-performance networks
As packets flow through the VSS Monitoring vBroker™and Protector™appliances, each packet is time-stamped at the time of capture to create a permanent time record —precise to the nanosecond —that is permanently embedded in the packet […] The time stamp is inserted as an 8-byte data between the payload and the cyclic redundancy check (CRC) value at the end of the packet.14
7.6. Time Stamps Time stamps, their precisions and all that can be quite confusing. This section will provide you with information about what’s going on while Wireshark processes time stamps.
While packets are captured, each packet is time stamped as it comes in. These time stamps will be saved to the capture file, so they also will be available for (later) analysis.
So where do these time stamps come from? While capturing, Wireshark gets the time stamps from the libpcap (WinPcap) library, which in turn gets them from the operating system kernel. If the capture data is loaded from a capture file, Wireshark obviously gets the data from that file.
7.6.1. Wireshark internals
The internal format that Wireshark uses to keep a packet time stamp consists of the date (in days since 1.1.1970) and the time of day (in nanoseconds since midnight). You can adjust the way Wireshark displays the time stamp data in the packet list, see the “Time Display Format” item in the Section 3.7, “The “View” Menu” for details.
While reading or writing capture files, Wireshark converts the time stamp data between the capture file format and the internal format as required.
While capturing, Wireshark uses the libpcap (WinPcap) capture library which supports microsecond resolution. Unless you are working with specialized capturing hardware, this resolution should be adequate.
IP Packet Delay Variation Metric for IP Performance Metrics (IPPM)
This document refers to a metric for variation in delay of packets across Internet paths. The metric is based on the difference in the One-Way-Delay of selected packets. This difference in delay is called “IP Packet Delay Variation (ipdv)“.15
Timestamp-resolution Problem of Traffic Analysis on High Speed Networks
Tcpdump and Wireshark are commonly used tools for analyzing network traffic between two communicating endpoints over the internet. Capture capabilities of these tools are based on the libpcap library. This library’s 1.0.x version only supports 10−6 second native timestamp resolution, however on Gigabit Ethernet and faster network speeds nanosecond resolution would be preferred.
Gigamon’s time stamp technology together with Riverbed Shark performance analytics, further reduce mean time to resolution (MTTR) by synchronizing time stamps across the network for performance troubleshooting and enable network managers to quickly determine where in the network problems are occurring. […] Gigamon’s time stamping technology for the GigaVUE H Series, including the GigaPORT-X12-TS, offers hardware-based nanosecond accurate time stamping enabling IT organizations to optimize networks for all types of traffic, including high frequency trading communications. Gigamon’s de-duplication removes duplicate packets - a result of port aggregation where the same packet is picked up at multiple places along its route - before they are unnecessarily processed or stored by tools for performance and accuracy enhancement.[Riverbed and Gigamon Partner to Enhance Network and Application Performance Management: www.riverbed.com]
SteelCentral Packet Analyzer Plus
Supports both pcap and pcap-ng files(default format) for Wireshark nanosecond granularity.16
SPAN Port Vs. TAP: The Latency Impact
The test results show that a SPAN port created more latency between the packets and additional latency when a packet was received. In conclusion, you should be aware of your SPAN port limitations and where TAPs might complement your toolbox.17
Understanding Queuing With Hierarchical Queueing Framework
The public pools of the mid-range routing series offers two possible locations to store packets: the Fast Switching buffers (F/S) and the normal pool buffers (depending on the IOS version, these buffers are used as the backup pool when the private interface pool is filled with packets). These buffers vary in size and access speed.18
Catalyst 3850: Troubleshooting Output drops
Output drops are generally a result of interface oversubscription caused by many to one or a 10gig to 1gig transfer. Interface buffers are a limited resource and can only absorb a burst up to a point after which packets will drop. Tuning the buffers can give you some cushion but it cannot guarantee a zero output drop scenario.19
Buffer Tuning for all Cisco Routers
Buffer tuning allows you to modify the way a router allocates buffers from its available memory, and helps prevent packet drops during a temporary burst of traffic.
To determine whether your router needs to have its buffer tuned, use the show interfaces and show buffers commands.20
Understanding Buffer Misses and Failures
[…] if an interface processor needs to pass a 20 byte packet to the RP, it “asks” for a Small buffer. If an interface processor needs to pass a 500 byte packet to the RP, it asks for a Middle buffer, and so forth. […] When IOS fails to get a Small buffer, it does not drop the packet. It increments the failed counter and falls through to the next level buffer, which is the Middle buffer and requests a buffer there. If it fails to get a Middle buffer, it requests the next level buffer, which is a Big buffer. This process continues until it hits the Huge buffer pool. If it fails to get a Huge buffer, then it drops the packet.21
TCP Pacing Revisited
We conclude that TCP pacing brings signicant benefits for many applications, and though paced flows sometimes suffer in performance when competing with non-paced flows, there are enough incentives for applications to migrate from using nonpaced TCP to paced TCP.22
- stackoverflow.com [return]
- Read nanosecond pcap file using libpcap:stackoverflow.com [return]
- Convert data frame with epoch timestamps to time-series with milliseconds:stackoverflow.com [return]
- Time display formats and time references:wireshark.org [return]
- Wireshark Time Stamps:wireshark.org [return]
- Timestamping: www.kernel.org [return]
- Microburst analysis for wireshark:osqa-ask.wireshark.org [return]
- Packet timing measurement:patents.google.com [return]
- Can ‘wireshark’ truly sense response in nansecond:stackoverflow.com [return]
- Wireshark Time display formats:wireshark.org [return]
- Nmap Packet Capture (NPcap):wireshark.org [return]
- Reference time from pcap header rather than keyframe:github.com [return]
- Solution Brief: www.riverbed.com [return]
- Solution Brief: www.riverbed.com [return]
- IP Packet Delay Variation Metric for IP Performance Metrics (IPPM):ietf.org [return]
- SteelCentral Packet Analyzer Plus: www.riverbed.com [return]
- SPAN Port Vs. TAP: The Latency Impact:networkcomputing.com [return]
- Understanding Queuing With Hierarchical Queueing Framework: www.cisco.com [return]
- Troubleshooting Output drops: www.cisco.com [return]
- Buffer Tuning for all Cisco Routers: www.cisco.com [return]
- Understanding Buffer Misses: www.cisco.com [return]
- TCP Pacing Revisited:citeseerx.ist.psu.edu [return]