Operator PacketDPDKSource

Primitive operator image not displayed. Problem loading file: ../../image/tk$com.teracloud.streams.network/op$com.teracloud.streams.network.source$PacketDPDKSource.svg

PacketDPDKSource is an operator for the Streams product that receives network packets from a supported ethernet adapter. The operator function and structure are similar to PacketLiveSource; see that operator's documentation for details of common functions and general background. The primary difference between these related operators is that PacketDPDKSource leverages the DPDK library for higher performance.

DPDK supports a variety of ethernet adapters and their drivers. The PacketDPDKSource operator has been tested with these adapters:

  • Mellanox ConnectX-3
  • Mellanox ConnectX-3 Pro
  • Intel e1000
  • Intel I350

DPDK stores network packets received by a supported ethernet adapter in memory that is directly accessible by the PacketDPDKSource operator, bypassing the Linux kernel. This reduces overhead and allows the PacketDPDKSource operator to ingest packets at a higher rate than the PacketLiveSource operator. The PacketDPDKSource operator requires a dedicated processor core, which must be specified explicitly, and which runs at 100% CPU utilization.

Output filters and attribute assignments are SPL expressions. They may use any of the built-in SPL functions and any of these functions:

The PacketDPDKSource operator can be configured to step quietly over 'jmirror' headers prepended to packets by Juniper Networks 'mirror encapsulation'.

DPDK setup

See Requirements for DPDK operators for system setup and preparation for building applications using DPDK operators.

Required Linux capabilities

This operator requires special privileges to receive 'raw' network packets from an ethernet interface.

The required Linux capabilities are:
  • CAP_DAC_READ_SEARCH+eip
  • CAP_IPC_LOCK+eip
  • CAP_SYS_ADMIN+eip
  • CAP_SYS_RAWIO+eip

See Running applications with operators requiring Linux capabilities for instructions to prepare the application or runtime environment.

Parallelization

DPDK can distribute network packets received by one ethernet adapter across multiple PacketDPDKSource operators. This can increase an application's packet rate by parallelizing its analytics. Note that each PacketDPDKSource operator requires a separate processor core, which runs at 100% CPU utiliziation.

DPDK requires that the total number of PacketDPDKSource operators be a power of 2, that is, the total number of PacketDPDKSource operators must be 1, 2, 4, 8, or 16. Packets received by the network adapter will be distributed across all of the operators, which will run in parallel. Packets are distributed according to a hash of their source and destination addresses and ports, so request and response packets that belong to the same TCP transaction are all sent to the same operator.

Promiscuous mode

Ethernet adapters normally ignore packets that are not addressed to them. However, when 'promiscious' mode is enabled, and adapter can capture all network packets on its ethernet segment, even those that are not addressed to it. This is sometimes referred to as "network sniffing". However, ethernet switches normally send only packets that are addressed to the adapter; 'promiscious' mode is useful only when a switch has been specifically configured to send packets that are not addressed to the adapter. This is sometimes referred to as "mirroring".

The PacketDPDKSource operator will enable 'promiscuous' mode when its promiscous parameter is set to true.

Example 


use com.teracloud.streams.network.source::*;
composite Main {
  type 
    PacketType =
      float64 captureTime,              // time that packet was captured, in seconds since Unix epoch
      uint64 packetNumber,              // sequence number of packet, as emitted by operator
      uint8 ipVersion,                  // IP version: 4 for IPv4, 6 for IPv6
      uint8 ipProtocol,                 // IP protocol: 0x01==1 for ICMP, 0x6==6 for TCP, 0x11==17 for UDP
      uint32 sourceAddress,             // IPv4 source address
      uint16 sourcePort,                // IP source port, or zero if not UDP or TCP packet
      uint32 destinationAddress,        // IPv4 destination address
      uint16 destinationPort,           // IP destination port, or zero if not UDP or TCP packet
      uint32 payloadLength,             // length of packet payload, excluding network headers
      uint32 packetLength,              // length of packet, including netwok headers
      blob packetData;                  // contents of packet captured, including network headers
    
    MetricType =
      float64 metricsTime,              // when metrics were recorded, in seconds since Unix epoch
      float64 averagePacketSize,        // average size of packets, including network headers
      float64 packetsProcessedPerSecond,// packets processed per second during last metrics interval
      float64 bytesProcessedPerSecond,  // bytes proceessed per second during last metrics interval
      float64 packetsDroppedPerSecond;  // packets dropped per second during last metrics interval

  graph

    ( stream<PacketType> PacketStream as PacketOut ;
      stream<MetricType> MetricStream as MetricOut )
      = PacketDPDKSource() {
        param
            nicPort: (int32)getSubmissionTimeValue("adapterPort", "0" );
            nicQueue: 0;
            lcoreMaster: (int32)getSubmissionTimeValue("dpdkMasterCore", "0" );
            lcore: (int32)getSubmissionTimeValue("dpdkIngestCore", "1" );
            buffersizes: getSubmissionTimeValue("dpdkBufferSizes", "100");
            promiscuous: true;
            metricsInterval: 1.0;
            outputFilters: true,              // output port 0 emits all packets processed
                           metricsUpdated();  // output port 1 emits operator metrics
                          
        output
          PacketOut:
              captureTime = (float64)CAPTURE_SECONDS() + (float64)CAPTURE_MICROSECONDS() / 1000000.0,
              packetNumber = packetsProcessed() - 1ul,
              ipVersion = IP_VERSION(),
              ipProtocol = IP_PROTOCOL(),
              sourceAddress = IPV4_SRC_ADDRESS(),
              destinationAddress = IPV4_DST_ADDRESS(),
              sourcePort = IP_SRC_PORT(),
              destinationPort = IP_DST_PORT(),
              payloadLength = PAYLOAD_LENGTH(),
              packetLength = PACKET_LENGTH(),
              packetData = PACKET_DATA();
          MetricOut:
              metricsTime = getTimestampInSecs(),
              averagePacketSize = (float64)metricsIntervalBytesProcessed() / (float64)metricsIntervalPacketsProcessed(),
              packetsProcessedPerSecond = (float64)metricsIntervalPacketsProcessed() / metricsIntervalElapsed(),
              bytesProcessedPerSecond = (float64)metricsIntervalBytesProcessed() / metricsIntervalElapsed(),
              packetsDroppedPerSecond = (float64)metricsIntervalPacketsDropped() / metricsIntervalElapsed();
    }
}

Summary

Ports
This operator has 0 input ports and 0 or more output ports.
Windowing
This operator does not accept any windowing configurations.
Parameters
This operator supports 11 parameters.

Required: lcore

Optional: buffersizes, dequeueThreadPinning, jMirrorCheck, lcoreMaster, metricsInterval, nicPort, nicQueue, outputFilters, promiscuous, rateLimit

Metrics
This operator reports 7 metrics.

Properties

Implementation
C++
Capabilities
CAP_DAC_READ_SEARCH+ep
CAP_IPC_LOCK+ep
CAP_SYS_ADMIN+ep
CAP_SYS_RAWIO+ep
Threading
Never - Operator never provides a single threaded execution context.

Output Ports

Assignments
This operator allows any SPL expression of the correct type to be assigned to output attributes.
Ports (0...)

The PacketDPDKSource operator requires one or more output ports:

Each output port will produce one output tuple for each packet received if the corresponding expression in the outputFilters parameter evaluates true, or if no outputFilters parameter is specified.

Output attributes can be assigned values with any SPL expression that evaluates to the proper type, and the expressions may include any of the PacketDPDKSource result functions. Output attributes that match input attributes in name and type are copied automatically.

Properties

Parameters

This operator supports 11 parameters.

Required: lcore

Optional: buffersizes, dequeueThreadPinning, jMirrorCheck, lcoreMaster, metricsInterval, nicPort, nicQueue, outputFilters, promiscuous, rateLimit

buffersizes

This optional parameter specifies how much buffer space, in megabytes, to allocate from available 2MB Linux 'hugepages' for each NUMA node, specified as a string containing comma-separated values. For the NUMA node the ethernet adapter is connected to, specify an even number of megabytes. For other NUMA nodes, if any, specify zero.

For example, on a machine with one NUMA node, specify "100" to allocate 100 megabytes of buffer space. For another example, on a machine with two NUMA nodes, where the ethernet adapter is connected to the second node, specify "0,100" to allocate 100 megabytes of buffer space on the second NUMA node. If multiple PacketDPDKSource operators are configured for the same network adapter, all operators should specify the same values for their buffersizes parameter.

To determine which NUMA node your ethernet adapter is connected to, enter the command cat /sys/class/net/$DPDK_INTERFACE/device/numa_node at a Linux prompt. To determine how much buffer space is available in the 'hugepages' for that NUMA node, enter the command grep -i -h hugepages /sys/devices/system/node/node*/meminfo at a Linux prompt.

If this parameter is not specified, DPDK will allocate all available 2MB Linux 'hugepages' for its buffer space.

Properties

dequeueThreadPinning

This optional parameter specifies a set of logical processor cores to pin the internal ring buffer dequeue thread to. This thread will run at 100% cpu utilization as it spins on the ring buffer tail.

By default, this thread runs unpinned.

Properties

jMirrorCheck

This optional parameter takes an expression of type 'boolean' that specifies whether or not the operator should check for Juniper Networks 'jMirror' headers on packets, and step over them when found.

The default value is 'false'.

Properties

lcore

This required parameter specifies a logical processor core for the DPDK packet receive thread. The core specified must belong to the NUMA node the ethernet adapter is connected to. The receive thread will run the specified core at 100% CPU utilization.

If multiple PacketDPDKSource operators are configured for the same ethernet adapter, each operator must specify a different value with its lcoreMaster parameter. The total number of PacketDPDKSource operators must be a power of 2, that is, the total number of PacketDPDKSource operators must be 1, 2, 4, 8, or 16. Packets received by the network adapter will be distributed across all of the operators, which will run in parallel.

To determine how many processor cores your machine has, and which cores are connected to which NUMA nodes, execute the lscpu command at a Linux prompt. To determine which NUMA node your ethernet adapter is connected to, look in the file '/sys/class/net/$DPDK_INTERFACE/device/numa_node'.

Properties

lcoreMaster

This optional parameter specifies a logical processor core for the DPDK master thread. This thread is not used for receiving packets, and does not have high CPU utiliztion. If the machine has more than one NUMA node, specify a core on a node that does not have the ethernet adapter connected to it.

When multiple PacketDPDKSource operators are configured for the same ethernet adapter, all operators must specify the same value with their lcoreMaster parameters.

To determine how many processor cores your machine has, and which cores are connected to which NUMA nodes, execute the lscpu command at a Linux prompt.

Properties

metricsInterval

This optional parameter takes an expression of type float64 that specifies the interval, in seconds, for sending operator metrics to the Streams runtime. If the value is zero or less, the operator will not report metrics to the runtime, and the output assigment functions for libpcap statistics will be zero.

The default value is '10.0'.

Properties

nicPort

This optional parameter specifies which port of the ethernet adapter the operator will receive network packets from. This applies only to adapters with multiple ports.

The default value is '0' if this parameter is not specified.

Properties

nicQueue

This optional parameter specifies which queue of received packets the operator will receive. This applies only when multiple PacketDPDKSource operators are configured for the same ethernet adapter. The value must be between 0 and 'N-1', where N is the total number of operators, and each operator must specify a different queue, so that all queues are connected to an operator.

Note that DPDK requires the total number of queues and operators be a power of 2, that is, the total number of queues and operators must be 1, 2, 4, 8, or 16. DPDK will distribute packets across all of the queues, and therefore across all of the operators.

The default value is '0' if this parameter is not specified.

Properties

outputFilters

This optional parameter takes a list of SPL expressions that specify which packets should be emitted by the corresponding output port. The number of expressions in the list must match the number of output ports, and each expression must evaluate to a boolean value. The output filter expressions may include any of the PacketDPDKSource result functions.

The default value of the outputFilters parameter is an empty list, which causes all packets processed to be emitted by all output ports.

Properties

promiscuous

This optional parameter takes an expression of type 'boolean' that specifies whether or not 'promiscuous' mode should be enabled on the network interface.

The default value is 'false'.

Properties

rateLimit

This optional parameter takes an expression of type 'float64' that specifies the maximum number of times per second the RATE_LIMITED() function returns false. This can be used to limit the rate of packets sent to an output port when used in a filter expression.

The default value is '1000.0'.

Properties

Code Templates

PacketDPDKSource

stream<${outputScheme}> ${outputStream} = com.teracloud.streams.network.source::PacketDPDKSource() {
  param
      lcore: ${logicalCpuCore};
      ${parameter}: ${parameterExpression}
  output
      ${outputStream}: ${attributeAssignments}
}

Metrics

maxQueueDepthSWCurrent - Counter

This metric reports the high water mark of the first software queue.

nBytesProcessedCurrent - Counter

This metric counts number of bytes of packet data processed by the operator.

nBytesReceivedCurrent - Counter

This metric counts the number of bytes received by the ethernet adapter, as counted by DPDK.

nPacketsDroppedCurrent - Counter

This metric counts the number of packets dropped by the ethernet adapter, as counted by DPDK.

nPacketsDroppedSWCurrent - Counter

This metric counts number of packets dropped by the initial software queue.

nPacketsProcessedCurrent - Counter

This metric counts number of packets processed by the operator.

nPacketsReceivedCurrent - Counter

This metric counts the number of packets received by the ethernet adapter, as counted by DPDK.

Libraries

Functions for parsing network headers.
Include Path: ../../impl/include
Streams DPDK 'glue' library 'libstreams_source.so'.
Library Name: streams_source
Library Path: ../../impl/src/source/dpdk/build/lib
Include Path: ../../impl/src/source/dpdk/streams_source