Operator PacketDPDKSource

Primitive operator image not displayed. Problem loading file: ../../image/tk$com.teracloud.streams.network/op$com.teracloud.streams.network.source$PacketDPDKSource.svg

PacketDPDKSource is an operator for the Streams product that receives network packets from a supported ethernet adapter. The operator function and structure are similar to PacketLiveSource; see that operator's documentation for details of common functions and general background. The primary difference between these related operators is that PacketDPDKSource leverages the DPDK library for higher performance.

DPDK supports a variety of ethernet adapters. The PacketDPDKSource operator has been tested with these adapters:

  • Mellanox ConnectX-3
  • Mellanox ConnectX-3 Pro
  • Intel e1000
  • Intel I350

DPDK stores network packets received by a supported ethernet adapter in memory that is directly accessible by the PacketDPDKSource operator, bypassing the Linux kernel. This reduces overhead and allows the PacketDPDKSource operator to ingest packets at a higher rate than the PacketLiveSource operator. The PacketDPDKSource operator requires a dedicated processor core, which must be specified explicitly, and which runs at 100% CPU utilization.

Output filters and attribute assignments are SPL expressions. They may use any of the built-in SPL functions and any of these functions:

The PacketDPDKSource operator can be configured to step quietly over 'jmirror' headers prepended to packets by Juniper Networks 'mirror encapsulation'.

This operator is part of the network toolkit. To use it in an application, include this statement in the SPL source file: 


use com.teracloud.streams.network.source::*;

parallelization

DPDK can distribute network packets received by one ethernet adapter across multiple PacketDPDKSource operators. This can increase an application's packet rate by parallelizing its analytics. Note that each PacketDPDKSource operator requires a separate processor core, which runs at 100% CPU utiliziation.

DPDK requires that the total number of PacketDPDKSource operators be a power of 2, that is, the total number of PacketDPDKSource operators must be 1, 2, 4, 8, or 16. Packets received by the network adapter will be distributed across all of the operators, which will run in parallel. Packets are distributed according to a hash of their source and destination addresses and ports, so request and response packets that belong to the same TCP transaction are all sent to the same operator.

promiscuous mode

Ethernet adapters normally ignore packets that are not addressed to them. However, when 'promiscious' mode is enabled, and adapter can capture all network packets on its ethernet segment, even those that are not addressed to it. This is sometimes referred to as "network sniffing". However, ethernet switches normally send only packets that are addressed to the adapter; 'promiscious' mode is useful only when a switch has been specifically configured to send packets that are not addressed to the adapter. This is sometimes referred to as "mirroring".

The PacketDPDKSource operator will enable 'promiscuous' mode when its 'promiscous' parameter is set to true.

setup for DPDK

The DPDK library requires some setup that is specific to the hardware and software configuration of your machine. The notes below offer general guidelines for this setup, and examples for an Intel processor with a Mellanox ethernet adapter. You will need to adapt the guidelines to match your configuration.

You will need 'root' privileges for some steps below. They are indicated by the sudo command in the examples.

Linux dependencies

DPDK depends upon additional Linux software packages, beyond those required for Streams. In particular, it requires libraries that match the specific version of the Linux kernel. To install these libraries, type these commands at a prompt: 


sudo yum install pciutils
sudo yum install glibc-devel
sudo yum install libibverbs-devel
sudo yum install "kernel-devel-$( uname -r )"

If you will be using a Mellanox ethernet adapter, you will also need to download and install OFED drivers from Mellanox.

gathering system information for DPDK

To configure the PacketDPDKSource operator, you will need to know how many processors the machine has, and how they are distributed amoung the machine's NUMA nodes. Use this command to display that information: 


lscpu

You will also need the PCI bus address of the adapter, the Linux interface names of the adapter's ports, and the NUMA node number the adapter is connected to.

For example, ind the PCI bus address of a Mellanox ethernet adapter with this command: 


/usr/sbin/lspci | grep -i mellanox | gawk '{print $1}'

To determine the Linux interface names of the adapter's ports, enter this command at a prompt: 


ls -l /sys/class/net | grep $DPDK_DEVICE | gawk '{print $9}'

and then, to determine the NUMA node number the adapter is connected to, enter this command at a prompt: 


cat /sys/class/net/$DPDK_INTERFACE/device/numa_node

configuring a Linux user group for DPDK

You will need a Linux user group for DPDK. To create the user group and add your own user account to it, enter these commands at a prompt: 


sudo groupadd dpdk
sudo usermod -a -G dpdk $USER
sudo cat "@dpdk - memlock unlimited" >/etc/security/limits.conf

You will need to log off and log back on for this to take effect. To verify that your user account is a member of the group, enter this command at a prompt: 


id

The PacketDPDKSource operator will need read/write permission for the Linux interface for the adapter's port. To grant that permission to all user accounts that are members of the 'dpdk' user group, enter these commands at a prompt: 


sudo chown root:dpdk /sys/class/infiniband/mlx4_0/device/net/$DPDK_INTERFACE/flags
sudo chmod 666 /sys/class/infiniband/mlx4_0/device/net/$DPDK_INTERFACE/flags

configuring Linux 'hugepages' for DPDK

DPDK uses Linux 'hugepages' for its buffers. These buffers must be allocated in the memory connected to the same NUMA node as the ethernet adapter, and may use either 2MB or 1GB 'hugepages'. You must create mount points for the 'hugepages' in the Linux file system and grant DPDK read/write permissions for them.

For example, to allocate 500 2MB 'hugepages' for DPDK buffers, enter these commands at a prompt: 


sudo mkdir /dev/hugepages-2MB 
sudo mount -t hugetlbfs nodev /dev/hugepages-2MB -o pagesize=2MB
sudo chmod 770 /dev/hugepages-2MB
sudo chown root:dpdk /dev/hugepages-2MB
sudo /usr/bin/bash -c "echo 500 > /sys/devices/system/node/node$DPDK_NODE/hugepages/hugepages-2048kB/nr_hugepages"

For another example, to allocate one 1GB 'hugepages' for DPDK buffers, enter these commands at a prompt: 


sudo mkdir /dev/hugepages-1GB 
sudo mount -t hugetlbfs nodev /dev/hugepages-1GB -o pagesize=1GB 
sudo chmod 770 /dev/hugepages-1GB
sudo chown root:dpdk /dev/hugepages-1GB
sudo /usr/bin/bash -c "echo 1 > /sys/devices/system/node/node$DPDK_NODE/hugepages/hugepages-1048576kB/nr_hugepages"

To verify that 'hugepages' have been allocated, enter this command at a prompt: 


grep -i -h hugepages /sys/devices/system/node/node*/meminfo

setting environment variables for DPDK

DPDK requires several environment variables. To set them, enter these commands at a prompt, or append them to your $HOME/.bashrc script: 


export RTE_SDK=$HOME/dpdk-17.11.3
[[ $( uname -m ) == "x86_64" ]] && export RTE_TARGET=x86_64-native-linuxapp-gcc
[[ $( uname -m ) == "ppc64le" ]] && export RTE_TARGET=ppc_64-power8-linuxapp-gcc
[[ -x /usr/sbin/lspci ]] && export DPDK_DEVICE=$( /usr/sbin/lspci | grep -i mellanox | gawk '{print $1}' )
[[ -n $DPDK_DEVICE ]] && export DPDK_INTERFACES=( $( ls -l /sys/class/net | grep $DPDK_DEVICE | gawk '{print $9}' ) )
[[ -n $DPDK_INTERFACES ]] && export DPDK_INTERFACE=${DPDK_INTERFACES[0]}

After adding these commands to your $HOME/.bashrc file, you will need to log off and log back on for them to take effect. To verify that the environment variables are set, enter this command at a prompt: 


export | grep -E 'RTE_|DPDK_'

installing the DPDK library

The PacketDPDKSource operator depends upon DPDK library version 17.11.3, available from http://dpdk.org. The library must be configured for your machine and ethernet adapter, and compiled from source code.

To get the DPDK source code, enter these lines at a command prompt: 


mkdir -p $RTE_SDK
wget http://fast.dpdk.org/rel/dpdk-17.11.3.tar.xz
tar -x -v -f dpdk-17.11.3.tar.xz -C $RTE_SDK --strip-components=1

To configure DPDK for your machine, enter these lines at a command prompt: 


cd $RTE_SDK
make config T=$RTE_TARGET O=$RTE_TARGET

To configure DPDK for a Mellonox ethernet adapter, enter these lines at a command prompt: 


cd $RTE_SDK/$RTE_TARGET
sed -i.bak s/CONFIG_RTE_BUILD_SHARED_LIB=./CONFIG_RTE_BUILD_SHARED_LIB=n/ ./.config
sed -i.bak s/CONFIG_RTE_LIBRTE_MLX4_PMD=./CONFIG_RTE_LIBRTE_MLX4_PMD=y/ ./.config

To compile the DPDK source code into a library, enter these lines at a command prompt: 


cd $RTE_SDK/$RTE_TARGET
EXTRA_CFLAGS=-fPIC make

installing the Streams DPDK 'glue' library

The PacketDPDKSource operator is part of the network toolkit for Streams, which contains a 'glue' library for DPDK. To get the toolkit and compile the 'glue' library, enter these commands at a prompt: 


mkdir -p $HOME/git
cd $HOME/git
git clone git@github.com:IBMStreams/streamsx.network.git
cd $HOME/git/streamsx.network
ant toolkit
EXTRA_CFLAGS=-fPIC ant dpdkglue

compiling SPL applications with DPDK

Streams SPL applications that include the PacketDPDKSource operator must be compiled with additional options for the GNU compiler and linker, like this: 


sc *streams-compiler-options* "--cxx-flags=-fPIC -m64 -pthread -fopenmp" "--ld-flags=-Wl,-L -Wl,$RTE_SDK/$RTE_TARGET/lib -Wl,--no-as-needed -Wl,-export-dynamic -Wl,--whole-archive -Wl,-ldpdk -Wl,-libverbs -Wl,--start-group -Wl,-lrt -Wl,-lm -Wl,-ldl -Wl,--end-group -Wl,--no-whole-archive -Wl,-Map=app.map -Wl,--cref" *other-streams-compiler-options*

loose ends

TODO: what kernel modules (besides ib_uverbs) need to be loaded for Mellanox devices?

TODO: can master thread run on the 'other' NUMA core ?

TODO: default for 'lcoreMaster' parameter is '-1' ... what does that mean?

TODO: what value should be specified for parameter 'nicPort'?

TODO: DPDK fails if only one PacketDPDKSource operator is configured!

TODO: DPDK fails if not executed as 'root'!

TODO: DPDK fails to allocate contiguous bufferspace when plenty is available!

TODO: DPDK hangs or throws exception on termination!

TODO: DPDK fails with 4.2, works with Streams 4.1!

Summary

Ports
This operator has 0 input ports and 0 or more output ports.
Windowing
This operator does not accept any windowing configurations.
Parameters
This operator supports 11 parameters.

Required: lcore

Optional: buffersizes, dequeueThreadPinning, jMirrorCheck, lcoreMaster, metricsInterval, nicPort, nicQueue, outputFilters, promiscuous, rateLimit

Metrics
This operator reports 7 metrics.

Properties

Implementation
C++
Threading
Never - Operator never provides a single threaded execution context.

Output Ports

Assignments
This operator allows any SPL expression of the correct type to be assigned to output attributes.
Ports (0...)

The PacketDPDKSource operator requires one or more output ports:

Each output port will produce one output tuple for each packet received if the corresponding expression in the outputFilters parameter evaluates true, or if no outputFilters parameter is specified.

Output attributes can be assigned values with any SPL expression that evaluates to the proper type, and the expressions may include any of the PacketDPDKSource result functions. Output attributes that match input attributes in name and type are copied automatically.

Properties

Parameters

This operator supports 11 parameters.

Required: lcore

Optional: buffersizes, dequeueThreadPinning, jMirrorCheck, lcoreMaster, metricsInterval, nicPort, nicQueue, outputFilters, promiscuous, rateLimit

buffersizes

This optional parameter specifies how much buffer space, in megabytes, to allocate from available 2MB Linux 'hugepages' for each NUMA node, specified as a string containing comma-separated values. For the NUMA node the ethernet adapter is connected to, specify an even number of megabytes. For other NUMA nodes, if any, specify zero.

For example, on a machine with one NUMA node, specify "100" to allocate 100 megabytes of buffer space. For another example, on a machine with two NUMA nodes, where the ethernet adapter is connected to the second node, specify "0,100" to allocate 100 megabytes of buffer space on the second NUMA node. If multiple PacketDPDKSource operators are configured for the same network adapter, all operators should specify the same values for their 'buffersizes' parameter.

To determine which NUMA node your ethernet adapter is connected to, enter the command cat /sys/class/net/$DPDK_INTERFACE/device/numa_node at a Linux prompt. To determine how much buffer space is available in the 'hugepages' for that NUMA node, enter the command grep -i -h hugepages /sys/devices/system/node/node*/meminfo at a Linux prompt.

If this parameter is not specified, DPDK will allocate all available 2MB Linux 'hugepages' for its buffer space.

Properties

dequeueThreadPinning

This optional parameter specifies a set of logical processor cores to pin the internal ring buffer dequeue thread to. This thread will run at 100% cpu utilization as it spins on the ring buffer tail.

By default, this thread runs unpinned.

Properties

jMirrorCheck

This optional parameter takes an expression of type 'boolean' that specifies whether or not the operator should check for Juniper Networks 'jMirror' headers on packets, and step over them when found.

The default value is 'false'.

Properties

lcore

This required parameter specifies a logical processor core for the DPDK packet receive thread. The core specified must belong to the NUMA node the ethernet adapter is connected to. The receive thread will run the specified core at 100% CPU utilization.

If multiple PacketDPDKSource operators are configured for the same ethernet adapter, each operator must specify a different value with its 'lcoreMaster' parameter. The total number of PacketDPDKSource operators must be a power of 2, that is, the total number of PacketDPDKSource operators must be 1, 2, 4, 8, or 16. Packets received by the network adapter will be distributed across all of the operators, which will run in parallel.

To determine how many processor cores your machine has, and which cores are connected to which NUMA nodes, execute the lscpu command at a Linux prompt. To determine which NUMA node your ethernet adapter is connected to, look in the file '/sys/class/net/$DPDK_INTERFACE/device/numa_node'.

Properties

lcoreMaster

This optional parameter specifies a logical processor core for the DPDK master thread. This thread is not used for receiving packets, and does not have high CPU utiliztion. If the machine has more than one NUMA node, specify a core on a node that does not have the ethernet adapter connected to it.

When multiple PacketDPDKSource operators are configured for the same ethernet adapter, all operators must specify the same value with their 'lcoreMaster' parameters.

To determine how many processor cores your machine has, and which cores are connected to which NUMA nodes, execute the lscpu command at a Linux prompt.

Properties

metricsInterval

This optional parameter takes an expression of type float64 that specifies the interval, in seconds, for sending operator metrics to the Streams runtime. If the value is zero or less, the operator will not report metrics to the runtime, and the output assigment functions for libpcap statistics will be zero.

The default value is '10.0'.

Properties

nicPort

This optional parameter specifies which port of the ethernet adapter the operator will receive network packets from. This applies only to adapters with multiple ports.

The default value is '0' if this parameter is not specified.

Properties

nicQueue

This optional parameter specifies which queue of received packets the operator will receive. This applies only when multiple PacketDPDKSource operators are configured for the same ethernet adapter. The value must be between 0 and 'N-1', where N is the total number of operators, and each operator must specify a different queue, so that all queues are connected to an operator.

Note that DPDK requires the total number of queues and operators be a power of 2, that is, the total number of queues and operators must be 1, 2, 4, 8, or 16. DPDK will distribute packets across all of the queues, and therefore across all of the operators.

The default value is '0' if this parameter is not specified.

Properties

outputFilters

This optional parameter takes a list of SPL expressions that specify which packets should be emitted by the corresponding output port. The number of expressions in the list must match the number of output ports, and each expression must evaluate to a boolean value. The output filter expressions may include any of the PacketDPDKSource result functions.

The default value of the outputFilters parameter is an empty list, which causes all packets processed to be emitted by all output ports.

Properties

promiscuous

This optional parameter takes an expression of type 'boolean' that specifies whether or not 'promiscuous' mode should be enabled on the network interface.

The default value is 'false'.

Properties

rateLimit

This optional parameter takes an expression of type 'float64' that specifies the maximum number of times per second the RATE_LIMITED() function returns false. This can be used to limit the rate of packets sent to an output port when used in a filter expression.

The default value is '1000.0'.

Properties

Metrics

maxQueueDepthSWCurrent - Counter

This metric reports the high water mark of the first software queue.

nBytesProcessedCurrent - Counter

This metric counts number of bytes of packet data processed by the operator.

nBytesReceivedCurrent - Counter

This metric counts the number of bytes received by the ethernet adapter, as counted by DPDK.

nPacketsDroppedCurrent - Counter

This metric counts the number of packets dropped by the ethernet adapter, as counted by DPDK.

nPacketsDroppedSWCurrent - Counter

This metric counts number of packets dropped by the initial software queue.

nPacketsProcessedCurrent - Counter

This metric counts number of packets processed by the operator.

nPacketsReceivedCurrent - Counter

This metric counts the number of packets received by the ethernet adapter, as counted by DPDK.

Libraries

Functions for parsing network headers.
Include Path: ../../impl/include
Streams DPDK 'glue' library 'libstreams_source.a'.
Library Name: streams_source
Library Path: ../../impl/src/source/dpdk/build/lib
Include Path: ../../impl/src/source/dpdk/streams_source