Preparing a host to use DPDK

Getting an installation

Red Hat provides a DPDK RPM as part of their AppStreams repository, but users may need to build their own DPDK install if the OS-provided version does not include a specific device driver.

The OS-provided DPDK can be obtained by the following command: 

sudo dnf install dpdk-devel dpdk-tools

If the OS-provided DPDK is not sufficient, an install can be compiled from source. See the DPDK Getting Started Guide for Linux for required software tools and libraries to do the build. Be sure that any needed libraries and drivers will be built during the build configuration phase.

Installing additional software

If you built your own DPDK installation, run the following to install software libraries that many DPDK libraries rely on: 

sudo dnf install pciutils rdma-core-devel libibumad libibverbs-devel libarchive-devel infiniband-diags

If you will be using a Mellanox ethernet adapter, you will also need to download and install OFED drivers from Mellanox.

Configuring a Linux user group for DPDK

You will need a Linux user group for DPDK. To create the user group and add your own user account to it, run the following commands: 


sudo groupadd dpdk
sudo usermod -a -G dpdk $USER
sudo cat "@dpdk - memlock unlimited" >/etc/security/limits.conf

You will need to log off and log back on for this to take effect. To verify that your user account is a member of the group, enter this command at a prompt: 


id

The DPDK operators need read-write permission for the Linux interface. To grant that permission to all user accounts that are members of the 'dpdk' user group, run the following commands: 


sudo chown root:dpdk /sys/class/infiniband/mlx4_0/device/net/$DPDK_INTERFACE/flags
sudo chmod 664 /sys/class/infiniband/mlx4_0/device/net/$DPDK_INTERFACE/flags

Configuring Linux Hugepages for DPDK

DPDK uses Linux hugepages for its buffers. These buffers must be allocated in the memory connected to the same NUMA node as the ethernet adapter, and may use either 2MB or 1GB hugepages. You must create mount points for the hugepages in the Linux file system and grant DPDK read/write permissions for them.

Hugepages can be allocated at runtime or boot time of the OS. The steps below outline allocating hugepages during runtime. If you want to setup hugepages to be enabled across boots, see DPDK's Use of Hugepages in the Linux Environment

IMPORTANT: The hugepage allocation commands below will divide the number of hugepages equally across all NUMA nodes (assuming there is enough memory on all NUMA nodes). If you want to only allocate hugepages on individual NUMA nodes, replace /sys/kernel/mm with /sys/devices/system/node/nodeX/ where X is the NUMA node number.

For example, to allocate 500 2MB 'hugepages' for DPDK buffers, run the following commands: 


sudo /usr/bin/bash -c "echo 500 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages"
sudo mkdir /dev/hugepages-2MB 
sudo mount -t hugetlbfs nodev /dev/hugepages-2MB -o pagesize=2MB
sudo chmod 770 /dev/hugepages-2MB
sudo chown root:dpdk /dev/hugepages-2MB

For another example, to allocate one 1GB 'hugepages' for DPDK buffers on NUMA node 0, run the following commands: 


sudo /usr/bin/bash -c "echo 1 > /sys/devices/system/node/node0/hugepages/hugepages-1048576kB/nr_hugepages"
sudo mkdir /dev/hugepages-1GB
sudo mount -t hugetlbfs nodev /dev/hugepages-1GB -o pagesize=1GB
sudo chmod 770 /dev/hugepages-1GB
sudo chown root:dpdk /dev/hugepages-1GB

To verify that 'hugepages' have been allocated, run the following command: 


grep -i -h hugepages /sys/devices/system/node/node*/meminfo

Setup Linux drivers and bind network ports

See Linux Drivers for instructions on how to load kernel drivers and bind a NIC port to the drivers.