Working with NVIDIA BlueField-2 DPU

I have recently been trying to set up a BlueField-2 DPU and found that not only is documentation scattered and sometimes incomplete, the few resources on setting up this product are often behind a paywall (!!). So, another quick start guide would be helpful.

If you’re not familiar with a data processing unit (DPU), you can think of it as a CPU (some ARM cores) with hardware accelerators sitting on the network interface card (NIC) itself, which is a component plugged into a server (the host).

The BlueField-2 DPU is interesting in that it has different modes of operation. In the Separated Host mode (“NIC mode”), it functions as a regular NIC. In the Embedded CPU Function Ownership (ECPF) mode (“DPU mode”), it has the additional functionality that all incoming packets to the NIC will first reach the ARM cores before reaching the host. See the below architecture diagram for the DPU mode.

Architecture for the BlueField-2 DPU in the DPU mode. Source: https://docs.nvidia.com/doca/sdk/nvidia+bluefield+modes+of+operation/index.html

The following section assumes that you have already set up administrative access to the DPU (refer to this guide).

Using the NIC mode

NVIDIA already provides an official guide for switching between the modes of operation. However, I found it somewhat incomplete. What I did to make the NIC mode work:

1. Switch to Separated Host mode on the host. Run:

mst start

# Get the dev address
sudo mst status -v

# Set the mode
sudo mlxconfig -d /dev/mst/<device> s INTERNAL_CPU_MODEL=0

And then, reboot the host for the config to apply.

2. Repeat step 1 on the DPU, i.e. run all the commands, and also reboot the DPU for changes to apply. Note, this step was not written on the official guide.

3. On both the host and the DPU, verify that the right mode is now used:

mst start
mlxconfig -d /dev/mst/<device> q | grep -i model
# Output should be SEPARATED_HOST

4. Unlike stated on the guide, do not remove the OVS bridge.

5. Finally, test that the NIC mode works. You can try pinging between two connected interfaces, e.g. from this server, via the interface using the DPU as a NIC, to another server. (If you ping via a loopback cable to another interface on the same server, remember to create a separate namespace).

Using the DPU mode

Similarly, set the INTERNAL_CPU_MODEL=1 on both the host and the DPU, and restart both for the changes to take effect. You can follow the Nvidia guide or my previous section. Don’t mess with the OVS bridge settings.

I found this diagram useful for understanding the various interfaces / “ports” as known in DPDK on the DPU. This diagram slightly different from the previous diagram on DPU architecture as it doesn’t show the eSwitch. I also believe the OVS should be placed into the ARM server, as it is running on the ARM server.

Interfaces and representors on the DPU and host. Source: https://docs.nvidia.com/doca/sdk/nvidia+doca+allreduce+application+guide/index.html

Another relevant resource for understanding the representors model is here. However, I don’t think this doc is sufficiently detailed as it tries to cover too much in a limited space.

Now, with knowledge of the above interfaces, simply assign an IP address to the physical interface on the DPU, i.e. enp3s0f0s0, and you can ping from the host interfaces to it.

Running from the DPU, using DPDK testpmd and the rte_flow API, you can even capture packets on specific flows destined to the host via the DPU, or destined to the DPU from the host, or vice versa. An easy way to achieve this is to define flow rules for a specific DPDK port which this flow will go through, although you can also do more by matching IP src/dst addresses. An example is:

flow create <port no> ingress pattern eth / ipv4 / end actions count / drop / end

This quick reference is helpful. You can ignore the advanced functionalities of redirecting flows, just drop packets if you only want to check flow path.

Conclusion

In this post, we went through how to set the Bluefield-2 DPU to either the NIC or DPU mode, and check connectivity in both. We also tried to capture flows from within the DPU, using the different interfaces exposed in the DPU. Future posts on this topic may come if we build interesting applications on the DPU!