For those of us who run Kubernetes on-premise on physical hardware, it is entirely possible that not all your nodes have the same hardware. For memory and CPU cores, Kubernetes magically does the right thing and each node advertises how many cores and how much memory it has, and workloads are scheduled on a node that has sufficient resources.
For other hardware (commonly GPUs) or if you need to match specific CPU features, there is an excellent tool called Node Feature Discovery (we’ll call it NFD for short) which scans your node hardware and labels the Kubernetes nodes with various key/value pairs – some are boolean flags while others are quantities. You can see your labels with the following command. The output is long, so I have truncated it here.
$ kubectl get node kube04 -o jsonpath='{ .metadata.labels }' | jq
{
# Examples of CPU features (e.g. instruction sets)
"feature.node.kubernetes.io/cpu-cpuid.ADX": "true",
"feature.node.kubernetes.io/cpu-cpuid.AESNI": "true",
"feature.node.kubernetes.io/cpu-cpuid.AVX": "true",
"feature.node.kubernetes.io/cpu-cpuid.AVX2": "true",
# Example of other system values
"feature.node.kubernetes.io/cpu-hardware_multithreading": "false",
"feature.node.kubernetes.io/cpu-model.family": "6",
"feature.node.kubernetes.io/cpu-model.id": "94",
"feature.node.kubernetes.io/cpu-model.vendor_id": "Intel",
"feature.node.kubernetes.io/kernel-version.full": "6.5.0-28-generic",
"feature.node.kubernetes.io/kernel-version.major": "6",
"feature.node.kubernetes.io/kernel-version.minor": "5",
"feature.node.kubernetes.io/kernel-version.revision": "0",
"feature.node.kubernetes.io/pci-0300_8086.present": "true",
"feature.node.kubernetes.io/storage-nonrotationaldisk": "true",
"feature.node.kubernetes.io/system-os_release.ID": "ubuntu",
"feature.node.kubernetes.io/system-os_release.VERSION_ID": "22.04"
}My specific use case is to inspect the network hardware and label nodes according to the speed of their network adapters. I have a physical cluster of 6 bare metal nodes and I’m halfway through upgrading them from gigabit Ethernet to 2.5G Ethernet. I would like the ability to pin specific bandwidth-hungry workloads to the nodes with the 2.5G networking.
NFD doesn’t include labels related to networking out of the box, but it does have access to a lot of extra information behind the scenes, and it provides several CRDs to enable you to access them.
NFD has a concept of Features, which is a piece of information that NFD has discovered about a node. The list of all currently available features is available in the docs, and it shows that there is a feature called network.device which includes the speed of each network interface, in Mbps, as a string.
The other piece of the puzzle is to “match” this feature by creating a NodeFeatureRule, which allows you to set conditions based on Features and use them to generate labels to be applied to the node. Here is a simple NodeFeatureRule that I have written to add labels for the network speed, based on which network interfaces are up.
apiVersion: nfd.k8s-sigs.io/v1alpha1
kind: NodeFeatureRule
metadata:
name: network-speed
spec:
rules:
- name: "network speed 10M"
labels:
"feature.node.kubernetes.io/network-10M": "true"
matchFeatures:
- feature: network.device
matchExpressions:
operstate: {op: In, value: ["up"]}
speed: {op: In, value: ["10"]}
- name: "network speed 100M"
labels:
"feature.node.kubernetes.io/network-100M": "true"
matchFeatures:
- feature: network.device
matchExpressions:
operstate: {op: In, value: ["up"]}
speed: {op: In, value: ["100"]}
- name: "network speed 1G"
labels:
"feature.node.kubernetes.io/network-1G": "true"
matchFeatures:
- feature: network.device
matchExpressions:
operstate: {op: In, value: ["up"]}
speed: {op: In, value: ["1000"]}
- name: "network speed 2.5G"
labels:
"feature.node.kubernetes.io/network-2.5G": "true"
matchFeatures:
- feature: network.device
matchExpressions:
operstate: {op: In, value: ["up"]}
speed: {op: In, value: ["2500"]}
- name: "network speed 5G"
labels:
"feature.node.kubernetes.io/network-5G": "true"
matchFeatures:
- feature: network.device
matchExpressions:
operstate: {op: In, value: ["up"]}
speed: {op: In, value: ["5000"]}
- name: "network speed 10G"
labels:
"feature.node.kubernetes.io/network-10G": "true"
matchFeatures:
- feature: network.device
matchExpressions:
operstate: {op: In, value: ["up"]}
speed: {op: In, value: ["10000"]}
With this applied to my cluster, I get the following output when I check my node labels:
$ kubectl get node kube04 -o jsonpath='{ .metadata.labels }' | jq | grep network
"feature.node.kubernetes.io/network-2.5G": "true",
$ kubectl get node kube07 -o jsonpath='{ .metadata.labels }' | jq | grep network
"feature.node.kubernetes.io/network-1G": "true",Now I can pin my high-bandwidth workloads on the nodes with 2.5G networking by using a nodeSelector like:
spec:
nodeSelector:
feature.node.kubernetes.io/network-2.5G: true
If you want some of this goodness on your cluster, I’ve published my NodeFeatureRule on GitHub.
Jonathan Gazeley 