FH 7.2 change request

After some testing with 2024.w04, LITEON, Quectel RM500Q, Ubuntu 22.04.3 LTS (GNU/Linux 5.15.0-1050-realtime x86_64) on Xeon(R) Gold 6442Y, I would suggest to consider the following changes:

1. Add configurable system_core - for systems where CPU 0 cannot be reasonable isolated from kernel threads. (DPDK uses system_core for its control threads: rte_mp_handle, eal-intr-thread,iavf-event-thread).
2. Add an explicit affinity assignment of ru_thread after initialization of XRAN library.

+
+  cpu = sched_getcpu();
+
+  if (cpu != ru->ru_thread_core)
+  {
+    cpu_set_t cpuset;
+    CPU_ZERO(&cpuset);
+    CPU_SET(ru->ru_thread_core, &cpuset);
+    //printf("AAA2 cpu %d instead of %d\n", cpu, ru->ru_thread_core);
+    int ret = pthread_setaffinity_np(pthread_self(), sizeof(cpu_set_t), &cpuset);
+    AssertFatal(ret == 0, "Error in pthread_getaffinity_np(): ret: %d, errno: %d", ret, errno);
+  }
+
   if (setup_RU_buffers(ru)!=0) {
     printf("Exiting, cannot initialize RU Buffers\n");
     exit(-1);

2. XRAN library: initialize EAL without telemetry to skip creation of another or two (DPDK 21.11.5) DPDK control thread(s)

adva@tfsrv01:~/phy/fhi_lib/lib$ git diff ethernet/ethdi.c
diff --git a/fhi_lib/lib/ethernet/ethdi.c b/fhi_lib/lib/ethernet/ethdi.c
index b6ba257..b53325f 100644
--- a/fhi_lib/lib/ethernet/ethdi.c
+++ b/fhi_lib/lib/ethernet/ethdi.c
@@ -335,6 +335,7 @@ xran_ethdi_init_dpdk_io(char *name, const struct xran_io_cfg *io_cfg,
     uint16_t count;

     char *argv[] = { name, core_mask, "-n2", iova_mode, socket_mem, socket_limit, "--proc-type=auto",
+        "--no-telemetry",
         "--file-prefix", name, "-a0000:00:00.0", bbdev_wdev, bbdev_vdev};

added 5G-NR gNB labels

Dear Andrew, thanks a lot!

I agree for 1 & 3. Regarding 2, following remarks:

• we should set this in the oran fhlib library, into which we call from ru_thread() using the function openair0_transport_load() just above. The reason being that so far this only seems necessary for 7.2; many other radios, e.g., USRP, use all cores floating, so we should not force it for that use case
• in fact, such code snippet was present in earlier versions, but I did not really understand why, so I commented it. The code is actually still there, see commented function set_main_core() in radio/fhi_72/oran-init.c. Sorry, I did not realize this earlier.

I have an open MR !2559 (merged), if you don't mind, I add it there tomorrow and will inform you when you when it is done.

• setting it in the fhlib library is not possible as such, as we don't have the ru_thread_core variable handy
• when starting the ru_thread in start_RU_proc(), we already set the priority. I am confused, why do we need it twice, do you have any info on that?

for the time being, I add the code that you suggested with an additional check ru_thread_core > -1

I dont have an explanation, but it seems like after eal_init the affinity of parent thread (ru_thread) is changed. It might be a local problem - I have observed it in one system at every run (see printout, planning to verify on another.

mentioned in merge request !2559 (merged)

@asergeev

(DPDK uses system_core for its control threads: rte_mp_handle, eal-intr-thread,iavf-event-thread).

do you have more information on that? I cannot find either how xran tells DPDK which is the system_core, nor do I find references to the system_core used in DPDK online. I will add it, but I still don't understand why that is necessary

mentioned in commit c5211b0f

mentioned in commit b32b9e83

mentioned in commit b5ba3a69

DPDK eal_init usually is called from main thread, which is not supposed to be a packet processing thread. In our case DPDK decides (compute_ctrl_threads_cpuse) to spread its control threads following the affinity of current thread(ru_thread). Running ru_thread and DPDK control threads on the same CPU causes problem on my setup.

@asergeev I still see that ru_thread is initializing eal-intr-thread, iavf-event-thread and rte_mp_handle.

Task Name:	nr-softmodem, State:	S (sleeping), Cpus_allowed_list:	0-31
Task Name:	MAC_STATS, State:	S (sleeping), Cpus_allowed_list:	0-31
Task Name:	RLC queue, State:	S (sleeping), Cpus_allowed_list:	0-31
Task Name:	PDCP data ind, State:	S (sleeping), Cpus_allowed_list:	0-31
Task Name:	pdcp_timer, State:	S (sleeping), Cpus_allowed_list:	0-31
Task Name:	TASK_SCTP, State:	S (sleeping), Cpus_allowed_list:	0-31
Task Name:	TASK_NGAP, State:	S (sleeping), Cpus_allowed_list:	0-31
Task Name:	TASK_GNB_APP, State:	S (sleeping), Cpus_allowed_list:	0-31
Task Name:	TASK_RRC_GNB, State:	S (sleeping), Cpus_allowed_list:	0-31
Task Name:	TASK_GTPV1_U, State:	S (sleeping), Cpus_allowed_list:	0-31
Task Name:	ru_thread, State:	R (running), Cpus_allowed_list:	9
Task Name:	eal-intr-thread, State:	S (sleeping), Cpus_allowed_list:	9
Task Name:	rte_mp_handle, State:	S (sleeping), Cpus_allowed_list:	9
Task Name:	fh_main_poll-1, State:	R (running), Cpus_allowed_list:	1
Task Name:	fh_rx_bbdev-2, State:	R (running), Cpus_allowed_list:	2
Task Name:	iavf-event-thre, State:	S (sleeping), Cpus_allowed_list:	9
Task Name:	Tpool0_18, State:	S (sleeping), Cpus_allowed_list:	18
Task Name:	Tpool1_19, State:	S (sleeping), Cpus_allowed_list:	19
Task Name:	Tpool2_20, State:	S (sleeping), Cpus_allowed_list:	20
Task Name:	Tpool3_21, State:	S (sleeping), Cpus_allowed_list:	21
Task Name:	Tpool4_22, State:	S (sleeping), Cpus_allowed_list:	22
Task Name:	Tpool5_23, State:	S (sleeping), Cpus_allowed_list:	23
Task Name:	Tpool6_24, State:	S (sleeping), Cpus_allowed_list:	24
Task Name:	Tpool7_25, State:	S (sleeping), Cpus_allowed_list:	25
Task Name:	L1_rx_thread, State:	R (running), Cpus_allowed_list:	8
Task Name:	L1_stats, State:	S (sleeping), Cpus_allowed_list:	0-31

This is my system with the following command line:

taskset -ac 1-16 ./nr-softmodem -O ../../../targets/PROJECTS/GENERIC-NR-5GC/CONF/gnb.sa.band78.273prb.fhi72.4x4-liteon.conf --sa --reorder-thread-disable 1

io_core = 4 worker_cores = (2); ru_thread_core = 6; io_cfg->system_core = 1

adva@tfsrv01:~$ sudo ps -o cpuid,comm,pid,spid,pcpu,pri -Tp `pidof nr-softmodem` | sort -n
CPUID COMMAND             PID    SPID %CPU PRI
    1 RLC queue         59837   59867  0.0  19
    1 TASK_GNB_APP      59837   59878  0.0  90
    1 Tpool1_-1         59837   59900  0.9 137
    1 eal-intr-thread   59837   59888  0.0 137
    1 iavf-event-thre   59837   59892  0.0 137
    1 nr-softmodem      59837   59837  0.0  19
    1 nr-softmodem      59838   59838  0.0  19
    1 pdcp_timer        59837   59869  0.3  19
    1 rte_mp_handle     59837   59889  0.0 137
    2 fh_rx_bbdev-2     59837   59890 99.9 138
    3 MAC_STATS         59837   59866  0.0  42
    3 TASK_RRC_GNB      59837   59881  0.0  90
    4 fh_main_poll-4    59837   59891 99.9 138
    5 Tpool4_-1         59837   59909  0.9 137
    6 ru_thread         59837   59887 99.9 137
    7 TASK_SCTP         59837   59872  0.0  90
    7 Tpool3_-1         59837   59906  0.9 137
    8 L1_rx_thread      59837   59921 39.6 137
    9 PDCP data ind     59837   59868  0.0  19
    9 Tpool2_-1         59837   59903  0.9 137
   10 Tpool5_-1         59837   59912  0.9 137
   11 L1_stats          59837   59924  0.0  41
   12 Tpool7_-1         59837   59918  0.9 137
   13 TASK_NGAP         59837   59875  0.0  90
   14 TASK_GTPV1_U      59837   59884  0.0  90
   15 Tpool0_-1         59837   59897  0.9 137
   16 Tpool6_-1         59837   59915  0.9 137

@asergeev You are running OAI using taskset whereas we don't do that. That is why we have the difference. I don't know if what you are doing is good or not because the threads we don't pin will take the threadpool threads. I don't know how it will affect the KPIs. As you can see the difference in your and my output.

Can you provide the output of the below script it will print like what I did above. This will provide allowed affinity for each thread.

You need to provide nr-softmodem id as it is the parent.

#!/bin/sh

for file in /proc/$1/task/*
do
   name=$(cat $file/status | grep Name)
   state=$(cat $file/status | grep State )
   cpus=$(cat $file/status | grep 'Cpus_allowed_list')
   check=$(cat $file/status | grep -o running )
   echo "Task $name, $state, $cpus"
done

closed with merge request !2564 (merged)

mentioned in commit 3f3a9869

reopened

I reopen to track this final CPU affinity problem

adva@tfsrv01:~$ ./print-cpu.sh 61574
Task Name:      nr-softmodem, State:    S (sleeping), Cpus_allowed_list:        1-16
Task Name:      MAC_STATS, State:       S (sleeping), Cpus_allowed_list:        1-16
Task Name:      RLC queue, State:       S (sleeping), Cpus_allowed_list:        1-16
Task Name:      PDCP data ind, State:   S (sleeping), Cpus_allowed_list:        9
Task Name:      pdcp_timer, State:      S (sleeping), Cpus_allowed_list:        1-16
Task Name:      TASK_SCTP, State:       S (sleeping), Cpus_allowed_list:        1-16
Task Name:      TASK_NGAP, State:       S (sleeping), Cpus_allowed_list:        1-16
Task Name:      TASK_GNB_APP, State:    S (sleeping), Cpus_allowed_list:        1-16
Task Name:      TASK_RRC_GNB, State:    S (sleeping), Cpus_allowed_list:        1-16
Task Name:      TASK_GTPV1_U, State:    S (sleeping), Cpus_allowed_list:        1-16
Task Name:      ru_thread, State:       R (running), Cpus_allowed_list: 6
Task Name:      eal-intr-thread, State: S (sleeping), Cpus_allowed_list:        1
Task Name:      rte_mp_handle, State:   S (sleeping), Cpus_allowed_list:        1
Task Name:      fh_rx_bbdev-2, State:   R (running), Cpus_allowed_list: 2
Task Name:      fh_main_poll-4, State:  R (running), Cpus_allowed_list: 4
Task Name:      iavf-event-thre, State: S (sleeping), Cpus_allowed_list:        1
Task Name:      Tpool0_-1, State:       S (sleeping), Cpus_allowed_list:        1-16
Task Name:      Tpool1_-1, State:       S (sleeping), Cpus_allowed_list:        1-16
Task Name:      Tpool2_-1, State:       S (sleeping), Cpus_allowed_list:        1-16
Task Name:      Tpool3_-1, State:       S (sleeping), Cpus_allowed_list:        1-16
Task Name:      Tpool4_-1, State:       S (sleeping), Cpus_allowed_list:        1-16
Task Name:      Tpool5_-1, State:       S (sleeping), Cpus_allowed_list:        1-16
Task Name:      Tpool6_-1, State:       S (sleeping), Cpus_allowed_list:        1-16
Task Name:      Tpool7_-1, State:       S (sleeping), Cpus_allowed_list:        1-16
Task Name:      L1_rx_thread, State:    R (running), Cpus_allowed_list: 8
Task Name:      L1_stats, State:        S (sleeping), Cpus_allowed_list:        1-16

@schmidtr so the behaviour on our machines is not the same. In our case ru_thread and dpdk threads are using the same CPU core. We need to see how to use system_core to force the use of other cores than core 0.

Could you share info about CPU, Linux distro and kernel version of your machine?

@asergeev It is the same as the one mentioned in 7.2 tutorial.

Ubuntu 22.04.2 LTS (5.15.0-1038-realtime) AMD EPYC 9374F 32-Core Processor, 128GB

I am planning to to isolate DPDK threads from all others. Going to create a dedicated branch and the merge request.