Draft: pthread_create, RT scheduler, and address sanitizer are incompatible in Ubuntu 18.04

When we run with the RT scheduler and the address sanitizer in Ubuntu 18.04, pthread_create will hang and never create threads. This MR adds a warning such that other users are informed about this issue

OAI RAN-Container-Parent build (2115): Your merge request has none of the mandatory labels:

• BUILD-ONLY
• 4G-LTE
• 5G-NR
• CI

Not performing CI due to lack of labels

OAI RAN-Container-Parent build (2115): failed (https://jenkins-oai.eurecom.fr/job/RAN-Container-Parent/2115/)

OAI RAN-CI-develop build (3502): Your merge request has none of the mandatory labels:

• BUILD-ONLY
• 4G-LTE
• 5G-NR
• CI

Not performing CI due to lack of labels

OAI RAN-CI-develop build (3502): failed (https://open5glab.eurecom.fr:8083/jenkins/job/RAN-CI-develop/3502/)

added 5G-NR label

changed title from pthread_create, RT scheduler, and address sanitizer are incompatible in Ubuntu 18.04 to WIP: pthread_create, RT scheduler, and address sanitizer are incompatible in Ubuntu 18.04

changed title from WIP: pthread_create, RT scheduler, and address sanitizer are incompatible in Ubuntu 18.04 to pthread_create, RT scheduler, and address sanitizer are incompatible in Ubuntu 18.04

OAI RAN-CI-develop build (3503): Some Changed files in Merge Request DO NOT follow OAI Formatting Rules

Can you please write the steps to reproduce the original issue, i.e., your Ubuntu version, the exact kernel, ASan version, gcc version, libc version, and command line?

Is there a difference to !1530 (closed) except the modifications of the flag?

OAI RAN-Container-Parent build (2116): failed (https://jenkins-oai.eurecom.fr/job/RAN-Container-Parent/2116/)

OAI RAN-CI-develop build (3503): passed (https://open5glab.eurecom.fr:8083/jenkins/job/RAN-CI-develop/3503/)

changed title from pthread_create, RT scheduler, and address sanitizer are incompatible in Ubuntu 18.04 to WIP: pthread_create, RT scheduler, and address sanitizer are incompatible in Ubuntu 18.04

changed title from WIP: pthread_create, RT scheduler, and address sanitizer are incompatible in Ubuntu 18.04 to pthread_create, RT scheduler, and address sanitizer are incompatible in Ubuntu 18.04

OAI RAN-Container-Parent build (2128): failed (https://jenkins-oai.eurecom.fr/job/RAN-Container-Parent/2128/)

added 89 commits

• 26065bde...d7dae8fa - 88 commits from branch develop
• 12dec436 - Merge branch 'eurecom-develop' into ubuntu_18_rt_add_san_incompatibility

Compare with previous version

@schmidtr, I apologize for not getting back to your comments sooner. Steps to reproduce the original issue include:

• Operating System: Ubuntu 18.04.5 LTS

• Kernel: Linux 5.4.0-110-generic

• gcc: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0

• Build OAI Binary with: --sanitize-address --run-with-gdb RelWithDebInfo --verbose-compile

• Run proxy_testscript.py in either LTE or SA mode case

  ~/multi-ue-proxy/proxy_testscript.py -u 1 --mode lte

  ~/multi-ue-proxy/proxy_testscript.py -u 1 --mode sa

• Manually launching for SA:

NRUE: sudo -E LD_LIBRARY_PATH=~/openairinterface5g/cmake_targets/ran_build/build: taskset --cpu-list 2 ./ran_build/build/nr-uesoftmodem -O ../ci-scripts/conf_files/episci/proxy_nr-ue.nfapi.conf --nokrnmod 1 --nfapi 5 --node-number 2 --emulate-l1 --log_config.global_log_options level,nocolor,time,thread_id --sa

gNB: sudo -E LD_LIBRARY_PATH=~/openairinterface5g/cmake_targets/ran_build/build: taskset --cpu-list 1 ./ran_build/build/nr-softmodem -O ../ci-scripts/conf_files/episci/proxy_gnb.band78.sa.fr1.106PRB.usrpn310.conf --nfapi 2 --noS1 --emulate-l1 --log_config.global_log_options level,nocolor,time,thread_id --sa

There is no difference to !1530 (closed) except the modifications of the flag and the rebased commit history.

OAI RAN-Container-Parent build (2157): failed (https://jenkins-oai.eurecom.fr/job/RAN-Container-Parent/2157/)

OAI RAN-CI-develop build (3525): Some Changed files in Merge Request DO NOT follow OAI Formatting Rules

OAI RAN-CI-develop build (3525): passed (https://open5glab.eurecom.fr:8083/jenkins/job/RAN-CI-develop/3525/)

i don't like very much the approach, and i never got this issue even if i compile and run with sanitze addr often

Nevertheless i agree on one general goal: be able to run with normal Linux user (not root) when we don't need real time

Here i'm a bit lost with Episci work, because, desipite it is simulation, it is claimed to run in some real time constrains. This obviously require to run without sanitize addr and with Linux scheduler in RT classes (FIFO or RR)

Anyway, i propose to add a command line parameter --non-root that will be used in the code to remove all privileged calls: real time scheduler, set_latency, set_affinity

added 1 commit

• 10a8cfc4 - Removing #define for RT scheuler option

Compare with previous version

OAI RAN-Container-Parent build (2193): failed (https://jenkins-oai.eurecom.fr/job/RAN-Container-Parent/2193/)

One general remark: instead of checking that flag in the functions, it would be better to not even call these functions. So I suggest to take out the if(non_root) in the functions set_sched() (here is btw the problem that if the nice level is too low, it might also not work), set_priority(), set_thread_priority() in vnf/pnf_sim.

To avoid the double-negation, maybe despite calling the option --not_root, maybe it could be called root (root_prio?) internally. This is optional.

requested review from @schmidtr

changed milestone to %REVIEW_IN_PROGRESS

added 1 commit

• e0e643a0 - Removing functional changes and maintaining comment

Compare with previous version

added 1 commit

• 8a0d55ac - Reverting whitespace changes

Compare with previous version

resolved all threads

Reverted all changes except for the explaination in the CMakeLists.txt. I think the comment is sufficient. Also, three LOG_As were demoted to Ds and I added them back here.

OAI RAN-Container-Parent build (2195): failed (https://jenkins-oai.eurecom.fr/job/RAN-Container-Parent/2195/)

OAI RAN-Container-Parent build (2196): failed (https://jenkins-oai.eurecom.fr/job/RAN-Container-Parent/2196/)

OAI RAN-CI-develop build (3561): Some Changed files in Merge Request DO NOT follow OAI Formatting Rules

OAI RAN-CI-develop build (3561): failed (https://open5glab.eurecom.fr:8083/jenkins/job/RAN-CI-develop/3561/)

OAI RAN-CI-develop build (3563): Some Changed files in Merge Request DO NOT follow OAI Formatting Rules

OAI RAN-CI-develop build (3563): passed (https://open5glab.eurecom.fr:8083/jenkins/job/RAN-CI-develop/3563/)

OAI RAN-CI-develop build (3564): Some Changed files in Merge Request DO NOT follow OAI Formatting Rules

OAI RAN-CI-develop build (3564): passed (https://open5glab.eurecom.fr:8083/jenkins/job/RAN-CI-develop/3564/)

added 1 commit

• be2910cd - add a warning for ubuntu 18.04

Compare with previous version

OAI RAN-CI-develop build (3566): Some Changed files in Merge Request DO NOT follow OAI Formatting Rules

OAI RAN-CI-develop build (3566): failed (https://open5glab.eurecom.fr:8083/jenkins/job/RAN-CI-develop/3566/)

OAI RAN-Container-Parent build (2198): failed (https://jenkins-oai.eurecom.fr/job/RAN-Container-Parent/2198/)

marked this merge request as draft

changed the description

changed milestone to %REVIEW_COMPLETED_AND_APPROVED

added 2 commits

• 7b75305e - pthread_create, RT scheduler, and address sanitizer are incompatible in Ubuntu 18.04
• a73e500d - Add a warning for ubuntu 18.04 in build_oai

Compare with previous version

**Code Review by : @schmidtr **

• IMPACT on FUNCTIONAL CODE
  • The code change does NOT respect architecture/protocol split
  • Abnormal exits are NOT properly handled
  • Coding Style Issues
    • Added DEAD Code
    • Improper logging
    • Added useless debug code
    • Duplication of an existing function
    • Indentation is not respected
    • Insufficient in-code comments / code readability
    • Any other coding style issue
  • A new tool/lib/package has been introduced (blocking, subject to negotiation)
• TESTING (The Merge Request requires additional testing)
  • Additional testing is present in the Merge Request (Best)
  • The contributor provides a framework/hints for testing
  • New testing means need to be developed
• DOCUMENTATION (The Merge Request requires additional documentation)
  • Feature Set documentation update needed
    • DONE
    • NOT DONE
  • Wiki tutorial / in-repo usage documentation needed
    • DONE
    • NOT DONE
  • Added/Modified/Removed script/build/runtime option(s)
    • Help UPDATED
    • Help NOT UPDATED

Recommendations:

to (1) say whether it is accepted and (2) say what should still be done to finalize the merge request

• Gives the user a warning when using U18 with Asan -> accepted

P.S. I squashed all commits in one to not pollute the git history more than necessary...

changed milestone to %OK_TO_BE_MERGED

mentioned in merge request !1591 (merged)

mentioned in commit 12a23fb1