-
Sagar Arora authored
- Fixing the tutorials for release v1.4.0
Sagar Arora authored- Fixing the tutorials for release v1.4.0
- 1. Building images in debug mode
- 1.1 Building the image with code inside (Only for development purpose)
- 1.1.1 Building a Developer Image
- 1.1.2 Mounting Code As Volume
- 2. Debuggers deployment of core network functions
- 2.1 Deploying as a process (baremetal deployment prefered by developers)
- 2.2 Docker environment (Recommeneded)
- 2.2.1 Prerequisites
- 2.2.2 Creating entrypoint files
- 2.2.3 Healthchecks
- 2.2.4 Creating docker-compose
- 2.2.5 Playing with docker-compose
- 3. Basic debugging
- 4. How to report an issue?
![]() |
OpenAirInterface 5G Core Network Deployment: How to debug the Network Functions? |
Currently there is no special tool for debugging the problems of core network components. Though there are certain methods which we are using in our team to diagnose an issue.
This page content expects you to read deployment pre-requisites and try to deploy a mini or basic normal deployment.
TABLE OF CONTENTS
- Building images in debug mode
- Debuggers deployment of core network functions
- Basic debugging
- How to report an issue?
1. Building images in debug mode
By default all the dockerfiles present in any network function repository (AMF, SMF, NRF, UPF, UDR, UDM, AUSF) are built with release
tag. In the release
mode the logging information is limited. This is done to reduce the image size and have a better performance. In beginning when a user is trying to understand the functioning of core network it is recommended to build the images in debug mode
. This way user will have more logs and can have better understanding. To build any core network image in debug mode follow the below steps after cloning the network function repository, the example is for AMF,
# clone amf repository
$ git clone -b <prefered_branch or develop> https://gitlab.eurecom.fr/oai/cn5g/oai-cn5g-amf.git
$ cd oai-cn5g-amf/docker/
# Depending on the environment where the image will be used choose the correct dockerfile
$ vi/vim/nano/subl Dockerfile.amf.ubuntu18
# replace the line RUN ./build_amf --clean --Verbose --build-type Release --jobs with
# RUN ./build_amf --clean --Verbose --build-type Debug --jobs
The same can be done for baremetal deployment of any core network function, just build with Debug
tag in ./build_amf
. There will be no change in running the core network function. Everything will be the same, now when the network function is started there will be extra logs with [debug]
tag apart from the normal [info ]
1.1 Building the image with code inside (Only for development purpose)
If you are interested in doing development you can leave the code inside the container. This is good for developers, they can code in docker-environment. They can even mount the code as a volume so that they can use their prefered editor (though vim/nano/vi are the best).
1.1.1 Building a Developer Image
Below example is only for AMF you need to repeat it for all network functions
$: git clone -b <prefered_branch or develop> https://gitlab.eurecom.fr/oai/cn5g/oai-cn5g-amf.git
$: docker build -f oai-cn5g-amf/docker/Dockerfile.amf.ubuntu18 --target oai-amf-builder --tag oai-amf-builder:develop --no-cache oai-cn5g-amf/
This will build your image and later you can use this image with below command,
$: docker run --privileged -d --name oai-amf-development oai-amf-builder:develop sleep infinity
$: docker exec -it oai-amf-development bash
# You will be inside the container
But in this approach you have to code inside the container using vi/vim/nano no graphical interface, but if you want a graphical interface then you can mount the code
1.1.2 Mounting Code As Volume
Below example is only for AMF you need to repeat it for all network functions
$: git clone -b <prefered_branch or develop> https://gitlab.eurecom.fr/oai/cn5g/oai-cn5g-amf.git /openair-amf
$: docker run --privileged -d --name oai-amf-development --volume openair-amf:/openair-amf ubuntu:bionic sleep infinity
$: docker exec -it oai-amf-development bash
# below command is same for all network functions
$: sudo apt update && apt install psmisc software-properties-common git vim nano vi gnupg
# now you can peform the build
$: cd /openair-amf/build/scripts
$: /build_amf --install-deps --force
$: /build_amf --clean --Verbose --build-type Debug --jobs
Now you are ready, start developing and testing
2. Debuggers deployment of core network functions
It is really important to safely keep the logs and configuration of core network components in case of error. If they are deployed in baremetal the logs and the configuration can be easily retrived based on how the core network is started. Here are some tips related to running/deploying core network in different environment,
2.1 Deploying as a process (baremetal deployment prefered by developers)
- In case of all in one sort of process deployment there can some conflicting dependencies between different components of core network. These conflicts have to be resolved on case by case bases by the user. You can follow the wiki of each network function for bare-metal installation for amf follow this.
- Once all the core-network components are build in debug mode with there dependencies store the logs in a file rather than printing on the terminal.
- All the configuration file should be stored in one place so that it is easy to reterive.
# storing logs and configuration in your prefered location
nohup /usr/local/bin/amf -c /tmp/oai-cn-5g/config/amf.conf -o >> /tmp/oai-cn-5g/logs/amf.log 2>&1
The above command will launch the network function in background and all the logs can be seen using tail -100f /tmp/oai-cn-5g/logs/amf.log
. In case to stop the component kill the its process process using pkill <component-name>
or ps -eaf
2.2 Docker environment (Recommeneded)
Using docker environment for deployment and development is the prefered environment because there it is easy to have dedicated working environment for each network component. It is lightweight and easy to manage. The docker-compose provided in tutorials is good for learning how the OAI core network works and how to use it. But if the user wants to change some parameters which are not variable or not allowed using docker-compose then it is hard to use docker-compose approach. If the user wants to provide their own configuration file then it is better to change the docker-compose. Follow the below steps to create a new developer/debugger specific docker-compose,
2.2.1 Prerequisites
-
Build the docker-images in debug mode following the previous section
-
Create a new folder
oai-docker-compose
-
In the
oai-docker-compose
folder createconfs
,logs
,entrypoints
andhealthchecks
folder -
In the
confs
folder copy the configuration files for each component you want to use. The configuration files are located hereFile Name Repository Location amf.conf (Gitlab) cn5g/oai-cn5g-amf etc/amf.conf smf.conf (Gitlab) cn5g/oai-cn5g-smf etc/smf.conf nrf.conf (Gilab) cn5g/oai-cn5g-nrf etc/nrf.conf spgw_u.conf (Github) OPENAIRINTERFACE/openair-spgwu-tiny etc/spgw_u.conf -
Create empty log files for the component you want to use in the
logs
folder usingtouch
command example -
Copy oai_db.sql make user database depending on the IMSI, LTKEY and Opc. New user entry can be added after this line
# Creating directory structure
$ mkdir ~/oai-docker-compose
$ mkdir -p ~/oai-docker-compose/confs
$ mkdir -p ~/oai-docker-compose/logs
$ mkdir -p ~/oai-docker-compose/entrypoints
$ mkdir -p ~/oai-docker-compose/healthchecks
# Creating empty log files
$ touch ~/oai-docker-compose/logs/amf.log
$ touch ~/oai-docker-compose/logs/smf.log
$ touch ~/oai-docker-compose/logs/nrf.log
$ touch ~/oai-docker-compose/logs/spgwu.log
# Copying the configuration files, if you have yours you can use that else copy from the repository and make changes manually
$ cp ~/oai-cn5g-amf/etc/amf.conf ~/oai-docker-compose/confs/
$ cp ~/oai-cn5g-smf/etc/smf.conf ~/oai-docker-compose/confs/
$ cp ~/oai-cn5g-nrf/etc/nrf.conf ~/oai-docker-compose/confs/
$ cp ~/openair-spgwu-tiny/etc/spgwu.conf ~/oai-docker-compose/confs/
2.2.2 Creating entrypoint files
The example of amf entrypoint.sh is below for other network functions it is analogus.
#!/bin/bash
set -eumb
echo "Running amf to check logs use tail -100f ~/oai-docker-compose/logs/amf.log"
exec nohup /usr/local/bin/amf -c /openair-amf/etc/amf.conf -o >> /openair-amf/etc/amf.log 2>&1
in the above replace amf with nrf,smf to create entrypoints for nrf and smf for spgwu it is slightly different
echo "Running spgwu to check logs use tail -100f ~/oai-docker-compose/logs/spgw_u.log"
exec nohup /usr/local/bin/spgwu -c /openair-spgwu-tiny/etc/spgw_u.conf -o >> /openair-spgwu-tiny/etc/spgw_u.log 2>&1
Create entrypoints for all the network functions which are required.
2.2.3 Healthchecks
The healthchecks can be directly used from here, copy them in the healthchecks
folder.
2.2.4 Creating docker-compose
To run this docker-compose the network demo-oai-public-net
should be created. To know how to create the network follow
version: '3.8'
services:
oai-nrf:
container_name: oai-nrf
image: oai-nrf:develop
entrypoint: ["/openair-nrf/bin/nrf-entrypoint.sh"]
restart: always
volumes:
- ./healthchecks/nrf-healthcheck.sh:/tmp/nrf-healthcheck.sh
- ./confs/nrf.conf:/openair-nrf/etc/nrf.conf
- ./entrypoints/nrf-entrypoint.sh:/openair-nrf/bin/nrf-entrypoint.sh
- ./logs/nrf.log:/openair-nrf/etc/nrf.log:rw
healthcheck:
test: /bin/bash -c "/tmp/nrf-healthcheck.sh"
interval: 10s
timeout: 15s
retries: 5
networks:
public_net:
ipv4_address: 192.168.70.130
mysql:
container_name: mysql
image: mysql:5.7
restart: always
volumes:
- ./oai_db.sql:/docker-entrypoint-initdb.d/oai_db.sql
- ./healthchecks/mysql-healthcheck.sh:/tmp/mysql-healthcheck.sh
depends_on:
- oai-nrf
environment:
- TZ=Europe/Paris
- MYSQL_DATABASE=oai_db
- MYSQL_USER=test
- MYSQL_PASSWORD=test
- MYSQL_ROOT_PASSWORD=linux
healthcheck:
test: /bin/bash -c "/tmp/mysql-healthcheck.sh"
interval: 10s
timeout: 5s
retries: 5
networks:
public_net:
ipv4_address: 192.168.70.131
oai-amf:
container_name: oai-amf
image: oai-amf:develop
entrypoint: ["/openair-amf/bin/amf-entrypoint.sh"]
restart: always
depends_on:
- mysql
volumes:
- ./healthchecks/amf-healthcheck.sh:/tmp/amf-healthcheck.sh
- ./confs/amf.conf:/openair-amf/etc/amf.conf
- ./entrypoints/amf-entrypoint.sh:/openair-amf/bin/amf-entrypoint.sh
- ./logs/amf.log:/openair-amf/etc/amf.log:rw
healthcheck:
test: /bin/bash -c "/tmp/amf-healthcheck.sh"
interval: 10s
timeout: 15s
retries: 5
networks:
public_net:
ipv4_address: 192.168.70.132
oai-smf:
container_name: oai-smf
entrypoint: ["/openair-smf/bin/smf-entrypoint.sh"]
restart: always
image: oai-smf:develop
depends_on:
- oai-amf
volumes:
- ./healthchecks/smf-healthcheck.sh:/tmp/smf-healthcheck.sh
- ./entrypoints/smf-entrypoint.sh:/openair-smf/bin/smf-entrypoint.sh
- ./confs/smf.conf:/openair-smf/etc/smf.conf
- ./logs/smf.log:/openair-smf/etc/smf.log:rw
healthcheck:
test: /bin/bash -c "/tmp/smf-healthcheck.sh"
interval: 10s
timeout: 5s
retries: 5
networks:
public_net:
ipv4_address: 192.168.70.133
oai-spgwu:
container_name: "oai-spgwu"
image: oai-spgwu:develop
entrypoint: ["/openair-spgwu-tiny/bin/spgwu-entrypoint.sh"]
restart: always
depends_on:
- oai-smf
cap_add:
- NET_ADMIN
- SYS_ADMIN
cap_drop:
- ALL
privileged: true
volumes:
- ./healthchecks/spgwu-healthcheck.sh:/tmp/spgwu-healthcheck.sh
- ./entrypoints/spgwu-entrypoint.sh:/openair-spgwu-tiny/bin/spgwu-entrypoint.sh
- ./confs/spgw_u.conf:/openair-spgwu-tiny/etc/spgw_u.conf
- ./logs/spgw_u.log:/openair-spgwu-tiny/etc/spgw_u.log:rw
healthcheck:
test: /bin/bash -c "/tmp/spgwu-healthcheck.sh"
interval: 10s
timeout: 5s
retries: 5
networks:
public_net:
ipv4_address: 192.168.70.134
networks:
public_net:
external:
name: demo-oai-public-net
2.2.5 Playing with docker-compose
# start docker-compose
$: docker-compose -p <project-name> -f <file-name> up -d
# if made changes in the conf files located in ./confs/ restart the container/service
$: docker-compose -p <project-name> -f <file-name> restart <service-name>
# force recreate a service
$: docker-compose -p <project-name> -f <file-name> up -d <service-name> --force-create
# incase the code is present in side the container and some changes are made then just restart the container never remove
$: docker-compose -p <project-name> -f <file-name> restart <service-name>
# stop the containers/service
$: docker-compose -p <project-name> -f <file-name> stop <service-name>
# remove the deployment
$: docker-compose -p <project-name> -f <file-name> down -t 0
Network components configuration is present in ~/oai-docker-compose/confs/
the logs are present in ~/oai-docker-compose/logs/
. There will be only one log file and it will container huge amount of logs. If needed this can also be rotated to avoid having one bulky file. To make it rotating make changes in the entrypoint.sh script
3. Basic debugging
- Building the images in debug mode will provide more information about UE attach-detach process.
- Capturing packets to understand message flow between the components and encapsulation-de-capsulation
- The captured packets can be compared with the end-to-end pcaps which are available in this tutorial
- Check the UE subscription information is available in the Mysql database and the OPC is correctly configured in AMF.
4. How to report an issue?
To report an issue regarding any-component of CN5G or attach-detach procedure follow the below procedure,
- Share the testing scenario, what the test is trying to achieve
- Debug logs of the 5GCN components and packet capture/tcpdump of the 5GCN components. Depending on where the packets are captured take care of interface on which the packets are captured. Also it will be nice to capture packets using a filter
ngap || http || pfcp || gtp
- If you have an issue with testing then you can send an email at openair5g-cn@lists.eurecom.fr with the configuration files, log files in debug mode and pcaps with appropriate filters.
- You can also report an issue or create bug directly on gitlab, to create an account on our gitlab please follow only create account part from here