DEBUG_5G_CORE.md

OpenAirInterface 5G Core Network Deployment: How to debug the Network Functions?

Currently there is no special tool for debugging the problems of core network components. Though there are certain methods which we are using in our team to diagnose an issue.

This page content expects you to have read the deployment pre-requisites and try to deploy a mini or basic normal deployment.

TABLE OF CONTENTS

1. Building images in debug mode
2. Debuggers deployment of core network functions
3. Basic debugging
4. How to report an issue?

1. Building images in debug mode

By default all the dockerfiles present in any network function repository (AMF, SMF, NRF, UPF) are built with release tag. In the release mode the logging information is limited. This is done to reduce the image size and have a better performance. In beginning when a user is trying to understand the functioning of core network it is recommended to build the images in debug mode. This way user will have more logs and can have better understanding. To build any core network image in debug mode follow the below steps after cloning the network function repository, the example is for AMF,

# clone amf repository 
$ git clone -b <prefered_branch or develop> https://gitlab.eurecom.fr/oai/cn5g/oai-cn5g-amf.git
$ cd oai-cn5g-amf/docker/
# Depending on the environment where the image will be used choose the correct dockerfile
$ vi/vim/nano/subl Dockerfile.amf.ubuntu18
# replace the line RUN ./build_amf --clean --Verbose --build-type Release --jobs with 
# RUN ./build_amf --clean --Verbose --build-type Debug --jobs

The same can be done for baremetal deployment of any core network function, just build with Debug tag in ./build_amf. There will be no change in running the core network function. Everything will be the same, now when the network function is started there will be extra logs with [debug] tag apart from the normal [info ]

1.1 Building the image with code inside (Only for development purpose)

If you are interested in doing development you can leave the code inside the container. This is good for developers, they can code in docker-environment. They can even mount the code as a volumne so that they can use their prefered editor (though vim/nano/vi are the best). To do this the dockerfile has to be edited and only the BUILDER IMAGE of the file is required the rest TARGET IMAGEpart including CMD and ENTERYPOINT should be removed.

2. Debuggers deployment of core network functions

It is really important to safely keep the logs and configuration of core network components in case of error. If they are deployed in baremetal the logs and the configuration can be easily retrived based on how the core network is started. Here are some tips related to running/deploying core network in different environment,

2.1 Deploying as a process (baremetal deployment prefered by developers)

1. In case of all in one sort of process deployment there can some conflicting dependencies between different components of core network. These conflicts have to be resolved on case by case bases by the user.
2. Once all the core-network components are build in debug mode with there dependencies store the logs in a file rather than printing on the terminal.
3. All the configuration file should be stored in one place so that it is easy to reterive.

# storing logs and configuration in your prefered location
nohup /usr/local/bin/amf -c /tmp/oai-cn-5g/config/amf.conf -o >> /tmp/oai-cn-5g/logs/amf.log 2>&1

The above command will launch the network function in background and all the logs can be seen using tail -100f /tmp/oai-cn-5g/logs/amf.log. In case to stop the component kill the its process process using pkill <component-name> or ps -eaf

2.2 Docker environment (Recommeneded)

Using docker environment for deployment and development is the prefered environment because there it is easy to have dedicated working environment for each network component. It is lightweight and easy to manage. The docker-compose provided in earlier tutorials is good for learning how the OAI core network works and how to use it. But if the user wants to change some parameters which are not variable or not allowed using docker-compose then it is hard to use docker-compose approach. If the user wants to provide their own configuration file then it is better to change the docker-compose. Follow the below steps to create a new developer/debugger specific docker-compose,

2.2.1 Prerequisites

1. Build the docker-images in debug mode following the previous section

2. Create a new folder oai-docker-compose

3. In the oai-docker-compose folder create confs, logs, entrypoints and healthchecks folder

4. In the confs folder copy the configuration files for each component you want to use. The configuration files are located here

   File Name	Repository	Location
   amf.conf	(Gitlab) cn5g/oai-cn5g-amf	etc/amf.conf
   smf.conf	(Gitlab) cn5g/oai-cn5g-smf	etc/smf.conf
   nrf.conf	(Gilab) cn5g/oai-cn5g-nrf	etc/nrf.conf
   spgw_u.conf	(Github) OPENAIRINTERFACE/openair-spgwu-tiny	etc/spgw_u.conf

5. Create empty log files for the component you want to use in the logs folder using touch command example

6. Copy oai_db.sql make user database depending on the IMSI, LTKEY and Opc. New user entry can be added after this line

# Creating directory structure
$ mkdir ~/oai-docker-compose
$ mkdir -p ~/oai-docker-compose/confs
$ mkdir -p ~/oai-docker-compose/logs
$ mkdir -p ~/oai-docker-compose/entrypoints
$ mkdir -p ~/oai-docker-compose/healthchecks 

# Creating empty log files
$ touch ~/oai-docker-compose/logs/amf.log
$ touch ~/oai-docker-compose/logs/smf.log
$ touch ~/oai-docker-compose/logs/nrf.log
$ touch ~/oai-docker-compose/logs/spgwu.log

# Copying the configuration files, if you have yours you can use that else copy from the repository and make changes manually
$ cp ~/oai-cn5g-amf/etc/amf.conf ~/oai-docker-compose/confs/
$ cp ~/oai-cn5g-smf/etc/smf.conf ~/oai-docker-compose/confs/
$ cp ~/oai-cn5g-nrf/etc/nrf.conf ~/oai-docker-compose/confs/
$ cp ~/openair-spgwu-tiny/etc/spgwu.conf ~/oai-docker-compose/confs/

2.2.2 Creating entrypoint files

The example of amf entrypoint.sh is below for other network functions it is analogus.

#!/bin/bash
set -eumb

echo "Running amf to check logs use tail -100f ~/oai-docker-compose/logs/amf.log"
exec nohup /usr/local/bin/amf -c /openair-amf/etc/amf.conf -o >> /openair-amf/etc/amf.log 2>&1

in the above replace amf with nrf,smf to create entrypoints for nrf and smf for spgwu it is slightly different

echo "Running spgwu to check logs use tail -100f ~/oai-docker-compose/logs/spgw_u.log"
exec nohup /usr/local/bin/spgwu -c /openair-spgwu-tiny/etc/spgw_u.conf -o >> /openair-spgwu-tiny/etc/spgw_u.log 2>&1

Create entrypoints for all the network functions which are required.

2.2.3 Healthchecks

The healthchecks can be directly used from here, copy them in the healthchecks folder.

2.2.4 Creating docker-compose

To run this docker-compose the network demo-oai-public-net should be created. To know how to create the network follow

version: '3.8'
services:
    oai-nrf:
        container_name: oai-nrf
        image: oai-nrf:develop
        entrypoint: ["/openair-nrf/bin/nrf-entrypoint.sh"]
        restart: always
        volumes:
            - ./healthchecks/nrf-healthcheck.sh:/tmp/nrf-healthcheck.sh
            - ./confs/nrf.conf:/openair-nrf/etc/nrf.conf
            - ./entrypoints/nrf-entrypoint.sh:/openair-nrf/bin/nrf-entrypoint.sh
            - ./logs/nrf.log:/openair-nrf/etc/nrf.log:rw
        healthcheck:
            test: /bin/bash -c "/tmp/nrf-healthcheck.sh"
            interval: 10s
            timeout: 15s
            retries: 5
        networks:
            public_net:
                ipv4_address: 192.168.70.130
    mysql:
        container_name: mysql
        image: mysql:5.7
        restart: always
        volumes:
            - ./oai_db.sql:/docker-entrypoint-initdb.d/oai_db.sql
            - ./healthchecks/mysql-healthcheck.sh:/tmp/mysql-healthcheck.sh
        depends_on:
            - oai-nrf
        environment:
            - TZ=Europe/Paris
            - MYSQL_DATABASE=oai_db
            - MYSQL_USER=test
            - MYSQL_PASSWORD=test
            - MYSQL_ROOT_PASSWORD=linux
        healthcheck:
            test: /bin/bash -c "/tmp/mysql-healthcheck.sh"
            interval: 10s
            timeout: 5s
            retries: 5
        networks:
            public_net:
                ipv4_address: 192.168.70.131
    oai-amf:
        container_name: oai-amf
        image: oai-amf:develop
        entrypoint: ["/openair-amf/bin/amf-entrypoint.sh"]
        restart: always
        depends_on:
            - mysql
        volumes:
            - ./healthchecks/amf-healthcheck.sh:/tmp/amf-healthcheck.sh
            - ./confs/amf.conf:/openair-amf/etc/amf.conf
            - ./entrypoints/amf-entrypoint.sh:/openair-amf/bin/amf-entrypoint.sh
            - ./logs/amf.log:/openair-amf/etc/amf.log:rw
        healthcheck:
            test: /bin/bash -c "/tmp/amf-healthcheck.sh"
            interval: 10s
            timeout: 15s
            retries: 5
        networks:
            public_net:
                ipv4_address: 192.168.70.132
    oai-smf:
        container_name: oai-smf
        entrypoint: ["/openair-smf/bin/smf-entrypoint.sh"]
        restart: always
        image: oai-smf:develop
        depends_on:
            - oai-amf
        volumes:
            - ./healthchecks/smf-healthcheck.sh:/tmp/smf-healthcheck.sh
            - ./entrypoints/smf-entrypoint.sh:/openair-smf/bin/smf-entrypoint.sh
            - ./confs/smf.conf:/openair-smf/etc/smf.conf
            - ./logs/smf.log:/openair-smf/etc/smf.log:rw
        healthcheck:
            test: /bin/bash -c "/tmp/smf-healthcheck.sh"
            interval: 10s
            timeout: 5s
            retries: 5
        networks:
            public_net:
                ipv4_address: 192.168.70.133
    oai-spgwu:
        container_name: "oai-spgwu"
        image: oai-spgwu:develop
        entrypoint: ["/openair-spgwu-tiny/bin/spgwu-entrypoint.sh"]
        restart: always
        depends_on:
            - oai-smf
        cap_add:
            - NET_ADMIN
            - SYS_ADMIN
        cap_drop:
            - ALL
        privileged: true
        volumes:
            - ./healthchecks/spgwu-healthcheck.sh:/tmp/spgwu-healthcheck.sh
            - ./entrypoints/spgwu-entrypoint.sh:/openair-spgwu-tiny/bin/spgwu-entrypoint.sh
            - ./confs/spgw_u.conf:/openair-spgwu-tiny/etc/spgw_u.conf
            - ./logs/spgw_u.log:/openair-spgwu-tiny/etc/spgw_u.log:rw
        healthcheck:
            test: /bin/bash -c "/tmp/spgwu-healthcheck.sh"
            interval: 10s
            timeout: 5s
            retries: 5
        networks:
            public_net:
                ipv4_address: 192.168.70.134
networks:
    public_net:
        external:
           name: demo-oai-public-net

2.2.5 Playing with docker-compose

# start docker-compose
docker-compose -p <project-name> -f <file-name> up -d
# if made changes in the conf files located in ./confs/ restart the container/service
docker-compose -p <project-name> -f <file-name> restart <service-name>
# force recreate a service 
docker-compose -p <project-name> -f <file-name> up -d <service-name> --force-create
# incase the code is present in side the container and some changes are made then just restart the container never remove
docker-compose -p <project-name> -f <file-name> restart <service-name>
# stop the containers/service
docker-compose -p <project-name> -f <file-name> stop <service-name>
# remove the deployment
docker-compose -p <project-name> -f <file-name> down

Network components configuration is present in ~/oai-docker-compose/confs/ the logs are present in ~/oai-docker-compose/logs/. There will be only one log file and it will container huge amount of logs. If needed this can also be rotated to avoid having one bulky file. To make it rotating make changes in the entrypoint.sh script

3. Basic debugging

1. Building the images in debug mode will provide more information about UE attach-detach process.
2. Capturing packets to understand message flow between the components and encapsulation-de-capsulation
3. The captured packets can be compared with the end-to-end pcaps which are available in this tutorial
4. Check the UE subscription information is available in the Mysql database and the OPC is correctly configured in AMF.

4. How to report an issue?

To report an issue regarding any-component of CN5G or attach-detach procedure follow the below procedure,

1. Share the testing scenario, what the test is trying to achieve
2. Debug logs of the 5GCN components and packet capture/tcpdump of the 5GCN components. Depending on where the packets are captured take care of interface on which the packets are captured. Also it will be nice to capture packets using a filter ngap || http || pfcp || gtp
3. If you have an issue with testing then you can send an email at openair5g-cn@lists.eurecom.fr with the configuration files, log files in debug mode and pcaps with appropriate filters.
4. In case you want to report a bug in the code of any network function then you can do it directly via gitlab.