From 196e0a2c85ddadafaa0a63ad30e4ec62827ccd3c Mon Sep 17 00:00:00 2001 From: Robert Schmidt <robert.schmidt@openairinterface.org> Date: Wed, 8 Nov 2023 16:35:08 +0100 Subject: [PATCH] Rewrite F1 documentation --- doc/F1-design.md | 442 +++++++++++++++++++++++++++-------------------- 1 file changed, 256 insertions(+), 186 deletions(-) diff --git a/doc/F1-design.md b/doc/F1-design.md index 7e491f53461..2666856a426 100644 --- a/doc/F1-design.md +++ b/doc/F1-design.md @@ -12,204 +12,44 @@ </tr> </table> -# 1. Introduction +[[_TOC_]] -We use OpenAirInterface source code, and it's regular deployment scheme. +# Introduction -F1 specification is in 3GPP documents: TS 38.470 header document and all documents listed in chapter 2 References of TS 38.470. +The F1 interface is the functional split of 3GPP between the CU (centralized +unit: PDCP, RRC, SDAP) and the DU (distributed unit: RLC, MAC, PHY). It is +standardized in TS 38.470 - 38.473 for 5G NR. No equivalent for 4G exists. -This document explains how the source code works. +We assume that each DU handles only one cell. Multiple DUs connected to one CU +are supported. Mobility over F1 is not yet supported. -Offered features and deployment instructions are described in: -https://gitlab.eurecom.fr/oai/openairinterface5g/-/wikis/f1-interface +# Control plane status (F1-C) -# 2. Common multi-threading architecture # +## Implementation Status -The CU and DU interfaces are based on ITTI threads (see oai/common/utils/ocp_itti/itti.md) +Note that OAI uses F1 "internally". That means, that even if you run a +monolithic gNB, the internal information exchange uses F1. You can therefore +expect that everything working in a monolithic deployment should also work in +F1. The current implementation is based on R16.3. -All OAI upper layers use the ITTI framework to run isolated threads dedicated to one feature. -The lower layers adopt case by case specific software architecture. +Note that NSA does NOT follow this logic yet, and does not work with F1. -```mermaid -graph TD -Linux[configuration file] --> APP[gNB_app_task] -Socks[Linux UDP] --> GTP[nr_gtpv1u_gNB_task] -APP --> CU -APP --> DU -A[Linux SCTP] -->|sockets| SCTP(OAI itti thread TASK_SCTP) -SCTP-->|TASK_NGAP| NG[ngap_gNB_task] -NG-->|TASK_SCTP| SCTP -SCTP-->|TASK_DU_F1| DU[F1AP_DU_task] -SCTP-->|TASK_CU_F1| CU[F1AP_CU_task] -CU-->|TASK_RRC_GNB| RRC -RRC -->|non ITTI| PHY[OAI L1+L2] -RRC -->|API calls to create tunnels| GTP -PHY --> |user plane| GTP -CU --> |TASK_GTPV1_U| GTP -DU --> |TASK_GTPV1_U| GTP -``` -A "task" is a Linux thread running infinite waiting loop on one ITTI queue - -app task manages the initial configuration - -sctp task mamanges the SCTP Linux sockets - -CU code runs in the task F1AP_CU_task. It stores it's private data context in a static variable. No mutex is used, as the single interface to CU ITTI tast in incoming ITTI messages. CU task is a single ITTI queue, but multiple CU could exist in one Linux process, using the "instance id" (called "module id" also) to create separate contextual information for each CU instance. - -DU code runs in the task F1AP_DU_task. The design is similar to CU task - -All GTP-U tunnels are managed in a Linux Thread, that have partially ITTI design: - -1. tunnel creation/deletion is by C API functions direct calls (a mutex protects it) -2. outgoing packets are pushed in a ITTI queue to the gtp thread -3. incoming packets are sent to the tunnel creator using a C callback (the callback function is given in tunnel creation order). The callback should not block - - -# 3. CU F1AP messages # - -## startup - -CU thread starts when the CU starts. It opens listening socket on F1AP_PORT_NUMBER by sending the appropriate message to TASK_SCTP. - -It will accept the first incoming connection on this port - -## F1 SETUP - -DU has connected, and sent this message. - -The CU receives, decode it and send relevant information to RRC thread by message F1AP_SETUP_REQ - -## F1 SETUP response - -RRC sends two messages to CU F1 task: the information to encode the F1 setup response and a second message to encode F1AP_GNB_CU_CONFIGURATION_UPDATE - -Upon reception of these two messages the CU task encodes and sends the messages to the DU - -## UE entry - -When CCH channel decoding finds a new UE identity, it call rrc_gNB_generate_RRCSetup() that will send F1AP_DL_RRC_MESSAGE to the CU task. - -The CU encodes and sends to DU DLRRCMessageTransfer - -## UE attach/detach - -When rrc_gNB_process_RRCReconfigurationComplete() occurs, the RRC sends to CU task the message F1AP_UE_CONTEXT_SETUP_REQ. The CU task encodes the F1 message UEContextSetup and send it to the DU over F1AP. - -This function rrc_gNB_process_RRCReconfigurationComplete() also creates the GTP-U user plane bearers. - -## F1AP incoming messages - -SCTP tasks sends a ITTI message SCTP_DATA_IND to the CU task. - -A array of functions pointers and the F1AP standard identifier "procedureCode", the CU calls the appropriate function - -Hereafter the most significant messages processing - -### CU_handle_F1_SETUP_REQUEST - -Transcodes information to the same message toward RRC (F1AP_SETUP_REQ) - -### CU_handle_INITIAL_UL_RRC_MESSAGE_TRANSFER - -Transcodes information to the same message toward RRC (NR_RRC_MAC_CCCH_DATA_IND) - -### CU_handle_UL_RRC_MESSAGE_TRANSFER - -Encode and send data to PDCP (calling pdcp_data_ind ()) for processing UL data. - -# DU F1 Messages - -## Startup - -The task "gNB app" after rzading the configuration file, sends a first message F1AP_SETUP_REQ to DU task - -Using this message, the uniq DU task (Linux Thread) creates a DU instance context memory space, calls SCTP task to create a socket to the CU. - -When it receives from the SCTP task the socket creation success, the DU task encodes+sends the F1 setup message to the CU. - -## F1AP_INITIAL_UL_RRC_MESSAGE - -When MAC layer push UL data to DU task, the DU sends to CU the message InitialULRRCMessageTransfer - -## F1AP_UL_RRC_MESSAGE - -In case of SRB bearer, the RLC layer use this message to push rlc bearer payload to DU task, that forward to CU - -## pdcp_data_ind - -This function call in RLC, when it is a DRB will be replaced by F1-U transport to CU - -## F1AP incoming messages - -SCTP tasks sends a ITTI message SCTP_DATA_IND to the DU task. - -A array of functions pointers and the F1AP standard identifier "procedureCode", the DU calls the appropriate function - -Hereafter the most significant messages processing - -### DU_send_F1_SETUP_RESPONSE - -Decode the CU message, then encode a message to TASK_GNB_APP. The thread "APP" extracts the system information block and reconfigure the DU with: du_extract_and_decode_SI(), configure_gnb_du_mac() - -### DU_handle_gNB_CU_CONFIGURATION_UPDATE - -No developped +The following messages are currently implemented: -### DU_handle_UE_CONTEXT_SETUP_REQUEST +- F1 Setup Request and F1 Setup Response/Failure +- Initial UL RRC, UL/DL RRC Message Transfer +- F1 UE Context management messages for the "good case" + * Setup Request/Response + * Modification Request/Response + * Release Request/Command/Complete -Push data to RLC layer for transmisstion in DL (to the UE) by calling enqueue_rlc_data_req() +All other messages are not implemented or not working reliably: -### DU_handle_DL_RRC_MESSAGE_TRANSFER +- F1 CU/DU configuration updade +- Paging messages +- Warning Message transmissions -Depending on the content, adds UE context in DU low layers and/or send data to the UE with rlc_data_req() - -# F1-U messages - -The data path by itself doesn't make any difficulty: -1. for UL, rlc calls pdcp_data_ind (DRB) or sends a message to RRC (SRB). So, we can replace pdcp_data_ind() by du_pdcp_data_ind() that will push the message in the GTP tunnel. In CU, assuming the tunnel is in place, the right call back decapsulate the transport (cu_ul_recv()), then calls pdcp_data_ind() in CU -The SRB are sent in DU RRC tasks via itti F1AP_UL_RRC_MESSAGE (see above the processing) -2. For DL, pdcp (in CU for DRBs) calls rlc_data_req (by a decoupling queue and thread rlc_data_req_thread, so function du_rlc_data_req() or enqueue_rlc_data_req()). in DU, assuming the gtp-u tunnel exists, a reception call back (du_dl_recv()) decapsulate the gtp-u packet and calls du_rlc_data_req() - -Hereafter the processing design, that doesn't require to setup a new context storage, as we can use the GTP-U internal tables: rnti+rb=>teid for outgoing gtp-u packets and teid=>rnti+rb for incoming gtp-u packets - -In CU case, the DRB tunnel to DU and the tunnel on N3 have the same key (rnti, rb id), but they run in two different GTP-U instances. -Each instance binds on diffrent sockets. - -For F1-U, both DU and CU binds on the full quadruplet (IP source, port source, IP destination, port destination) from the configuration file parameters: -local_s_address, remote_s_address,local_s_portd, remote_s_portd - -These 4 values are in: -CU: at cell level (same level as nr_cellid parameter), they are read if tr_s_preference = "f1"; is set -DU: in block MACRLCs, if tr_n_preference = "f1"; is set in this block - -## tunnels setup -In GTP-U, TS 29.281 specifies a option header (NR RAN Container This extension header may be transmitted in a G-PDU over the X2-U, Xn-U and F1-U user plane interfaces), but it is not mandatory optional header (as is the N3 one). - -So, we can use this header a something reliable with other vendors implementation. - -In DL, we need to associate the packet with: rnti, rb (radio bearer) and "mui" (a OAI internal id that is optional and may be removed soon) to call rlc_data_req() - -In UL, we need also the RNTI, the rb - -So, we need to create a gtp-u tunnel for each rb over F1, then manage in CU and DU the association between a uniq TEid and the pair(rnti, rb). This is already what we have in existing OAI GTP-U layer interface. - -In F1AP, for each "DRB to Be Setup Item IEs", we have a field TNL (transport network layer) to set a specific GTP tunnel (@IP, TEid, udp port is always 2152). This is the same for each message related to DRBs. (F1AP carries the SRB payload inside F1AP). - -So, For each F1AP containing DRB setup/modification/deletion, the related GTP-U tunnels will be modified one to one. -The exception is the intialisation of new tunnel: in the call to tunnel creation, we need to send the remote TEID, but we don't know yet if we are the initial source. In this case, we issue a dummy (0xFFFF) remote teid; when we receive the remote answer, we get the source teid, that we can inform GTP-U with (using update tunnel). - -## processing on GTP-U incoming message - -Du_dl_recv() or cu_ul_recv() are called because we have set this callbacks in creat/update tunnel calls to gtp-u. As gtp-u maintains a pair(rnti,rbid) associated to each tunnel, the functions can be very simple stateless callback that calls pdcp_data_ind() or rlc_data_req() to get back in normal processing path - -## CU/DU outgoing - -DU rlc north output goes to du_pdcp_data_ind() that push a outgoing packet request to gtp-u, using the rnti+rb as ids. GTP-U must have the tunnel existing, then it simply implement the same as in N3 for exemple. - -CU pdcp south output calls cu_rlc_data_ind() that do the same symetric processing. - - -# High-level F1-C code structure +## High-level F1-C code structure The F1 interface is used internally between CU (mostly RRC) and DU (mostly MAC) to exchange information. In DL, the CU sends messages as defined by the @@ -252,3 +92,233 @@ Monolithic: mac_rrc_dl_direct.c | | | | +-------------+ ``` + +# Data plane status (F1-U) + +F1-U uses GTP-U for information exchange. The GTP protocol uses some extensions +for NR-U protocol to report buffer status, but this is not properly tested and +might malfunction. One limitation is that the current implementation +acknowledges each packet individually. + +Multiple DUs at one CU are supported in the data plane (as in the control plane). + +# How to run + +As mentioned earlier, OAI uses F1 internally. It is always compiled in. +To start CU/DU, you use `./nr-softmodem` with the appropriate configuration +files, for instance: + +``` +sudo cmake_targets/ran_build/build/nr-softmodem --sa -O ci-scripts/conf_files/gnb-cu.sa.band78.106prb.conf +sudo cmake_targets/ran_build/build/nr-softmodem --sa -O ci-scripts/conf_files/gnb-du.sa.band78.106prb.rfsim.conf +``` + +These files are tested in the CI, and are configured for use in docker, +see [this `docker-compose` file](../ci-scripts/yaml_files/5g_f1_rfsimulator/docker-compose.yaml). + +## Configuration of F1 IP/port information + +For a local deployment, you should update the following fields. +We assume that the CU will bind on `192.168.70.129` towards the core, +`127.0.0.3` towards the DU, and the DU `127.0.0.4` towards the CU. + +In the CU file: +- Update the `gNBs.[0].amf_ip_address` and `gNBs.[0].NETWORK_INTERFACES` + section towards the core (typically, OAI CN is configured to provide a docker + bridge on `192.168.70.129` and the AMF is on `192.168.70.132`): + - `gNBs.[0].amf_ip_address.[0].ipv4 192.168.70.132` + - `gNBs.[0].NETWORK_INTERFACES.GNB_IPV4_ADDRESS_FOR_NG_AMF 192.168.70.129` + - `gNBs.[0].NETWORK_INTERFACES.GNB_IPV4_ADDRESS_FOR_NGU 192.168.70.132` +- Set `gNBs.[0].tr_s_preference` (transport south-bound) to `f1` +- Update the `gNBs.[0].local_s_address` (CU-local south address) for the F1-C + south-bound interface: `127.0.0.3` +- Note: the `gNBs.[0].remote_s_address` (CU-remote/DU south address) is + ignored, but we recommend to put `0.0.0.0` ("any") +- Ports should match the ones in the DU config, but for simplicity and + standards-conformity, simply set all to 2152: + - `local_s_portc` in CU should match `remote_n_portc` in DU + - `local_s_portd` in CU should match `remote_n_portd` in DU + - `remote_s_portc` in CU should match `local_n_portc` in DU + - `remote_s_portd` in CU should match `local_n_portd` in DU + +In the DU file: +- Set `MACRLCs.[0].tr_n_preference` to `f1` +- Update `MACRLCs.[0].local_n_address` (local north-bound address of the DU) to + `127.0.0.4` +- Update `MACRLCs.[].remote_n_address` (remote north-bound address of the CU) + to `127.0.0.3` + +Note: all `local_*_if_name` parameters are ignored. + +## Configuration of multiple DUs + +Upon F1 Setup Request of a new DU, the CU cross-checks that + +- no DUs have the same gNB-DU IDs, taken from `gNBs.[0].gNB_ID`, and +- no cells have the same NR Cell ID, taken from `gNBs.[0].nr_cellid`, and +- no cells have the same physical cell IDs, taken from `gNBs.[0].servingCellConfigCommon.[0].physCellId`. + +If any of these are the same, the CU will reject the DU with an F1 Setup +Failure. Thus, you should make sure that in the DU's configs, these parameters +are set differently. + +You have to of course make sure that the local interface of the DU +(`MACRLCs.[0].local_n_address`) is different as well. + +Assuming you use RFsim, you should make the RFsimulator server side (typically +the gNB) bind on different hosts (`rfsimulator.serverport`). + +# Code documentation + +## Common multi-threading architecture + +The CU and DU interfaces are based on ITTI threads (see `common/utils/ocp_itti/itti.md`) +adopted by all OAI upper layers to run isolated threads dedicated to one feature. + +```mermaid +graph TD +Linux[configuration file] --> APP[gNB_app_task] +Socks[Linux UDP] --> GTP[nr_gtpv1u_gNB_task] +APP --> CU +APP --> DU +A[Linux SCTP] -->|sockets| SCTP(OAI itti thread TASK_SCTP) +SCTP-->|TASK_NGAP| NG[ngap_gNB_task] +NG-->|TASK_SCTP| SCTP +SCTP-->|TASK_DU_F1| DU[F1AP_DU_task] +SCTP-->|TASK_CU_F1| CU[F1AP_CU_task] +CU-->|TASK_RRC_GNB| RRC +RRC -->|non ITTI| PHY[OAI L1+L2] +RRC -->|API calls to create tunnels| GTP +PHY --> |user plane| GTP +CU --> |TASK_GTPV1_U| GTP +DU --> |TASK_GTPV1_U| GTP +``` + +A "task" is a Linux thread running an infinite waiting loop on one ITTI queue. + +The `app` task manages the initial configuration; the `sctp` task manages the +SCTP Linux sockets. + +The CU encoding/decoding runs in the task executing `F1AP_CU_task()`. It stores its private data context in a static variable. +The DU encoding/decoding runs in the task executing `F1AP_DU_task()`. The design is similar to CU task. + +All GTP-U tunnels are managed in a Linux Thread, that have partially ITTI design: + +1. tunnel creation/deletion is by C API functions direct calls (a mutex protects it) +2. outgoing packets are pushed in a ITTI queue to the gtp thread +3. incoming packets are sent to the tunnel creator using a C callback (the callback function is given in tunnel creation order). The callback should not block + + +## F1-C messages towards the CU + +The CU thread starts when the CU starts. It opens listening socket on the +configuration-specified IP/port by sending the appropriate message to +`TASK_SCTP`. It will accept any incoming connection and forward any F1 message +to RRC. + +You can trace any message by looking up the handler +`cu_task_handle_sctp_data_ind()`, which ultimately triggers a corresponding +ITTI message sent to RRC. All ITTI messages arriving are dispatched in the ITTI +message handling function in `rrc_gNB.c`. For your convenience, find below a +table of same messages that might arrive from the DU and how they are +forwarded. + +After handling messages, the RRC uses a callback that determines if messages go +directly to the MAC or are sent to the CU ITTI task (see +[above](#high-level-f1-c-code-structure)) + +You might also want to consult TS 38.401 regarding the message exchange. + +| Incoming F1 message | ITTI message to RRC | RRC handler | Comments | +|---------------------|---------------------|-------------|----------| +| F1 Setup Request | `F1AP_SETUP_REQ` | `rrc_gNB_process_f1_setup_req()` | will trigger either a response or failure | +| Initial UL RRC Message Transfer | `F1AP_INITIAL_UL_RRC_MESSAGE` | `rrc_gNB_process_initial_ul_rrc_message()` | first message from UE, follows up with RRC Setup or Reestablishment | +| UL RRC Message Transfer | `F1AP_UL_RRC_MESSAGE` | `rrc_gNB_decode_dcch()` | UL RRC messages contain DCCH | +| UE Context Response | `F1AP_UE_CONTEXT_SETUP_RESP` | `rrc_CU_process_ue_context_setup_response()` | does not trigger a follow-up, as UE sends Security Response | +| UE Context Modification Response | `F1AP_UE_CONTEXT_MODIFICATION_RESP` | `rrc_CU_process_ue_context_modification_response()` | triggers Reconfiguration if CellGroup in answer | +| UE Context Modification Required | `F1AP_UE_CONTEXT_MODIFICATION_REQUIRED` | `rrc_CU_process_ue_modification_required()` | triggers Reconfiguration | +| UE Context Release Request | `F1AP_UE_CONTEXT_RELEASE_REQ` | `rrc_CU_process_ue_context_release_request()` | RRC will trigger UE release | +| UE Context Release Complete | `F1AP_UE_CONTEXT_RELEASE_COMPLETE` | `rrc_CU_process_ue_context_release_complete()` | frees UE Context, signals to NGAP | + +## F1-C Messages towards the DU + +The task "gNB app", after reading the configuration file, sends a message +`F1AP_DU_REGISTER_REQ` to the DU task. This message contains network +configuration for the establishment of the SCTP connection, as well as the F1 +Setup Request to be sent to the CU. + +Using the network configuration, the DU task sets up an SCTP connection via the +SCTP task. When it receives from the SCTP task the socket creation success, +the DU task encodes and sends the F1 setup message to the CU. + +Similarly as for the CU, you can trace any message by looking up the handler +`du_task_handle_sctp_data_ind()`. Unlike in the CU, the decoders do NOT send an +ITTI message, but directly call into the scheduler via a message handling +function. This handler acquires the scheduler mutex and executes the +corresponding message handling functionality. All handlers are in +`mac_rrc_dl_handler.c`. + +After handling messages, the MAC uses a callback that determines if messages +are sent back to RRC via ITTI, or are sent to the DU ITTI task (see +[above](#high-level-f1-c-code-structure)) + +You might also want to consult TS 38.401 regarding the message exchange. + +| Incoming F1 message | MAC handler | Comments | +|---------------------|-------------|----------| +| F1 Setup Response | `f1_setup_response()` | the DU does not start the radio before receiving this message | +| DL RRC Message Transfer | `dl_rrc_message_transfer()` | Processing timer started if reconfiguration is present[^footnote-reconfig] | +| UE Context Setup Request | `ue_context_setup_request()` | | +| UE Context Modification Request | `ue_context_modification_request()` | | +| UE Context Modification Confirm | `ue_context_modification_confirm()` | | +| UE Context Modification Refuse | `ue_context_modification_refuse()` | Will trigger release request | +| UE Context Release Command | `ue_context_release_command()` | Starts timer for release if UE in sync | + +[^footnote-reconfig]: The DU does not decode RRC messages, and does not know + whether an RRC message is a reconfiguration. However, the spec mandates that + a reconfiguration has to be triggered if the CU receives a CellGroupConfig, + originating at the DU. See also flag `expect_reconfiguration`. + +## F1-U messages + +### General + +In the DU in UL, RLC checks in `deliver_sdu()` if we are operating in split +mode, and either (direct) calls `pdcp_data_ind` (DRB) or (f1ap) sends an +`GTPV1U_TUNNEL_DATA_REQ` ITTI message to the GTP task. + +In the CU in UL, assuming the tunnel is in place, GTP decapsulates the packet +and calls the callback `cu_f1u_data_req()`, which calls `pdcp_data_ind()` in CU. + +In the CU in DL, the PDCP function `deliver_pdu_drb_gnb()` either (direct) calls +into the RLC via `enqueue_rlc_data_req()`, or (f1ap) sends a +`GTPV1U_TUNNEL_DATA_REQ` ITTI message to the GTP task. + +In the DU in DL, assuming the GTP-U tunnel exists, GTP decapsulates the packet +and calls the reception call back `du_rlc_data_req()`, which calls +`enqueue_rlc_data_req()` in turn. + +## Tunnel Setup + +In GTP-U, TS 29.281 specifies a optional header (NR RAN Container). This +extension header may be transmitted in a G-PDU over the X2-U, Xn-U and F1-U user +plane interfaces, but it is not mandatory. + +Note that the GTP-U uses internal tables. Instead of exposing TEIDs, it uses the +UE ID (RNTI, RRC UE ID) and Radio Bearer ID to map to a TEID for outgoing GTP-U +packets. Upon receiving, it does the reverse and calls the callbacks. In the CU +case, the DRB tunnel to DU and the tunnel on N3 have the same keys (RRC UE ID, +RB ID), but they run in two different GTP-U instances. Each instance binds on +diffrent sockets, which is the reason you cannot have the same interface for +F1-U and N3/NG-U interfaces (this can be considered a design flaw). + +In F1AP, for each "DRB to Be Setup Item IEs", we have a field TNL (transport +network layer) to set a specific GTP tunnel (@IP, TEid). This is the same for +each message related to DRBs. + +So, For each F1AP containing DRB setup/modification/deletion, the related GTP-U +tunnels will be modified one to one. The exception is the intialisation of new +tunnel: in the call to tunnel creation, we need to send the remote TEID, but we +don't know yet if we are the initial source. In this case, we issue a dummy +(0xFFFF) remote TEID; when we receive the remote answer, we get the source teid, +with which the GTP-U endpoint is updadet. -- GitLab