DEPLOY_SA5G_HC.md 13.4 KB
Newer Older
arora's avatar
arora committed
1 2 3 4 5 6 7 8 9 10 11 12 13 14
<table style="border-collapse: collapse; border: none;">
  <tr style="border-collapse: collapse; border: none;">
    <td style="border-collapse: collapse; border: none;">
      <a href="http://www.openairinterface.org/">
         <img src="./images/oai_final_logo.png" alt="" border=3 height=50 width=150>
         </img>
      </a>
    </td>
    <td style="border-collapse: collapse; border: none; vertical-align: center;">
      <b><font size = "5">OpenAirInterface 5G Core Network Deployment using Helm Charts</font></b>
    </td>
  </tr>
</table>

arora's avatar
arora committed
15 16 17 18 19 20 21


The cloud native network functions in production will be deployed using a production grade container orchestrator like kubernetes or Openshift. OAI 5g Core network is being designed to keep up with the latest cloud native deployment scenarios. This release of helm charts is focused on deploying each Cloud-native Network Function(CNF) individually.

![Helm Chart Deployment](./images/helm_diag.png)


arora's avatar
arora committed
22 23 24 25 26
**Reading time: ~40mins**

**Tutorial replication time: ~1h30mins**


arora's avatar
arora committed
27 28 29
**TABLE OF CONTENTS**

1.  [Description](#1-description)
arora's avatar
arora committed
30
2.  [Building Images](#2-building-images)
arora's avatar
arora committed
31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46
3.  [Configuring Helm Charts](#3-configuring-helm-charts)
4.  [Deploying Helm Charts](#4-deploying-helm-charts)


## 1. Description

The helm charts can be used on any production grade kubernetes cluster. Currently they are only tested on our inhouse Openshift cluster the cluster information can be found below.

| Software                 | Version                             |
|:------------------------ |:----------------------------------- |
| Openshift Client Version | 4.4.10                              |
| Kubernetes Version       | Kubernetes Version: v1.17.1+45f8ddb |
| helm                     | v3.5.3                              |

### Pre-requisite

arora's avatar
arora committed
47
The cluster on which these helm charts will be deployed should have RBAC and [Multus CNI](https://github.com/k8snetworkplumbingwg/multus-cni). Multus is necessary to provide multiple interfaces to AMF and UPF/SPGWU. 
arora's avatar
arora committed
48 49 50

## 2. Building Images

arora's avatar
arora committed
51
The base image used by network function is dependent on the operating system it will be running on. If it is a debian (ubuntu) based cluster then base image will be ubuntu. If it is a RPM (core-os) based cluster then base images will UBI8. Follow the tutorial on [how to build images](./BUILD_IMAGES.md) depending on the cluster/worker-node operating system. In case of Ubuntu based worker nodes, the images can be pulled from [docker-hub](./RETRIEVE_OFFICIAL_IMAGES.md).
arora's avatar
arora committed
52 53 54

## 3. Configuring Helm Charts

arora's avatar
arora committed
55
Clone the helm chart repository from gitlab repository
arora's avatar
arora committed
56 57

```
arora's avatar
arora committed
58 59
$ git clone https://gitlab.eurecom.fr/oai/cn5g/oai-cn5g-fed.git
$ git checkout v1.1.0 #check the tag v1.1.0
arora's avatar
arora committed
60 61 62 63 64
$ cd charts
$ ls charts
mysql  oai-amf  oai-nrf  oai-smf  oai-spgwu-tiny
```

arora's avatar
arora committed
65
Helm chart of every network function looks similar and has the below structure. Only the chart of mysql database is different and the NRF helm chart has an extra pvc.yaml to create a presistant volume for storing tcpdump.
arora's avatar
arora committed
66 67 68 69 70

```
Network_function/
├── Chart.yaml
├── templates
arora's avatar
arora committed
71 72 73 74 75 76 77 78
│             ├── configmap.yaml
│             ├── deployment.yaml
│             ├── _helpers.tpl
│             ├── multus.yaml
│             ├── NOTES.txt
│             ├── rbac.yaml
│             ├── serviceaccount.yaml
|             └── service.yaml
arora's avatar
arora committed
79 80 81 82 83
└── values.yaml 

1 directory, 10 files
```

arora's avatar
arora committed
84
All the configurable parameters for a particular commit/release are mentioned in the `values.yaml` file. These parameters will keep on changing in the future depending on the nature of development and features. 
arora's avatar
arora committed
85

arora's avatar
arora committed
86
**NOTE**: If there is a need to edit a specific configuration parameter that is not configurable via the helm-chart values.yaml file then it has to be changed at the time of building images.
arora's avatar
arora committed
87

arora's avatar
arora committed
88
Depending on the namespace where these charts will be instantiated change the value of `namespace` parameter in `values.yaml`. All the network function related configurable parameters are in the sections `config` of the `values.yaml`. To understand the usage and description of each network function configuration parameter refer their [wiki page](https://gitlab.eurecom.fr/oai/cn5g/oai-cn5g-amf/-/wikis/home).
arora's avatar
arora committed
89

arora's avatar
arora committed
90
Create a namespace where the helm-charts will be deployed, in our environment we deploy them in `oai` namespace. To create a namespace use the below command on your cluster, 
arora's avatar
arora committed
91

arora's avatar
arora committed
92 93 94 95 96 97
```bash
# needs a user which has the right to create namespaces
$ kubectl create ns oai
or 
$ oc new-project oai
```
arora's avatar
arora committed
98

arora's avatar
arora committed
99

arora's avatar
arora committed
100
### 3.1 Networking related information
arora's avatar
arora committed
101

102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118
Network functions communicate based on **kubernetes service concept**, the network functions are using FQDN of other network functions to use their service. 

*For example: AMF registers with NRF using NRF FQDN (`oai-nrf-svc.oai.svc.cluster.local`). This way we can get rid of any static ip-address configuration. Though we are still keeping the fields where we are mentioning ip-addresses for example `nrfIpv4Addr` in amf values.yaml but that is not being used if `USE_FQDN_DNS` is set to `true`*

In our environment to reduce complexity and reduce static ip-address allocation we are providing only one interface to each pod (default Kubernetes CNI) except AMF and UPF where we provide two interfaces because gNB or gNB emulator is not running in the cluster environment. N1/N2/NGAP and N3/GTP-U interface are provided using multus CNI to AMF and UPF pod.

**Note**: Each network function can be configured with multiple interfaces using multus CNI if needed. In the `values.yaml` of each network function there are appropriate comments. There is a section `multus` in every value.yaml


```
## Example from ../charts/oai-amf/values.yaml
multus:
  create: false
  n4IPadd: "192.168.18.178"
  n4Netmask: "24"
  n4Gw: "192.168.18.129"
```
arora's avatar
arora committed
119 120 121 122


### 3.2 Configuring AMF

arora's avatar
arora committed
123
Open the [values.yaml](../charts/oai-amf/values.yaml) to configure the required parameters. There are many parameters which can be configured. Below are some important parameters,  
arora's avatar
arora committed
124
```
arora's avatar
arora committed
125
namespace: "oai"
arora's avatar
arora committed
126

arora's avatar
arora committed
127 128 129
nfimage:
  registry: local
  repository: rdefosseoai/oai-amf # image name either locally present or in a public/private repository
arora's avatar
arora committed
130
  version: v1.1.0 # image tag
arora's avatar
arora committed
131 132 133 134 135 136 137 138 139
  # pullPolicy: IfNotPresent or Never or Always
  pullPolicy: Always

tcpdumpimage:
  registry: local
  repository: corfr/tcpdump
  version: latest
  #pullPolicy: IfNotPresent or Never or Always
  pullPolicy: Always
arora's avatar
arora committed
140 141 142

# configure these values based on gNB/emulator configuration
config: 
arora's avatar
arora committed
143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159
  mcc: "208"
  mnc: "95"
  regionId: "128"
  amfsetId: "1"
  servedGuamiMcc0: "208"
  servedGuamiMnc0: "95"
  servedGuamiRegionId0: "128"
  servedGuamiAmfSetId0: "1"
  servedGuamiMcc1: "460"
  servedGuamiMnc1: "11"
  servedGuamiRegionId1: "10"
  servedGuamiAmfSetId1: "1"
  plmnSupportMcc: "208"
  plmnSupportMnc: "95"
  plmnSupportTac: "0xa000"
  sst0: "222"
  sd0: "123"
arora's avatar
arora committed
160 161
  sst1: "111"
  sd1: "124"
arora's avatar
arora committed
162 163 164 165 166 167 168 169 170 171 172 173
```

The mysql database is pre-configured with some subscriber information. If there is new subscriber information then it should be configured in the mysql database. The subscriber PLMN information should match with gNB and AMF. 

```
  mySqlServer: "mysql"
  mySqlUser: "root"
  mySqlPass: "linux"
  mySqlDb: "oai_db"
  operatorKey: "63bfa50ee6523365ff14c1f45f88737d" (should be the same in mysql)
```

arora's avatar
arora committed
174 175 176

Configure `readinessProbe` and `livenessProbe` by default they are `true` to switch them off change the value with `false`

arora's avatar
arora committed
177 178
There are more parametes which can be configured like extra interfaces and resource usage by network function, please refer [values.yaml](../charts/oai-amf/values.yaml). Infront of every parameter there is a comment. 

arora's avatar
arora committed
179 180
### 3.3 Configuring SMF

arora's avatar
arora committed
181
Open [values.yaml](../charts/oai-smf/values.yaml) to configure SMF configuration. DNS configuration which will be shared with the UE based on the DNS used in your environment, it can be 8.8.8.8 or 4.4.4.4 if you don't know your DNS. 
arora's avatar
arora committed
182 183

```
arora's avatar
arora committed
184 185
namespace: "oai" # namespace where SMF will be deployed 

arora's avatar
arora committed
186 187 188 189 190 191 192 193 194 195 196 197 198
nfimage:
  registry: local
  repository: rdefosseoai/oai-smf
  version: v1.1.0
  #pullPolicy: IfNotPresent or Never or Always
  pullPolicy: Always

tcpdumpimage:
  registry: local
  repository: corfr/tcpdump
  version: latest
  #pullPolicy: IfNotPresent or Never or Always
  pullPolicy: Always
arora's avatar
arora committed
199 200

config:
arora's avatar
arora committed
201 202 203 204
  dnsIpv4Address: ""
  dnsSecIpv4Address: ""
```

arora's avatar
arora committed
205 206
Currenly DNN related information and UE network related information can only be changed at the time of building the SMF network function. Refer [Configure the Containers](./CONFIGURE_CONTAINERS.md) to understand how it can be done. 

arora's avatar
arora committed
207 208
Configure `readinessProbe` and `livenessProbe` by default they are `true` to switch them off change the value with `false`

arora's avatar
arora committed
209 210 211 212 213 214 215 216 217
There are more parametes which can be configured like extra interfaces and resource usage by network function, please refer [values.yaml](../charts/oai-smf/values.yaml). Infront of every parameter there is a comment. 

### 3.4 Configuring UPF

Open [values.yaml](../charts/oai-spgwu-tiny/values.yaml) to configure UPF/SPGWU configuration

```
namespace: "oai" # namespace where SMF will be deployed 

arora's avatar
arora committed
218 219 220 221 222 223 224 225 226 227 228 229 230
nfimage:
  registry: local
  repository: rdefosseoai/oai-spgwu-tiny
  version: v1.1.2
  # pullPolicy: IfNotPresent or Never or Always
  pullPolicy: Always

tcpdumpimage:
  registry: local
  repository: corfr/tcpdump
  version: latest
  #pullPolicy: IfNotPresent or Never or Always
  pullPolicy: Always
arora's avatar
arora committed
231 232 233 234 235 236 237 238 239

config:
  gwId: 1
  mnc: 208 # should match with AMF and SMF and gNB information
  mcc: 95 # should match with AMF and SMF and gNB information
  realm: "3gpp.org" 
  pidDirectory: "/var/run"
```

arora's avatar
arora committed
240 241
Configure `readinessProbe` and `livenessProbe` by default they are `true` to switch them off change the value with `false`

arora's avatar
arora committed
242 243 244 245 246 247
There are more parametes which can be configured like extra interfaces and resource usage by network function, please refer [values.yaml](../charts/oai-spgwu-tiny/values.yaml). Infront of every parameter there is a comment. 

### 3.5 Configuring NRF

NRF configuration is straight forward, most of configurable parameters have comment infront of it for explaination. Refer the [values.yaml](../charts/oai-nrf/values.yaml)

arora's avatar
arora committed
248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266
```
namespace: "oai"

# NF image
nfimage:
  registry: local
  repository: rdefosseoai/oai-nrf
  version: v1.1.0
  # pullPolicy: IfNotPresent or Never or Always
  pullPolicy: Always

tcpdumpimage:
  registry: local
  repository: corfr/tcpdump
  version: latest
  # pullPolicy: IfNotPresent or Never or Always
  pullPolicy: Always
```

arora's avatar
arora committed
267 268
Configure `readinessProbe` and `livenessProbe` by default they are `true` to switch them off change the value with `false`

arora's avatar
arora committed
269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286
### 3.6 Configuring subscriber data in MYSQL

Currently the MYSQL is configured with below subscriber data,

```
IMSI - 208950000000030-34
IMEI - 55000000000001
Secret Key (K) - 0x0C0A34601D4F07677303652C0462535B
OPc - 0x63bfa50ee6523365ff14c1f45f88737d
```

To configure new subscriber add a new entry in [values.yaml](../charts/mysql/values.yaml) line 345, add a new entry after 

```
INSERT INTO `users` VALUES (imsi,'380561234567',imei,NULL,'PURGED',50,40000000,100000000,47,0000000000,1,0xkey,0,0,0x40,'ebd07771ace8677a',0xOpC);
```

In above statement change `imsi`, `imei`, `key` and `opc` with appropriate value according to your environment. 
arora's avatar
arora committed
287 288 289 290 291

## 4. Deploying Helm Charts

Helm charts have an order of deployment for the proper configuration of core network. 

292
`mysql --> nrf --> amf --> smf --> upf(spgwu)`
arora's avatar
arora committed
293 294 295 296

Once the configuration is finished the charts can be deployed with a user who has the rights to

1. Create RBAC
arora's avatar
arora committed
297 298
2. Run pod with privileged and anyuid scc
3. Create multus binds
arora's avatar
arora committed
299 300 301

```
$ helm install mysql mysql/
arora's avatar
arora committed
302
# wait for the pod to be ready 
arora's avatar
arora committed
303
$ helm install amf oai-amf/
arora's avatar
arora committed
304
# wait for the pod to be ready 
arora's avatar
arora committed
305
$ helm install nrf oai-nrf/
arora's avatar
arora committed
306
# wait for the pod to be ready 
arora's avatar
arora committed
307
$ helm install smf oai-smf/
arora's avatar
arora committed
308
# wait for the pod to be ready 
arora's avatar
arora committed
309
$ helm install upf oai-spgwu-tiny/
arora's avatar
arora committed
310
# wait for the pod to be ready 
arora's avatar
arora committed
311
$ helm list 
arora's avatar
arora committed
312
NAME  NAMESPACE       REVISION  UPDATED                                   STATUS    CHART                 APP VERSION
arora's avatar
arora committed
313 314 315 316 317 318 319 320 321 322 323 324
amf   oai-5g-develop  1         2021-08-02 14:45:20.055915967 +0200 CEST  deployed  oai-amf-1.1.0         1.1.0      
mysql oai-5g-develop  1         2021-08-02 13:19:21.141268411 +0200 CEST  deployed  mysql-1.6.9           5.7.30     
nrf   oai-5g-develop  1         2021-08-02 14:39:05.615418329 +0200 CEST  deployed  oai-nrf-1.1.0         1.1.0      
smf   oai-5g-develop  1         2021-08-02 14:52:53.573249685 +0200 CEST  deployed  oai-smf-1.1.0         1.1.0      
upf   oai-5g-develop  1         2021-08-02 14:49:48.741260605 +0200 CEST  deployed  oai-spgwu-tiny-1.1.2  1.1.2  
$ kubectl get pods
NAME                              READY   STATUS    RESTARTS   AGE
mysql-5dd98b7d97-gh4bz            1/1     Running   0          20m
oai-amf-7bb898fc58-56pr5          2/2     Running   0          10m
oai-nrf-859b987c48-8v94s          2/2     Running   0          16m
oai-smf-678bbc965f-whdr6          2/2     Running   0          2m46s
oai-spgwu-tiny-6c4d68fd45-mpv5v   2/2     Running   0          5m51s
Rohan's avatar
Rohan committed
325
```
arora's avatar
arora committed
326

arora's avatar
arora committed
327 328 329 330 331 332 333 334 335 336 337 338 339
## 4.1 How to check if the Core network is properly configured? 

Check the logs `smf` and `upf` to see that the PFCP session is properly configured, 

```
$ kubectl oai-smf-678bbc965f-whdr6 smf | grep 'Received N4 ASSOCIATION SETUP RESPONSE from an UPF'
[2021-08-02T14:52:57.695110] [smf] [smf_n4 ] [info ] Received N4 ASSOCIATION SETUP RESPONSE from an UPF
$ kubectl logs oai-spgwu-tiny-6c4d68fd45-mpv5v spgwu | grep 'Received SX HEARTBEAT REQUEST' | wc -l
60 (should be more than 1)
```

This will verify that `smf` and `upf` have successfully registered to `nrf` and there is a PFCP session. 

arora's avatar
arora committed
340
Now go ahead and use OAI-gNB/dsTest/gNBSIM or any other gNB or emulator to test the deployed core network.