No need to get container name, we work directly with docker services.

In CI we sometimes encouter fail of the gNB/UE deployment (and fail of
the pipeline) caused by unsuccsessful initialization of the USRP N310.
Restart the container if the health check fails during initialization.

Introduce a configurable parameter (from XML) to set max number of
attempts for the container deployment.

Store logs from failed deployment attempts.

Rework GetContainerHealth() to work with docker compose and services,
instead of using docker and containers. Return health and message about
deployment status (used for console and HTML logging) for a given service.

- pass parameter directly instead of indirectly via class
- harmonize to svr_id to read through common code in the next commits
- pass images to express which images will be deleted
- refactor function to use "with" for cls_cmd, and return success (or
  not)

- pass parameter directly instead of indirectly via class
- call it simply "images", to harmonize with clean-function later
- harmonize to svr_id as in the previous commit
- rewrite function to use with-statement with cls_cmd
- optionally pass a tag to use (which will be used in a test, and can be
  used to pull flexric)
- Introduce Pull_Image() which does the actual pulling, can pull from
  any registry, and will be reused in the case of pulling from cluster.

- pass parameter directly instead of indirectly via class
- harmonize to svr_id to harmonize parameter with parameters in the next
  commit (and towards a future harmonization of directly taking node
  names instead of "IDs")

Raphael Defosseux authored 4 months ago and Robert Schmidt committed 4 months ago

This will be used to pull images with the tag "develop" (i.e., without a
commit ID).

Raphael Defosseux authored 4 months ago and Robert Schmidt committed 4 months ago

Some commands might use that folder before Ping/Iperf, in which it has
been created.

In the last commits, SSHconnection class has been removed from
cls_containerize.py. We therefore don't need the username and password,
as cls_cmd uses certificates only. Remove the usage of the username and
the password, and harmonize the use of GetCredentials().

Recent refactoring replaced the CreateWorkspace function call with a
dedicated XML step (for clarity, and because it avoids needless
recreation of workspaces). In the proxy function, the same
CreateWorkspace xml step has been added, cloning the OAI RAN repo, when
we actually need the repository for the proxy. Hence, fix this by
calling the CreateWorkspace() function manually, but only when
necessary, to build the proxy. (It would only be a problem if the
pipeline would rebuild the proxy, which did not happen because not much
work is happening on the proxy repo).

Further, refactoring and simplication of the proxy build function were
done:

- never tag proxy with develop, that makes no sense
- overwrite CreateWorkspace() info after having created and used the
  workspace (the "if" to build the proxy)
- simplify copying of logs
- add simple unit test
- remove check for ubuntu; the dependcy is "docker". Simplify the code
  to not obscure with a class variable.

Remove CheckAndAddRoute(), as it is complicated to read, uses a
multitude of hostname/IP/password combinations. Replace with scripts on
the machines, which, as of this commit, are:

- porcepix: ~/after-reboot.sh
- ofqot: ~/after-reboot.sh
- nepes: ~/after-reboot.sh
- obelix: ~/how-to-resetup-obelix-out-of-reboot.sh

Since commit 041b2b2d, the CI main loop
iterates over all steps, even if there has been a problem during an
earlier XML step. In all cases, UndeployObject() is marked to always
run, as we always need to make sure that all deployed services are
terminated. UndeployObject() always recovers logs. Hence, it's not
necessary to do it here as well; it leads to duplicated logs.

This is visible in all pipeline with a failed deployment: the logs
appear twice.

Signed-off-by: Raphael Defosseux <raphael.defosseux@eurecom.fr>

pyflakes warns about various errors, addressed here. For cls_cluster.py,
remove (unknown) sys.exit and replace through Exception, as sys.exit()
has the unwanted effect that the CI script would stop.  Instead, raise
an Exception that would show the problem in the HTML, and ensures that
the script runs until the end.

These are the warnings flagged by pyflakes.

    cls_cluster.py:297:4: undefined name 'sys'

    cls_containerize.py:39:1: 'subprocess' imported but unused
    cls_containerize.py:41:1: 'threading' imported but unused
    cls_containerize.py:43:1: 'multiprocessing.Process' imported but unused
    cls_containerize.py:43:1: 'multiprocessing.Lock' imported but unused
    cls_containerize.py:43:1: 'multiprocessing.SimpleQueue' imported but unused
    cls_containerize.py:49:1: 'cls_cluster as OC' imported but unused
    cls_containerize.py:50:1: redefinition of unused 'cls_cmd' from line 42

    cls_module.py:28:1: 'os' imported but unused
    cls_module.py:29:1: 'sys' imported but unused
    cls_module.py:34:1: 'subprocess' imported but unused
    cls_module.py:35:1: 'datetime.datetime' imported but unused
    cls_module.py:211:11: undefined name 'sshconnection'

    cls_oaicitest.py:37:1: 'pexpect' imported but unused
    cls_oaicitest.py:47:1: 'cls_cluster as OC' imported but unused
    cls_oaicitest.py:54:1: 'matplotlib.pyplot as plt' imported but unused
    cls_oaicitest.py:55:1: 'numpy as np' imported but unused

    cls_oai_html.py:34:1: 'sys' imported but unused
    cls_oai_html.py:41:1: 'multiprocessing.Process' imported but unused
    cls_oai_html.py:41:1: 'multiprocessing.Lock' imported but unused
    cls_oai_html.py:41:1: 'multiprocessing.SimpleQueue' imported but unused

    cls_physim.py:37:1: 'multiprocessing.SimpleQueue' imported but unused

    cls_static_code_analysis.py:39:1: 'time' imported but unused
    cls_static_code_analysis.py:40:1: 'multiprocessing.Process' imported but unused
    cls_static_code_analysis.py:40:1: 'multiprocessing.Lock' imported but unused
    cls_static_code_analysis.py:40:1: 'multiprocessing.SimpleQueue' imported but unused

    epc.py:41:1: 'multiprocessing.Process' imported but unused
    epc.py:41:1: 'multiprocessing.Lock' imported but unused
    epc.py:41:1: 'multiprocessing.SimpleQueue' imported but unused

    main.py:60:1: 'pexpect' imported but unused
    main.py:66:1: 'datetime' imported but unused
    main.py:68:1: redefinition of unused 'subprocess' from line 63
    main.py:70:1: 'multiprocessing.Process' imported but unused
    main.py:70:1: 'multiprocessing.Lock' imported but unused
    main.py:70:1: 'multiprocessing.SimpleQueue' imported but unused

    provideUniqueImageTag.py:2:1: 'os' imported but unused

Use a script, which can be tested independently of CI code (by running
it on the host on which it will be used), and print the directory that
has been prepared in the HTML.  Also, in the description, use the
infinitive/imperative(?) of the verb to harmonize with other steps.

This reimplements the CI deployment functions, and makes them a bit more
robust.  Concretely:

- Introduce a generic "deployment tag" option that can be used to
  override the tag to use for specific images. By default, it is the
  current "branch-commitID[0:8]" tag name (requires the change from
  function ImageTagToUse() to CreateTag(), impacting also pull/push
  image functions)
- Avoid sed for image names, and use an .env file that docker-compose
  picks up automatically; the deployment analyzes a potentially existing
  .env file and updates instead of simply overriding. For instance, some
  pipelines might use -asan images for the gNB and "normal" (non-asan)
  images for UE, and a simple overwriting might make the -asan image
  name tag disappear for the gNB, resulting in deployment failures).
  Finally, undeployment removes the .env file, so that there are no
  modifications in the repository after undeployment.
- Redo the previous behavior of autodetecting asan, and use that (the
  current function always uses asan, no matter what)
- Remove deployKind/displayedNewTags globals, as they are not necessary
- Make the usedImage output in HTML slimmer
- On undeployment, print service names next to undeploy analysis, and
  return success/failure.
- Make the functions generally less verbose and easier to read

Note that as of and only in this commit, deployment does not work, as
all the YAML files have not been updated to work with this updated
version.  The next commit adds tests for the new deployment, and updates
one YAML file (also being used in the tests). The follow-up commit then
modifies all YAML files.

Use the previous commit on returning success from each task execution
function, and implement handling of errors in the main loop. Add a
script with which this is visible using Custom_Commands.

Signed-off-by: Raphael Defosseux <raphael.defosseux@eurecom.fr>

Signed-off-by: Raphael Defosseux <raphael.defosseux@eurecom.fr>

Bipin Adhikari authored 8 months ago and Raphael Defosseux committed 6 months ago

- Introduced additional helper functions: CopyinContainerLog, GetRunningServices, CheckLogs.