Draft: Proposal to use robot framework to generate CI HTML report

Infos

This branch shows how we could use Robot Framework (RF) to make our CI testing easier.

Changes

• Refactored and renamed checkTutorial (Robot Framework requires a Python class which is named the same as the file)
• Created very simple RF test

Notes

• The changes in checkTutorial are not really necessary, but I wanted to have a nicer output in the robot framework log
• I'd actually prefer if each executed command is treated as keyword in RF, I kind of emulated that by logging each command as log keyword

Advantages

• When we follow this approach, we can delete (and especially we don't have to maintain) all the "generateHTMLReport" scripts
• Also, it is easily extensible and robust
• We could e.g. define our own tests in robot directly instead of or additionally to the tutorials:
  • Have a Setup which deploys our CN, calling the Library core-network.py
  • Make our tests (e.g. ping, wget) -> best written in Python and just called from robot
  • Have a Teardown to undeploy CN, calling the Library core-network.py
  • This would allow us to test different deployments which are not described in the tutorials (e.g. different slicing configurations, etc)
• We don't reinvent the wheel
• When we want to have the table of used images, we can easily include that in the report, without having to write our own HTML

As you can see, integrating robot with Python is very easy, so we would not really have to learn the (horrible) syntax of robot framework.

Disadvantages

• The RF syntax is just weird (but when we just call Python scripts its fine)
• New framework to learn
• Pytest is more maintainable than RF (but both are more maintainable compared to our current solution)

Example

This log.html shows how a successful and a failed test looks like.

Alternatives

Pytest using the pytest-html plugin

How to execute

First, install robotframework:

pip3 install robotframework

Then, just execute robot or alternatively python3 -m robot.run:

# "." executes all the robot files in the current directory
python3 -m robot.run .  

Note: It should also work to call python3 -m robot.run check_tutorials.robot, but that somehow did not work for me

Details can be found in the Robot Framework user guide.