Commit d48a2dea authored by Daniele Venzano's avatar Daniele Venzano

Documentation update

parent ca09172f
......@@ -13,29 +13,21 @@ Development repository
----------------------
Development happens at `Eurecom's GitLab repository <https://gitlab.eurecom.fr/zoe/main>`_. The GitHub repository is a read-only mirror.
The choice of GitLab over GitHub is due to the CI pipeline that we set-up to test Zoe. Please note the issue tracking happens on GitHub.
The choice of GitLab over GitHub is due to the CI pipeline that we set-up to test Zoe. Please note the issue tracking for the Zoe project happens on GitHub.
Bug reports and feature requests
--------------------------------
Bug reports and feature requests are handled through the GitHub issue system at: `https://github.com/DistributedSystemsGroup/zoe/issues <https://github.com/DistributedSystemsGroup/zoe/issues>`_
The issue system should be used for only for bug reports or feature requests. If you have more general questions, you need help setting up Zoe or running some application, please use the mailing list.
Mailing list
------------
The mailing list: `http://www.freelists.org/list/zoe <http://www.freelists.org/list/zoe>`_
Use the mailing list to stay up-to-date with what other developers are working on, to discuss and propose your ideas. We prefer small and incremental contributions, so it is important to keep in touch with the rest of the community to receive feedback. This way your contribution will be much more easily accepted.
Code and documentation contributions
------------------------------------
To contribute code and/or documentation you should follow this workflow:
1. announce your idea on the mailing list, to prevent duplicated work
2. fork the Zoe repository via GitHub (if you don't already have a fork)
1. check the issue tracker on GitHub to see if someone is already working on your idea
2. open a new issue stating your idea and how you wish to implement it
2. fork the Zoe repository via Eurecom's GitLab
3. create a branch that will hold your changes
4. ... develop and debug ...
5. when you are ready propose your changes on the mailing list
......@@ -59,4 +51,12 @@ In general, if your code passes pylint, run with our configuration file with a 1
Code quality and tests
^^^^^^^^^^^^^^^^^^^^^^
Contributors can setup their own CI pipeline following the quality guidelines (:ref:`quality`). At a bare minimum all code should be tested via the `run_tests.sh` script available in the root of the repository. Accepted contributions will be run through the CI pipeline at Eurecom before being published on the public repository.
Before committing, all code should be tested via the `run_tests.sh` script available in the root of the repository.
All contributions to the codebase are centralised into a repository at Eurecom. There, every commit (on any branch) triggers a continuous integration pipeline that verifies code quality and runs tests. Only commits and merges on the master branch for which the CI succeeds are pushed to the public repository.
A description of the CI pipeline is available in the :ref:`ci-gitlab` page.
Sphinx documentation is tested with the ``doc8`` tool with default options.
Refer to the :ref:`integration-test` documentation for details on integration testing.
......@@ -76,7 +76,6 @@ Development and contributing to the project
developer/index
architecture
quality
contributing
External resources
......
.. _proxy:
Accessing Zoe through a reverse proxy
=====================================
The Zoe web interface and REST API can be exposed through a reverse proxy. Additionally some support for exposing the web interfaces of running executions is also available, with some constraints.
Configuration
-------------
* reverse-proxy-path : path-portion of the external URL in case Zoe is exposed by the reverse proxy under a path (ex.: /zoe)
* websocket_base : Base URL for websocket connections (ex.: ws://<server_address>)
Accessing ZApps through the Træfik reverse proxy
------------------------------------------------
Zoe contains support for dynamically updating a reverse proxy for giving access to users without exposing internal IP addresses. This support is experimental and comes with several limitations:
* Only web interfaces can be exposed via a reverse proxy
* Only Træfik with a ZooKeeper dynamic configuration backend is supported
* Usually the web application running in the Zoe execution must be informed of the external URL it is exposed with. Zoe exposes an environment variable with this information, but it is up to the ZApp implementation to pass the correct options to the web application.
Configuration
^^^^^^^^^^^^^
* traefik-zk-ips : ZooKeeper addresses for storing dynamic configuration for træfik (ex.: ``z1:2181,z2:2181,z3:2181``)
* traefik-base-url : Base path used in reverse proxy URLs generated for træfik (default is ``/zoe/proxy/``)
ZApp description
^^^^^^^^^^^^^^^^
In the JSON description of ZApps, ports that need to be exposed through the reverse proxy need to have the ``proxy`` property set to ``true``. The property is optional and defaults to false, so by default no ports will be exposed via the reverse proxy.
Environment variables
^^^^^^^^^^^^^^^^^^^^^
ZApps can use the ``REVERSE_PROXY_PATH_<port number>`` environment variable to configure correctly the URL routing of web applications they contain. The value of these variables will be the concatenation of ``traefik-base-url`` and the unique key generated at runtime for each proxied port.
Access ZApps through Ingress Controller on Kubernetes
=====================================================
-----------------------------------------------------
Overview
--------
^^^^^^^^
* We can access Zapps through a web proxy, so we do not need to open too many ports due to security reasons.
* This can be achieved when Zoe runs on Kubernetes by the support of an Ingress Controller.
* Automate the process of creating an ingress for a servive created by Zoe.
* Services which are exposed in Zapp can be accessed through the proxy url, which has the following format: ``http://servicename-executionid-deploymentname.proxy-path``
Requirements
------------
^^^^^^^^^^^^
* A Kubernetes cluster which has:
* Zoe
......@@ -19,7 +55,7 @@ Requirements
* kubernetes-auto-ingress.
How it works
------------
^^^^^^^^^^^^
1. Zoe configuration file:
* ``--proxy-path``: the **ServerName** field in apache2 virtualhost configuration
......@@ -31,10 +67,10 @@ How it works
3. kubernetes-auto-ingress:
* Currently, the process to submit an Ingress resource to the Ingress controller is mannually done by cluster admins. kubernetes-auto-ingress automates this process. Every services have the labels "auto-ingress/enabled" is "enabled" will be automatically attached with the associated ingress resources.
* Currently, the process to submit an Ingress resource to the Ingress controller is manually done by cluster admins. kubernetes-auto-ingress automates this process. Every services have the labels "auto-ingress/enabled" is "enabled" will be automatically attached with the associated ingress resources.
References
----------
^^^^^^^^^^
* Kubernetes Ingress: https://kubernetes.io/docs/concepts/services-networking/ingress/
* NGINX Ingress Controller: https://github.com/kubernetes/ingress/tree/master/controllers/nginx
* kubernetes-auto-ingress: https://github.com/hxquangnhat/kubernetes-auto-ingress
.. _quality:
Testing and quality assurance
=============================
Every commit that hits the master branch on the public repository of Zoe has to pass a testing gate.
All contributions to the codebase are centralised into an internal repository at Eurecom. There, every commit (on any branch) triggers a continuous integration pipeline that verifies code quality and runs tests. Only commits and merges on the master branch for which the Jenkins build succeeds are pushed to the public repository.
GitHub has been configured to protect the master branch on the `Zoe repository <https://github.com/DistributedSystemsGroup/zoe>`_. It will accept only pushes that are marked with a status check. This, together with Jenkins pushing only successful builds, guarantees that the quality of the published code respects our standards.
A description of the CI pipeline is available in the :ref:`ci-gitlab` page.
Sphinx documentation is tested with the ``doc8`` tool with default options.
Refer to the :ref:`integration-test` documentation for details on integration testing.
......@@ -8,8 +8,9 @@ Quotas enforce resource limits to users. A quota can be assigned to multiple use
Quotas can be set on the following resources:
* concurrent_executions : maximum number of concurrent executions in an active state
* memory : maximum amount of memory a user can reserve in total, across all its active executions (not yet implemented)
* cores : maximum amount of cores a user can reserve in total, across all its active executions (not yet implemented)
* memory : maximum amount of memory a user can reserve in total, across all its active executions
* cores : maximum amount of cores a user can reserve in total, across all its active executions
* runtime_limit : maximum time an execution is permitted to run
A default quota is always available:
......@@ -17,5 +18,6 @@ A default quota is always available:
* concurrent executions: 5
* memory: 32GB
* cores: 20
* runtime_limit: 24 hours
This default quota can be modified, but not deleted. More quotas can be created via the zoe_admin.py command.
This default quota can be modified, but not deleted. More quotas can be created via the zoe_admin.py command or the web interface.
......@@ -21,4 +21,4 @@ By default three roles are created:
* superuser : has can_see_status, can_access_api, can_customize_resources and can_access_full_zapp_shop
* user : has no capabilities
Zoe will refuse to delete or modify the admin role.
Zoe will refuse to delete or modify the admin role. Other roles can be created and modificed via the zoe-admin.py tool or the web interface.
......@@ -5,7 +5,7 @@ Users
Zoe has a flexible user management system. All users that need access to Zoe need to have an entry created in the Zoe user database through the command-line utility (zoe-admin.py) or the web interface.
When the entry is being created, the administrator can choose an authentication source, that can be different for each user. Currently the following sources are available:
The administrator can choose an authentication source for each user. Currently the following sources are available:
* internal : the password is stored in Zoe
* LDAP(+SASL) : authentication is performed by contacting an external LDAP server
......@@ -16,5 +16,22 @@ More backends can be developed, the authentication system is designed to be plug
Each user has a :ref:`roles` and a :ref:`quotas` associated.
By default Zoe has an admin user (password admin), created during the first startup. While deploying Zoe, this user must be disabled or its password changed. The default password is a security risk.
By default Zoe has an admin user (with password "admin"), created during the first startup. While deploying Zoe, this user must be disabled or its password changed. The default password is a security risk.
Experimental OAuth2 support
--------------------------
This version of Zoe includes an experimental OAuth2 client implementation that points to a GitLab instance in Eurecom's infrastructure. To use it with your own OAuth2 provider, you need to modify the source code under ``zoe_api/auth/requests_oauth2/services.py``, ``zoe_api/rest_api/user.py`` and ``zoe_api/web/start.py``.
Users that login with OAuth2 can access only the web interface (no oauth2 authentication has been implemented for the API). The user entry in the Zoe database is created automatically and a workspace is created too when the first execution is launched.
The following options are available in the Zoe configuration:
* oauth-client-id: OAuth2 client ID as generated by your identity provider
* oauth-client-secret: OAuth2 client secret as generated by your identity provider
* oauth-redirect-uri: Full URL of the Zoe API OAuth callback as visible from clients (e.g. in front of a reverse proxy)
* oauth-role: Role to assign to new users authenticated via OAuth2
* oauth-quota: Quota to assign to new users authenticated via OAuth2
* oauth-create-workspace-script: Full path to a script that creates user workspace, Zoe will call using sudo and pass username and fs_id as arguments
The user Zoe runs with needs to be able to run the workspace creation script via sudo.
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment