Commit 9354a7b9 authored by Daniele Venzano's avatar Daniele Venzano
Browse files

First round of documentation updates

parent fa5030ab
......@@ -28,12 +28,12 @@ Workspaces:
* ``workspace-deployment-path`` : path appended to the ``workspace-base-path`` to distinguish this deployment. If left unspecified it is equal to the deployment name
* ``workspace-base-path = /mnt/zoe-workspaces`` : Base directory where user workspaces will be created. This directory should reside on a shared filesystem visible by all hosts where containers will be run.
* ``fs-group-id = 5001`` : Group ID to use for all Zoe users in workspace files
Metrics:
* ``influxdb-dbname = zoe`` : Name of the InfluxDB database to use for storing metrics
* ``influxdb-url = http://localhost:8086`` : URL of the InfluxDB service (ex. )
* ``influxdb-enable = False`` : Enable metric output toward influxDB
* ``kairosdb-enable = false`` : Enable gathering of usage metrics recorded in KairosDB
* ``kairosdb-url = http://localhost:8090`` : URL of KairosDB REST API
Service logs (see: :ref:`logging`):
......@@ -60,31 +60,24 @@ API options:
Master options:
* ``api-listen-uri = tcp://*:4850`` : ZeroMQ server connection string, used for the master listening endpoint
* ``kairosdb-enable = false`` : Enable gathering of usage metrics recorded in KairosDB
* ``kairosdb-url = http://localhost:8090`` : URL of KairosDB REST API
* ``overlay-network-name = zoe`` : name of the pre-configured Docker overlay network Zoe should use (Swarm backend)
* ``max-core-limit = 16`` : maximum amount of cores a user is able to reserve
* ``max-memory-limit = 64`` : maximum amount of memory a user is able to reserve
* ``no-user-edit-limits-web = False`` : if set to true, users are NOT allowed to modify ZApp reservations via the web interface
* ``additional-volumes = <none>`` : list of additional volumes to mount in every service, for every ZApp (ex. /mnt/data:data,/mnt/data_n:data_n)
Authentication:
* ``auth-type = text`` : Authentication type (text, ldap or ldapsasl)
* ``auth-file = zoepass.csv`` : Path to the CSV file containing user,pass,role lines for text authentication
* ``ldap-server-uri = ldap://localhost`` : LDAP server to use for user authentication
* ``ldap-bind-user = ou=something,dc=any,dc=local`` : LDAP user for binding to the server
* ``ldap-bind-password = mysecretpassword`` : Password for the bind user
* ``ldap-base-dn = ou=something,dc=any,dc=local`` : LDAP base DN for users
* ``ldap-admin-gid = 5000`` : LDAP group ID for admins
* ``ldap-user-gid = 5001`` : LDAP group ID for users
* ``ldap-guest-gid = 5002`` : LDAP group ID for guests
* ``ldap-group-name = gidNumber`` : LDAP user attribute that contains the group names/IDs
Scheduler options:
* ``scheduler-class = <ZoeElasticScheduler>`` : Scheduler class to use for scheduling ZApps (default: elastic scheduler)
* ``scheduler-policy = <FIFO | SIZE>`` : Scheduler policy to use for scheduling ZApps (default: FIFO)
* ``placement-policy = <waterfill | random | average>`` : how containers should be placed on hosts (default: average)
ZApp shop:
......@@ -92,15 +85,7 @@ ZApp shop:
Back-end choice:
* ``backend = <DockerEngine|Swarm|Kubernetes>`` : cluster back-end to use to run ZApps, default is DockerEngine
Swarm back-end options:
* ``backend-swarm-url = zk://zk1:2181,zk2:2181,zk3:2181`` : connection string to the Swarm API endpoint. Can be expressed by a plain http URL or as a zookeeper node list in case Swarm is configured for HA.
* ``backend-swarm-zk-path = /docker`` : ZooKeeper path used by Docker Swarm
* ``backend-swarm-tls-cert = cert.pem`` : Docker TLS certificate file
* ``backend-swarm-tls-key = key.pem`` : Docker TLS private key file
* ``backend-swarm-tls-ca = ca.pem`` : Docker TLS CA certificate file
* ``backend = <DockerEngine|Kubernetes>`` : cluster back-end to use to run ZApps, default is DockerEngine
Kubernetes back-end:
......
.. _contributing:
.. _contributing:
Contributing to Zoe
===================
......
......@@ -38,7 +38,6 @@ Variables
To run the tests a number of variables need to be set from the GitLab interface:
* REGISTRY_PASSWORD: the password used for authenticating with the registry via docker login
* SSH_PRIVATE_KEY: private key to be used to deploy via rsync the staging build
* STAGING_IP: IP/hostname of the staging server
* WEB_STAGING_PATH: path for the web interface on the staging server
......
......@@ -7,51 +7,7 @@ As a developer you can:
- call Zoe from your own software: :ref:`Zoe REST API documentation <rest-api>`
- create ot modify ZApps: :ref:`howto_zapp`
- contribute to Zoe: keep reading
Contributing to Zoe
-------------------
Zoe is open source and all kinds of contributions are welcome.
Zoe is licensed under the terms of the Apache 2.0 license.
Bugs, issues and feature requests
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
`Zoe issue tracker <https://github.com/DistributedSystemsGroup/zoe/issues>`_
Testing beta code
^^^^^^^^^^^^^^^^^
The ``HEAD`` of the master branch represents the latest version of Zoe. Automatic tests are performed before code is merged into master, but human feedback is invaluable. Clone the repository and report on the `mailing list <http://www.freelists.org/list/zoe>`_ or on the `issue tracker <https://github.com/DistributedSystemsGroup/zoe/issues>`_.
Code changes and pull requests
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
**When you contribute code, you affirm that the contribution is your original work and that you license the work to the project under the project’s open source license. Whether or not you state this explicitly, by submitting any copyrighted material via pull request, email, or other means you agree to license the material under the project’s open source license and warrant that you have the legal authority to do so.**
To contribute code and/or documentation you should follow this workflow:
1. announce your idea on the mailing list, to prevent duplicated work
2. fork the Zoe repository via GitHub (if you don't already have a fork)
3. ... develop and debug ...
4. when you are ready propose your changes with a pull request
Zoe maintainers will review your code, give constructive feedback and eventually accept the code and merge.
Contributors can setup their own CI pipeline following the quality guidelines (:ref:`quality`). At a bare minimum all code should be tested via the `run_tests.sh` script available in the root of the repository. Accepted contributions will be run through the full Zoe CI pipeline before being merged in the public repository.
Repository contents
^^^^^^^^^^^^^^^^^^^
- `docs`: Sphinx documentation used to build these pages
- `scripts`: scripts for deployment and testing
- `zoe_api`: the front-end Zoe process that provides the REST API
- `zoe_cmd`: Command-line client
- `zoe_lib`: library, contains common modules needed by the api and the master processes
- `zoe_master`: the back-end Zoe process schedules and talks to the containerization system
- `contrib`: supervisord config files and sample ZApps
- contribute to Zoe: :ref:`contributing`
Internal module/class/method documentation
......
......@@ -15,7 +15,45 @@ In case the request causes an error, an appropriate HTTP status code is returned
With an error message detailing the kind of error that happened.
Some endpoints require credentials for authentication. For now the API uses straightforward HTTP Basic authentication. In case credentials are missing or wrong a 401 status code will be returned.
Some endpoints require credentials for authentication. The API uses HTTP Basic authentication. After the first successful authentication, a cookie will be returned in the response headers. The cookie can be used as an authentication token for the subsequent requests.
Login endpoint
--------------
Get back a cookie for further authentication/authorization with other api endpoints instead of using HTTP basic username, password
Request::
curl -u 'username:password' -c zoe_cookie.txt http://bf5:8080/api/<api_version>/login
Will return a JSON document, like this::
{
"user": {
"fs_uid": 10000,
"email": "user@domain",
"quota_id": 1,
"auth_source": "pam",
"role_id": 1,
"id": 8,
"priority": 0,
"username": "user33",
"enabled": true
}
}
And the file named ``zoe_cookie.txt`` contains the cookie information.
Pass this cookie on each api request which requires authentication.
Example::
curl -b zoe_cookie.txt http://bf5:8080/api/<api_version>/execution
Note:
- For zoe web interface, we require cookie_based mechanism for authentication/authorization.
- Every unauthorized request will be redirected to **http://<hostname>:8080/login**
- After a successful login, a cookie will be saved in the browser for further authentication/authorization purpose.
Info endpoint
-------------
......@@ -64,7 +102,7 @@ Execution details
Request (GET)::
curl -u 'username:password' http://bf5:8080/api/<api_version>/execution/<execution_id>
curl -b zoe_cookie.txt http://bf5:8080/api/<api_version>/execution/<execution_id>
Where:
......@@ -110,7 +148,7 @@ This endpoint terminates a running execution.
Request (DELETE)::
curl -X DELETE -u 'username:password' http://bf5:8080/api/<api_version>/execution/<execution_id>
curl -X DELETE -b zoe_cookie.txt http://bf5:8080/api/<api_version>/execution/<execution_id>
If the request is successful an empty response with status code 200 will be returned.
......@@ -120,7 +158,7 @@ This endpoint deletes an execution from the database, terminating it if it is ru
Request (DELETE)::
curl -u 'username:password' http://bf5:8080/api/<api_version>/execution/delete/<execution_id>
curl -b zoe_cookie.txt http://bf5:8080/api/<api_version>/execution/delete/<execution_id>
If the request is successful an empty response with status code 200 will be returned.
......@@ -131,7 +169,7 @@ This endpoint will list all executions belonging to the calling user. If the use
Request (GET)::
curl -u 'username:password' http://bf5:8080/api/<api_version>/execution
curl -b zoe_cookie.txt http://bf5:8080/api/<api_version>/execution
Will return a JSON document like this::
......@@ -161,7 +199,7 @@ Starting from verion 0.7 of the API, the execution list can be filtered.
You need to pass via the URL (GET parameters) the criteria to be used for filtering, for example::
curl -u 'username:password' http://bf5:8080/api/<api_version>/execution?status=terminated\&limit=1
curl -b zoe_cookie.txt http://bf5:8080/api/<api_version>/execution?status=terminated\&limit=1
Valid criteria that can be used are:
......@@ -183,13 +221,13 @@ Start execution
Request (POST)::
curl -X POST -u 'username:password' --data-urlencode @filename http://bf5:8080/api/<api_version>/execution
curl -X POST -b zoe_cookie.txt --data-urlencode @filename http://bf5:8080/api/<api_version>/execution
Needs a JSON document passed as the request body::
{
"application": <zapp json>,
'name': "experiment #33"
"name": "experiment #33"
}
Where:
......@@ -212,7 +250,7 @@ Execution endpoints
Request (GET)::
curl -X GET -u 'username:password' http://bf5:8080/api/<api_version>/execution/endpoints/<execution_id>
curl -b zoe_cookie.txt http://bf5:8080/api/<api_version>/execution/endpoints/<execution_id>
Will return a JSON list like this::
......@@ -237,7 +275,7 @@ Service details
Request::
curl -u 'username:password' http://bf5:8080/api/<api_version>/service/<service_id>
curl -b zoe_cookie.txt http://bf5:8080/api/<api_version>/service/<service_id>
Will return a JSON document like this::
......@@ -274,7 +312,7 @@ Service standard output and error
Request::
curl -u 'username:password' http://bf5:8080/api/<api_version>/service/logs/<service_id>
curl -b zoe_cookie.txt http://bf5:8080/api/<api_version>/service/logs/<service_id>
Will stream the service instance output, starting from the time the service started. It will close the connection when the service exits.
......@@ -334,31 +372,327 @@ Where:
The actual content of the response may vary between different Zoe releases.
Login endpoint
User endpoints
--------------
Get back a cookie for further authentication/authorization with other api endpoints instead of using raw username, password
These endpoints modify the user tables in Zoe. For more information about users, check :ref:`users`.
Get from ID
^^^^^^^^^^^
Request::
curl -u 'username:password' -c zoe_cookie.txt http://bf5:8080/api/<api_version>/login
curl -b zoe_cookie.txt http://bf5:8080/api/<api_version>/user/<user_id>
Will return a JSON document, like this::
{
"role": "admin",
"uid": "admin"
"user": {
"fs_uid": 10000,
"email": "user@domain",
"quota_id": 1,
"auth_source": "pam",
"role_id": 1,
"id": 8,
"priority": 0,
"username": "user33",
"enabled": true
}
}
And a file named ``zoe_cookie.txt`` contains the cookie information.
Create
^^^^^^
Pass this cookie on each api request which requires authentication.
Request::
Example::
curl -X POST -b zoe_cookie.txt --data-urlencode @filename http://bf5:8080/api/<api_version>/user
curl -b zoe_cookie.txt http://bf5:8080/api/<api_version>/execution
Needs a JSON document passed as the request body::
Note:
{
"username": <new username>,
"email": <email>,
"role_id": <ID of an existing role>,
"quota_id": <ID of an existing quota>,
"auth_source": <authentication method>
}
- For zoe web interface, we require cookie_based mechanism for authentication/authorization.
- Every unauthorized request will be redirected to **http://<hostname>:8080/login**
- After a successful login, a cookie will be saved in the browser for further authentication/authorization purpose.
Will return a JSON document like this::
{
"user_id": 23
}
Where:
* ``user_id`` is the ID of the new user just created.
Please note that to set the password a second request to the update user endpoint needs to be performed.
Search
^^^^^^
Request::
curl -b zoe_cookie.txt http://bf5:8080/api/<api_version>/user?username=<username>
The following filters can be used:
* username
* email
* priority
* enabled
* auth_source
* role_id
* quota_id
Will return a JSON document like this::
{
"33": {
... user object ...
},
"44": {
... user object ...
}
}
Delete
^^^^^^
Request::
curl -X DELETE -b zoe_cookie.txt http://bf5:8080/api/<api_version>/user/<user_id>
If the request is successful an empty response with status code 200 will be returned.
Update
^^^^^^
Request::
curl -X POST -b zoe_cookie.txt http://bf5:8080/api/<api_version>/user/<user_id>
Needs a JSON document passed as the request body::
{
"username": <new username>,
"password": <new password>,
"email": <email>,
"role_id": <ID of an existing role>,
"quota_id": <ID of an existing quota>,
"auth_source": <authentication method>
}
The document should contain only the fields to update.
Role endpoints
--------------
These endpoints modify the role tables in Zoe. For more information about roles, check :ref:`roles`.
Get from ID
^^^^^^^^^^^
Request::
curl -b zoe_cookie.txt http://bf5:8080/api/<api_version>/role/<role_id>
Will return a JSON document, like this::
{
"role": {
"can_change_config": true,
"can_delete_executions": true,
"name": "admin",
"can_see_status": true,
"can_access_full_zapp_shop": true,
"id": 1,
"can_operate_others": true,
"can_customize_resources": true,
"can_access_api": true
}
}
Create
^^^^^^
Request::
curl -X POST -b zoe_cookie.txt --data-urlencode @filename http://bf5:8080/api/<api_version>/role
Needs a JSON document passed as the request body::
{
"name": <name of the new role>,
"can_change_config": <true|false>,
"can_delete_executions": <true|false>,
"can_see_status": <true|false>,
"can_access_full_zapp_shop": <true|false>,
"can_operate_others": <true|false>,
"can_customize_resources": <true|false>,
"can_access_api": <true|false>
}
Will return a JSON document like this::
{
"role_id": 23
}
Where:
* ``role_id`` is the ID of the new role just created.
Search
^^^^^^
Request::
curl -b zoe_cookie.txt http://bf5:8080/api/<api_version>/role?name=<role name>
The following filters can be used:
* name
Will return a JSON document like this::
{
"3": {
... role object ...
},
"44": {
... role object ...
}
}
Delete
^^^^^^
Request::
curl -X DELETE -b zoe_cookie.txt http://bf5:8080/api/<api_version>/role/<role_id>
If the request is successful an empty response with status code 200 will be returned.
Update
^^^^^^
Request::
curl -X POST -b zoe_cookie.txt @filename http://bf5:8080/api/<api_version>/role/<role_id>
Needs a JSON document passed as the request body::
{
"name": <name of the new role>,
"can_change_config": <true|false>,
"can_delete_executions": <true|false>,
"can_see_status": <true|false>,
"can_access_full_zapp_shop": <true|false>,
"can_operate_others": <true|false>,
"can_customize_resources": <true|false>,
"can_access_api": <true|false>
}
The document should contain only the fields to update.
Quota endpoints
---------------
These endpoints modify the quota tables in Zoe. For more information about quotas, check :ref:`quotas`.
Get from ID
^^^^^^^^^^^
Request::
curl -b zoe_cookie.txt http://bf5:8080/api/<api_version>/quota/<user_id>
Will return a JSON document, like this::
{
"quota": {
"concurrent_executions": 5,
"cores": 5,
"id": 1,
"name": "default",
"memory": 34359738368
}
}
Create
^^^^^^
Request::
curl -X POST -b zoe_cookie.txt --data-urlencode @filename http://bf5:8080/api/<api_version>/quota
Needs a JSON document passed as the request body::
{
"name": <name of the new quota>,
"concurrent_executions": <maximum number of running executions>,
"memory": <maximum amount of memory reserved across all running executions>,
"cores": <maximum amount of cores reserved across all running executions>
}
Will return a JSON document like this::
{
"quota_id": 23
}
Where:
* ``quota_id`` is the ID of the new quota just created.
Search
^^^^^^
Request::
curl -b zoe_cookie.txt http://bf5:8080/api/<api_version>/quota?name=<quota name>
The following filters can be used:
* name
Will return a JSON document like this::
{
"3": {
... quota object ...
},
"44": {
... quota object ...
}
}
Delete
^^^^^^
Request::
curl -X DELETE -b zoe_cookie.txt http://bf5:8080/api/<api_version>/quota/<quota_id>
If the request is successful an empty response with status code 200 will be returned.
Update
^^^^^^
Request::
curl -X POST -b zoe_cookie.txt http://bf5:8080/api/<api_version>/quota/<quota_id>
Needs a JSON document passed as the request body::
{
"name": <name of the new quota>,
"concurrent_executions": <maximum number of running executions>,
"memory": <maximum amount of memory reserved across all running executions>,
"cores": <maximum amount of cores reserved across all running executions>
}
The document should contain only the fields to update.
......@@ -50,10 +50,13 @@ Main documentation
:maxdepth: 1
install
kube_backend
config_file
users
roles
quotas
logging
proxy
kube_backend
Zoe applications
^^^^^^^^^^^^^^^^
......@@ -61,7 +64,6 @@ Zoe applications
.. toctree::
:maxdepth: 1
zapps/classification
zapps/howto_zapp
zapps/zapp_format
......@@ -75,8 +77,6 @@ Development and contributing to the project
developer/index
architecture
quality
motivations
vision
contributing
External resources
......
.. _motivation:
The motivation behind Zoe
=========================
The fundamental idea of Zoe is that a user who wants run data analytics applications should not be bothered by systems details, such as how to configure the amount of RAM a Spark Executor should use, how many cores are available in the system or even how many worker nodes should be used to meet an execution deadline.
Moreover final users require a lot of flexibility, they want ot test new analytics systems and algorithms as soon as possible, without having to wait for some approval procedure to go through the IT department. Zoe proposes a flexible model for applications descriptions: their management can be left entirely to the final user, but they can also prepared very quickly in all or in part by an IT department, who sometimes is more knowledgeable of resource limits and environment variables. We also plan to offer a number of building blocks (Zoe Frameworks) that can be composed to make Zoe Applications.
Finally we feel that there is a lack of solutions in the field of private clouds, where resources are not infinite and data layers (data-sets) may be shared between different users. All the current Open Source solutions we are aware of target the public cloud use case and try, more or less, to mimic what Amazon and other big names are doing in their data-centers.
Zoe strives to satisfy the following requirements:
* easy to use for the end-user
* easy to manage for the system administrator, easy to integrate in existing data-centers/clouds/VM deployments
* short (a few seconds) reaction times to user requests or other system events
* smart queuing and scheduling of applications when resources are critical