* ``workspace-deployment-path`` : path appended to the ``workspace-base-path`` to distinguish this deployment. If left unspecified it is equal to the deployment name
* ``workspace-base-path = /mnt/zoe-workspaces`` : Base directory where user workspaces will be created. This directory should reside on a shared filesystem visible by all hosts where containers will be run.
* ``fs-group-id = 5001`` : Group ID to use for all Zoe users in workspace files
Metrics:
* ``influxdb-dbname = zoe`` : Name of the InfluxDB database to use for storing metrics
* ``influxdb-url = http://localhost:8086`` : URL of the InfluxDB service (ex. )
* ``kairosdb-enable = false`` : Enable gathering of usage metrics recorded in KairosDB
* ``kairosdb-url = http://localhost:8090`` : URL of KairosDB REST API
Service logs (see: :ref:`logging`):
...
...
@@ -60,31 +60,24 @@ API options:
Master options:
* ``api-listen-uri = tcp://*:4850`` : ZeroMQ server connection string, used for the master listening endpoint
* ``kairosdb-enable = false`` : Enable gathering of usage metrics recorded in KairosDB
* ``kairosdb-url = http://localhost:8090`` : URL of KairosDB REST API
* ``overlay-network-name = zoe`` : name of the pre-configured Docker overlay network Zoe should use (Swarm backend)
* ``max-core-limit = 16`` : maximum amount of cores a user is able to reserve
* ``max-memory-limit = 64`` : maximum amount of memory a user is able to reserve
* ``no-user-edit-limits-web = False`` : if set to true, users are NOT allowed to modify ZApp reservations via the web interface
* ``additional-volumes = <none>`` : list of additional volumes to mount in every service, for every ZApp (ex. /mnt/data:data,/mnt/data_n:data_n)
Authentication:
* ``auth-type = text`` : Authentication type (text, ldap or ldapsasl)
* ``auth-file = zoepass.csv`` : Path to the CSV file containing user,pass,role lines for text authentication
* ``ldap-server-uri = ldap://localhost`` : LDAP server to use for user authentication
* ``ldap-bind-user = ou=something,dc=any,dc=local`` : LDAP user for binding to the server
* ``ldap-bind-password = mysecretpassword`` : Password for the bind user
* ``ldap-base-dn = ou=something,dc=any,dc=local`` : LDAP base DN for users
* ``ldap-admin-gid = 5000`` : LDAP group ID for admins
* ``ldap-user-gid = 5001`` : LDAP group ID for users
* ``ldap-guest-gid = 5002`` : LDAP group ID for guests
* ``ldap-group-name = gidNumber`` : LDAP user attribute that contains the group names/IDs
Scheduler options:
* ``scheduler-class = <ZoeElasticScheduler>`` : Scheduler class to use for scheduling ZApps (default: elastic scheduler)
* ``scheduler-policy = <FIFO | SIZE>`` : Scheduler policy to use for scheduling ZApps (default: FIFO)
* ``placement-policy = <waterfill | random | average>`` : how containers should be placed on hosts (default: average)
ZApp shop:
...
...
@@ -92,15 +85,7 @@ ZApp shop:
Back-end choice:
* ``backend = <DockerEngine|Swarm|Kubernetes>`` : cluster back-end to use to run ZApps, default is DockerEngine
Swarm back-end options:
* ``backend-swarm-url = zk://zk1:2181,zk2:2181,zk3:2181`` : connection string to the Swarm API endpoint. Can be expressed by a plain http URL or as a zookeeper node list in case Swarm is configured for HA.
* ``backend-swarm-zk-path = /docker`` : ZooKeeper path used by Docker Swarm
The ``HEAD`` of the master branch represents the latest version of Zoe. Automatic tests are performed before code is merged into master, but human feedback is invaluable. Clone the repository and report on the `mailing list <http://www.freelists.org/list/zoe>`_ or on the `issue tracker <https://github.com/DistributedSystemsGroup/zoe/issues>`_.
Code changes and pull requests
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
**When you contribute code, you affirm that the contribution is your original work and that you license the work to the project under the project’s open source license. Whether or not you state this explicitly, by submitting any copyrighted material via pull request, email, or other means you agree to license the material under the project’s open source license and warrant that you have the legal authority to do so.**
To contribute code and/or documentation you should follow this workflow:
1. announce your idea on the mailing list, to prevent duplicated work
2. fork the Zoe repository via GitHub (if you don't already have a fork)
3. ... develop and debug ...
4. when you are ready propose your changes with a pull request
Zoe maintainers will review your code, give constructive feedback and eventually accept the code and merge.
Contributors can setup their own CI pipeline following the quality guidelines (:ref:`quality`). At a bare minimum all code should be tested via the `run_tests.sh` script available in the root of the repository. Accepted contributions will be run through the full Zoe CI pipeline before being merged in the public repository.
Repository contents
^^^^^^^^^^^^^^^^^^^
- `docs`: Sphinx documentation used to build these pages
- `scripts`: scripts for deployment and testing
- `zoe_api`: the front-end Zoe process that provides the REST API
- `zoe_cmd`: Command-line client
- `zoe_lib`: library, contains common modules needed by the api and the master processes
- `zoe_master`: the back-end Zoe process schedules and talks to the containerization system
- `contrib`: supervisord config files and sample ZApps
@@ -15,7 +15,45 @@ In case the request causes an error, an appropriate HTTP status code is returned
With an error message detailing the kind of error that happened.
Some endpoints require credentials for authentication. For now the API uses straightforward HTTP Basic authentication. In case credentials are missing or wrong a 401 status code will be returned.
Some endpoints require credentials for authentication. The API uses HTTP Basic authentication. After the first successful authentication, a cookie will be returned in the response headers. The cookie can be used as an authentication token for the subsequent requests.
Login endpoint
--------------
Get back a cookie for further authentication/authorization with other api endpoints instead of using HTTP basic username, password
The fundamental idea of Zoe is that a user who wants run data analytics applications should not be bothered by systems details, such as how to configure the amount of RAM a Spark Executor should use, how many cores are available in the system or even how many worker nodes should be used to meet an execution deadline.
Moreover final users require a lot of flexibility, they want ot test new analytics systems and algorithms as soon as possible, without having to wait for some approval procedure to go through the IT department. Zoe proposes a flexible model for applications descriptions: their management can be left entirely to the final user, but they can also prepared very quickly in all or in part by an IT department, who sometimes is more knowledgeable of resource limits and environment variables. We also plan to offer a number of building blocks (Zoe Frameworks) that can be composed to make Zoe Applications.
Finally we feel that there is a lack of solutions in the field of private clouds, where resources are not infinite and data layers (data-sets) may be shared between different users. All the current Open Source solutions we are aware of target the public cloud use case and try, more or less, to mimic what Amazon and other big names are doing in their data-centers.
Zoe strives to satisfy the following requirements:
* easy to use for the end-user
* easy to manage for the system administrator, easy to integrate in existing data-centers/clouds/VM deployments
* short (a few seconds) reaction times to user requests or other system events
* smart queuing and scheduling of applications when resources are critical
