Commit a27ec6ce authored by Daniele Venzano's avatar Daniele Venzano

Documentation update

parent c8b49392
......@@ -2,7 +2,10 @@
## Version 2017.06
* ZApp description format revision, use JSON schemas
* ZApp description format revision
* use JSON schemas to validate ZApps
* Expand the execution list API, adding filtering capabilities to limit the number of results returned
* Several minor bug fixes
## Version 2017.03
......
Zoe - Container-based Analytics as a Service
============================================
Zoe Analytics - Container-based Analytics as a Service
======================================================
Zoe provides a simple way to provision any kind of data analytics applications.
Zoe Analytics provides a simple way to provision any kind of data analytics applications.
Resources:
......@@ -10,8 +10,4 @@ Resources:
- Mailing list: http://www.freelists.org/list/zoe
- Issue tracker: https://github.com/DistributedSystemsGroup/zoe/issues
Zoe applications (ZApps):
- Repository: https://github.com/DistributedSystemsGroup/zoe-applications
Zoe is licensed under the terms of the Apache 2.0 license.
......@@ -9,15 +9,15 @@ The main Zoe Components are:
* zoe api: the Zoe frontend, offering a web interface and a REST API
* zoe: command-line client
The Zoe master is the core component of Zoe and communicates with the clients by using an internal ZeroMQ-based protocol. This protocol is designed to be robust, using the best practices from ZeroMQ documentation. A crash of the Api or of the Master process will not leave the other component inoperable, when the faulted process restarts, work will restart where it was left.
The Zoe master is the core component of Zoe and communicates with the clients by using an internal ZeroMQ-based protocol. This protocol is designed to be robust, using the best practices from ZeroMQ documentation. A crash of the Api or of the Master process will not leave the other component inoperable, and when the faulted process restarts, work will restart where it was left.
In this architecture we moved all the state bookkeeping out to Postgres database. With Zoe we try very hard not to reinvent the wheel and the internal state system we had in the previous architecture iteration was starting to show its limits.
In this architecture all state is kept in a Postgres database. With Zoe we try very hard not to reinvent the wheel and the internal state system we had in the previous architecture iteration was starting to show its limits.
Users submit *execution requests*, composed by a name and an *application description*. The frontend process (Zoe web) informs the Zoe Master that a new execution request is available for execution.
Users submit *execution requests*, composed by a name and an *application description*. The frontend process (Zoe api) informs the Zoe Master that a new execution request is available for execution.
Inside the Master, a scheduler keeps track of available resources and execution requests, and applies a
scheduling policy to decide which requests should be satisfied as soon as possible and which ones can be deferred for later.
The master also talks to Docker Swarm to create and destroy containers and to read monitoring information used to schedule applications.
The master also talks to a container orchestrator (Docker Swarm for example) to create and destroy containers and to read monitoring information used to schedule applications.
Application descriptions
------------------------
......@@ -25,6 +25,4 @@ Application descriptions are at the core of Zoe. They are likely to evolve in ti
Application descriptions are composed of a set of generic attributes that apply to the whole Zoe Application (abbreviated in ZApp) and a list of Zoe Frameworks. Each Framework is composed by Zoe Services, that describe actual Docker containers. The composition of Frameworks and Services is described by a dependency tree.
The Zoe Service descriptions are strictly linked to the Docker images they use, as they specify environment variables and commands to be executed. We successfully used third party images, demonstrating the generality of Zoe's approach, but in general prefer to build our own.
You can have a look to example applications by going to the `zoe-applications <https://github.com/DistributedSystemsGroup/zoe-applications>`_ repository.
The Zoe Service descriptions are strictly linked to the Docker images they use, as they specify environment variables and commands to be executed. We successfully used third party images, demonstrating the generality of Zoe's approach.
......@@ -3,9 +3,9 @@
Zoe configuration
=================
Zoe can be configured by files, environment variables or commandline options. The configuration directives listed in this file can be specified by any of the three methods. Use the ``--help`` command-line option to have more details on the format od environment variables and precedence rules.
Zoe can be configured by files, environment variables or commandline options. The configuration directives listed in this file can be specified by any of the three methods. Use the ``--help`` command-line option to have more details on the format of environment variables and precedence rules.
Also the directive ``--write-config <filename>`` is available: it will generate a configuration file with all options set to the default values.
The directive ``--write-config <filename>`` is also available: it will generate a configuration file with all options set to the default values.
The Zoe config file have a simple format of ``<option name> = <value>``. Dash characters can be use for comments.
All Zoe processes use one single configuration file, called zoe.conf. It is searched in the current working directory and in ``/etc/zoe/``.
......@@ -23,10 +23,10 @@ Common options:
* ``influxdb-url = http://localhost:8086`` : URL of the InfluxDB service (ex. )
* ``influxdb-enable = False`` : Enable metric output toward influxDB
* ``workspace-base-path = /mnt/zoe-workspaces`` : Base directory where user workspaces will be created. This directory should reside on a shared filesystem visible by all Docker hosts.
* ``overlay-network-name = zoe`` : name of the pre-configured Docker overlay network Zoe should use
* ``overlay-network-name = zoe`` : name of the pre-configured Docker overlay network Zoe should use (Swarm backend)
* ``backend = Swarm`` : ' Name of the backend to enable and use
Database options:
PostgresQL database options:
* ``dbname = zoe`` : DB name
* ``dbuser = zoe`` : DB user
......
......@@ -10,11 +10,7 @@ To better work together we have established some rules on how to contribute.
Bug reports and feature requests
--------------------------------
Bug reports and feature requests are handled through the GitHub issue system.
For Zoe itself, issues should be reported here: `https://github.com/DistributedSystemsGroup/zoe/issues <https://github.com/DistributedSystemsGroup/zoe/issues>`_
For the sample Zoe applications we provide, reports should be sent here: `https://github.com/DistributedSystemsGroup/zoe-applications/issues <https://github.com/DistributedSystemsGroup/zoe-applications/issues>`_
Bug reports and feature requests are handled through the GitHub issue system at: `https://github.com/DistributedSystemsGroup/zoe/issues <https://github.com/DistributedSystemsGroup/zoe/issues>`_
The issue system should be used for only for bug reports or feature requests. If you have more general questions, you need help setting up Zoe or running some application, please use the mailing list.
......@@ -38,6 +34,20 @@ To contribute code and/or documentation you should follow this workflow:
Zoe maintainers will review your code, give constructive feedback and eventually perform a pull and a merge.
Coding style
^^^^^^^^^^^^
Zoe code conforms to Python's `PEP8 coding style rules <https://www.python.org/dev/peps/pep-0008/>`_, with a few variations, detailed below:
* No line length limit
* All modules, classes, methods and functions must have a docstring
* The docstring for private methods is optional
* Names for unused variables and function parameters must end with ``_``
We also relaxed a number of pylint tests, check the ``.pylintrc`` file at the root of the source repository for details.
In general, if your code passes pylint, run with our configuration file with a 10/10 mark, it is ok and matches Zoe's coding rules.
Code quality and tests
^^^^^^^^^^^^^^^^^^^^^^
......
Internal API enpoint
====================
Internal API endpoint
=====================
.. automodule:: zoe_api.api_endpoint
:members:
......@@ -23,6 +23,7 @@ Using the Docker executor, we configured our runner with these options::
url = "https://gitlab.eurecom.fr/ci"
token = "<your-token-here>"
executor = "docker"
environment = ["DOCKER_AUTH_CONFIG={ \"auths\": { \"docker-registry:5000\": { \"auth\": \"some-base64-string\" } } }", "DOCKER_REGISTRY=docker-registry:5000"]
[runners.docker]
image = "docker:latest"
disable_cache = false
......@@ -37,7 +38,6 @@ Variables
To run the tests a number of variables need to be set from the GitLab interface:
* DOCKER_REGISTRY: the URL of the registry
* REGISTRY_PASSWORD: the password used for authenticating with the registry via docker login
* SONARQUBE_SERVER_URL: the URL of the SonarQube server
* SONARQUBE_USER: the SonarQube user
......
.. _scheduler:
The elastic scheduler
=====================
The elastic scheduler, available since the 2016.03 release, is able to use the information about elastic services encoded in ZApp descriptions to make efficient use of the available resources. The algorithm, along with a performance evaluation, is described in detail in this paper: `Flexible Scheduling of Distributed Analytic Applications <https://arxiv.org/abs/1611.09528>`_.
Scheduler classes
=================
......
......@@ -21,7 +21,7 @@ To use the Zoe command-line interface, first of all you have to define three env
export ZOE_USER=joe # User name
export ZOE_PASS=joesecret # Password
Now you can check that you are up and running with this command::
Now you can check if you are up and running with this command::
./zoe.py info
......@@ -53,7 +53,6 @@ Main documentation
kube_backend
config_file
logging
monitoring
proxy
Zoe applications
......@@ -65,7 +64,6 @@ Zoe applications
zapps/classification
zapps/howto_zapp
zapps/zapp_format
zapps/contributing
Development and contributing to the project
......
......@@ -24,6 +24,10 @@ Zoe components:
Zoe is written in Python and uses the ``requirements.txt`` file to list the package dependencies needed for all components of Zoe. Not all of them are needed in all cases, for example you need the ``kazoo`` library only if you use Zookeeper to manage Swarm high availability.
Optional components:
* :ref:`api-manager-label`
Overview
--------
......@@ -83,3 +87,13 @@ Currently this is the recommended procedure, once the initial Swarm setup has be
5. Start running ZApps!
In case of troubles, check the logs for errors. Zoe basic functionality can be tested via the ``zoe.py stats`` command. It will query the ``zoe-api`` process, that in turn will query the ``zoe-master`` process.
.. _api-manager-label:
API Manager
-----------
To provide TLS termination, authentication, load balancing, metrics, and other services to the Zoe API, you can use an API manager in front of the Zoe API. For example:
* Tyk: https://tyk.io/tyk-documentation/get-started/with-tyk-on-premise/
* Kong: https://getkong.org/docs/0.10.x/proxy/
.. _monitoring:
Monitoring interface
====================
Zoe has a built-in metrics generator able to send data to InfluxDB. By default it is disabled, it can be enabled by using the ``influxdb-*`` options available in the master configuration file. Metrics are generated for a number of internal events, listed below, and can be used to monitor Zoe performance and aliveness.
Please note that Zoe does not involve itself with container metrics, to gather container statistics you need to use third party tools able to talk directly to Docker. For example Telegraf, from InfluxData, is able to retain all the label associated with a container, thus producing very useful per-container metrics.
REST API metrics
----------------
These metrics report the latency measured during all API calls, as seen from the Zoe Master process.
service_time
^^^^^^^^^^^^
Time in milliseconds taken to service a request. The tags associated with the request will add more details:
* action: get, post, delete, ...
* user_id: user identifier of the authenticated user that performed the request
......@@ -16,11 +16,11 @@ Zoe strives to satisfy the following requirements:
* short (a few seconds) reaction times to user requests or other system events
* smart queuing and scheduling of applications when resources are critical
Kubernetes, OpenStack Sahara, Mesos and YARN are the projects that, each in its own way, come near Zoe, without solving the needs we have.
Kubernetes, OpenStack Sahara, Mesos and YARN are the projects that, each in its own way, come near Zoe, without solving the distributed analytics problem.
Kubernetes (Borg)
-----------------
Kubernetes is a very complex system, both to deploy and to use. It takes some of the architectural principles from Google Borg and targets data centers with vast amounts of resources. We feel that while Kubernetes can certainly run analytic services in containers, it does so at a very high complexity cost for smaller setups. Moreover, in our opinion, certain scheduler choices in how preemption is managed do not apply well to environments with a limited set of users and compute resources, causing a less than optimal resource usage.
Kubernetes
----------
Kubernetes is a very complex system, both to deploy and to use. It takes some of the architectural principles from Google Borg and targets data centers with vast amounts of resources. We feel that while Kubernetes can certainly run analytic services in containers, it does so at a very high complexity cost for smaller setups. Moreover, certain scheduler choices in how preemption is managed do not apply well to environments with a limited set of users and compute resources, causing a less than optimal resource usage.
OpenStack Sahara
----------------
......
......@@ -22,7 +22,7 @@ SonarQube
`SonarQube <https://www.sonarqube.org/>`_ is a code quality tool that performs a large number of static tests on the codebase. It applies rules from well-known coding standards like Misra, Cert and CWE and generates a quality report.
We configured our test pipeline to fail if the code quality of new commits is below the following rules:
We configured our test pipelines to fail if the code quality of new commits is below the following rules:
* Coverage less than 80%
* Maintainability worse than A
......@@ -43,15 +43,14 @@ Refer to the :ref:`integration-test` documentation for details.
Zapp image vulnerability scan
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
We use **clair**, a vulnerability static analyzer for Containers, from **CoreOS** to analyze Zoe docker image before using it.
We use `clair <https://github.com/coreos/clair>`_, a vulnerability static analyzer for Containers from CoreOS, to analyze the Zoe Docker image before using it.
If the base image you are using to build Zoe has too many vulnerabilities, you could choose another images which have less vulnerabilities.
The result after analyzing would be on the **console output** of the Jenkins job for Zoe. Insert the script below into the Zoe's Jenkins job to do the Clair analysis, all the necessary files could be found on ``ci/clair`` folder::
To scan the Zoe image (or any other Docker image) with clair, you can use the script below::
export imageID=`docker image inspect <your-registry-address>/zoe:$BUILD_ID | grep "Id" | awk -F ' ' '{print $2}' | awk -F ',' '{print $1}' | awk -F '"' '{print $2}'`
docker exec clair_clair analyzer $imageID
To start Clair and eventually add it to your Jenkins pipeline, check the files under the ``ci/clair`` folder.
RAML API description
--------------------
......
......@@ -5,8 +5,6 @@ Roadmap
We, the main developers of Zoe, are an academic research team. As such we have limited resources and through collaborations with other universities and private companies our aim is to do research and advance the state of the art. Our roadmap reflects this and pushes more on large-scale topics than on specific features.
The first priority for Zoe is to mature a stable and modular architecture on which advanced features can be built. Most of the work that is going into version 0.10.x is related to this point.
Scheduler architectures and resource allocation
-----------------------------------------------
......
......@@ -18,9 +18,9 @@ The users of Zoe
Deviation from the current ZApp terminology
-------------------------------------------
In the current Zoe implementation (0.10.x) ZApps are self-contained descriptions of a set of cooperating processes. They get submitted once, to request the start-up of an execution. This fits well the model of a single spark job or of a throw-away jupyter notebook.
In the current Zoe implementation ZApps are self-contained descriptions of a set of cooperating processes. They get submitted once, to request the start-up of an execution. This fits well the model of a single spark job or of a throw-away jupyter notebook.
We need to revise a bit this terminology: ZApps remain the top level, user-visible entity. The ZApp building blocks, analytic engines or support tools, are called frameworks. A framework, by itself, cannot be run. It lacks configuration or a user-provided binary to execute, for example. Each framework is composed of one or more processes, that come with some metadata of the needed configuration and a container image.
ZApps are at the top level, they are the user-visible entity. The ZApp building blocks, analytic engines or support tools, are called frameworks. A framework, by itself, cannot be run. It lacks configuration or a user-provided binary to execute, for example. Each framework is composed of one or more processes, that come with some metadata of the needed configuration and a container image.
A few examples:
......
.. _zapps_contributing:
Contributing ZApps
------------------
Zoe applications are maintained in the `zoe-applications <https://github.com/DistributedSystemsGroup/zoe-applications>`_ repository, feel free to fork it and generate pull requests for new applications, frameworks and services.
Check also the :ref:`howto_zapp` document for help on building ZApps from the already-available services and the :ref:`zapp_format`.
.. _zoe_fe:
Deploying the Angular FrontEnd
==============================
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment