Commit 5422016b authored by Daniele Venzano's avatar Daniele Venzano

Update README and documentation landing page

Reduce the content of the README so that it does not easily get outdated. Rewrite the landing page, taking Spark as an example, to be more direct and concise. Remove the introduction for the developer documentation, as it is verbose and confusing, clarify also the developer documentation landing page.
Move the Kubernetes backend page to the top level as it is about setup, not development.
parent 57ec2f7a
Zoe - Container-based Analytics as a Service
============================================
Zoe provides a simple way to provision data analytics applications.
Zoe provides a simple way to provision any kind of data analytics applications.
Resources:
- Main website: http://zoe-analytics.eu
- Website: http://zoe-analytics.eu
- Documentation: http://docs.zoe-analytics.eu
- How to install: :ref:`install`
- Mailing list: http://www.freelists.org/list/zoe
- Issue tracker: https://github.com/DistributedSystemsGroup/zoe/issues
Zoe applications:
Zoe applications (ZApps):
- Repository: https://github.com/DistributedSystemsGroup/zoe-applications
- How to create a ZApp: :ref:`howto_zapp`
Contents
--------
.. toctree::
:maxdepth: 2
install
config_file
logging
monitoring
architecture
quality
vision
motivations
roadmap
contributing
Zoe applications
----------------
:ref:`modindex`
.. toctree::
:maxdepth: 2
zapps/classification
zapps/howto_zapp
zapps/zapp_format
zapps/contributing
Developer documentation
-----------------------
:ref:`modindex`
.. toctree::
:maxdepth: 1
developer/index
Contacts
========
`Zoe website <http://zoe-analytics.eu>`_
The main discussion area for issues, questions and feature requests is the `GitHub issue tracker <https://github.com/DistributedSystemsGroup/zoe/issues>`_.
Zoe is licensed under the terms of the Apache 2.0 license.
......@@ -3,27 +3,69 @@
Developer documentation
=======================
Repository contents
As a developer you can:
- call Zoe from your own software: :ref:`Zoe REST API documentation <rest-api>`
- create ot modify ZApps: :ref:`howto_zapp`
- contribute to Zoe: keep reading
Contributing to Zoe
-------------------
- `contrib`: supervisord config files
- `docs`: Sphinx documentation
- `scripts`: Scripts used to test Zoe images outside of Zoe
Zoe is open source and all kinds of contributions are welcome.
Zoe is licensed under the terms of the Apache 2.0 license.
Bugs, issues and feature requests
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
`Zoe issue tracker <https://github.com/DistributedSystemsGroup/zoe/issues>`_
Testing beta code
^^^^^^^^^^^^^^^^^
The ``HEAD`` of the master branch represents the latest version of Zoe. Automatic tests are performed before code is merged into master, but human feedback is invaluable. Clone the repository and report on the `mailing list <http://www.freelists.org/list/zoe>`_ or on the `issue tracker <https://github.com/DistributedSystemsGroup/zoe/issues>`_.
Code changes and pull requests
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
**When you contribute code, you affirm that the contribution is your original work and that you license the work to the project under the project’s open source license. Whether or not you state this explicitly, by submitting any copyrighted material via pull request, email, or other means you agree to license the material under the project’s open source license and warrant that you have the legal authority to do so.**
To contribute code and/or documentation you should follow this workflow:
1. announce your idea on the mailing list, to prevent duplicated work
2. fork the Zoe repository via GitHub (if you don't already have a fork)
3. ... develop and debug ...
4. when you are ready propose your changes with a pull request
Zoe maintainers will review your code, give constructive feedback and eventually accept the code and merge.
Contributors can setup their own CI pipeline following the quality guidelines (:ref:`quality`). At a bare minimum all code should be tested via the `run_tests.sh` script available in the root of the repository. Accepted contributions will be run through the full Zoe CI pipeline before being merged in the public repository.
Repository contents
^^^^^^^^^^^^^^^^^^^
- `docs`: Sphinx documentation used to build these pages
- `scripts`: scripts for deployment and testing
- `zoe_api`: the front-end Zoe process that provides the REST API
- `zoe_cmd`: Command-line client
- `zoe_lib`: Client-side library, contains also some modules needed by the observer and the master processes
- `zoe_master`: The core of Zoe, the server process that listens for client requests and creates the containers on Swarm
- `zoe_api`: The web client interface
- `zoe_lib`: library, contains common modules needed by the api and the master processes
- `zoe_master`: the back-end Zoe process schedules and talks to the containerization system
- `contrib`: supervisord config files and sample ZApps
:ref:`modindex`
Internal module/class/method documentation
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. toctree::
:maxdepth: 2
:maxdepth: 1
introduction
rest-api
auth
rest-api
api-endpoint
master-api
scheduler
backend
stats
:ref:`modindex`
General design decisions
========================
In this architecture we overturned our previous decision of keeping state internal, with periodic checkpointing.
State is kept in Postgres and shared among the different Zoe components. For a distributed system an external database simplifies enormously many common situation, with transactions and strong guarantees of consistency.
User management is left out of Zoe as much as possible. User authentication backends provide just a minimum of information for Zoe: a user ID and a role. Zoe does not manage creation, deletion, passwords, etc.
Zoe is distributed and uses threads to keep the APIs responsive at all times.
Object naming
-------------
Database IDs are used to identify executions and services. Container names within Docker Swarm must be unique, we decided to produce names that give some information to the administrator who looks at the output of ``docker ps`` instead of using opaque UUIDs. In addition, these same names are exposed by standard monitoring tools.
.. include:: ../README.rst
.. _main_index:
Zoe - Container-based Analytics as a Service
============================================
Zoe provides a simple way to provision data analytics applications. It hides the complexities of managing resources, configuring and deploying complex distributed applications on private clouds. Zoe is focused on data analysis applications, such as `Spark <http://spark.apache.org/>`_ or `Tensorflow <https://www.tensorflow.org/>`_. A generic, very flexible application description format lets you easily describe any kind of data analysis application.
Downloading
-----------
Get Zoe from the `GitHub repository <https://github.com/DistributedSystemsGroup/zoe>`_. Stable releases are tagged on the master branch and can be downloaded from the `releases page <https://github.com/DistributedSystemsGroup/zoe/releases>`_.
Zoe is written in Python 3.4+ and requires a number of third-party packages to function. Deployment scripts for the supported back-ends, install and setup instructions are available in the :ref:`installation guide <install>`.
Quick tutorial
--------------
To use the Zoe command-line interface, first of all you have to define three environment variables::
export ZOE_URL=http://localhost:5000 # address of the zoe-api instance
export ZOE_USER=joe # User name
export ZOE_PASS=joesecret # Password
Now you can check that you are up and running with this command::
./zoe.py info
It will return some version information, by querying the zoe-api and zoe-master processes.
Zoe applications are passed as JSON files. A few sample ZApps are available in the ``contrib/zoeapps/`` directory. To start a ZApp use the following command::
./zoe.py start joe-spark-notebook contrib/zoeapps/eurecom_aml_lab.json
ZApp execution status can be checked this way::
./zoe.py exec-ls # Lists all executions, past and present
./zoe.py exec-get <execution id> # Inspects an execution
Where ``execution id`` is the ID of the ZApp execution to inspect, taken from the ``exec-ls`` command.
Where to go from here
---------------------
Main documentation
^^^^^^^^^^^^^^^^^^
.. toctree::
:maxdepth: 1
install
kube_backend
config_file
logging
monitoring
proxy
Zoe applications
^^^^^^^^^^^^^^^^
.. toctree::
:maxdepth: 1
zapps/classification
zapps/howto_zapp
zapps/zapp_format
zapps/contributing
Development and contributing to the project
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. toctree::
:maxdepth: 1
developer/index
architecture
quality
motivations
vision
contributing
External resources
^^^^^^^^^^^^^^^^^^
- `Zoe Homepage <http://zoe-analytics.eu>`_
- `Issue tracker <https://github.com/DistributedSystemsGroup/zoe/issues>`_
- `ZApp repository <https://github.com/DistributedSystemsGroup/zoe-applications>`_
Zoe is licensed under the terms of the Apache 2.0 license.
......@@ -18,16 +18,16 @@ How it works
1. Zoe configuration file:
* ``--backend``: put Kubernetes instead of Docker Swarm
* ``--kube-config-file``: the configuration file of kubernetes cluster
* ``--kube-config-file``: the configuration file of Kubernetes cluster
2. Zoe:
* Zapp Description:
* Add new field: ``replicas``, if users doesn't specify this value, the default value for each service would be ``1``.
* In field ``require_resources``, the ``cores`` field can be float.
* Idea:
* Idea:
* Create each **replication controller** per each service of a Zapp. Replication Controller assures to have at least a number of **replicas** (pod) always running.
* Create a Kubernetes **service** per each **replication controller**, which has the same **labels** and **label selectors** with the associated **replication controller**. The service would help the zapp service be exposed to the network by exposing the same port of the service on all kubernetes nodes.
......@@ -37,5 +37,5 @@ References
* Kubernetes: https://kubernetes.io/
* Kubernetes Replication Controller : https://kubernetes.io/docs/user-guide/replication-controller/
* Kubernetes Service: https://kubernetes.io/docs/user-guide/services/
* Kubernetes Limit and Request: https://kubernetes.io/docs/user-guide/compute-resources/
* Kubernetes Limit and Request: https://kubernetes.io/docs/user-guide/compute-resources/
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment