Commit ed287d9a authored by Daniele Venzano's avatar Daniele Venzano

Merge from master branch

parents e1db183a e3fe2eec
# Zoe Changelog
## Version 0.20.0
## Version 0.20.0 - in progress
* The scheduler implementation is now configurable
* New elastic scheduler added, the default remains the one available in previous releases
......
Zoe - Container-based Analytics as a Service
============================================
Zoe provides a simple way to provision data analytics applications using Docker Swarm.
Zoe provides a simple way to provision data analytics applications using containers.
Resources:
- Main website: http://zoe-analytics.eu
- Documentation: http://docs.zoe-analytics.eu
- How to install: http://zoe-analytics.readthedocs.org/en/latest/install.html
- Main website: http://zoe-analytics.eu
- Documentation: http://docs.zoe-analytics.eu
- How to install: http://docs.zoe-analytics.eu/en/latest/install.html
- Project roadmap and guidelines: https://github.com/DistributedSystemsGroup/zoe/wiki
Zoe applications can be easily created by users, we provide several examples in the `zoe-applications https://github.com/DistributedSystemsGroup/zoe-applications`_ repository to get you started.
Other Zoe resources:
- Zoe applications: https://github.com/DistributedSystemsGroup/zoe-applications
- Zoe logger: https://github.com/DistributedSystemsGroup/zoe-logger
- Development and API documentation: http://docs.zoe-analytics.eu/en/latest/developer/index.html
A note on the master branch
---------------------------
We are currently redesigning Zoe with a new architecture, so the master branch is unstable and changes very rapidly.
The latest stable version is maintained under the 0.9.7-stable branch. All the documentation currently refers to this stable version, unless otherwise noted.
A note on releases and the master branch
----------------------------------------
The Zoe master branch is tested before being published and only commits that pass the tests are published on this repository. A description of the CI pipeline is here: http://docs.zoe-analytics.eu/en/latest/quality.html
Official releases are tagged with their version number in the repository and changes for each released version are summarised in the `ChangeLog https://github.com/DistributedSystemsGroup/zoe/blob/master/CHANGELOG.md`_ file.
Repository contents
-------------------
......@@ -34,11 +35,9 @@ Repository contents
- `zoe_master`: The core of Zoe, the server process that listens for client requests and creates the containers on Swarm
- `zoe_api`: The web client interface
|Travis build| |Documentation Status|
|Documentation Status|
Zoe is licensed under the terms of the Apache 2.0 license.
.. |Documentation Status| image:: https://readthedocs.org/projects/zoe-analytics/badge/?version=latest
:target: https://readthedocs.org/projects/zoe-analytics/?badge=latest
.. |Travis build| image:: https://travis-ci.org/DistributedSystemsGroup/zoe.svg
:target: https://travis-ci.org/DistributedSystemsGroup/zoe
......@@ -20,7 +20,7 @@ To better understand what we mean by "analytic service", here are a few examples
A number of predefined applications for testing and customization can be found at the `zoe-applications <https://github.com/DistributedSystemsGroup/zoe-applications>`_ repository.
Have a look at the :ref:`vision` and at the :ref:`roadmap` to see what we are currently planning and feel free to `contact us <daniele.venzano@eurecom.fr>`_ via email or through the `GitHub issue tracker <https://github.com/DistributedSystemsGroup/zoe/issues>`_ to pose questions or suggest ideas and new features.
Have a look at the :ref:`vision` and at the `roadmap <https://github.com/DistributedSystemsGroup/zoe/wiki>`_ to see what we are currently planning and feel free to `contact us <daniele.venzano@eurecom.fr>`_ via email or through the `GitHub issue tracker <https://github.com/DistributedSystemsGroup/zoe/issues>`_ to pose questions or suggest ideas and new features.
A note on terminology (needs to be updated)
-------------------------------------------
......@@ -46,7 +46,6 @@ Contents
quality
vision
motivations
roadmap
contributing
Zoe applications
......
.. _roadmap:
Roadmap
=======
We, the main developers of Zoe, are an academic research team. As such we have limited resources and through collaborations with other universities and private companies our aim is to do research and advance the state of the art. Our roadmap reflects this and pushes more on large-scale topics than on specific features.
The first priority for Zoe is to mature a stable and modular architecture on which advanced features can be built. Most of the work that is going into version 0.10.x is related to this point.
Scheduler architectures and resource allocation
-----------------------------------------------
In parallel to classic, stable and well known schedulers (FIFO), we plan to design and implement within Zoe novel approaches to application scheduling and resource allocation. This includes:
* Optimistic, pessimistic, distributed, centralized schedulers
* Distributed or centralized schedulers
Scheduling policies
-------------------
While the FIFO policy is fine for many settings, is it not the most efficient way of managing work that can be done concurrently. Many decades of scheduling literature point in all sorts of directions, some of which can find new applications in analytic systems:
* Appropriate management of batch Vs interactive Vs streaming analytic applications
* Deadline scheduling for streaming frameworks
* Size-based scheduling better utilization and smaller response times
Dynamic resource allocation
---------------------------
Users are usually bad guessers on how many resources a particular application will need. We all have a tendency of overestimating resource reservations to make sure there is some headroom for unplanned spikes. This overestimation causes low utilization and non-efficient resource usage: with better reservation and allocation mechanisms that can adapt at runtime, more work could be done with the same resources.
* Resize dynamically running applications in terms of number of services
* Resize dynamically running applications in terms of memory and cores allocated for each service
Fault tolerance
---------------
Any modern system must be able to cope with faults and failures of any kind. Zoe is currently built around state of the art mechanisms for fault tolerance, but this does not stop us from further investigating fault tolerance mechanisms both for Zoe itself and for the applications it runs.
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment