Commit 6ff13872 authored by Sabino Papagna's avatar Sabino Papagna

Repository creation with new FE in angular2

parents
# Created by .ignore support plugin (hsz.mobi)
### Python template
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
# C extensions
*.so
# Distribution / packaging
.Python
env/
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib64/
parts/
sdist/
var/
*.egg-info/
.installed.cfg
*.egg
# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec
# Installer logs
pip-log.txt
pip-delete-this-directory.txt
# Unit test / coverage reports
htmlcov/
.tox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*,cover
# Translations
*.mo
*.pot
# Django stuff:
*.log
# Sphinx documentation
docs/_build/
# PyBuilder
target/
.idea/
state.zoe
/zoe*.conf
zoepass.csv
This diff is collapsed.
language: python
python:
- "3.4"
- "3.5"
install:
- pip install -r requirements.txt
- pip install -r requirements_tests.txt
script:
- pylint *.py zoe_*
- doc8 docs/
after_script:
- mypy -s *.py zoe_*
# Zoe Changelog
## Version 0.10.1
* Service discovery API endpoint: simple, read-only, unauthenticated access to a list DNS names of services. Needed for frameworks that need a list of hosts for configuration, can be used by scripts in the images.
* Add `start -s` option to the Zoe commandline client to have it wait for execution termination and stream the log meanwhile.
* Add the `workspace-deployment-path` option in the configuration file. It should be used when the workspace path should be build not with the deployment name, but with something else.
* All errors generated by docker while creating containers are now considered as fatal, except the "not enough free resource" one. When a fatal error is generated, Zoe will not try to start the execution again.
* Error messages generated by Docker are now exposed to the user
## Version 0.10.0
* This version is the start of a new series of releases moving toward a new architecture
* Only one overlay network (created by the sysadmin) will be used by all Zoe executions. We provide Dockerfiles for SOCKS and SSHd containers that can be used to give users access inside the Zoe overlay network. These containers will no longer be managed by Zoe.
* Move the REST API in the web process and add a ZeroMQ-based API between the web and the master.
* Move all user management in the web process.
* Use Postgresql to store the state
* Rename the zoe-web component in zoe-api to better describe its new role
* User authentication can be performed by a CSV text file or via LDAP
## Version 0.9.7
* Check application description version during validation.
* Bump application description version to 2 since we are going to make some major changes in the format
* Add fields `total_count`, `essential_count` and `startup_order` to service descriptions.
* Comment code related to the parsing of cluster statistics from Swarm. The data returned by API changes too often and parsing it consistently is too complex. A bug is open on the Swarm issue tracker.
## Version 0.9.6
* Workspaces: Zoe now supports starting containers with a directory from the hosts mounted as a volume. The directory is private for the user and is not wiped when the Zoe execution terminates. There are some issues with file permissions when the container images define users with arbitrary UIDs.
* Add an error status and error message visible to the user when Zoe fails starting an execution
* Add a Docker Compose file to help test deployments
* The gateway container image is now configurable
* Move applications to their own repository (zoe-applications)
* Move the Zoe logger code into its own repository
* Update the stats module to the latest Docker version. Since the Python Docker API refuses to provide machine readable output, we have to stay locked to Docker versions
* Add date and time to log output produced by Zoe processes
* Expand the web interface, users can now list, start, terminate and restart executions
FROM python:3.4
MAINTAINER Daniele Venzano <venza@brownhat.org>
RUN mkdir -p /opt/zoe
WORKDIR /opt/zoe
RUN apt-get update && apt-get install -y libldap2-dev libsasl2-dev && apt-get clean
COPY . /opt/zoe
RUN pip install --no-cache-dir -r requirements.txt
RUN echo 'admin,admin,admin' > /opt/zoe/zoepass.csv
VOLUME /etc/zoe/
RUN python3 ./zoe-api.py --write-config /etc/zoe/zoe.conf
This diff is collapsed.
Zoe - Container-based Analytics as a Service
============================================
Zoe provides a simple way to provision data analytics applications using Docker Swarm.
Resources:
- Main website: http://zoe-analytics.eu
- Documentation: http://docs.zoe-analytics.eu
- How to install: http://zoe-analytics.readthedocs.org/en/latest/install.html
Zoe applications can be easily created by users, we provide several examples in the `zoe-applications https://github.com/DistributedSystemsGroup/zoe-applications`_ repository to get you started.
Other Zoe resources:
- Zoe applications: https://github.com/DistributedSystemsGroup/zoe-applications
- Zoe logger: https://github.com/DistributedSystemsGroup/zoe-logger
A note on the master branch
---------------------------
We are currently redesigning Zoe with a new architecture, so the master branch is unstable and changes very rapidly.
The latest stable version is maintained under the 0.9.7-stable branch. All the documentation currently refers to this stable version, unless otherwise noted.
Repository contents
-------------------
- `contrib`: supervisord config files
- `docs`: Sphinx documentation
- `scripts`: Scripts used to test Zoe images outside of Zoe
- `zoe_cmd`: Command-line client
- `zoe_lib`: Client-side library, contains also some modules needed by the observer and the master processes
- `zoe_master`: The core of Zoe, the server process that listens for client requests and creates the containers on Swarm
- `zoe_api`: The web client interface
|Travis build| |Documentation Status|
Zoe is licensed under the terms of the Apache 2.0 license.
.. |Documentation Status| image:: https://readthedocs.org/projects/zoe-analytics/badge/?version=latest
:target: https://readthedocs.org/projects/zoe-analytics/?badge=latest
.. |Travis build| image:: https://travis-ci.org/DistributedSystemsGroup/zoe.svg
:target: https://travis-ci.org/DistributedSystemsGroup/zoe
#!/usr/bin/env bash
set -e
export ZOE_USER=admin
export ZOE_PASS=changeme
export ZOE_URL=http://localhost:4850
echo "Get statistics"
./zoe.py stats
echo "Get superuser"
./zoe.py user-get zoeadmin
echo "Create a new user"
./zoe.py user-new --name test --password test --role guest
echo "Delete a user"
./zoe.py user-rm test
echo "Export an application template"
./zoe.py pre-app-export hdfs > /tmp/zoe-hdfs.json
[program:zoe-master]
command=/usr/bin/python3 /home/ubuntu/zoe/zoe-master.py
directory=/home/ubuntu/zoe
autostart=true
autorestart=true
startretries=3
stderr_logfile=/var/log/zoe/master.err.log
stdout_logfile=/var/log/zoe/master.out.log
user=ubuntu
[program:zoe-web]
command=/usr/bin/python3 /home/ubuntu/zoe/zoe-web.py
directory=/home/ubuntu/zoe
autostart=true
autorestart=true
startretries=3
stderr_logfile=/var/log/zoe/web.err.log
stdout_logfile=/var/log/zoe/web.out.log
user=ubuntu
[program:zoe-observer]
command=/usr/bin/python3 /home/ubuntu/zoe/zoe-observer.py
directory=/home/ubuntu/zoe
autostart=true
autorestart=true
startretries=3
stderr_logfile=/var/log/zoe/observer.err.log
stdout_logfile=/var/log/zoe/observer.out.log
user=ubuntu
{
"name": "aml-lab",
"priority": 512,
"requires_binary": false,
"services": [
{
"docker_image": "192.168.45.252:5000/zoerepo/spark-master",
"environment": [
[
"SPARK_MASTER_IP",
"spark-master-{execution_name}-{user_name}-{deployment_name}-zoe.{user_name}-{deployment_name}-zoe"
]
],
"monitor": false,
"name": "spark-master",
"networks": [],
"ports": [
{
"is_main_endpoint": false,
"name": "Spark master web interface",
"path": "/",
"port_number": 8080,
"protocol": "http"
}
],
"required_resources": {
"memory": 536870912
}
},
{
"docker_image": "192.168.45.252:5000/zoerepo/spark-worker",
"environment": [
[
"SPARK_WORKER_CORES",
"6"
],
[
"SPARK_WORKER_RAM",
"11273240064"
],
[
"SPARK_MASTER_IP",
"spark-master-{execution_name}-{user_name}-{deployment_name}-zoe.{user_name}-{deployment_name}-zoe"
],
[
"SPARK_LOCAL_IP",
"spark-worker0-{execution_name}-{user_name}-{deployment_name}-zoe.{user_name}-{deployment_name}-zoe"
]
],
"monitor": false,
"name": "spark-worker0",
"networks": [],
"ports": [
{
"is_main_endpoint": false,
"name": "Spark worker web interface",
"path": "/",
"port_number": 8081,
"protocol": "http"
}
],
"required_resources": {
"memory": 12884901888
}
},
{
"docker_image": "192.168.45.252:5000/zoerepo/spark-jupyter-notebook",
"environment": [
[
"SPARK_MASTER",
"spark://spark-master-{execution_name}-{user_name}-{deployment_name}-zoe.{user_name}-{deployment_name}-zoe:7077"
],
[
"SPARK_EXECUTOR_RAM",
"11273240064"
],
[
"SPARK_DRIVER_RAM",
"2147483648"
],
[
"NB_USER",
"{user_name}"
],
[
"NAMENODE_HOST",
"hdfs-namenode.hdfs"
]
],
"monitor": true,
"name": "spark-jupyter",
"networks": [],
"ports": [
{
"is_main_endpoint": false,
"name": "Spark application web interface",
"path": "/",
"port_number": 4040,
"protocol": "http"
},
{
"is_main_endpoint": true,
"name": "Jupyter Notebook interface",
"path": "/",
"port_number": 8888,
"protocol": "http"
}
],
"required_resources": {
"memory": 4294967296
}
},
{
"docker_image": "192.168.45.252:5000/zoerepo/spark-worker",
"environment": [
[
"SPARK_WORKER_CORES",
"6"
],
[
"SPARK_WORKER_RAM",
"11273240064"
],
[
"SPARK_MASTER_IP",
"spark-master-{execution_name}-{user_name}-{deployment_name}-zoe.{user_name}-{deployment_name}-zoe"
],
[
"SPARK_LOCAL_IP",
"spark-worker1-{execution_name}-{user_name}-{deployment_name}-zoe.{user_name}-{deployment_name}-zoe"
]
],
"monitor": false,
"name": "spark-worker1",
"networks": [],
"ports": [
{
"is_main_endpoint": false,
"name": "Spark worker web interface",
"path": "/",
"port_number": 8081,
"protocol": "http"
}
],
"required_resources": {
"memory": 12884901888
}
}
],
"version": 1,
"will_end": false
}
{
"name": "openmpi-helloworld",
"priority": 512,
"requires_binary": true,
"services": [
{
"command": "",
"docker_image": "192.168.45.252:5000/zoerepo/openmpi-worker",
"environment": [],
"monitor": false,
"name": "mpiworker0",
"ports": [],
"required_resources": {
"memory": 1073741824
},
"volumes": []
},
{
"command": "",
"docker_image": "192.168.45.252:5000/zoerepo/openmpi-worker",
"environment": [],
"monitor": false,
"name": "mpiworker1",
"ports": [],
"required_resources": {
"memory": 1073741824
},
"volumes": []
},
{
"command": "",
"docker_image": "192.168.45.252:5000/zoerepo/openmpi-worker",
"environment": [],
"monitor": false,
"name": "mpiworker2",
"ports": [],
"required_resources": {
"memory": 1073741824
},
"volumes": []
},
{
"command": "",
"docker_image": "192.168.45.252:5000/zoerepo/openmpi-worker",
"environment": [],
"monitor": false,
"name": "mpiworker3",
"ports": [],
"required_resources": {
"memory": 1073741824
},
"volumes": []
},
{
"command": "mpirun -np 4 --hostfile ./mpi-helloworld/mpihosts ./mpi-helloworld/MPI_Hello",
"docker_image": "192.168.45.252:5000/zoerepo/openmpi-worker",
"environment": [],
"monitor": true,
"name": "mpirun",
"ports": [],
"required_resources": {
"memory": 1073741824
},
"volumes": []
}
],
"version": 1,
"will_end": true
}
{
"name": "spark-submit",
"priority": 512,
"requires_binary": true,
"services": [
{
"docker_image": "192.168.45.252:5000/zoerepo/spark-master",
"environment": [
[
"SPARK_MASTER_IP",
"spark-master-{execution_name}-{user_name}-{deployment_name}-zoe.{user_name}-{deployment_name}-zoe"
]
],
"monitor": false,
"name": "spark-master",
"networks": [],
"ports": [
{
"is_main_endpoint": false,
"name": "Spark master web interface",
"path": "/",
"port_number": 8080,
"protocol": "http"
}
],
"required_resources": {
"memory": 4294967296
}
},
{
"command": "--class fr.eurecom.dsg.WordCount wc.jar WC-GBig hdfs://hdfs-namenode.hdfs/datasets/gutenberg/gutenberg_small.txt hdfs://hdfs-namenode.hdfs/tmp/cntwdc1",
"docker_image": "192.168.45.252:5000/zoerepo/spark-submit",
"environment": [
[
"SPARK_MASTER_IP",
"spark-master-{execution_name}-{user_name}-{deployment_name}-zoe.{user_name}-{deployment_name}-zoe"
],
[
"SPARK_EXECUTOR_RAM",
"6442450944"
],
["WS_DIR", "wordcount"]
],
"monitor": true,
"name": "spark-submit",
"networks": [],
"ports": [
{
"is_main_endpoint": false,
"name": "Spark application web interface",
"path": "/",
"port_number": 4040,
"protocol": "http"
}
],
"required_resources": {
"memory": 4294967296
}
},
{
"docker_image": "192.168.45.252:5000/zoerepo/spark-worker",
"environment": [
[
"SPARK_WORKER_CORES",
"4"
],
[
"SPARK_WORKER_RAM",
"6978272768"
],
[
"SPARK_MASTER_IP",
"spark-master-{execution_name}-{user_name}-{deployment_name}-zoe.{user_name}-{deployment_name}-zoe"
],
[
"SPARK_LOCAL_IP",
"spark-worker0-{execution_name}-{user_name}-{deployment_name}-zoe.{user_name}-{deployment_name}-zoe"
]
],
"monitor": false,
"name": "spark-worker0",
"networks": [],
"ports": [
{
"is_main_endpoint": false,
"name": "Spark worker web interface",
"path": "/",
"port_number": 8081,
"protocol": "http"
}
],
"required_resources": {
"memory": 8589934592
}
},
{
"docker_image": "192.168.45.252:5000/zoerepo/spark-worker",
"environment": [
[
"SPARK_WORKER_CORES",
"4"
],
[
"SPARK_WORKER_RAM",
"6978272768"
],
[
"SPARK_MASTER_IP",
"spark-master-{execution_name}-{user_name}-{deployment_name}-zoe.{user_name}-{deployment_name}-zoe"
],
[
"SPARK_LOCAL_IP",
"spark-worker1-{execution_name}-{user_name}-{deployment_name}-zoe.{user_name}-{deployment_name}-zoe"
]
],
"monitor": false,
"name": "spark-worker1",
"networks": [],
"ports": [
{
"is_main_endpoint": false,
"name": "Spark worker web interface",
"path": "/",
"port_number": 8081,
"protocol": "http"
}
],
"required_resources": {
"memory": 8589934592
}
},
{
"docker_image": "192.168.45.252:5000/zoerepo/spark-worker",
"environment": [
[
"SPARK_WORKER_CORES",
"4"
],
[
"SPARK_WORKER_RAM",
"6978272768"
],
[
"SPARK_MASTER_IP",
"spark-master-{execution_name}-{user_name}-{deployment_name}-zoe.{user_name}-{deployment_name}-zoe"
],
[
"SPARK_LOCAL_IP",
"spark-worker2-{execution_name}-{user_name}-{deployment_name}-zoe.{user_name}-{deployment_name}-zoe"
]
],
"monitor": false,
"name": "spark-worker2",
"networks": [],
"ports": [
{
"is_main_endpoint": false,
"name": "Spark worker web interface",
"path": "/",
"port_number": 8081,
"protocol": "http"
}
],
"required_resources": {
"memory": 8589934592
}
}
],
"version": 1,
"will_end": false
}
[doc8]
ignore-path=docs/_build
ignore=D001
version: '2'
services:
postgres:
image: postgres
gateway-socks:
image: zoerepo/gateway-socks
networks:
- zoe
zoe-api:
image: zoerepo/zoe
command: python3 zoe-api.py --debug --swarm ${SWARM_URL} --deployment-name compose --master-url tcp://zoe-master:4850 --dbuser postgres --dbhost postgres --dbname postgres
ports:
- "8080:5001"
depends_on:
- postgres
zoe-master:
image: zoerepo/zoe
ports:
- "4850:4850"
volumes:
- /etc/zoe:/etc/zoe
- /opt/zoe-workspaces:/mnt/zoe-workspaces
command: python3 zoe-master.py --debug --swarm ${SWARM_URL} --deployment-name compose --dbuser postgres --dbhost postgres --dbname postgres
depends_on:
- zoe-api
networks:
zoe:
driver: bridge
# Makefile for Sphinx documentation
#
# You can set these variables from the command line.
SPHINXOPTS = -b coverage
SPHINXBUILD = sphinx-build
PAPER =
BUILDDIR = _build
# User-friendly check for sphinx-build
ifeq ($(shell which $(SPHINXBUILD) >/dev/null 2>&1; echo $$?), 1)
$(error The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed, then set the SPHINXBUILD environment variable to point to the full path of the '$(SPHINXBUILD)' executable. Alternatively you can add the directory with the executable to your PATH. If you don't have Sphinx installed, grab it from http://sphinx-doc.org/)
endif
# Internal variables.
PAPEROPT_a4 = -D latex_paper_size=a4
PAPEROPT_letter = -D latex_paper_size=letter
ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
# the i18n builder cannot share the environment and doctrees with the others