Commit 5d65736a authored by Daniele Venzano's avatar Daniele Venzano

Update the documentation with the new configuration options and the shared...

Update the documentation with the new configuration options and the shared filesystem for workspaces
parent eccabd82
.. _config_file:
Zoe config files
================
Zoe configuration
=================
Zoe can be configured by files, environment variables or commandline options. The configuration directives listed in this file can be specified by any of the three methods. Use the ``--help`` command-line option to have more details on the format od environment variables and precedence rules.
Also the directive ``--write-config <filename>`` is available: it will generate a configuration file with all options set to the default values.
Zoe config files have a simple format of ``<option name> = <value>``. Dash characters can be use for comments.
Each Zoe component has its own configuration file, as described in the following sections:
Each Zoe component has its own configuration directives, as described in the following sections:
zoe-master.conf
---------------
* ``debug = <true|false>`` : enable or disable debug log output
* ``swarm = zk://zk1:2181,zk2:2181,zk3:2181`` : connection string to the Swarm API endpoint. Can be expressed by a plain http URL or as a zookeeper node list in case Swarm is configured for HA.
* ``private-registry = <address:port>`` : address of the private registry containing Zoe images
* ``state-dir = /var/lib/zoe`` : Directory where all state and other binaries (execution logs) are saved.
* ``zoeadmin-password = changeme`` : Password for the zoeadmin user
* ``deployment-name = devel`` : name of this Zoe deployment. Can be used to have multiple Zoe deployments using the same Swarm (devel and prod, for example)
......@@ -21,14 +24,15 @@ zoe-master.conf
* ``influxdb-enable = False`` : Enable metric output toward influxDB
* ``passlib-rounds = 60000`` : Number of hashing rounds for passwords, has a sever performance impact on each API call
* ``gelf-address = udp://1.2.3.4:1234`` : Enable Docker GELF log output to this destination
* ``workspace-base-path = /mnt/zoe-workspaces`` : Base directory where user workspaces will be created. This directory should reside on a shared filesystem visible by all Docker hosts.
* ``guest-gateway-image-name`` : Docker image for guests gateway container (ex.: zoerepo/guest-gateway). The default image contains an ssh-based SOCKS proxy.
* ``user-gateway-image-name`` : Docker image for users gateway container (ex.: zoerepo/guest-gateway). The default image contains an ssh-based SOCKS proxy.
zoe-observer.conf
-----------------
* ``debug = <true|false>`` : enable or disable debug log output
* ``swarm = zk://zk1:2181,zk2:2181,zk3:2181`` : connection string to the Swarm API endpoint. Can be expressed by a plain http URL or as a zookeeper node list in case Swarm is configured for HA.
* ``zoeadmin-password = changeme`` : Password for the zoeadmin user
* ``deployment-name = devel`` : name of this Zoe deployment. Can be used to have multiple Zoe deployments using the same Swarm (devel and prod, for example)
* ``master-url = http://<address:port>`` : address of the Zoe Master REST API
* ``spark-activity-timeout = <seconds>`` : number of seconds to wait before an inactive Spark cluster is automatically terminated, this is done only for guest users
* ``loop-time = 300`` : time in seconds between successive checks for idle applications that can be automatically terminated
......@@ -46,8 +50,4 @@ zoe-logger.conf
This component is optional.
* ``debug = <true|false>`` : enable or disable debug log output
* ``deployment-name = devel`` : name of this Zoe deployment. Can be used to have multiple Zoe deployments using the same Swarm (devel and prod, for example)
* ``influxdb-dbname = zoe`` : Name of the InfluxDB database to use for storing metrics
* ``influxdb-url = http://localhost:8086`` : URL of the InfluxDB service (ex. )
* ``influxdb-enable = False`` : Enable metric output toward influxDB
* ``kafka-broker = 1.2.3.4:9092``: Address of the Kafka broker to send logs to
......@@ -5,7 +5,7 @@ Zoe components:
* Master
* Observer
* logger
* logger (optional)
* web client
* command-line client
......@@ -13,6 +13,17 @@ Zoe is written in Python and uses the ``requirements.txt`` file to list the pack
Zoe is a young software project and we foresee it being used in places with wildly different requirements in terms of IT organization (what is below Zoe) and user interaction (what is above Zoe). For this reason we are aiming at providing a solid core of features and a number of basic external components that can be easily customized. For example, the Spark idle monitoring feature is useful only in certain environments and it is implemented as an external service, that can be customized of takes as an example to build something different.
There is an experimental configuration file for Docker Compose, if you want to try it. It will run Zoe and its components inside Docker containers. It needs to be customized with the address of your Swarm master, the port mappings and the location of a shared filesystem.
Overview
--------
The applications run by Zoe, usually, expose a number of interfaces (web, rest and others) to the user. Docker Swarm does not provide an easy way to manage this situation, the prt can be statically allocated, by the public IP address is chosen arbitrarily by Swarm and there is no discovery mechanism (DNS) exposed to the outside of Swarm.
To work around this problem Zoe creates a gateway container for each user. The image used for this gateway container is configurable. The default one, downloaded from the Docker hub contains an ssh-based SOCKS proxy that the user must configure in his/her browser to be able to access the services run by Zoe executions.
Zoe requires a shared filesystem, visible from all Docker hosts. Some Zoe Applications (for example spark submit, MPI) require user-provided binaries to run. Zoe creates and maintains for each user a workspace directory on this shared filesystem. The user can access the directory from outside Zoe and put the files required for his/her application. We are evaluating whether to integrate into the Zoe web client some kind of web interface for accessing the workspace directory.
Requirements
------------
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment