Commit 4f575398 authored by Daniele Venzano's avatar Daniele Venzano

Remove the zoe-logger code, as it now lives in a separate repository, update the documentation.

parent dc0b7474
......@@ -11,6 +11,12 @@ Resources:
Zoe applications can be easily created by users, we provide several examples in the `zoe-applications https://github.com/DistributedSystemsGroup/zoe-applications`_ repository to get you started.
Other Zoe resources:
- Zoe applications: https://github.com/DistributedSystemsGroup/zoe-applications
- Zoe logger: https://github.com/DistributedSystemsGroup/zoe-logger
Repository contents
-------------------
......@@ -19,7 +25,6 @@ Repository contents
- `scripts`: Scripts used to test Zoe images outside of Zoe
- `zoe_cmd`: Command-line client
- `zoe_lib`: Client-side library, contains also some modules needed by the observer and the master processes
- `zoe_logger`: Optional Kafka producer for Docker container logs
- `zoe_observer`: The Observer process that monitors Swarm and informs the master of various events
- `zoe_master`: The core of Zoe, the server process that listens for client requests and creates the containers on Swarm
- `zoe_web`: The web client interface
......
......@@ -43,11 +43,3 @@ zoe-web.conf
* ``listen-address`` : address Zoe will use to listen for incoming connections to the web interface
* ``listen-port`` : port Zoe will use to listen for incoming connections to the web interface
* ``master-url = http://<address:port>`` : address of the Zoe Master REST API
zoe-logger.conf
---------------
This component is optional.
* ``debug = <true|false>`` : enable or disable debug log output
* ``kafka-broker = 1.2.3.4:9092``: Address of the Kafka broker to send logs to
......@@ -5,7 +5,7 @@ Zoe components:
* Master
* Observer
* logger (optional)
* logger (optional, see https://github.com/DistributedSystemsGroup/zoe-logger)
* web client
* command-line client
......
Container logs
==============
By default Zoe does not involve itself with the output from container processes. The logs can be retrieved with the usual Docker command ``docker logs`` while a container is alive and then they are lost forever when the container is deleted.
By default Zoe does not involve itself with the output from container processes. The logs can be retrieved with the usual Docker command ``docker logs`` while a container is alive and then they are lost forever when the container is deleted. This solution however does not scale very well: users need to used docker commandline tools and when containers produce lots of output Docker will fill-up the disk of whatever Docker host the container is running in.
Using the ``gelf-address`` option of the Zoe Master process, Zoe can configure Docker to send the container outputs to an external destination in GELF format. GELF is the richest format supported by Docker and can be ingested by a number of tools such as Graylog and Logstash. When that option is set all containers created by Zoe will send their output (standard output and standard error) to the destination specified.
Docker is instructed to add all Zoe-defined tags to the GELF messages, so that they can be aggregated by Zoe Execution, Zoe User, etc.
Docker is instructed to add all Zoe-defined tags to the GELF messages, so that they can be aggregated by Zoe execution, Zoe user, etc.
Zoe also provides a Zoe Logger process, in case you prefer to use Kafka in your log pipeline. Each container output will be sent to its own topic, that Kafka will conserve for seven days by default. With Kafka you can also monitor the container output in real-time, for example to debug your container images running in Zoe. In this case GELF is converted to syslog-like format for easier handling
Zoe also provides a Zoe Logger process, in case you prefer to use Kafka in your log pipeline. Each container output will be sent to its own topic, that Kafka will retain for seven days by default. With Kafka you can also monitor the container output in real-time, for example to debug your container images running in Zoe. In this case GELF is converted to syslog-like format for easier handling.
The logger process is very small and simple, you can modify it to suit your needs and convert logs in any format to any destination you prefer.
The logger process is very small and simple, you can modify it to suit your needs and convert logs in any format to any destination you prefer. It lives in its own repository, here: https://github.com/DistributedSystemsGroup/zoe-logger
If you are interested in sending container output to Kafka, please make your voice heard at `this Docker issue https://github.com/docker/docker/issues/21271`_ for a more production-friendly Docker-Kafka integration.
#!/usr/bin/python3
import sys
import argparse
from kafka import KafkaConsumer
parser = argparse.ArgumentParser(description='Zoe Kafka log viewer')
parser.add_argument('kafka_address', help='Address of the Kafka broker')
parser.add_argument('--list-logs', action='store_true', help='List all the available service logs')
parser.add_argument('--topic', help='Service name to fetch and monitor for activity')
args = parser.parse_args()
consumer = KafkaConsumer(bootstrap_servers=args.kafka_address)
if args.list_logs:
for topic in consumer.topics():
if topic[0] != '_':
print(topic)
sys.exit(0)
consumer.subscribe(pattern=args.topic)
consumer.poll(1)
consumer.seek_to_beginning()
try:
for msg in consumer:
print(msg.value.decode('utf-8'))
except KeyboardInterrupt:
print('showlog exiting...')
#!/usr/bin/python3
# Copyright (c) 2016, Daniele Venzano
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from zoe_logger.entrypoint import main
if __name__ == '__main__':
main()
# Copyright (c) 2015, Daniele Venzano
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from zoe_lib.configargparse import ArgumentParser, Namespace
config_paths = [
'zoe-logger.conf',
'/etc/zoe/zoe-logger.conf'
]
_conf = None
def load_configuration(test_conf=None):
global _conf
if test_conf is None:
argparser = ArgumentParser(description="Zoe Logger - Container Analytics as a Service Swarm Log Manager component",
default_config_files=config_paths,
auto_env_var_prefix="ZOE_LOGGER_",
args_for_setting_config_path=["--config"],
args_for_writing_out_config_file=["--write-config"])
argparser.add_argument('--debug', action='store_true', help='Enable debug output')
argparser.add_argument('--kafka-broker', help='Address of the Kafka broker to send logs to', default='localhost:9092')
opts = argparser.parse_args()
if opts.debug:
argparser.print_values()
_conf = opts
else:
_conf = test_conf
def get_conf() -> Namespace:
return _conf
#!/usr/bin/python3
# Copyright (c) 2016, Daniele Venzano
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import logging
import socketserver
import json
import gzip
import kafka
from zoe_logger.config import load_configuration, get_conf
log = logging.getLogger("main")
class GELFUDPHandler(socketserver.DatagramRequestHandler):
def handle(self):
data = self.rfile.read()
data = gzip.decompress(data)
data = json.loads(data.decode('utf-8'))
service_id = '.'.join([data['_zoe.service.name'], data['_zoe.execution.name'], data['_zoe.owner'], data['_zoe.deployment_name']])
log_line = ' '.join([str(data['timestamp']), data['host'], service_id, data['short_message']])
self.server.kafka_producer.send(topic=service_id, value=log_line.encode('utf-8'))
# log.debug(log_line)
class ZoeLoggerUDPServer(socketserver.ThreadingUDPServer):
def __init__(self, server_address, handler_class, kafka_producer):
self.allow_reuse_address = True
super().__init__(server_address, handler_class)
self.kafka_producer = kafka_producer
def udp_listener(kafka_producer):
while True:
try:
server = ZoeLoggerUDPServer(("0.0.0.0", 12201), GELFUDPHandler, kafka_producer)
server.serve_forever()
except:
log.exception('Exception in UDP listener')
def setup_kafka():
return kafka.KafkaProducer(bootstrap_servers=get_conf().kafka_broker)
def main():
"""
The entrypoint for the zoe-observer script.
:return: int
"""
load_configuration()
args = get_conf()
if args.debug:
logging.basicConfig(level=logging.DEBUG)
else:
logging.basicConfig(level=logging.INFO)
logging.getLogger('kafka').setLevel(logging.WARN)
kafka_producer = setup_kafka()
udp_listener(kafka_producer)
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment