Commit 1f878fe3 authored by Daniele Venzano's avatar Daniele Venzano

Update ZApp shop sample, delete unmaintained Jenkins CI

parent 4bfcfba4
# Zoe Continuous Integration
----------------------------
# Overview
- Integrate Zoe repository to Jenkins and SonarQube
- Each commit to Zoe repository trigger a build at Jenkins:
- Run SonarQube Scanner to analyze the codebase
- Create two containers for zoe-master, zoe-api
- Run integration test [testing rest api]
- Build new images if no errors happen
- Deploy Zoe with latest images
# Software Stack
- Jenkins - version 2.7.4
- SonarQube - version 6.1
# Configuration
### Jenkins: all the configurations in this section is configured on Jenkins Server
- Required:
- Plugins: Github plugin, SonarQube Plugin, Quality Gates, Email Plugin (optional), Cobertura Coverage Report (optional)
- Softwares: Java, Python, Docker
- Go to **Manage Jenkins**, then **Global Tool Configuration** to setup Java SDK, SonarQube Scanner
- SonarQube server configuration: this aims to connect Jenkins and SonarQube together
- Go to **Manage Jenkins** then **Configure System**
- SonarQube servers: input name, server URL, server version, **sever authentication token** (created on SonarQube Server)
- Quality Gates configuration:
- Go to **Manage Jenkins** then **Configure System**
- Quality Gates: input name, server URL, username and password to login into SonarQube server
- Github Servers configuration:
- Go to **Manage Jenkins** then **Configure System**
- Github: **Add Github Server**, the API URL would be ``https://api.github.com``. The credentials creation is well defined in the document of Github plugin:
- You can create your own [personal access token](https://github.com/settings/tokens/new) in your account GitHub settings.
- Token should be registered with scopes:
- admin:repo_hook - for managing hooks (read, write and delete old ones)
- repo - to see private repos
- repo:status - to manipulate commit statuses
- In Jenkins create credentials as «Secret Text», provided by Plain Credentials Plugin
- Create credentials for Github account: this is similar when you want to [connect to Github over SSH](https://help.github.com/articles/connecting-to-github-with-ssh/), here, beside adding your public key to Github, you also need to add your private key to Jenkins.
- Create SSH key pair on the machine run Jenkins:
- Add public key to Github
- Add private key to Jenkins credentials
- Create new item as a **freestyle project**: this aims to create a Jenkins job with the github repository
- General
- Select Github project
- Insert project URL
- Source Code Management
- Select **Git**
- Repositories
- Repository URL: use **SSH URL** of Github repository
- Credentials: select the one created above
- Build Triggers
- For Github plugin with version before 1.25.1: Select **Build when a change is pushed to Github**
- For Github plugin with version from 1.25.1: Select **GitHub hook trigger for GITScm polling**
- Build
- Add **Execute SonarQube Scanner** to do SonarQube Analysis
- Add **Quality Gates** to break the build when the SonarQube quality gate is not passed
- Add **Execute Shell** to run script for testing, deploying. Please refer to the Appendix section for the script.
- Post-build Actions [Optional]
- Add **Publish Covetura Coverage Report** for getting report from coverage
Due to the shell script in Appendix, the xml file generated by coverage is located at ``test`` folder, so, we should put ``**/tests/coverage.xml`` as the input of the field **Cobertura xml report pattern**.
- Add **E-mail Notification** for notifying when jobs finish
### Github
- Add new SSH key (the one created on Jenkins server)
- Go to the project (which is integrated to Jenkins) settings
- Integration & Services
- Add Service, choose **Jenkins (Github plugin)**
- Add Jenkins hook url
- For github plugin, this one would have the format: http://your-jenkins.com/github-webhook/
- In case your Jenkins doens't expose to the world, try [**ngrok**](https://ngrok.com/)
### SonarQube: all the configurations in this section is configured on SonarQube Server
- On **Administration**, go to **My Account**, then **Security**
- Generate Tokens, copy this and paste to **server authentication token** on Jenkins configuration
- The project needs to provides **sonar-properties** file in the repo: [Analyzing with SonarQube Scanner](http://docs.sonarqube.org/display/SCAN/Analyzing+with+SonarQube+Scanner)
# Appendix
----------
# Sonar properties files
Take a look at sonar-project.properties files at branch [**i34-t7.4.1**](https://github.com/DistributedSystemsGroup/zoe-kpmg/tree/i34-t7.4.1), in root, ``zoe_api``, ``zoe_master``, ``zoe_lib``, ``zoe_fe`` folders.
# Execute Shell Script
Script to execute ci in zoe-kpmg branch i34-t7.4.1
Push this script inside the execute shell script of Jenkins job you created above
```
# Run Style checker for Sphinx RST documentation
doc8 docs/
# Build new container images
python3 ci/zoeci.py 1 tcp://192.168.12.2:2375 192.168.12.2:5000/zoe:$BUILD_ID
# Deploy new zoe with the above images for testing
python3 ci/zoeci.py 0 tcp://192.168.12.2:2375 ci/docker-compose-test.yml 192.168.12.2:5000/zoe:$BUILD_ID
# Run integration test
cd tests
coverage run -p basic_auth_success_test.py
coverage run -p cookie_auth_success_test.py
coverage combine
coverage xml
cd ..
# Rebuild new frontend
cd zoe_fe
ng build --env=prod --output-path=build/prod/
cd ..
cp -r zoe_fe/build/prod .
tar -cvf prod.tar prod
# Push the built images above to local registry
python3 ci/zoeci.py 2 tcp://192.168.12.2:2375 192.168.12.2:5000/zoe:$BUILD_ID
# Redeploy zoe with new images
python3 ci/zoeci.py 0 tcp://192.168.12.2:2375 ci/docker-compose-prod.yml 192.168.12.2:5000/zoe:$BUILD_ID
```
FROM golang:1.8-alpine
MAINTAINER Quang-Nhat Hoang-Xuan <hxquangnhat@gmail.com>
VOLUME /config
EXPOSE 6060 6061
ENV GOPATH /go
RUN apk add --no-cache git bzr rpm xz && \
go get -v github.com/coreos/clair/cmd/clair && \
go install -v github.com/coreos/clair/cmd/clair && \
mv /go/bin/clair /clair && \
go install -v github.com/coreos/clair/contrib/analyze-local-images && \
mv /go/bin/analyze-local-images /bin/analyzer && \
rm -rf /go /usr/local/go
RUN apk update && \
apk add ca-certificates wget && \
update-ca-certificates
RUN wget https://get.docker.com/builds/Linux/x86_64/docker-17.03.0-ce.tgz && \
tar -xvf docker-17.03.0-ce.tgz && \
mv docker/docker /bin && \
rm -rf docker docker-17.03.0-ce.tgz
ENTRYPOINT ["/clair"]
# Copyright 2015 clair authors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# The values specified here are the default values that Clair uses if no configuration file is specified or if the keys are not defined.
clair:
database:
# PostgreSQL Connection string
# http://www.postgresql.org/docs/9.4/static/libpq-connect.html
type: pgsql
options:
source:
postgresql://postgres:password@clair_postgres:5432?sslmode=disable
# Number of elements kept in the cache
# Values unlikely to change (e.g. namespaces) are cached in order to save prevent needless roundtrips to the database.
cacheSize: 16384
api:
# API server port
port: 6060
# Health server port
# This is an unencrypted endpoint useful for load balancers to check to healthiness of the clair server.
healthport: 6061
# Deadline before an API request will respond with a 503
timeout: 900s
# 32-bit URL-safe base64 key used to encrypt pagination tokens
# If one is not provided, it will be generated.
# Multiple clair instances in the same cluster need the same value.
paginationKey:
# Optional PKI configuration
# If you want to easily generate client certificates and CAs, try the following projects:
# https://github.com/coreos/etcd-ca
# https://github.com/cloudflare/cfssl
servername:
cafile:
keyfile:
certfile:
updater:
# Frequency the database will be updated with vulnerabilities from the default data sources
# The value 0 disables the updater entirely.
interval: 2h
notifier:
# Number of attempts before the notification is marked as failed to be sent
attempts: 3
# Duration before a failed notification is retried
renotifyInterval: 2h
http:
# Optional endpoint that will receive notifications via POST requests
endpoint:
# Optional PKI configuration
# If you want to easily generate client certificates and CAs, try the following projects:
# https://github.com/cloudflare/cfssl
# https://github.com/coreos/etcd-ca
servername:
cafile:
keyfile:
certfile:
# Optional HTTP Proxy: must be a valid URL (including the scheme).
proxy:
version: '2'
services:
postgres:
container_name: clair_postgres
image: postgres:latest
environment:
POSTGRES_PASSWORD: password
clair:
container_name: clair_clair
image: hxquangnhat/clair:latest
depends_on:
- postgres
ports:
- "6060-6061:6060-6061"
links:
- postgres
volumes:
- /tmp:/tmp
- ./clair_config:/config
- /var/run/docker.sock:/var/run/docker.sock
command: [-config, /config/config.yaml]
#!/usr/bin/env python3
# Copyright (c) 2016, Quang-Nhat Hoang-Xuan
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Backend Deploy."""
import time
import yaml
import docker
from utils.docker_container_parameter import DockerContainerParameter
class ZoeBackendDeploy():
def __init__(self, dockerUrl, dockerComposePath, dockerVersion):
self.cli = docker.DockerClient(base_url=dockerUrl, version=dockerVersion)
self.zoe_api = ''
self.zoe_master = ''
self.zoe_opts = []
self.docker_compose_file = dockerComposePath
self.typeDeploy = 1 if 'prod' in dockerComposePath else 0
self.parse_docker_compose()
self.previousImage = ''
def parse_docker_compose(self):
content = ''
with open(self.docker_compose_file, 'r') as stream:
content = list(yaml.load(stream)['services'].items())
for i in range(0, 2):
opts = DockerContainerParameter()
if 'api' in content[i][0]:
self.zoe_api = content[i][0]
else:
self.zoe_master = content[i][0]
opts.set_name(content[i][0])
opts.set_hostname(content[i][0])
details = content[i][1]
opts.set_image(details['image'])
opts.set_command(details['command'])
if 'volumes' in details:
opts.set_volumes(details['volumes'])
if 'ports' in details:
opts.set_ports(details['ports'])
if 'logging' in details:
logging = details['logging']
if logging['driver'] == 'gelf':
opts.set_gelf(logging['options']['gelf-address'])
self.zoe_opts.append(opts)
def create_container(self, opts, image):
"""Create a container."""
ports = {}
volumes = {}
opts.set_image(image)
if len(opts.get_ports()) != 0:
ports_list = opts.get_ports()[0]
for p in ports_list:
splitted = p.split(":")
ports[splitted[0] + '/tcp'] = int(splitted[1])
if len(opts.get_volumes()) != 0:
volumes_list = opts.get_volumes()[0]
for v in volumes_list:
splitted = v.split(":")
bind = {'bind': splitted[1], 'mode': 'rw'}
volumes[splitted[0]] = bind
if opts.get_gelf() != '':
log_config = {
"type": "gelf",
"config": {
'gelf-address': opts.gelf_address
}
}
else:
log_config = {
"type": "json-file",
"config": {}
}
cont = self.cli.containers.create(image=opts.get_image(),
command=opts.get_command(),
hostname=opts.get_hostname(),
log_config=log_config,
name=opts.get_name(),
tty=True,
network_disabled=False,
network_mode='bridge',
ports=ports,
volumes=volumes)
print('Created ' + opts.get_name() + ' container')
return cont
def export_master_ip(self):
try:
master = self.cli.containers.get(self.zoe_master)
ip_zoe_master = master.attrs['NetworkSettings']['Networks']['bridge']['IPAddress']
hostEntry = ip_zoe_master + '\t' + self.zoe_master
add_to_host = 'bash -c "echo ' + "'" + hostEntry + "'" + ' >> /etc/hosts"'
api = self.cli.containers.get(self.zoe_api)
api.exec_run(add_to_host)
except Exception as ex:
print(ex)
def deploy(self, image):
"""deploy with docker-compose, if success, return, else, fallback"""
try:
retcode = 1
for s in [self.zoe_api, self.zoe_master]:
res = self.cli.containers.list(all=True, filters={'name': s})
if len(res) > 0:
for r in res:
name = r.name
if self.typeDeploy == 0:
if name == s:
r.remove(force=True)
else:
if name == s:
imgID = r.attrs['Image']
self.previousImage = self.cli.images.get(imgID).tags[0]
r.remove(force=True)
print('Removed ' + name + ' container')
print('deploying with ' + image)
for opts in self.zoe_opts:
ctn = self.create_container(opts, image)
print('Deploying zoe backend...')
# start zoe_api
api_c = self.cli.containers.get(self.zoe_api)
api_c.start()
time.sleep(5)
api_c = self.cli.containers.get(self.zoe_api)
print('Started latest ' + self.zoe_api + ' container...')
if not api_c.attrs['State']['Running']:
retcode = -1
time.sleep(5)
# start zoe_master
master_c = self.cli.containers.get(self.zoe_master)
master_c.start()
time.sleep(5)
master_c = self.cli.containers.get(self.zoe_master)
print('Started latest ' + self.zoe_master + ' container...')
if not master_c.attrs['State']['Running']:
retcode = 0
# export zoe-master ip address to hosts file of zoe-api
self.export_master_ip()
except Exception as ex:
print(ex)
retcode = 0
pass
return retcode
def fallback(self, image):
return
#!/usr/bin/env python3
# Copyright (c) 2016, Quang-Nhat Hoang-Xuan
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Frontend deploy."""
import docker
class ZoeFrontendDeploy():
def __init__(self, dockerUrl, apache2, dockerVersion):
self.src = 'prod.tar'
self.srcBackup = 'backup.tar'
self.dst = '/var/www/'
self.dstBackup = '/var/www/prod'
self.cli = docker.DockerClient(base_url=dockerUrl, version=dockerVersion)
self.apache2 = apache2
return
def deploy(self):
""" Put new frontend folder behind apache2 container """
try:
retcode = 1
#do backup
container = self.cli.containers.get(self.apache2)
strm, stat = container.get_archive(path=self.dstBackup)
filebackup = open(self.srcBackup, 'wb')
filebackup.write(strm.read())
filedata = open(self.src, 'rb').read()
res = container.put_archive(path=self.dst, data=filedata)
if res is False:
retcode = 0
except Exception as ex:
print(ex)
retcode = 0
return retcode
def fallback(self):
try:
container = self.cli.containers.get(self.apache2)
retcode = 1
filebackup = open(self.srcBackup, 'rb').read()
res = container.put_archive(path=self.dst, data=filebackup)
if res is False:
retcode = 0
except Exception as ex:
print(ex)
retcode = 0
return retcode
version: '2'
services:
zoe-api:
image: hxquangnhat/zoe:k8s-0.2
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /media/sdb/kubernetes/data/config:/opt/zoe/kube.conf
command: python3 zoe-api.py --debug --backend Kubernetes --deployment-name prod --dbuser postgres --dbhost 192.168.12.2 --dbport 5432 --dbname postgres --dbpass postgres --master-url tcp://zoe-master:4850 --auth-type text --proxy-path zoe.fsdna.on.kpmg.de --listen-port 5001
ports:
- "5001:5001"
zoe-master:
image: hxquangnhat/zoe:k8s-0.2
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /mnt/zoe-workspaces:/mnt/zoe-workspaces
- /media/sdb/kubernetes/data/config:/opt/zoe/kube.conf
command: python3 zoe-master.py --debug --backend Kubernetes --deployment-name prod --dbuser postgres --dbhost 192.168.12.2 --dbport 5432 --dbname postgres --dbpass postgres --auth-type text --proxy-path zoe.fsdna.on.kpmg.de --listen-port 5001
ports:
- "4850:4850"
depends_on:
- zoe-api
version: '2'
services:
zoe-api:
image: 192.168.12.2:5000/zoe:reducetime
volumes:
- /var/run/docker.sock:/var/run/docker.sock
command: python3 zoe-api.py --debug --backend-swarm-url consul://192.168.12.2 --deployment-name prod --dbuser postgres --dbhost 192.168.12.2 --dbport 5432 --dbname postgres --dbpass postgres --overlay-network-name my-net --master-url tcp://zoe-master:4850 --auth-type ldap --ldap-server-uri ldap://172.17.0.6 --ldap-base-dn ou=users,dc=example,dc=com --proxy-path zoe.fsdna.on.kpmg.de
ports:
- "5001:5001"
logging:
driver: "gelf"
options:
gelf-address: "udp://192.168.12.2:5004"
tag: "zoe-api"
zoe-master:
image: 192.168.12.2:5000/zoe:reducetime
volumes:
- /var/run/docker.sock:/var/run/docker.sock
command: python3 zoe-master.py --debug --backend-swarm-url consul://192.168.12.2 --deployment-name prod --dbuser postgres --dbhost 192.168.12.2 --dbport 5432 --dbname postgres --dbpass postgres --overlay-network-name my-net --auth-type ldap --ldap-server-uri ldap://ldapker.example.com --ldap-base-dn ou=users,dc=example,dc=com --proxy-path zoe.fsdna.on.kpmg.de
ports:
- "4850:4850"
depends_on:
- zoe-api
logging:
driver: "gelf"
options:
gelf-address: "udp://192.168.12.2:5004"
tag: "zoe-api"
version: '2'
services:
zoe-api-test:
image: hxquangnhat/zoe:k8s-0.2
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /media/sdb/kubernetes/data/config:/opt/zoe/kube.conf
command: python3 zoe-api.py --debug --backend Kubernetes --deployment-name test --dbuser postgres --dbhost 192.168.12.2 --dbport 4321 --dbname postgres --dbpass postgres --overlay-network-name my-net --master-url tcp://zoe-master-test:4850 --auth-type text --proxy-path zoe.fsdna.on.kpmg.de --listen-port 5100
ports:
- "5100:5100"
zoe-master-test:
image: hxquangnhat/zoe:k8s-0.2
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /mnt/zoe-workspaces:/mnt/zoe-workspaces
- /media/sdb/kubernetes/data/config:/opt/zoe/kube.conf
command: python3 zoe-master.py --debug --backend Kubernetes --deployment-name test --dbuser postgres --dbhost 192.168.12.2 --dbport 4321 --dbname postgres --dbpass postgres --overlay-network-name my-net --auth-type text --proxy-path zoe.fsdna.on.kpmg.de --listen-port 5100
depends_on:
- zoe-api-test
version: '2'
services:
zoe-api-test:
image: 192.168.12.2:5000/zoe:wi9
volumes:
- /var/run/docker.sock:/var/run/docker.sock
command: python3 zoe-api.py --debug --backend-swarm-url consul://192.168.12.2 --deployment-name test --dbuser postgres --dbhost 192.168.12.2 --dbport 4321 --dbname postgres --dbpass postgres --overlay-network-name my-net --master-url tcp://zoe-master-test:4850 --auth-type text --listen-port 5100
ports:
- "5100:5100"
zoe-master-test:
image: 192.168.12.2:5000/zoe:wi9
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /mnt/zoe-workspaces:/mnt/zoe-workspaces
command: python3 zoe-master.py --debug --backend-swarm-url consul://192.168.12.2 --deployment-name test --dbuser postgres --dbhost 192.168.12.2 --dbport 4321 --dbname postgres --dbpass postgres --overlay-network-name my-net --auth-type text --listen-port 5100
depends_on:
- zoe-api-test
#!/usr/bin/env bash
CI_BUILD_REF=${CI_BUILD_REF:-manual}
ZOE_COMMON_OPTIONS="--debug --backend-swarm-url http://localhost:2375 --deployment-name test${CI_BUILD_REF} --dbuser postgres --dbhost localhost --dbport 5432 --dbname postgres --dbpass postgres --master-url tcp://localhost:4850 --auth-type text --proxy-type none --listen-port 5100 --workspace-base-path /tmp"
if [ z"${CI_BUILD_REF}" = z"manual" ]; then
echo "Start Postgres"
docker run --name postgres-${CI_BUILD_REF} -d -p 5432:5432 postgres:9.3
sleep 5
fi
mkdir /tmp/test${CI_BUILD_REF}
python3 create_db_tables.py ${ZOE_COMMON_OPTIONS}
echo "Start Zoe API process"
python3 zoe-api.py ${ZOE_COMMON_OPTIONS} --log-file /tmp/zoe-api-${CI_BUILD_REF}.log &
API_PID=$!
sleep 2
echo "Start Zoe Master process"
python3 zoe-master.py ${ZOE_COMMON_OPTIONS} --log-file /tmp/zoe-master-${CI_BUILD_REF}.log &
MASTER_PID=$!
sleep 2
cd tests
coverage run basic_auth_success_test.py localhost:5100
cd ..
echo "<============== MASTER LOGS ======================>"
cat /tmp/zoe-master-${CI_BUILD_REF}.log
rm -f /tmp/zoe-master-${CI_BUILD_REF}.log
echo "<================ API LOGS =======================>"
cat /tmp/zoe-api-${CI_BUILD_REF}.log
rm -f /tmp/zoe-api-${CI_BUILD_REF}.log
kill ${API_PID} ${MASTER_PID}
rm -Rf /tmp/test${CI_BUILD_REF}
if [ z"${CI_BUILD_REF}" = z"manual" ]; then
echo "Stop Postgres"
docker rm -f postgres-${CI_BUILD_REF}
fi
#!/usr/bin/env python3
# Copyright (c) 2016, Quang-Nhat Hoang-Xuan
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Container Parameter class"""
from typing import Iterable
class DockerContainerParameter:
""" Class holding Docker Container configuration """
def __init__(self):
self.image = ''
self.volumes = []
self.command = ""
self.name = ''
self.ports = []
self.network_name = 'bridge'
self.gelf_address = ''
self.restart = True
self.hostname = ''
def set_gelf(self, gelf_address):
""" setter gelf_address """
self.gelf_address = gelf_address
def get_gelf(self) -> str:
""" getter gelf_addres """
return self.gelf_address
def set_ports(self, ports):
""" setter ports """
self.ports.append(ports)
def get_ports(self) -> Iterable[str]:
""" getter ports """
return self.ports
def set_image(self, image):
""" setter image """
self.image = image
def get_image(self) -> str: