Commit 041ed5b3 authored by Daniele Venzano's avatar Daniele Venzano
Browse files

Testing and fixing the Spark ZApp

parent 4f59b711
Pipeline #10404 passed with stages
in 10 minutes and 18 seconds
......@@ -155,6 +155,7 @@ def spark_jupyter_notebook_service(mem_limit, worker_mem_limit):
:rtype: dict
"""
driver_ram = (2 * 1024 ** 3)
worker_ram = mem_limit - (1024 ** 3) - (512 * 1025 ** 2)
service = {
'name': "spark-jupyter",
'image': NOTEBOOK_IMAGE,
......@@ -180,6 +181,7 @@ def spark_jupyter_notebook_service(mem_limit, worker_mem_limit):
'environment': [
["SPARK_MASTER", "spark://{dns_name#spark-master0}:7077"],
["SPARK_DRIVER_RAM", str(driver_ram)],
["SPARK_WORKER_RAM", str(worker_ram)],
["SPARK_HOME", "/opt/spark"],
["HADOOP_USER_NAME", "{user_name}"],
["PYTHONHASHSEED", "42"]
......
......@@ -39,7 +39,7 @@ ENV SHELL /bin/bash
ENV LC_ALL en_US.UTF-8
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US.UTF-8
ENV PYTHONPATH $SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.10.4-src.zip
ENV PYTHONPATH $SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.10.7-src.zip
ENV PYSPARK_PYTHON=/opt/conda/bin/python
RUN cd /tmp && \
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment