This page describes how to set up Docker containers for development purposes.
docker-compose -f docker-compose-dev.yml build docker-compose -f docker-compose-dev.yml up
This builds and runs the docker containers. You can now connect to localhost:28080 to see your Invenio installation. The admin user does not have any password.
This will mirror the current source code directory into the Docker container, so make sure to delete all *.pyc-files before. They might not be compatible with the Python version and libraries of Docker image.
If you are using boot2docker you need to set up port forwarding by running the following command in a new terminal:
boot2docker ssh -vnNT \ -Llocalhost:29200:localhost:29200 \ -Llocalhost:28080:localhost:28080 \ -Llocalhost:26379:localhost:26379 \ -Llocalhost:25673:localhost:25673 \ -Llocalhost:25672:localhost:25672 \ -Llocalhost:23306:localhost:23306
You have to run this after Invenio booted up. Do not stop it while you are working with Invenio. Otherwise the port forwarding gets stopped. You have to restart the forwarding when your restart the Docker containers. The process can be stopped using CTRL-C.
Should you require a fresh installation and therefore wipe all your instance data, run:
docker-compose -f docker-compose-dev.yml rm -f
6.5.2. Interactive sessions¶
For interactive sessions inside the container simply attach a shell to the running instance:
docker exec -it invenio_web_1 bash
Some tools (mostly ncurses-based ones) require the ability to detect the used terminal. To enable this, set the TERM environment variable to the correct value as a first command inside the container, e.g. by running:
Do not forget to run bibsched to schedule the processing of uploaded records. You can put it into automatic mode if you like.
The docker-compose-dev.yml enables Werkzeug, a debugger that automatically kicks in whenever an error occurs. Stacktraces and debugger terminal are available via web interface.
You can also insert a tracepoint into the code to start an interactive debugger session:
import ipdb; ipdb.set_tracepoint()
Furthermore you can debug MySQL at localhost:23306, Elasticsearch at localhost:29200, RabbitMQ via localhost:25672 (webinterface at localhost:25673) and Redis at localhost:26379. You might want to use flower for celery debugging and analysis as well. Just run the following command to open the webinterface at port 5555:
celery flower --broker=amqp://guest:guest@localhost:25672//
Should you require additional information about the behaviour of the different containers as well as the contained processes and their interaction with the system and other processes, the usage of classical Linux tools like Wireshark and sysdig might be helpful.
6.5.4. Code changes and live reloading¶
This section does not apply to OS X, Windows and boot2docker as these systems are not properly supported by the used watchdog mechanism. When you are using one of these setups, you have to restart the Docker containers to reload the code and templates.
As long as you do not add new requirements (python and npm) and only change files inside the invenio package, it is not required to rebuild the docker images. Code changes are mirrored to the containers. If Flask supports it, on your system it will automatically reload the application when changes are detected. This sometimes might lead to timeouts in your browser session. Do not worry about this, but be aware to only save files when you are ready for reloading.
As of this writing changing template files do not lead to application reloading and do not purge caches. As a workaround you can simple alter one of the python files, e.g. by using touch.
Changing the Python source files will invalidate stored bytecode files. For security reasons, these bytecode files can only be recreated by the root user. This can be done via:
docker exec -it -u root invenio_web_1 python -O -m compileall .
6.5.5. Building documentation¶
You can also use the Docker container to build the documentation. This can be done by attaching to running container:
docker exec -it invenio_web_1 sphinx-build -nW docs docs/_build/html
This needs do be done in a running or initialized container because it requires that Invenio is set up correctly. Otherwise, the script will break because of missing access rights.
6.5.6. Running tests¶
You can also run tests using the Docker containers. Wait until the containers finished setup and the webservice is running. Then use:
docker exec -it invenio_web_1 python setup.py test
Running the test requires the deactivation of redirection
debugging. You can achieve this by setting the configuration
variable DEBUG_TB_INTERCEPT_REDIRECTS = False. (Done for you by
default if you use
You might want to use build distribute overlays using Docker. Instead of creating an entire new image and rewrite everything from scratch, you can the Invenio Docker image. Start by building the image from a branch or release of your choice:
cd src/invenio docker build -t invenio:2.0 .
Now go to your overlay and create a Dockerfile that suits your needs, e.g:
# based on the right Invenio base image FROM invenio:2.0 # get root rights again USER root # optional: # add new packages # (update apt caches, because it was cleaned from the base image) # RUN apt-get update && \ # apt-get -qy install whatever_you_need # optional: # add new packages from pip # RUN pip install what_suits_you # optional: # add new packages from npm # RUN npm update && \ # npm install fun # optional: # make even more modifications # add content ADD . /code-overlay WORKDIR /code-overlay # fix requirements.txt and install additional dependencies RUN sed -i '/inveniosoftware\/invenio[@#]/d' requirements.txt && \ pip install -r requirements.txt --exists-action i # build RUN python setup.py compile_catalog # optional: # do some cleanup # step back # in general code should not be writeable, especially because we are using # `pip install -e` RUN mkdir -p /code-overlay/src && \ chown -R invenio:invenio /code-overlay && \ chown -R root:root /code-overlay/invenio_demosite && \ chown -R root:root /code-overlay/scripts && \ chown -R root:root /code-overlay/setup.* && \ chown -R root:root /code-overlay/src # finally step back again USER invenio
Notice that this Dockerfile must be located in the directory of your overlay. For a full working example, please see invenio-demosite. Here is how to build the demo site:
cd ~/private/src/invenio git checkout maint-2.0 docker build -t invenio:2.0 . cd ~/private/src/invenio-demosite git checkout maint-2.0 docker-compose -f docker-compose-dev.yml build docker-compose -f docker-compose-dev.yml up
After all the daemons are up and running, you can populate the demo site with demo records:
docker exec -i -t -u invenio inveniodemosite_web_1 \ inveniomanage demosite populate \ --packages=invenio_demosite.base --yes-i-know
Done. Your Invenio overlay installation is now up and running.