3.1. How to create an Invenio overlay

CAVEAT LECTOR

The following overlay guide was written for Invenio v2.0 release series. It is to be updated for Invenio v3.0.x

TODO

The following items are still missing from this document, feel free to contribute to it.

  • celery/redis/honcho/flower
  • populate the data

Thanks

3.1.1. What is an overlay

Invenio is a library that enable the creation of an digital library but it has to be coupled with an overlay. The overlay will contain your configuration options, your desired look and feel, the extra invenio modules you’ve developed and are using.

3.1.2. Creating your first Invenio overlay

If you’ve already setup the Invenio developer’s environement itself, it will feel familiar. We are reproducing here the steps for Ubuntu LTS. Other distributions may be found in the previous link.

3.1.2.1. The global setup

Some softwares and libraries are required to work on your overlay. It’s mostly Python, MySQL, Redis as well as XML, XSLT and graphical libraries. Node.js is solely required for development purposes.

$ python --version
Python 2.7.5+
$ sudo apt-get update
$ sudo apt-get install build-essential git redis-server \
                       libmysqlclient-dev libxml2-dev libxslt-dev \
                       libjpeg-dev libfreetype6-dev libtiff-dev \
                       libffi-dev libssl-dev \
                       software-properties-common python-dev \
                       virtualenvwrapper
$ sudo pip install -U virtualenvwrapper pip
$ source .bashrc

# Install MySQL server, and keep the root password somewhere safe.
$ sudo apt-get install mysql-server

# Install node.js
$ curl -sL https://deb.nodesource.com/setup_4.x | sudo bash -
$ sudo apt-get install nodejs

3.1.2.2. The virtual environment

Python development usually recommends to work within a virtualenv, which creates an isolated environment where the libraries required by one will not intervene with the system ones or the ones of another system. We are using virtualenvwrapper but nothing prevents your from directly using virtualenv you are familiar with it.

$ mkvirtualenv myoverlay
(myoverlay)$ # we are in your overlay environment now and
(myoverlay)$ # can leave it using the deactivate command.
(myoverlay)$ deactivate
$ # Now join it back, recreating it would fail.
$ workon myoverlay
(myoverlay)$ # That's all there is to know about it.

3.1.2.3. The base of the overlay

Let’s dive in.

$ workon myoverlay
(myoverlay)$ cdvirtualenv
(myoverlay)$ mkdir -p src/myoverlay
(myoverlay)$ cd src/myoverlay
(myoverlay)$ edit setup.py

The setup.py file contains the definition of a python package. Having one means you can rely on existing tools like pip to package, install, deploy it later on. Here is its minimal content.

from setuptools import setup
from setuptools import setup, find_packages
packages = find_packages()

setup(
    name="My Overlay",
    version="0.1.dev0",
    url="http://inveniosoftware.org/",
    author="Invenio Software",
    author_email="invenio@inveniosoftware.org",
    description="My first overlay",
    packages=packages,
    install_requires=[
        "Invenio>=2"
    ],
    entry_points={
        "invenio.config": ["myoverlay = myoverlay.config"]
    }
)

Now we can install it in editable mode (-e), meaning you don’t have to reinstall it after each change

(myoverlay)$ pip install -e .

This will fetch the latest Invenio version published on PyPI. As a developer, you may instead want to use the development version of Invenio from GitHub. To do so, create a file called requirements.txt with the following content:

git+git://github.com/inveniosoftware/invenio@pu#egg=Invenio-dev
-e .

and install using:

(myoverlay)$ pip install -r requirements.txt

3.1.3. Configuration

As you’ve seen above, we defined an entry_point for myoverlay.config. It points to a module that will contain our configuration. So create your application.

src/
 │
 ├ myoverlay/
 │  │
 │  ├ base/
 │  │  │
 │  │  └ __init__.py
 │  │
 │  ├ __init__.py
 │  └ config.py
 │
 ├ requirements.txt
 └ setup.py

Put the required configuration into config.py.

CFG_SITE_LANGS = ["en"]

CFG_SITE_NAME = "My Overlay"
CFG_SITE_NAME_INTL = {
    "en": CFG_SITE_NAME
}

PACKAGES = [
    "myoverlay.base",
    # TODO list your packages here
    "invenio_base",
]

try:
    from myoverlay.instance_config import *
except ImportError:
    pass

3.1.3.1. Sensitive configuration

Other configuration elements like database username and password or the website url should not be put here as this file is not specific to the installation and may be put under a version control system such as Git or Subversion.

The configuration can be handled via the inveniomanage command line interface (or by editing the invenio.cfg file in the instance folder and reloading the application).

(myoverlay)$ inveniomanage config set create secret-key
# MySQL configuration
(myoverlay)$ inveniomanage config set CFG_DATABASE_NAME mysql-database
(myoverlay)$ inveniomanage config set CFG_DATABASE_USER mysql-user
# HOST configuration (for redirects, etc.)
(myoverlay)$ inveniomanage config set CFG_SITE_URL http://invenio.example.com
(myoverlay)$ inveniomanage config set CFG_SITE_SECURE_URL https://invenio.example.com
(myoverlay)$ inveniomanage config set DEBUG True
(myoverlay)$ inveniomanage config set ASSETS_DEBUG True

3.1.3.2. Database setup

(invenio)$ inveniomanage database init --user=root --password=$MYSQL_ROOT --yes-i-know
...
>>> Database has been installed.
(invenio)$ inveniomanage database create
...
>>> Tables filled successfully.

3.1.3.3. Assets

Most of the JavaScript and CSS libraries used are not bundled with invenio itself and needs to be downloaded via bower. Bower is configured using two files:

  • .bowerrc: tells where the assets are downloaded
  • bower.json: lists the dependencies to be downloaded

The bower.json can be automagically generated through the inveniomanage command. However, we will have to tell bower the path we want the libraries downloaded to. In most cases, it will be in the instance directory under the static/vendors path.

$ sudo su -c "npm install -g bower less clean-css requirejs uglify-js"
(myoverlay)$ cdvirtualenv var/invenio.base-instance
(myoverlay)$ echo '{"directory": "static/vendors"}' > .bowerrc
(myoverlay)$ inveniomanage bower > bower.json
(myoverlay)$ bower install

For invenio to see the static files from the myoverlay.base module, it needs to declare a Flask blueprint. Create the following file: myoverlay/base/views.py.

from flask import Blueprint

blueprint = Blueprint(
    "myoverlay",
    __name__,
    url_prefix="/",
    template_folder="templates",  # where your custom templates will go
    static_folder="static"        # where the assets go
)

The assets will now be collected into the instance static folder from your overlay, invenio itself and every libraries it uses.

(myoverlay)$ inveniomanage collect

3.1.4. Running

(myoverlay)$ inveniomanage runserver

3.1.5. Translations

Invenio comes with full internationalization and localization support based on Babel library and Flask-Babel. All strings you want to translate in your overlay have to be marked with _().

When you have all strings properly marked, it is time to prepare catalog that contains all these strings for tranlations to desired languages.

3.1.5.1. Configuration

First of all, you have to get into the source folder of your overlay and create a configuration file for Babel.

[python: **.py]
encoding = utf-8

[jinja2: **/templates/**]
encoding = utf-8
extensions = jinja2.ext.autoescape.jinja2.ext.with_

Save it as babel.cfg next to your setup.py. Before we run the extraction tool we need to add section to configure translation directory to setup.cfg.

[compile_catalog]
directory = myoverlay/base/translations/

[extract_messages]
output-file = myoverlay/base/translations/myoverlay.pot

[init_catalog]
input-file = myoverlay/base/translations/myoverlay.pot
output-dir = myoverlay/base/translations/

[update_catalog]
input-file = myoverlay/base/translations/myoverlay.pot
output-dir = myoverlay/base/translations/

3.1.5.2. Message Extraction

Then it’s time to run the Babel string extraction with given configuration:

(myoverlay)$ python setup.py extract_messages

3.1.5.3. Create Catalog for New Language

Once all translatable strings are extracted, one need to prepare catalogs for new languages. Following example shows how to prepare new catalog for French in PO (Portable Object) format.

(myoverlay)$ python setup.py init_catalog -l fr

Now edit the myoverlay/base/translations/fr/LC_MESSAGES/messages.po file as needed.

3.1.5.4. Compiling Catalog

Next step is to prepare MO (Machine Object) files in the format which is defined by the GNU gettext tools and the GNU translation project.

To compile the translations for use, pybabel integration with distutils helps again:

(myoverlay)$ python setup.py compile_catalog

If you install Invenio in development mode you must compile catalog also from the Invenio directory project.

Note

You should tell git to ignore your compliled translation by running:

$ echo \*.mo >> .gitignore

3.1.5.5. Updating Strings

It is pretty common that your strings in the code will change over the time. Pybabel provides support for updating the translation catalog with new strings or changing existing ones. What do you have to do? Create a new myoverlay.pot like above and then let pybabel merge the changes:

$ python setup.py update_catalog

3.1.6. Deployment

Deploying Invenio is almost a piece of cake using Fabric. The following step are inspired by the Flask documentation: Deploying with Fabric

3.1.6.1. Prerequisites

First, you need a server with remote access (SSH), where you’ve installed all the python dependencies (e.g. build-essentials, python-dev, libmysqlclient-dev, etc.).

Install fabric locally,

$ pip install fabric

and create a boilerplate fabfile.py:

import json

from fabric.api import *
from fabric.utils import error
from fabric.contrib.files import exists


env.user = 'invenio'  # remote username
env.directory = '/home/invenio/www'  # remote directory
env.hosts = ['yourserver']  # list of servers

3.1.6.2. Preparing the tarball

Before deploying anything, we need to locally prepare the python package to be installed. Thanks to our setup.py file, it’s very simple.

Beforehand, we have to generate the static assets into our static folder. By doing so, it’s not required to install anything related to node.js on your server (no bower, less, uglifyjs, etc.).

@task
def pack():
    """Create a new source distribution as tarball."""
    with open(".bowerrc") as fp:
        bower = json.load(fp)

    local("inveniomanage assets build --directory {directory}/gen"
          .format(**bower))
    return local("python setup.py sdist --formats=gztar", capture=False) \
        .succeeded

Try it:

$ fab pack
...
Done
$ ls dist/
My-Overlay-0.1.dev0.tar.gz

This is the package that will be installed on your server.

3.1.6.3. Creating the virtual environement

We love virtual environments. We recommend you to install each version into its own virtual env enabling quick rollbacks.

@task
def create_virtualenv():
    """Create the virtualenv."""
    package = local("python setup.py --fullname", capture=True).strip()
    venv = "{0}/{1}".format(env.directory, package)

    with cd(env.directory):
        if exists(package):
            return error("This version {0} is already installed."
                         .format(package))

        return run("virtualenv {0}".format(package)).succeeded

3.1.6.4. Installing the package

We can now upload the local tarball into the virtualenv, and install everything there.

@task
def install():
    """Install package."""
    package = local("python setup.py --fullname", capture=True).strip()
    venv = "{0}/{1}".format(env.directory, package)

    if not exists(venv):
        return error("Meh? I need a virtualenv first.")

    # Upload the package and put it into our virtualenv.
    put("dist/{0}.tar.gz".format(package), "/tmp/app.tgz")
    run("mkdir -p {0}/src".format(venv))
    with cd("{0}/src".format(venv)):
        run("tar xzf /tmp/app.tgz")
        run("rm -rf /tmp/app.tgz")

    # Jump into the virtualenv and install stuff
    with cd("{0}/src/{1}".format(venv, package)):
        success = run("{0}/bin/python setup.py install".format(venv)

        if success:
            # post install
            run("{0}/bin/inveniomanage collect".format(venv))
    return success

Combining all the three steps:

$ fab pack virtualenv install

3.1.6.5. Configuration

The setup doesn’t have the invenio.cfg file that is generated via inveniomanage config. You should do so manually.

3.1.6.6. Running the server

uWSGI is super simple and neat, all you need is two files. In the example below, we’ve installed two versions of our overlay and a symbolic link is pointing to the one we want to run.

$ ls www/
current -> My-Overlay-0.1
My-Overlay-0.1.dev1
My-Overlay-0.1.dev2
My-Overlay-0.1
wsgi.py
uwsgi.ini

Let’s create the wsgi.py file.

from invenio_base.factory import create_wsgi_app

application = create_wsgi_app()

And the µWSGI configuration:

[uwsgi]
http = 0.0.0.0:4000
master = true

processes = 4
die-on-term = true
vaccum = true

chdir = %d
virtualenv = %d/current/
module = wsgi:application
touch-reload = %d/wsgi.py

Let’s run it.

$ pip install uwsgi

$ uwsgi --ini uwsgi.ini
# or in daemon mode
$ uwsgi -d uwsgi.log --ini uwsgi.ini

If the new version causes troubles, going back to the old one is as fast as changing the symbolic link and restarting the WSGI server.

$ rm current
$ ln -s My-Overlay-0.1.dev1 current
$ touch wsgi.py

3.1.6.7. Dealing with versions

One good idea is to use symlink to point to your current virtualenv and run your overlay from there. Doing that via Fabric is left as an exercise to the reader.

When installing a new version, copying the invenio.cfg file over is the only requirements. Restarting the WSGI server is usually done by touch-ing the wsgi.py file.