Build a module

Invenio modules are independent, interchangeable components that add functionalities. Each module exposes its own APIs and uses APIs of other modules. A full invenio application consists of a set of modules, and can be easily customized by adding or removing specific modules.

A module is usually called:

  1. with plural noun, meaning “database (of things)”, for example invenio-records, invenio-tags, invenio-annotations,
  2. with singular noun, meaning “worker (using things)”, for example invenio-checker, invenio-editor.

The user interface and the REST API interface of a module may be split into separate modules, for example invenio-records-ui and invenio-records-rest, to clarify dependencies and offer an easier customization.

All modules have the same structure, which is defined in the cookiecutter-invenio-module template.

First steps

To create a new module, make sure you have cookiecutter installed and run the following command:

$ cookiecutter gh:inveniosoftware/cookiecutter-invenio-module
project_name [Invenio-FunGenerator]: Invenio-Foo
project_shortname [invenio-foo]:
package_name [invenio_foo]:
github_repo [inveniosoftware/invenio-foo]:
description [Invenio module that adds more fun to the platform.]:
author_name [CERN]:
author_email [info@inveniosoftware.org]:
year [2018]:
copyright_holder [CERN]:
copyright_by_intergovernmental [True]:
superproject [Invenio]:
transifex_project [invenio-foo]:
extension_class [InvenioFoo]:
config_prefix [FOO]:

The newly scaffolded module will have the following folder structure:

invenio-foo/
    docs/
    examples/
    invenio_foo/
        templates/invenio_foo/
        __init__.py
        config.py
        ext.py
        version.py
        views.py
    tests/
    *.rst
    run-tests.sh
    setup.py

These files are described in the sections below.

*.rst files

All these files are used by people who want to know more about your module (mainly developers).

  • README.rst is used to describe your module. You can see the short description written in the Cookiecutter here. You should update it with more details.
  • AUTHORS.rst should list all contributors to this module.
  • CHANGES.rst should be updated at every release and store the list of versions with the list of changes (changelog).
  • CONTRIBUTING.rst presents the rules to contribute to your module.
  • INSTALL.rst describes how to install your module.

setup.py

First, there is the setup.py file, one of the most important: this file is executed when you install your module with pip. If you open it, you can see several parts.

On the top, the list of the requirements:

  • For normal use.
  • For development.
  • For tests.

Depending on your needs, you can install only part of the requirements, or everything (pip install invenio-foo[all]).

Then, in the setup() function, you can find the description of your module with the values entered in cookiecutter. At the end, you can find the entrypoints section.

run-tests.sh

This is used to run a list of tests locally, to make sure that your module works as intended. It will generate the documentation, run pytest and any remaining checks.

docs folder

This folder contains the settings to generate documentation for your module, along with files where you can write the documentation. When you run the run-tests.sh script, it will create the documentation in HTML files in a sub-folder.

examples folder

Here you can find a small example of how to use your module. You can test it following the steps described in the Run the example application section.

tests folder

Here are all the tests for your application, that will be run when you execute the run-tests.sh script. If all these tests pass, you can safely commit your work.

See pytest-invenio for how to structure your tests.

invenio_foo folder

This folder has the name of your module, in lower case with the dash changed to an underscore. It contains the code of your module. You can add any code files here, organized as you wish.

The files that already exist are standard, and are covered in the following sections. A rule of thumb is that if you need multiple files for one action (for instance, 2 views: one for the API and a standard one), create a folder having the name of the file you want to split (here, a views folder with ui.py and api.py inside).

MANIFEST.in

This file lists all the files included in the sub-folders. It should be updated before the first commit.

config.py

All configuration variables should be declared in this file.

ext.py

This file contains a class that extends the Invenio application with your module. It registers the module during the initialization of the application and loads the default configuration from config.py.

version.py

File containing the version of your module.

views.py

Here you declare the views or endpoints you want to expose. By default, it creates a simple view on the root end point that renders a template.

templates

All your Jinja templates should be stored in this folder. A Jinja template is an HTML file that can be modified according to some parameters.

static

If your module contains JavaScript or CSS files, they should go in a folder called static. Also, if you want to group them in bundles, you should add a bundles.py file next to the static folder.

Install a module

First of all, create a virtualenv for the module:

$ mkvirtualenv my_venv

Installing the module is very easy, you just need to go to its root directory and pip install it:

(my_venv)$ cd invenio-foo/
(my_venv)$ pip install --editable .[all]

Some explanations about the command:

  • The --editable option is used for development. It means that if you change the files in the module, you won’t have to reinstall it to see the changes. In a production environment, this option shouldn’t be used.

  • The . is in fact the path to your module. As we are in the root folder of the module, we can just say here, which is what the dot means.

  • The [all] after the dot means we want to install all dependencies, which is common when developing. Depending on your use of the module, you can install only parts of it:

    • The default (nothing after the dot) installs the minimum to make the module run.
    • [tests] installs the requirements to test the module.
    • [docs] installs the requirements to build the documentation.
    • Some modules have extra options.

If you need multiple options, you can chain them: [tests,docs].

Run the tests

In order to run the tests, you need to have a valid git repository. The following steps need to be run only once. Go into the root folder of the module:

(my_venv)$ git init
(my_venv)$ git add --all
(my_venv)$ check-manifest --update

What we have done:

  • Change the folder into a git repository, so it can record the changes made to the files.
  • Add all the files to this repository.
  • Update the file MANIFEST.in (this file controls which files are included in your Python package when it is created and installed).

Now, we are able to run the tests:

(my_venv)$ ./run-tests.sh

Build the documentation

The documentation can be built with the run-tests.sh script, but you need to have the package installed with its tests requirements. If you just want to build the documentation, you will only need the docs requirements (see the Install a module section above). Make sure you are at the root directory of the module and run:

(my_venv)$ python setup.py build_sphinx

Open docs/_build/html/index.html in the browser and voilà, the documentation is there.

Run the example application

The example application is a minimal application that presents the features of your module. The example application is useful during development for testing. By default, it simply prints a welcome page. To try it, go into the examples folder and run:

(my_venv)$ ./app-setup.sh
(my_venv)$ ./app-fixtures.sh
(my_venv)$ export FLASK_APP=app.py FLASK_DEBUG=1
(my_venv)$ flask run

You can now open a browser and go to the URL http://localhost:5000/ where you should be able to see a welcome page.

To clean the server, run the ./app-teardown.sh script after stopping the server.

Publishing on GitHub

Before going further in the tutorial, we can publish your repository to GitHub. This allows to integrate a continuous integration system such as TravisCI and allows an easy publishing of your module to PyPI afterwards.

First, create an empty repository in your GitHub account. Be sure not to generate any .gitignore or README files, as our code already has them. If you don’t have a GitHub account, you can skip this step, it is only necessary if you plan to publish your module on PyPI.

Now, go into the root directory of your module, and run:

$ git remote add origin URL-OF-YOUR-GITHUB-REPO

We can commit and push the generated files:

$ git commit -am "Initial module structure"
$ git push --set-upstream origin master

Finally, we create a new branch to develop on it.

$ git checkout -b dev

Use the module in your application

Integrating a new module to a full Invenio application comes down to adding it as a dependency in the central Pipfile. In order to do that, you should have published your module on GitHub and run the following command from the root folder of your Invenio application:

$ pipenv install URL-OF-YOUR-GITHUB-REPO

pipenv will update the Pipfile and install your module in the virtual enviroment of your application.

If your module has been released on PyPI, you can install it in your application by running the following command:

$ pipenv install invenio-foo

Next steps

To learn more about the development process in Invenio, follow the next guide Developing with Invenio.