Build a module¶
Invenio modules are independent, interchangeable components that add functionalities. Each module exposes its own APIs and uses APIs of other modules. A full invenio application consists of a set of modules, and can be easily customized by adding or removing specific modules.
A module is usually called:
with plural noun, meaning “database (of things)”, for example
with singular noun, meaning “worker (using things)”, for example
The user interface and the REST API interface of a module may be split into separate modules,
invenio-records-rest, to clarify dependencies and
offer an easier customization.
All modules have the same structure, which is defined in the cookiecutter-invenio-module template.
To create a new module, make sure you have cookiecutter installed and run the following command:
$ cookiecutter gh:inveniosoftware/cookiecutter-invenio-module project_name [Invenio-FunGenerator]: Invenio-Foo project_shortname [invenio-foo]: package_name [invenio_foo]: github_repo [inveniosoftware/invenio-foo]: description [Invenio module that adds more fun to the platform.]: author_name [CERN]: author_email [firstname.lastname@example.org]: year : copyright_holder [CERN]: copyright_by_intergovernmental [True]: superproject [Invenio]: transifex_project [invenio-foo]: extension_class [InvenioFoo]: config_prefix [FOO]:
The newly scaffolded module will have the following folder structure:
invenio-foo/ docs/ examples/ invenio_foo/ templates/invenio_foo/ __init__.py config.py ext.py version.py views.py tests/ *.rst run-tests.sh setup.py
These files are described in the sections below.
All these files are used by people who want to know more about your module (mainly developers).
README.rstis used to describe your module. You can see the short description written in the Cookiecutter here. You should update it with more details.
AUTHORS.rstshould list all contributors to this module.
CHANGES.rstshould be updated at every release and store the list of versions with the list of changes (changelog).
CONTRIBUTING.rstpresents the rules to contribute to your module.
INSTALL.rstdescribes how to install your module.
First, there is the
setup.py file, one of the most important: this file is
executed when you install your module with pip. If you open it, you can see
On the top, the list of the requirements:
For normal use.
Depending on your needs, you can install only part of the requirements, or
pip install invenio-foo[all]).
Then, in the
setup() function, you can find the description of your module with
the values entered in cookiecutter. At the end, you can find the
This is used to run a list of tests locally, to make sure that your module works as intended. It will generate the documentation, run pytest and any remaining checks.
This folder contains the settings to generate documentation for your module,
along with files where you can write the documentation. When you run the
run-tests.sh script, it will create the documentation in HTML files in a
Here you can find a small example of how to use your module. You can test it following the steps described in the Run the example application section.
Here are all the tests for your application, that will be run when
you execute the
run-tests.sh script. If all these tests pass, you can
safely commit your work.
See pytest-invenio for how to structure your tests.
This folder has the name of your module, in lower case with the dash changed to an underscore. It contains the code of your module. You can add any code files here, organized as you wish.
The files that already exist are standard, and are covered
in the following sections. A rule of thumb is that if you need multiple
files for one action (for instance, 2
views: one for the API and a standard
one), create a folder having the name of the file you want to split (here, a
views folder with
This file lists all the files included in the sub-folders. It should be updated before the first commit.
All configuration variables should be declared in this file.
This file contains a class that extends the Invenio application
with your module. It registers the module during the initialization of the application
and loads the default configuration from
File containing the version of your module.
Here you declare the views or endpoints you want to expose. By default, it creates a simple view on the root end point that renders a template.
All your Jinja templates should be stored in this folder. A Jinja template is an HTML file that can be modified according to some parameters.
static. Also, if you want to group them in bundles,
you should add a
bundles.py file next to the
Install a module¶
First of all, create a virtualenv for the module:
$ mkvirtualenv my_venv
Installing the module is very easy, you just need to go to its root directory and pip install it:
(my_venv)$ cd invenio-foo/ (my_venv)$ pip install --editable .[all]
Some explanations about the command:
--editableoption is used for development. It means that if you change the files in the module, you won’t have to reinstall it to see the changes. In a production environment, this option shouldn’t be used.
.is in fact the path to your module. As we are in the root folder of the module, we can just say here, which is what the dot means.
[all]after the dot means we want to install all dependencies, which is common when developing. Depending on your use of the module, you can install only parts of it:
The default (nothing after the dot) installs the minimum to make the module run.
[tests]installs the requirements to test the module.
[docs]installs the requirements to build the documentation.
Some modules have extra options.
If you need multiple options, you can chain them:
Run the tests¶
In order to run the tests, you need to have a valid git repository. The following steps need to be run only once. Go into the root folder of the module:
(my_venv)$ git init (my_venv)$ git add --all (my_venv)$ check-manifest --update
What we have done:
Change the folder into a git repository, so it can record the changes made to the files.
Add all the files to this repository.
Update the file
MANIFEST.in(this file controls which files are included in your Python package when it is created and installed).
Now, we are able to run the tests:
Build the documentation¶
The documentation can be built with the
run-tests.sh script, but you need
to have the package installed with its tests requirements. If you just want
to build the documentation, you will only need the docs requirements (see
the Install a module section above). Make sure you are at the root directory
of the module and run:
(my_venv)$ python setup.py build_sphinx
docs/_build/html/index.html in the browser and voilà, the documentation is
Run the example application¶
The example application is a minimal application that presents the features of your
module. The example application is useful during development for testing.
By default, it simply prints a welcome page.
To try it, go into the
examples folder and run:
(my_venv)$ ./app-setup.sh (my_venv)$ ./app-fixtures.sh (my_venv)$ export FLASK_APP=app.py FLASK_DEBUG=1 (my_venv)$ flask run
You can now open a browser and go to the URL http://localhost:5000/ where you should be able to see a welcome page.
To clean the server, run the
./app-teardown.sh script after stopping the
Publishing on GitHub¶
Before going further in the tutorial, we can publish your repository to GitHub. This allows to integrate a continuous integration system such as TravisCI and allows an easy publishing of your module to PyPI afterwards.
First, create an empty repository in your GitHub account. Be sure not to generate any .gitignore or README files, as our code already has them. If you don’t have a GitHub account, you can skip this step, it is only necessary if you plan to publish your module on PyPI.
Now, go into the root directory of your module, and run:
$ git remote add origin URL-OF-YOUR-GITHUB-REPO
We can commit and push the generated files:
$ git commit -am "Initial module structure" $ git push --set-upstream origin master
Finally, we create a new branch to develop on it.
$ git checkout -b dev
Use the module in your application¶
Integrating a new module to a full Invenio application comes down to adding it as a dependency
in the central
Pipfile. In order to do that, you should have published your module on GitHub and
run the following command from the root folder of your Invenio application:
$ pipenv install URL-OF-YOUR-GITHUB-REPO
pipenv will update the
Pipfile and install your module in the virtual enviroment of your application.
If your module has been released on PyPI, you can install it in your application by running the following command:
$ pipenv install invenio-foo