Build a module¶
Invenio modules are independent, interchangeable components that add functionalities. Each module exposes its own APIs and uses APIs of other modules. A full invenio application consists of a set of modules, and can be easily customized by adding or removing specific modules.
A module is usually called:
- with plural noun, meaning “database (of things)”, for example
invenio-records
,invenio-tags
,invenio-annotations
, - with singular noun, meaning “worker (using things)”, for example
invenio-checker
,invenio-editor
.
The user interface and the REST API interface of a module may be split into separate modules,
for example invenio-records-ui
and invenio-records-rest
, to clarify dependencies and
offer an easier customization.
All modules have the same structure, which is defined in the cookiecutter-invenio-module template.
First steps¶
To create a new module, make sure you have cookiecutter installed and run the following command:
$ cookiecutter gh:inveniosoftware/cookiecutter-invenio-module
project_name [Invenio-FunGenerator]: Invenio-Foo
project_shortname [invenio-foo]:
package_name [invenio_foo]:
github_repo [inveniosoftware/invenio-foo]:
description [Invenio module that adds more fun to the platform.]:
author_name [CERN]:
author_email [info@inveniosoftware.org]:
year [2018]:
copyright_holder [CERN]:
copyright_by_intergovernmental [True]:
superproject [Invenio]:
transifex_project [invenio-foo]:
extension_class [InvenioFoo]:
config_prefix [FOO]:
The newly scaffolded module will have the following folder structure:
invenio-foo/
docs/
examples/
invenio_foo/
templates/invenio_foo/
__init__.py
config.py
ext.py
version.py
views.py
tests/
*.rst
run-tests.sh
setup.py
These files are described in the sections below.
*.rst files¶
All these files are used by people who want to know more about your module (mainly developers).
README.rst
is used to describe your module. You can see the short description written in the Cookiecutter here. You should update it with more details.AUTHORS.rst
should list all contributors to this module.CHANGES.rst
should be updated at every release and store the list of versions with the list of changes (changelog).CONTRIBUTING.rst
presents the rules to contribute to your module.INSTALL.rst
describes how to install your module.
setup.py¶
First, there is the setup.py
file, one of the most important: this file is
executed when you install your module with pip. If you open it, you can see
several parts.
On the top, the list of the requirements:
- For normal use.
- For development.
- For tests.
Depending on your needs, you can install only part of the requirements, or
everything (pip install invenio-foo[all]
).
Then, in the setup()
function, you can find the description of your module with
the values entered in cookiecutter. At the end, you can find the
entrypoints
section.
run-tests.sh¶
This is used to run a list of tests locally, to make sure that your module works as intended. It will generate the documentation, run pytest and any remaining checks.
docs folder¶
This folder contains the settings to generate documentation for your module,
along with files where you can write the documentation. When you run the
run-tests.sh
script, it will create the documentation in HTML files in a
sub-folder.
examples folder¶
Here you can find a small example of how to use your module. You can test it following the steps described in the Run the example application section.
tests folder¶
Here are all the tests for your application, that will be run when
you execute the run-tests.sh
script. If all these tests pass, you can
safely commit your work.
See pytest-invenio for how to structure your tests.
invenio_foo folder¶
This folder has the name of your module, in lower case with the dash changed to an underscore. It contains the code of your module. You can add any code files here, organized as you wish.
The files that already exist are standard, and are covered
in the following sections. A rule of thumb is that if you need multiple
files for one action (for instance, 2 views
: one for the API and a standard
one), create a folder having the name of the file you want to split (here, a
views
folder with ui.py
and api.py
inside).
MANIFEST.in¶
This file lists all the files included in the sub-folders. It should be updated before the first commit.
config.py¶
All configuration variables should be declared in this file.
ext.py¶
This file contains a class that extends the Invenio application
with your module. It registers the module during the initialization of the application
and loads the default configuration from config.py
.
version.py¶
File containing the version of your module.
views.py¶
Here you declare the views or endpoints you want to expose. By default, it creates a simple view on the root end point that renders a template.
templates¶
All your Jinja templates should be stored in this folder. A Jinja template is an HTML file that can be modified according to some parameters.
static¶
If your module contains JavaScript or CSS files, they should go in a folder called static
. Also, if you want to group them in bundles,
you should add a bundles.py
file next to the static
folder.
Install a module¶
First of all, create a virtualenv for the module:
$ mkvirtualenv my_venv
Installing the module is very easy, you just need to go to its root directory and pip install it:
(my_venv)$ cd invenio-foo/
(my_venv)$ pip install --editable .[all]
Some explanations about the command:
The
--editable
option is used for development. It means that if you change the files in the module, you won’t have to reinstall it to see the changes. In a production environment, this option shouldn’t be used.The
.
is in fact the path to your module. As we are in the root folder of the module, we can just say here, which is what the dot means.The
[all]
after the dot means we want to install all dependencies, which is common when developing. Depending on your use of the module, you can install only parts of it:- The default (nothing after the dot) installs the minimum to make the module run.
[tests]
installs the requirements to test the module.[docs]
installs the requirements to build the documentation.- Some modules have extra options.
If you need multiple options, you can chain them: [tests,docs]
.
Run the tests¶
In order to run the tests, you need to have a valid git repository. The following steps need to be run only once. Go into the root folder of the module:
(my_venv)$ git init
(my_venv)$ git add --all
(my_venv)$ check-manifest --update
What we have done:
- Change the folder into a git repository, so it can record the changes made to the files.
- Add all the files to this repository.
- Update the file
MANIFEST.in
(this file controls which files are included in your Python package when it is created and installed).
Now, we are able to run the tests:
(my_venv)$ ./run-tests.sh
Build the documentation¶
The documentation can be built with the run-tests.sh
script, but you need
to have the package installed with its tests requirements. If you just want
to build the documentation, you will only need the docs requirements (see
the Install a module section above). Make sure you are at the root directory
of the module and run:
(my_venv)$ python setup.py build_sphinx
Open docs/_build/html/index.html
in the browser and voilà, the documentation is
there.
Run the example application¶
The example application is a minimal application that presents the features of your
module. The example application is useful during development for testing.
By default, it simply prints a welcome page.
To try it, go into the examples
folder and run:
(my_venv)$ ./app-setup.sh
(my_venv)$ ./app-fixtures.sh
(my_venv)$ export FLASK_APP=app.py FLASK_DEBUG=1
(my_venv)$ flask run
You can now open a browser and go to the URL http://localhost:5000/ where you should be able to see a welcome page.
To clean the server, run the ./app-teardown.sh
script after stopping the
server.
Publishing on GitHub¶
Before going further in the tutorial, we can publish your repository to GitHub. This allows to integrate a continuous integration system such as TravisCI and allows an easy publishing of your module to PyPI afterwards.
First, create an empty repository in your GitHub account. Be sure not to generate any .gitignore or README files, as our code already has them. If you don’t have a GitHub account, you can skip this step, it is only necessary if you plan to publish your module on PyPI.
Now, go into the root directory of your module, and run:
$ git remote add origin URL-OF-YOUR-GITHUB-REPO
We can commit and push the generated files:
$ git commit -am "Initial module structure"
$ git push --set-upstream origin master
Finally, we create a new branch to develop on it.
$ git checkout -b dev
Use the module in your application¶
Integrating a new module to a full Invenio application comes down to adding it as a dependency
in the central Pipfile
. In order to do that, you should have published your module on GitHub and
run the following command from the root folder of your Invenio application:
$ pipenv install URL-OF-YOUR-GITHUB-REPO
pipenv
will update the Pipfile
and install your module in the virtual enviroment of your application.
If your module has been released on PyPI, you can install it in your application by running the following command:
$ pipenv install invenio-foo
Next steps¶
To learn more about the development process in Invenio, follow the next guide Developing with Invenio.