Version 3.2.0

Released 2019-12-20

We are proud to announce the release of Invenio v3.2.0.

Python compatibility

Invenio v3.2 supports Python 2.7 (until 2019-12-31), Python 3.5 and Python 3.6.

Getting started

See our Quickstart guide.

What’s new in Invenio v3.2?

Files bundle

We have released four new modules as part of the new Files bundle:

To understand more about how to use the new Files bundle see our integration guide in Integrating Files.

Elasticsearch v7 support

Invenio now supports Elasticsearch v7.

Marshmallow 3 compatibility

Invenio now support both Marshmallow v2 or v3. The support is done via a compatibility layer, that allows Invenio to work with either schemas from v2 or v3. This should allow users to upgrade to Invenio v3.2 without being forced to upgrade their Marshmallow schemas immediately to v3.

We advice all users to start planning an upgrade of their Marshmallow schemas from v2 to v3 as the upgrade is non-trivial and needs proper testing due to significant differences between Marshmallow v2 and v3.

Invenio will continue support for both Marshmallow v2 and v3 for a transition period to allow users to upgrade at their own pace. After the transition period Marshmallow v2 support will be deprecated and removed from Invenio.

For information about how to upgrade see Marshmallow v2/v3 compatibility.

Search index prefixing

Elasticsearch does not support the concept of virtual hosts and thus with previous versions of Invenio it was not possible to share an Elasticsearch cluster between multiple Invenio instances.

We have now added support for index/alias/template prefixing, so that all names can be prefixed with a string and thus allows multiple Invenio instances to share an Elasticsearch cluster. Note, that this is only name prefixing, thus technically two Invenio instances will can read each other indexes and thus must trust each other.

Read more about the new feature on https://invenio-search.readthedocs.io/en/latest/configuration.html#index-prefixing.

Minor changes in v3.2

Documentation improvements

We’ve improved documentation for a couple of modules:

Elasticsearch request library changed

Previously Invenio was using the Python library requests for connecting via HTTP to Elasticsearch. The library has now been changed to use urllib3 instead as recommended by Elasticsearch.

The old behavior can be restored by configuring the connection_class in the Elasticsearch client configuration:

from elasticsearch.connection import RequestsHttpConnection

SEARCH_CLIENT_CONFIG = dict(
    connection_class=RequestsHttpConnection
)

Hide webpack warnings

Webpack warnings are now by default hidden.

HTML sanitization improvements

Invenio-Formatter has a new template filter sanitize_html() to use for HTML sanitization.

The default configuration provided by Invenio-Config now sets default values for ALLOWED_HTML_TAGS and ALLOWED_HTML_ATTRS which is used by the bleach library for HTML sanitization. Please be careful if you change these values as it is very easy to open up for Cross-Site Scripting (XSS) attacks.

Sentry now supported via Sentry SDK instead of Raven

Invenio-Logging still has support for Raven, but being moved to Sentry SDK. To use the old Raven library set the following variable in your

SENTRY_SDK = False

Rate limiting improvements

There are now differentiated rate limits for authenticated (5000 per hour, 100 per minute) vs anonymous users (1000 per hour, 60 per minute). The limits can be changed via RATELIMIT_AUTHENTICATED_USER and RATELIMIT_GUEST_USER configuration variables. You can also setup per endpoint rate limits via the RATELIMIT_PER_ENDPOINT configuration variable.

There is also a new error template for when HTTP 429 (rate limited) error code is being returned. It can be changed via THEME_429_TEMPLATE configuration variable.

Records REST API improvements

Two new query parameters was added to the search REST API in order to make implementation of infinite scroll on a frontend application easier:

  • from: For pagination you can now use from (an item index) and size instead of page (a page index) and size.

  • aggs: The parameter can be used to completely turn-off computation of all aggregations (?aggs=) or select only specific aggregations to be show ?aggs=year).

CSV serializer

A new CSV serializer ( CSVSerializer) has been added to Invenio-Records-REST.

Improved indexing signal

Introduced a new method .dynamic_connect() on the signal before_record_index which allows conditional indexing. This is useful to e.g. only connect a receiver to a specific index.

Bug fixes

  • Invenio-OAuthClient: The next parameter which determines where to redirect after an successful authentication could not have query string parameters in the URL.

Deprecations in v3.2

Following list of features have been deprecated and will be removed in either Invenio v3.2 or Invenio v3.3:

Elasticsearch v2 and v5 support

We did not remove Elasticsearch v2 support in Invenio v3.2 as originally announced due to not having a user-friendly solution for upgrading Elasticsearch indexes. In Invenio v3.3 we plan to release the Invenio-Index-Migrator module which makes migrating/upgrading Elasticsearch indexes easy, and we thus plan to fully remove Elasticsearvh v2 and v5 support in either Invenio v3.3 or v3.4.

Both Elasticsearch v2 and v5 have reached end of life and no longer receives any bug or security fixes from Elastic.

Both the support in Invenio-Search for creating indexes for v2/v5 as well as any v2/v5 Elasticsearch mappings in other Invenio modules will be removed.

AMD/RequireJS (reminder from v3.1)

Invenio’s assets build system based on AMD/RequireJS will be removed in Invenio v3.3 or v3.4.

This involves e.g. the two CLI commands:

$ invenio npm
$ invenio assets build

Several Python modules in Invenio-Assets will be removed, including (but not limited to):

  • invenio_assets.npm

  • invenio_assets.filters

  • invenio_assets.glob

  • invenio_assets.proxies

Also, bundle definitions in other Invenio modules will be removed. These are usually located in bundles.py files, e.g.:

  • invenio_theme.bundles

Also, some static files will be removed from bundles, e.g.:

  • invenio_theme/static/js/*

  • invenio_theme/static/scss/*

AngularJS (reminder from v3.0)

In Invenio v3.0 we deprecated the AngularJS 1.4 application Invenio-Search-JS as AngularJS by that time was already outdated. We have selected React and SemanticUI as the replacement framework for AngularJS.

The new Webpack build system released in Invenio v3.1 is part of the strategy to move from AngularJS to React (note however that you can use Webpack with your favorite framework, including AngularJS).

We have started the rewrite of Invenio-Search-JS and have already released the first version of React-SearchKit which eventually will replace Invenio-Search-JS.

Features removed in v3.2

These following already deprecated features have been removed in Invenio v3.2.

Records CLI

The CLI in Invenio-Records module was removed.

DynamicPermission class

The DynamicPermission class was removed from Invenio-Access. The functionality provided by the class is still available via the Permission class.

Maintenance policy

Invenio v3.2 will be supported with bug and security fixes until the release of Invenio v3.4 and minimum until 2020-12-20.

See our Maintenance Policy.

What’s next?

In Invenio v3.3 we are planning to release the Statistics bundle including:

  • invenio-stats

    • Invenio module for statistical data processing and querying with support for collecting COUNTER Research Data Usage Metrics compliant statistics.

  • counter-robots

    • Library for COUNTER-compliant detection of machines and robots.

In addition to the Statistics bundle, we will also release final versions of the following two modules:

  • invenio-index-migrator

    • Elasticsearch index migrator for Invenio (aka zero down-time reindexing and index migration).