QGIS Processing, Model Designer and ETL Campaign

QGIS Processing offers a rich and expandable set of algorithms which can operate on spatial data, along with a powerful Model Designer which allows users to string together these algorithms to create custom workflows.

Since its introduction in QGIS 2, the Processing framework has seen an intensive amount of development and optimisation efforts. In recent QGIS releases it offers a very user-friendly way of performing complex spatial data processing tasks, all without requiring ANY expensive third-party tools or software licenses!

At North Road we are passionate about the QGIS Processing framework, and have invested considerable effort in this framework over the past 5 years. We’re proud to announce that our latest crowd-funding campaign is focused on further expanding the capabilities and flexibility of Processing and the Processing Model Designer!

This campaign is taking the form of a “à la carte” menu of Processing enhancements. These range from small “paper-cut” style fixes, through to larger architectural improvements, and are each individually priced accordingly. We are seeking backers who would like to fund these improvements and see them available in a future QGIS release!

How it works

Unlike a traditional crowd-funding campaign, we aren’t running this campaign as an “all or nothing” campaign. Rather, we are asking backers to pick individual enhancements from the menu below, and fund that enhancement’s development in full. In order to make this campaign affordable for a wide range of backers, we’ve included a huge range of enhancements which vary in price from smaller amounts to larger amounts.

We don’t have a pre-set deadline for this campaign either — as soon as an enhancement is funded, we’ll start work on it and implement it in time for the next QGIS release!

Funding an enhancement

In  order to fund one of the proposed enhancements, just email crowdfunding@north-road.com and let us know your details and which enhancement (or enhancements!) you want to sponsor. We will mark the enhancement as “sponsored” in the list below, and you’ll get full credit for funding its development!

Progress

Enhancements funded:

20 / 20

Available enhancements

The enhancements are broken down into three categories:

If you don’t see your particular wish-list enhancement described below, just email us and let us know!

Model Designer improvements

1. Remember model inputs when testing models – Funded!

During model creation it is common to test run a model many times. Unfortunately, every time a model is run from the QGIS Processing model designer the user must re-enter all the desired input parameter values. This repetition quickly gets tedious!

This enhancement would cause the input values to be remembered between these test runs. Whenever a model is run from within the Model Designer window, the input parameters would all be initially pre-populated with the values used during the last test run. (We’d even store these values within the model itself, so that they are persisted between different QGIS sessions).

Price: 200EU – Funded by Alta Ehf
See also:
https://github.com/qgis/QGIS/issues/27131

2. Copy and paste model components- Funded!

This change would allow model components (including both algorithms and inputs) to be copied and pasted. Components could be pasted within the same model, or into a different model. Allowing copy/paste within models would greatly speed up the model development process for users!

Price: 300EU – Funded
See also:
https://github.com/qgis/QGIS/issues/15101

3. Select and manipulate multiple model components at once – Funded!

Currently, the model designer only allows for a single component (an input or an algorithm) to be selected at once. This makes complex model development a frustrating process, as components can only be moved or deleted one-by-one. This enhancement would allow multiple components to be selected, either by holding the shift key while clicking on them or via a click-and-drag selection rectangle. If multiple components are selected, then pressing delete would delete all of them (after confirmation!), and moving them would also move all selected components.

Price: 350EU – Funded
See also:
https://github.com/qgis/QGIS/issues/15089

4. Add undo/redo support to designer – Funded!

This enhancement would see “undo” and “redo” support added to the model designer. Changes could be rolled back or re-done, allowing users to correct accidental model changes!

Price: 350EU – Funded
See also: https://github.com/qgis/QGIS/issues/15097

5. Resizing model components – Funded!

Currently, all model components are shown in the designer with a fixed size. This means that descriptions are often truncated, making it difficult to locate particular parts of a model:

This enhancement would permit model components to be freely resized, so that the whole description can be visible within the designer window.

Price: 250EU – Funded
See also: https://github.com/qgis/QGIS/issues/24189

6. Adding comments to a model – Funded

For readability of complex models it would be helpful for users to be able to add comment notes to their models. This change would add a tool for users to create comments boxes within their models. If a comment is attached to a particular component of a model, then we’d also ensure that the comment is included whenever a model is exported to a Python script.

Price: 150EU – Funded by Fisel + König
See also: https://github.com/qgis/QGIS/issues/22491

7. Create group boxes and bookmarks in a model – Funded!

To further improve the readability of complex models, this enhancement would add a new tool for drawing colored group boxes within models. The boxes would store their position and size, and would be drawn below other model components. They would allow users to manually group portions of their model into visual groups, and would also support an optional heading label (e.g. “Data retrieval steps”, “Cleanup and filter”, etc).

(Note that these boxes would be freely positioned and sized — they would not “attach” to any contained components and would be independently movable away from these contents if desired).

Price: 250EU – Funded
See also: https://github.com/qgis/QGIS/issues/15090

8. Allow unconnected model components – Funded

A common frustration with the QGIS model designer is that it constantly forces models to be valid, even work-in-progress models. This means it’s impossible to add a component to a model and only partially populate its settings — you have to fill in everything upfront. If you realise mid way through this that you need to make a change somewhere else in your model, it’s impossible to do without canceling and losing all the settings you’ve already configured.

This change would see the model validation deferred until run time. Models would be allowed to be temporarily invalid (e.g. having algorithm components which are only partially configured). The missing required values could be filled at a later stage, without losing any existing settings. If a non-valid model is run, the user will get a descriptive warning informing them of the changes they need to make in order to finish configuration of the model.

A new “validate model” button will also be added to the Model designer toolbar, allowing users to validate their models at any time during the design process.

Price: 600EU – Funded by QGIS Sweden User Group!
See also: https://github.com/qgis/QGIS/issues/24191

9. Conditional model branches – Funded!

This change would add an option for “conditional branches” in models. These would take the form of a (model-only) algorithm, which allows users to set an expression function to execute. The user can also define as many expected results from the expression as required. These would become “outputs” from the algorithm, and when the model is run branches matching the evaluated expression value would be executed. All algorithms connected to other output branches would be skipped.

This enhancement would allow users to create a logic branch using expressions such as:

count(“some_algorithm_output”) > 0

Price: 350EU – Funded by Andreas Neumann

10. Raising exceptions from models – Funded!

Often, it’s desirable to abort a model early, such as when a certain prerequisite condition is not correctly met. This enhancement would add a new, model-only algorithm “Raise exception”, which would immediately abort the model when triggered (with a user-defined message). This new algorithm is designed to be used in conjunction with the “conditional model branches” enhancement, and would allow the model to be abort if a certain run-time condition is not met.

Price: 50EU – Funded by Fisel + König

11. Logging messages from models – Funded!

This change would add a new, model-only algorithm “Log message”, which would push a user-defined message to the algorithm log window when the model is run. The message could include QGIS expressions (using the same format as elsewhere in QGIS, by embedding expressions within a “[% %]” block). E.g. the algorithm could be used to write the log message “Found [% @OUTPUT_FEATURES %] matching roads to process“, with the OUTPUT_FEATURES value dynamically inserted at run-time.

Price: 50EU – Funded

12. Snap model components to grid – Funded!

This change would add a new option to the Model designer window to allow model components to be snapped to a grid, allowing pixel-perfect alignment of your models!

Price: 250EU – Funded
See also:
https://github.com/qgis/QGIS/issues/29107, https://github.com/qgis/QGIS/issues/18690

13. Allow model inputs to be reordered – Funded!

Currently, all inputs defined for a model are shown in a quasi-random order when the model is run. Model designers cannot set a logical order for users to enter these parameter values, which can severely affect model usability.

This enhancement would rework the UI where inputs are created for a model, and allow them to be manually reordered by model designers.

Price: 350EU – Funded by NaturalGIS!
See also: https://github.com/qgis/QGIS/issues/27787

New algorithms

14. Filter features by geometry type – Funded!

A desirable feature to have in models is the ability to filter incoming features by geometry type, and either discard geometries of unwanted types or direct them to another part of the model. This enhancement would expose this functionality through two new algorithms:

“Filter features by geometry type”
This new algorithm would have an input for vector layer features, and separate outputs for points, lines, polygons and features without geometries. Incoming features would be routed to the appropriate output depending on their geometry type.

“Filter layers by type”
Similarly, this new algorithm would accept a map layer input parameter, and direct that layer to either a vector, raster or mesh output parameter depending on the layer type. It would allow map layers of different types to be directed to specific parts of models designed to handle input layers of that particular type.

Price: 150EU – Funded
See also: https://github.com/qgis/QGIS/issues/27141

15. “Save log” algorithm – Funded!

This new, model-only algorithm would allow writing the current Processing log contents to a specific text file. (Obviously, only the log contents up to the point in the model at which the Save Log algorithm is executed would be saved!). It is designed to allow logs to be permanently stored for later review.

Price:  100EU – Funded

General improvements

16. Connect directly to database layers without loading them in a project – Funded!

Currently, Processing algorithms can only operate on either disk-based layers or layers from the current project. This makes it impossible to run algorithms or models using a database layer as an input without first loading it into the current project.

This enhancement would expose a “Browser” based layer selection option for input layers when running algorithms, allowing for users to pick layers directly from any registered database server (including Postgres, SQL Server, Oracle, and even WFS layers!). Another benefit of this change would be that is allows inputs to be set to a layer from a specific Geopackage (or spatialite) file, or a specific sheet from a spreadsheet file. 

Specifically, this change would see the behavior of the “…” button which is shown next to a vector layer input for an algorithm would change from opening a file picker to instead showing a menu with the options “Select File” and “Browse for Layer”.

Price: 400EU – Funded
See also: https://github.com/qgis/QGIS/issues/27113

17. Write outputs direct to SQL Server tables – Funded!

While Processing currently allows saving outputs direct to a Postgres table, writing direct to other database formats such as SQL Server is not supported. Unfortunately, the current process used to save outputs to Postgres cannot be easily extended to other database servers. This enhancement would refactor and improve the current Postgres output writing, and expose the ability to write outputs direct to SQL Server tables. (Future work could extend this to also support direct outputs to Oracle tables).

Price: 2,500EU – Funded
See also: https://github.com/qgis/QGIS/issues/27440

18. Improved control when writing to a database – Funded!

Currently, when writing an output to a database table, QGIS always replaces the table. This change would expose new options for handling this, specifically allowing features to be appended to an existing table.

Price: 3,500EU – Funded
See also: https://github.com/qgis/QGIS/issues/27441

19. Limit the number of features to read from vector layer inputs, and other per-input settings – Funded!

When creating models it is desirable to test-run them using only a small subset of the features from an input layer, especially when an input layer contains many millions of features! This feature would see new, per-input options added for each of the vector layer inputs to an algorithm, allowing the behavior of each input to be tweaked.

Initially, we would expose settings for a maximum number of features to read from the layer (defaulting to reading all features), and a setting to control the invalid geometry handling for features read from that layer. This work would be designed to allow additional settings to be added in future, such as an “area of interest” filter or expression based filter on the inputs.

Price: 990EU – Funded
See also: https://github.com/qgis/QGIS/issues/28771

20. Allow execution of models and algorithms via a command line tool – Funded!

This enhancement covers creation of a new command-line executable which would allow for execution of processing algorithms outside of the QGIS GUI application, allowing for easy scripting of data manipulation via traditional batch files, shell scripts, and via operating system tools such as task scheduler.

Additionally, this command-line tool would make it easier for 3rd party applications to call and utilise Processing algorithms without the need to create custom PyQGIS scripts to achieve this.

See the full details of this enhancement at https://github.com/qgis/QGIS-Enhancement-Proposals/issues/140

Price: 990EU – Funded by QGIS Sweden User Group!

About us

This work will be undertaken by QGIS core contributor Nyall Dawson.

At North Road we invest back into the open-source geospatial community. We do large amounts of volunteer fixes and feature additions across the open source geospatial stack, so by investing in us you also help support us improve the open source GIS stack.

We use the software we develop daily, so we take pride in developing stable, polished code with extensive regression testing and refined workflows. We offer long-term support for the code we create, providing proactive fixes and improvements even after a project is complete. Our development record demonstrates the confidence you can place in us for timely delivery of this crowd funding campaign’s product.

Please contact us for any further details on this project or for contributing to this campaign.