While the use of open-source software is widely accepted and respected within the geospatial industry, organisations are often restricted in their ability to completely embrace these powerful new tools due to the need to utilise existing projects and data from proprietary software vendors.
This is particularly an issue with the ESRI software suite, where organisations may have hundreds (or thousands!) of existing map documents and layer styles which they currently cannot utilise outside of the ESRI software ecosystem. Unfortunately, ESRI software is often prohibitively expensive, yet organisations cannot completely transition to alternative software packages due to their dependence on accessing their pre-existing work.
Furthermore, many providers of spatial data only include ESRI specific layer formatting files with their data supplies. This leaves users with no means of utilising these official pre-defined styles in non-ESRI tools.
The good news: North Road have been conducting extensive research and development over the past 12 months, and are pleased to announce an upcoming product which will allow use of these previously restricted formats within the open-source software environment!
This document describes the upcoming conversion tool, what we have planned, and how you can access the tool by becoming a project sponsor.
Table of Contents
- Planned Functionality
- Open Source Pledge
- Backing the Project
- What Sponsors Get
- Guarantees and Risks
- Stage 1 – Symbology Conversion
- Stage 2 – Vector LYR file conversion
- Stage 3 – Raster LYR File Conversion
- Stage 4 – MXD Conversion
- Future Work
Breakdown of Planned Functionality
The conversion tool targets three ESRI specific binary file formats:
- “.style” databases: These database files store libraries of ESRI symbols (markers, lines, and fills), alongside other associated style objects such as color palettes, color ramps, legend presets, and text formats. Style databases also support library “tags”, allowing symbols to be organized by arbitrary, user-defined tag strings. We will support conversion of .style marker, line, and fill symbols, color palettes, and color ramps (as the other .style objects like legend presets have no current equivalent in QGIS). Within QGIS, style libraries (containing symbols and color ramps) are stored as .XML style library files, and color palettes are stored using the standard “.GPL” palette format (which are also compatible with other applications such as GIMP and Inkscape).
- “.lyr” files: These files contain both a vector or raster layer’s definition (including the file or database path to the layer’s source data) and the associated symbology and layer settings to apply to that layer. LYR files can be added to an ArcMap document to automatically load and style data in a single step. Within QGIS, the “QLR” file format offers an equivalent functionality.
- “.mxd” documents: These map document files contain groups of layers, their styles, and print layouts for exporting the layers. The QGIS equivalent is a “QGS” project file.
All three formats are proprietary, ESRI specific binary formats. No specifications describing the formats have been publicly released, and the only currently available tools for working with these formats require access to an ArcGIS installation. Additionally, current tools are severely limited in their conversion functionality and require outdated ArcGIS software installs.
In this project, North Road will continue work we begun in reverse engineering these binary formats and creating custom tools for extraction and conversion of these file types. Conversion tools for these ESRI binary formats will be available through the QGIS “SLYR” plugin, which will be available for QGIS versions 3.4 and later.
The plugin will offer rich integration within the QGIS application, by adding common functionality like support for direct drag and drop of the ESRI style/LYR/MXD formats to the QGIS application window.
Wherever possible, we will also be exposing the style/LYR/MXD conversion tools via QGIS “Processing Algorithms”. Implementing these tools as Processing algorithms will allow users to utilise the conversion functionality as steps in QGIS graphical models, batch processes, and from PyQGIS scripts and other plugins, greatly improving their flexibility and allowing them be incorporated into wider workflows.
The project consists of four stages (with each stage a pre-requisite for the functionality described in later stages).
Open Source Pledge
North Road are passionate believers in the power of open source software, and have a history of extensive contributions to the open-source geospatial software stack! While we fully intend to make the SLYR plugin open source and freely publish the style/LYR/MXD conversion tools, we also require financial backing in order to support the significant time required to completely reverse engineer these file formats and develop quality tools supporting their use outside of the ESRI software ecosystem. Accordingly, the specifications and file parsing library will initially be closed source and available to project sponsors only. Exactly six months after we hit the pledged sponsorship levels for each stage of the project, we will open-source that component of the code and freely publish the results under a permissive open source license.
Backing the Project
In order to fund the remaining research and development costs, North Road are opening this project to a limited number of “project sponsors”. Prices for project sponsors are a one-time cost:
- International (non-Australian) entities: €1950
- Australian entities: AU$3000 (+10% GST)
As a reward for early sponsors, the FIRST 10 SPONSORS ONLY will receive a discount rate of:
- International (non-Australian) entities: €1000
- Australian entities: AU$1600 (+10% GST)
To become a project sponsor and receive early access to the tools, email firstname.lastname@example.org for further details.
What Sponsors Get
Early Access to Tools
Project sponsors are granted immediate access to the in-development conversion tools. This includes the work completed to date for all project stages, and on-going access to the latest version of the tools as work continues on the remaining project stages. (Non-sponsors must wait until the open-source version of the tools are released which, as described under the “Open Source Pledge” section, occurs 6 months after each project stage is completed).
Becoming a project sponsor gives an organisation unlimited access to the latest version of the tools for a single physical workplace.
Additionally, project sponsors have access to priority support for the conversions tools from North Road.
Unfortunately, the three binary formats are heavily dependent on the software version they have been created with, and accordingly it is not uncommon to encounter versions of the .style/LYR/MXD files which have not yet been completely reverse engineered and supported by the SLYR conversion tools.
Project sponsors will have access to priority support, allowing them to send unsupported file versions to North Road for analysis and inclusion in the conversion tools ASAP. Non-sponsors will have no access to these support channels, and will only be able to convert versions of the files which are already handled by the conversion tools.
The support available to project sponsors will also extend to assistance with use and installation of the software.
Guarantees and Risks
All projects of this nature have associated risk. However, we are tagging this particular project as “low risk”, because:
- North Road have a long-standing history of delivering quality open source software. Our lead developer, Nyall Dawson, is one of the most prolific contributors to QGIS, individually responsible for over 9000 commits to the open source software stack.
- We’ve conducted extensive research into the target formats, and have previously published a beta tool for .style conversion (which is publicly available, and can be tested at https://github.com/nyalldawson/slyr). Based on our experiences with this tool, we performed proof of concept testing to determine that both LYR and MXD formats can be reverse engineered using a similar approach.
- Project sponsors will have constant access to the latest developmental versions of the tools, and can test and provide feedback on these as further development proceeds.
Stage 1 – Symbology Conversion
Sponsorship progress : 1 of 11 sponsors
Stage 1 of the project involves completing the conversion of ArcGIS style database symbology to QGIS style XML libraries. While SLYR currently offers conversion of a large range of ESRI symbol styles, some components are not currently supported. During stage 1 we will implement conversion of the most common remaining symbol types and settings, including implementing some missing features within QGIS itself.
Additionally, we will expose support for ESRI style databases within the QGIS browser panel. When the SLYR plugin is enabled, .style files will be shown when navigating through the browser. Double clicking these files will start the conversion of the .style file to a QGIS .xml style database, after first prompting users to enter a path for the output .xml file. (For QGIS 3.6 installs this will be enhanced, so that double-clicking a .style file will immediately open a “style browser” window showing the symbols and color ramps embedded inside that .style file!).
Missing symbol properties which will be added during stage 1 are:
Translation of ESRI gradient fill symbols
- Linear and circular gradient fill symbols will be converted directly to the equivalent QGIS linear or circular gradient fill symbol.
- “Buffered” fill symbols will be converted to the equivalent QGIS “Shapeburst” fill symbol.
Out of scope
Conversion of rectangular gradients will not be supported. QGIS does not currently have support for rectangular gradient fills, and their rare occurrence makes supporting them low priority.
Marker fill symbology
SLYR currently handles conversion of basic ESRI marker fill symbols to QGIS symbols. The remaining properties of ESRI marker fill symbols need to be implemented within QGIS, as QGIS does not currently support these settings.
We will add support for some of these missing features within QGIS, specifically, by adding support for x and y marker fill symbol offsets. Currently QGIS does not support these offsets, which can affect the accuracy of marker fill symbol conversion.
Out of scope
QGIS is also missing support for random marker fills, and currently only allows for regular marker fill patterns. Adding support for random marker fills to QGIS is non-trivial, and needs to be implemented as part of a larger work package implementing multiple types of random symbology features. Accordingly, Random marker fill symbols are considered out of scope and will be converted to a regular marker fill pattern only.
SLYR is missing support for some picture marker and picture fill settings. These conversions will be implemented as part of stage 1:
- Support for picture “transparent color” settings
- Support for flipped foreground/background colors
Additionally, in order to completely support conversion of ESRI picture fill symbols to QGIS image fill symbols we will implement support for picture fill separation x/y settings within QGIS.
We will also upgrade SLYR to take advantage of some new functionality available in QGIS 3.6, and utilise the new raster marker symbol type when converting ESRI picture marker symbols (instead of the current approach of converting these to SVG files with an embedded raster image), and additionally we will use the new QGIS 3.6 feature of allowing embedded raster images for raster fill symbols when converting ESRI picture fill symbols. These 3.6 specific features will be optional, with the current approach used when SLYR is run in QGIS 3.4.
Out of scope
Picture line symbols will not be supported and are considered out of scope. These are not yet supported within QGIS and adding support for these is non-trivial. Accordingly, they are being marked as out of scope in order to keep costs for this stage minimal.
Currently SLYR only supports conversion of line decorations which are at the start or ends of lines. During stage 1 we will implement conversion for decoration markers at positions other than the start and end of lines.
Fill outline properties
Support for conversion of fill symbol outlines with offsets, decorations or dashed line templates will be added.
Remaining missing conversions – out of scope
Upon completion of stage 1, some symbol types and properties will remain non-convertible. These are considered out of scope, for reasons outlined below:
These are not supported in QGIS and adding support is non-trivial. Accordingly they are being marked as out of scope to keep costs for this work package minimal.
In order to minimise the cost of this work package parsing of 3D symbols is not included. These should be possible to parse and convert to QGIS 3D symbology, but this work is left for a future follow-up project.
These are not currently supported within QGIS, and adding this functionality is non-trivial. While it may be possible to “fake” halo support by careful creation of symbol levels with wider outlines set, we would prefer to defer this support in favour of a proper implementation within QGIS.
Stage 2 – Vector LYR file conversion
Sponsorship progress : 0 of 30 sponsors (plus requirements for stage 1)
Stage 2 of the project involves adding direct support for LYR files within SLYR. This will allow common properties of LYR files to be read, including the layer paths and symbology. LYR support will not rely on any external tools (such as the mdbtools dependency required for .style file conversions), and will not be dependent on installation of any ArcGIS products.
Included in Stage 2:
Reverse engineering of LYR file format
During stage 2 we will reverse engineer vector LYR file components, e.g. layer paths, scale ranges, coordinate systems, unique values renderers, ranges renderers, etc. This will build off the symbology reverse engineering work completed in stage 1, so all symbol types and settings supported as part of stage 1 will work correctly when used in LYR files.
“Set Style from LYR File” Algorithm
A new SLYR Processing algorithm will be created, which will set an existing QGIS vector layer’s style to match the style from a lyr file, replacing the layer’s current renderer in-place.
On launching the algorithm, users will be asked to select a layer from the current project and a corresponding LYR file. The SLYR tool will extract the renderer settings from that file, including conversion of unique value and ranges renderers, and apply it to the selected project layer.
“LYR to QLR” Algorithm
An algorithm for direct conversion of LYR files to QGIS QLR files will also be added. On launching, this algorithm will prompt users to select a LYR file, and the filename for the generated QLR file.
QLR files are a direct QGIS equivalent of ESRI LYR files, storing both a layer’s source and symbology. This makes them conversion of LYR to QLR files a highly desirable function.
“LYR to QGIS Style XML” Algorithm
The final algorithm added during Stage 2 will be an algorithm which creates a QGIS Style XML database containing all symbols found within a specified LYR file. This will allow users to select a LYR file, and obtain a “dump” of all symbols found within that file. The resultant QGIS Style XML database can be opened within the QGIS Style Manager and the symbols saved to the local style database.
Direct Opening of Vector LYR Files
Stage 2 will also implement direct opening of vector LYR files within QGIS. This will be exposed in two ways:
- Dragging and dropping LYR files on the main QGIS window will convert the LYR file in the background and add the resultant data file to the current project, styled using the symbology and settings from the original LYR file.
- Additionally, we will expose support for LYR files within the QGIS browser panel. When the SLYR plugin is enabled, .lyr files will be shown when navigating through the browser. Double clicking these files will convert the LYR file in the background, and load the corresponding data file into the current project, styled using the symbology and settings from the LYR file.
Out of scope:
Vector layer labeling
Supporting vector label settings will be deferred to a follow up piece of work, as conversion of label settings to QGIS labeling requires some extensive additions to QGIS labeling engine.
Stage 3 – Raster LYR File Conversion
Sponsorship progress : 0 of 20 sponsors (plus requirements for stage 2)
Stage 3 of the project will consist of reverse engineering raster LYR file components, including raster layer symbology. Wherever equivalent settings within QGIS exist, the LYR file properties will be converted to their QGIS equivalent.
All the methods described in Stage 2 for working with vector LYR files within QGIS will be extended to also work for raster layers (e.g. support for dragging and dropping raster LYR files to QGIS, and exposing raster LYR files in the QGIS browser panel.)
During stage 3 we will also identify feature gaps within QGIS’ raster symbology functionality, and implement the missing functionality wherever it is trivial to do so. The remaining missing functionality will be addressed in future follow-up work.
Stage 4 – MXD Conversion
Sponsorship progress : 0 of 35 sponsors (plus requirements for stage 3)
Stage 4 of the project will consist of reverse engineering the ESRI MXD map document format. This document format includes sets of multiple layers (with their original styles), legend groups, and print layouts.
Deliverables for stage 4 consist of:
“Add Layers from MXD” algorithm
A new algorithm will be added to the Processing toolbox, allowing users to select an input MXD document. The algorithm will extract all layers from the document, and add them to the current project, keeping their original symbology from the MXD intact (with the exclusions already described in Stages 1-3).
“Extract Print Layouts from MXD” algorithm
This new algorithm will accept an MXD document input, and extract all print layouts contained within that document into either the current project or to QGIS QPT layout template files.
“Convert MXD to QGS” algorithm
This final algorithm is the “holy grail”! It will allow direct conversion of MXD files to their equivalent QGS project file, keeping as much of the original MXD settings as are translatable to QGIS project settings.
MXD Integration into QGIS application
Finally, we will implement similar handling for MXD documents as has been described in stage 2 for LYR files. Specifically,
- Support will be added allowing drag and drop of MXD documents to the QGIS window. This will convert the document in the background and open as an equivalent QGIS project.
- MXD documents will be shown within the QGIS browser panel, allowing them to be directly opened as new projects.
Potential future follow up work could build off stages 1-4 and add support for MPK map packages and LPK layer packages. These formats could potentially be exposed within the QGIS browser panel and support drag and drop to the QGIS main window, much like the LYR and MXD support added in stages 2 and 4 respectively.