Hello,

I found this wonderful datasource : https://beta.source.coop/repositories/vida/google-microsoft-open-buildings/description/ which provides merged building footprints from microsoft and google

the format of the data is either

• GeoParquet
• FlatGeobuf
• PMTiles

I searched in the documentation and the forum there are (almost) no reference of these formats.

with a online web service I got to transform geoparquet to geojson so I was able to check that the data provided is really what I need (for a very small sample of data, I tried with Andore! see attached files)

Now I need a stronger work stream to import this data in Manifold, I found this software which is able to convert geoparquet to geojson https://github.com/planetlabs/gpq; it got Andorre data converted in 1s, and Cameroon data converted in few minutes (600Mo to 4Gb)

gpq geojson output of Andorre was successfully imported in manifold9, but not Cameroon (log windows just says "invalid file type"

geopandas combined with arrow library in python can also convert parquet to geojson, worked fine for Andore, it took 30mn for Cameroon and ended with the same error "invalid file type"

How can I import these files in Manifold9

Attachments:
1441151880758558720.geojson
bf_small.parquet

I found that geojson is really not adapted to this kind of massive file (>2GB) so I am trying to convert them to gpkg with fiona and geopandas python packages. It works for small files, but for larger files it is very long (>30mn).

Any idea of how I can work this through?

Seems like you are hacking through the wilderness right now. Keep at it. I took a quick look at a couple of those formats. Have you tried a GDAL approach, as I think at least two of those file formats have GDAL drivers. Keep at it, let us know how you get on.

only GDAL 3.03 is loaded (I don't know why V3.04-3.07 have failed to load). FlatGeoBuf is opened by GDAL but only the columns headers, no records are imported

in Manifold 9 GDAL V3.03 do not recognize geoparquet. According to the documentation of GDAL, geoparquet is supported since V3.8

I can import bf_small.parquet into Manifold with GDAL 3.7.3 (gdal307.dll)

https://gdal.org/drivers/vector/parquet.html

(Geo)Parquet

New in version 3.5.

[...]

The GeoParquet 1.0.0 specification is supported since GDAL 3.8.0

Maintaining a most up to date and still supported GDAL/OGR dll-s for Manifold requires some care and patience. I try to share how I go about it, later.

nice! will it read from a URL, does the Manifold UI honour passing down the /vsicurl prefix?

/vsicurl/https://github.com/opengeospatial/geoparquet/raw/main/examples/example.parquet

https://github.com/mdsumner

No. Not directly

(root)::[example.parqet] Cannot open file.

I tried to put the link into .vrt file but I'm not sure what's missing.

(root)::[vrt] Driver: OGR_VRT (VRT - Virtual Datasource)

GDAL creates a table with 0 columns and 0 rows.

oh, I got it.

<OGRVRTLayer name="example">

the name "example" must match what datasource says. I had arbitrary name there.

ogrinfo helped to debug .vrt file.

ogrinfo C:\Downloads\example.vrt

INFO: Open of `C:\Downloads\example.vrt'

using driver `OGR_VRT' successful.

1: example_layerERROR 1: Failed to find layer 'example_layer' on datasource '/vsicurl/https://github.com/opengeospatial/geoparquet/raw/main/examples/example.parquet'.

(None)

Attachments:
example.vrt

Something I recently found helpful in getting remote parquet files to interop with Release 9 was this slight variation on what you've done with vrt.

Release 9 gdal data source component will respect the "FID" provided to it by gdal, and you can generate one on the fly and then declare it in your VRT syntax, like this (attached your example vrt with minor changes).

This way a BTREE index is auto-generated on the linked (or imported) dataset in 9, allowing you to alt-click on features and ctrl-click select/copy/whatever them, without any additional steps.

I had additionally hoped that linking such an OgrDataSource VRT file would enable View --> Refresh updating of the datasource, but it does not seem that this causes the GDAL driver to reach out to the remote file and provide the "freshest" rowset, unfortunately, so it's still a matter of closing 9 and re-opening, which is fast enough, at least.

Hope that's useful to someone.

Attachments:
example.vrt

That auto-fake FID trick is great!

Similar trick to make PostgreSQL views Alt-click pickable and Ctrl-click selectable is

--SQL 

CREATE VIEW ... AS

SELECT

 row_number() OVER ()::integer AS fid,

  ...

FROM

  ...

Knowing how keep GDAL dll updated would help a lot!

I give a simplified version which works very well if you do not need gdalplugins.

Use ​OSGeo4W network installer (link to osgeo4w-setup.exe).

I keep QGIS (and everything that comes with it) up to date at default location C:\OSGeo4W

When installing, just check qgis or qgis-ltr.

Osgeo4w installer installs everything required by QGIS, including GDAL.

Two manual steps to let Manifold find GDAL and GDAL find PROJ.

1) Add C:\OSGeo4W\bin to PATH variable

2) Set PROJ_LIB variable to C:\OSGeo4W\share\proj

Manifold should now find gdal307.dll from C:\OSGeo4W\bin

Attachments:
gdal_env.png
select_just_qgis.png

If you absolutely do not want QGIS, you can install only gdal307-runtime. You can also use a custom location.

Then adjust env. vars accordingly.

Attachments:
gdal307-runtime.png

Thank you, but it is not working on my laptop

I have installed OSGeo4W as you mentioned (qgis & qgis-ltr), it works well since I got an updated version of qgis (3.34) with gdal 3.08 (which works in importing geoparquet / FlatGeoBuf)

then I edited my environment variables as you can see below :

1) Add C:\OSGeo4W\bin to PATH variable

2) Set PROJ_LIB variable to C:\OSGeo4W\share\pro

but GDAL is not working in manifold, I still got messages "failed to load librairy"

What did I miss?

GDAL 3.08 is integrated to qGIS, so the importation can be done in few minutes in qGIS, then I can work on my data in qGIS and export it back to manifold (9 or 8). Much more easy than all the gdal installation work I am struggling with since few days.