Hi All

A long time since I've posted here. I still use Manifold for fairly trivial work - importing and manipulating tree survey data results. But my current verison of Manifold 8.0.30 does not import gpkg. Do I need manifold-9.0.175 for this, or will manifold-8.0.31 suffice.

Also, my current installation has Manifold 8.0.30 and Radian 9.0.163 as seperate applications. Does 9.0.175 rejoin the Radian and Manifold apps back together?

(I can't get import or link or new data source to a gpkg to work in Radian 9.0.163).

Thanks

Oisin

Use Release 9. See also the Release 9 FAQ.

The GeoPackage (GPKG) format is platform-independent, and is implemented as a SQLite database container, and can be used to store both vector and raster data. The format was defined by the Open Geospatial Consortium (OGC), and was published in 2014.

GeoPackage can be used to store the following in a SQLite database:

• vector features

• tile matrix sets of imagery and raster maps

• attributes (non-spatial data)

• extensions

so file -> Tool import ->export file compatible with Manifold 8

see GeoPackage Implementations ( Tools that support import and export )

here software that are free

CLI GDAL — GDAL documentation

GUI Welcome to the QGIS project! <---------------- will choose this

CLI SpatiaLite: SpatiaLite (gaia-gis.it)

---- add support for file formats compare to SQLite

-----after test a little need ubuntu for raster or WSL2 on windows 10 ( apt-get rather than exe msi )

GUI GeoTools The Open Source Java GIS Toolkit — GeoTools

Book about Science , cosmological model , Interstellar travels

Boyle surface fr ,en

But not compatible with M8.0.30.

A small note:

Manifold 8 does not support GPKG natively, correct. You have to use Manifold 9. But once you have Manifold 9, you can work with GPKG data from both Manifold 9 and 8. From 9, just naturally. From 8, by accessing GPKG through the ODBC driver of 9:

Example: Connect Through Manifold ODBC to a Third Party (the example is for GDB, but GPKG will work as well)

I ask myself what is really user case and relation between SQLite and GPKG ....already cover by manifold documentation

-GPKG format is implemented as a SQlitedatabase container

-SQLite is Open-Source, not Open-Contribution

-SpatialLite that is a SQLite database engine for vector

-QGIS 3 switched from Shapefile( shapefile ( proprietary but open format) to GeoPackage !

-SQLite file is like acces file ( *.accdb, *.accdc,*.accde, *.accdp) use by Microsoft Access.

=

So i try to find if some database container, database engine and GUI specific to SQLite can be install easely for CRUD,import,Export using vector ( spatialLite wiki gaia-gis.it, GPK), raster (Rasterlite /Gaia-gis, GPKG ) and text ! Most of the case we need to use Gdal or Qgis (use gdal) or Rasterlite ( use gdal) ... but not a lot tool specific for SQLite in this context. SQLiteStudio 21,6 mo don't support raster and vector ( see capture screen) some specific SQLite file ( SQLCipher, WxSQLite3and System.Data.SQLite. ! When read the licence of SQLite i think raster and vector is not the main purpose of SQLite "owner".

SQLite (engine format ) is like Matroska container for Video Audio title ...track content : SQLite define the container and GPK (Wikipedia spec ) is the content that support text , raster , vector under OGC ( Wikipedia , website).

=

After read the manifold doc about GPK and ESRI implementation is : does manifold 9 use gdal or standard ( dll) or its own implementation for manage GPKG content that should follow both OGC specification ?

it is really unfortunate that manifold don't support export SQLite but only GPKG .I think rename file extension should work (todo) after better understand what mean SQLite

regard's

some comments during this post

-project gaia that implement a raster extension for SQLite use for the file ItalyRail.atlas

-manifold don't support export SQL for backup using schema ( i need to use phpmyadmin for that ! )

-many things come from army research ( monaco algo , SQLite) and personal project ( SQLite , python , linux, git, gtk, Latext )

- SQLite ( GPKG GeoPackage Implementations) : about radian :

• Radian Studio is a new Windows data engineering environment for GIS and DBMS that blends geospatial and traditional data capabilities. Radian automatically runs fully many-core CPU parallel and also massively GPU parallel using thousands of cores as a GPU DBMS. Radian has built-in GeoPackage support for full read/write of vectors and rasters, including read/write of WKB curves, geometry with Z/M, GP11 extensions, rasters with pyramids and full coordinate system support including SRIDs for GPKG coordinate systems.

Attachments:
manifold_table_export.png
SQliteStudio_support-FileExtension.png

Book about Science , cosmological model , Interstellar travels

Boyle surface fr ,en

does manifold 9 use gdal or standard ( dll) or its own implementation for manage GPKG content that should follow both OGC specification ?

As you said, GPKG uses a SQLite container. SQLite is being updated constantly, so we access SQLite data using their reference code which we embed into EXT.DLL. We implement additional logic required for GPKG using our own code on top of that.

If you provide your own SQLITE3.DLL (by putting this DLL into one of the Manifold folders, or into one of the folders referenced in PATH), we access SQLite data using that DLL. We then implement additional logic required for GPKG using our own code on top of that.

it is really unfortunate that manifold don't support export SQLite but only GPKG

I went to check and we indeed do not provide an option to export (eg, a table) as an SQLite database, although we do provide an option to export (eg, a drawing) as a GPKG file. This seems like an unfortunate omission. We'll try to add it.

In the meantime, you can create a blank SQLite database, eg, using the SQLite command-line tool, link that, then copy and paste the data you want into it. Or you can create a blank SQLite database once, then create a copy of the SQLITE file every time you need one. Agree this is not terribly convenient.

Missed the first time:

manifold don't support export SQL for backup using schema

What are you looking to do? Perform a backup from the command window using SQL?

Modern versions of SQLite do support online backup, but I am not sure they allow performing backup via SQL. I imagine the best place to process a SQL statement for the backup would be in the host = Manifold in this case. We can add this, if this is desired. But I would first like to make sure that this is what is desired (and also what's wrong with making a copy of the file).

Oisin, welcome back. Go with version 9, it is very workable for what you are wanting. re:Radian. There may still be a v9 upgrade path for your Radian license, something you might look into. This would effectively join Radian & v9, but v8 is still a separate app. The current most flexible place to be is to have both v8 for a fallback, and charge forward learning v9. I recommend this. Right now, building on top of that engine new capabilities as well as rolling the GIS'y features we need to get back to refined layout output renders(image, pdf, print) is the track we are on. Recent additions this year have been lidar tools, vector editing refinement including traverse translations. Transform panel hit big last year along with Join table dialog. Catch up on those capabilities through the videos on the Manifold YouTube channel.

I use for a long time database on linux and I know that SQLite is use a lot inside Android. So i ask myself what mean all the words use around database .

Here a table that show how we can have acces to SQLite content using script ( dll) , driver ( dll with wizard) , exe ( CLI available by default with sql engine , GUI). For each implementation , people have to take care about Visual Studio version (VS), the 32/64 bit/ARM ( "any cpu" has now a new flag that say stay in 32 bit even the runtime is 64), the MS library ( ."NET" FW ,FW small device , .net Core , .NET) and library version. OBDC is the oldest ( like JET) technology not specific to microsoft OS and next technology use specific technology ( OLE and .NET Framework) except last technology : .NET ( multi OS). It seem that in the past, the free version of visual Studio can be use but it is not the case nowodays and developper have to pay for compile ( one SQLite driver are host by forum and are free) . VS Code should have limited fonctionnalities compare to VisualStudio ( only 64 bits ? and not too much advance management project tool).

It is hard to understand most of the time if new version of a product is done by make accessible old technology and in same time wrap ( marshalling) this old technology with the new technology or if some or whole part are create from scratch and are original work.

Linq is in a way an XPath way to target data store in table but ( data) behave like xml ! For achieve that user use SQL like syntax but using C# script ! No need to master XLST,SAX, XLINK or DOM ! .... So linq C# code can be convert to SQL and linq C# code can be use for search data formated in xml ( file , node object , database, file , .... ) !

Does gdal is use wrap internally by manifold to have acces to GPKG ( raster and vector) ?

Attachments:
SQLite_acces.png

Book about Science , cosmological model , Interstellar travels

Boyle surface fr ,en

some infography about the table above !!

Datasource Link work for File (files, file databases, scripts and text comments) and connected DB( native PostgreSQL or Oracle, generic MS such as ODBC,OLE DB , ADO, ADO.NET and web servers such as xml/json over http use by REST ,WFS, WMS, WMTS and TMS overpass implemented by Google map , bing map, osm )

Web Feature Service – Add features as an editable version of data that one would view via WMS.

Web Map Services – Add maps as a standard protocol for serving georeferenced map images over the Internet that are generated by a map server using data from a GIS database.

Web Map Tile Service – Add data as a pre-rendered georeferenced map tiles over the Internet

Tile Map Service – Loads rasters of maps like Open Street Maps, Mapbox, or Bing; these can serve as a basemap.

Attachments:
providers_drivers_TABLE_history.png

Book about Science , cosmological model , Interstellar travels

Boyle surface fr ,en

This is actually a great picture. I'm always confused about the realtionship between the ADO-s and OLE-s etc. That clears it up a bit.

I was just thinking again, what would be the recommended way to plug a custom datasource. For example, some file format that has its structure already defined with Kaitai Struct.

I remembered this. (I should publish the code of my failed ADO.NET provider experiment.)

The real deal would be 'infrastructure for pluggable dataports'. Suggestion sent :-)

I ll stay focus on SQLite and Manifold and Microsoft OS

about SQLite

-i discover now that SQLite container file is use for GPKG (*.gpkg, *.sqlite, *.stylx , *.db) but also Mapbox Vector Tiles MVT ( *.mbtiles ) !!

-in manifold doc there is information about Thrid party release Levels ( manifold.net) and a chapter about SQLite:

3.35.4 - This adds UPDATE FROM, ALTER TABLE DROP COLUMN, RETURNING, various improvements to the query planner, and some SQL math functions.

about gpkg

-geometry : one column for one table for GPKG specification / many for manifold

-How is manage export using manifold if table contain many geom column ?

-Does manifold support Vector ( import export) ?

-Raster : Does manifold support raster ( import export) ? no info at GPKG (manifold.net)

about mbtiles

-There is some discussion about mbtiles in Georeference forum ...

-no MVT file name list at Formats and Data Sources (manifold.net).

-mbtiles is name for GDAL chapter . Gdal seem t be every where and is a masterpiece in gis context !!

For those of you who would like to see the return of IMS in Manifold 9, I hope you will learn something new ( see SQLite.mht to open/use/read with old internet Explorer browser ..chromium forbidden) . Things are evolving slowly but surely on the web (my domain).

Download Internet Explorer 11 (64-Bit) from Official Microsoft Download Center

Internet Explorer 11 - Wikipedia use MSHTML - Wikipedia not Chromium (web browser) - Wikipedia

Attachments:
SQLite.mht

Book about Science , cosmological model , Interstellar travels

Boyle surface fr ,en

On GPKG:

Yes, a table in a MAP file might have multiple geometry fields while a table in a GPKG file might only have (zero or) one, per the GPKG spec. We adjust for the difference during all operations. If we are exporting a table with multiple geometry fields into a GPKG, we let one geometry field stay geometry in the GPKG's sense, and export other geometry fields as binary + mark them up as 'geometry' in metadata written into the GPKG. If we are then reading that GPKG file, we are treating the binary fields marked up as 'geometry' as geometry fields in Manifold.

Yes, we support rasters in GPKG. We support all of read / write / export.

On vector tiles:

We do not support vector tiles. We might support them one day (we actually do have a prototype dataport for them in one of the side branches). But, honestly, we are not sure vector tiles are such a great idea. As in, we completely agree with the desire to have something that would do for vector data what intermediate levels / pyramids do for raster data -- which is, whatever zoom you are looking at, you can paint something reasonable while only transferring a fixed amount of data. We just aren't sure vector tiles are a good way to do this.

It is after reading this post on the forum that I studied SQLite and gpkg. In a way I wanted to use them with C# code to save non-gis data. Since I use C# I wanted to know if there were any libraries available to connect to these files using SQL and have a lightweight flexible solution compared to a relational SQL search engine (rdbms). The common sense would have been to use being in the MS environment a Microsoft file storage mode ( MS Excel , MS access , MS SQL Server ) or others !

The SQLite /C# drivers I tested ( use nuget) are :

- Microsoft.Data.Sqlite from Microsoft ( work with SQLite team !? )

-System.Data.SQLite from SQLite ( official)

When you study these Drivers you notice that they use the same method names ( SQLiteConnection() , CreateCommand() ) but not the same namespace . Under MySQL you replace SQLite by MySQL ...

But to save text, SQLite is a solution even if there are restrictions. Under manifold , I can't modify sometime the schema and name the columns as I want ( I thought it was a bug but it was linked to SQLite ...thanks to the C# debugger ) . To avoid that, I work with a sourcedata table and a local table and I do delete, copy, paste from manifold and it's very efficient.

Book about Science , cosmological model , Interstellar travels

Boyle surface fr ,en

Regarding Microsoft and SQLite, I am not aware of any big involvement from Microsoft in SQLite code (there could have been some bug fixes for SQLite submitted by Microsoft employees, and that's probably it). But Windows 10 ships with SQLite DLL in the system folder (WINSQLITE3.DLL). So I guess some Windows utilities use SQLITE to store some of their data.

since in see using nuget a microsoft package about SQLite i was think microsoft is involve in SQLite ...Perhaps they simply create a package without implement it !

Overview - Microsoft.Data.Sqlite | Microsoft Docs compare to System.Data.SQLite: Home

regard's

Book about Science , cosmological model , Interstellar travels

Boyle surface fr ,en

Microsoft did not create SQLite. SQLite was created by Richard Hipp, starting in 2000. SQLite has become ubiquitous, and Microsoft uses it for some things.