Subscribe to this thread
Home - Cutting Edge / All posts - Manifold System +

10,239 post(s)
#11-Sep-21 15:40

SHA256: 7d75f68eda9af0d592a60d4a2621077375d360469f083e236dffe7cd21112dad

SHA256: ddda96262de327a0fbf8bb1d413ef759f17800c5125ca93edd270860224c672e

SHA256: 95473bc07abdc67c6a51baa19d632e16ebde45a8178d24d333c579a341024ea5

SHA256: 459bf84f949ea02684d5ad3facb81a1f43eae864511c8529274979325ecf6ea4

The new build for 9 contains a big set of fixes and improvements for databases, better error diagnostics for operations on data in MAP files, and a rework of query operations for tile masks.

The new build for 8 contains several small fixes, particularly, the new proposed fix for rendering big images using Make Image.

Cutting edge builds for 8 come with limitations that are slightly different from those for 9: there should be a working non-portable install of Manifold 8 on the same system (9 does not require this), third-party applications and web sites continue to use the non-portable install (with 9 you can specify which install to use). To run a cutting edge build, just unpack it and run one of the MANIFOLD.EXE files (either in BIN or in BIN64).


10,239 post(s)
#11-Sep-21 15:42

Changes for 9

Error messages for table design operations in MAP files are extended to include the names of fields / constraints / indexes relevant to the error. (For example: "Cannot rename field 'length', referenced by computed field 'length-km'.")

Error messages for table record operations in MAP files are extended to include the names of fields / constraints / indexes relevant to the error. (For example: "Cannot set values of one or more fields, rejected by index 'name-x'." Or: "Invalid value of key field 'x', null not allowed.")

The file filter for GDB is changed from '*.gdb;gdb' to just 'gdb'.

The file filter for GDBMOBILE is renamed from 'GEODATABASE Files' to 'GDB Mobile Files'.

Computing bounding box for geometry data on SQLITE with SPATIALITE or ESRI ST_GEOMETRY extensions uses ST_MinX/MinY/MaxX/MaxY functions for performance. (We used to call a different aggregate, but apparently we can rely on the simpler - and much faster - ones being there as well, so we switched.)

(Fix) ALTER TABLE no longer can rename the schema item of a wrong type. (Attempting to rename, eg, an index did not check whether the schema contains an index with the specified name, just that the schema contains an item with the specified name. Then the operation could rename an item of a different type, eg, a field. The change fails such a rename early.)

(Fix) Creating a new non-primary index on SQLITE adjusts the name of the index to make it unique.

Creating a new table on DB2 / Jet / MySQL / Oracle / PostgreSQL / SQL Server reduces the name of the table to obey the server limit.

(Fix) Creating an identity field on GPKG no longer forces the identity field to be autoincrementing.

(Fix) Creating an RTREE index on GPKG fails if the index references a field other than the primary key field. (Such an index violates the GPKG spec.)

(Fix) Data sources set to cache data in the parent database (eg, Bing Maps or another web image with the 'Saved cached data between session' option turned on, by default the option is turned off) no longer sometimes fail to work if the parent database is not a MAP file.

Data sources set to cache data in the parent database check the capabilities of the parent database and proceed without cache if the database does not support some of the required features (MAP is accepted, databases like SQL Server are accepted with a couple of minor limitations, eg, DB2 is rejected for vector caches because its default SRID requires X and Y values to be non-negative - we will find a workaround, GDB and similar data sources that look like databases but have many limitations are rejected).

Data sources set to cache data in the parent database use separate cache tables for each database schema. (Previously we were trying to use cache tables in any accessible schema, this was creating situations where different users with limited permissions would create their own cache tables - this is fine - and then users with extensive permissions would end up having to choose between multiple available cache tables and would have to make the choice basically at random - not fine. The change makes things much more predictable. Whichever schema you connect to by default is the one that we are going to use for the cache.)

Data sources set to cache data in the parent database continue working without cache when cache tables cannot be created or are inaccessible.

(Fix) BTREExxx indexes in SQLITE now correctly detect the ASC / DESC order for the key fields.

All databases implement the uniqueness requirements implied by BTREExxx indexes using UNIQUE indexes instead of UNIQUE constraints. (Using UNIQUE constraints was backfiring in some situations, putting UNIQUE onto indexes matches the semantics that we currently have more closely. BTREExxx indexes created by older versions of 9 will continue to work as before.)

Renamed query function: TileMaskExtract -> TileMask.

The TileMask query function now produces a numeric tile with 0/1 values (was producing a tile with boolean values).

The TileMaskReplace query function now accepts mask as a numeric tile (was accepting a tile with boolean values). Pixels whose mask is either missing or 0 are made invisible in the result. Other pixels are made visible. Pixels that were previously invisible but became visible are set to 0.

The NOT / AND / OR / XOR operators for tiles now accept numeric tiles with 0/* values (0 interpreted as FALSE, any other value interpreted as TRUE) and produce a numeric tile with 0/1 values.

The comparison operators for number/tile, tile/number, tile/tile combinations now produce a numeric tile with 0/1 values.

The Compare query function for number/tile, tile/number, tile/tile combinations now produces a numeric tile with -1/0/1 values. The TileCompare query function is removed (Compare now does everything it did and more).

The BETWEEN operator for tiles (at least one of the operands is a tile, operands that are not tiles are numbers) now produces a numeric tile with 0/1 values.

(Fix) Creating a new computed field or constraint no longer sometimes accepts an expression with extra tokens at the end (eg, 'b+100-', this was being accepted as 'b+100', now we complain about the extra '-' at the end).

End of list.


10,239 post(s)
#11-Sep-21 15:42

Changes for 8

There is support for cutting edge builds. The product name in install packages now includes a 3-digit version number, similarly to 9. Cutting edge builds require a regular build to be installed and reuse some of its services (see the first post in the thread). Cutting edge builds expire 3 months after the build date, warn about expiring soon 2 months after the build date.

(Fix) The Make Image dialog tries to work around the bug in GDI+ that shifts rendered lines slightly on big outputs by rendering data in portions of at most 128 MB.

Help menu commands are replaced with Documentation and Web Site, similarly to 9.

The application declares compatibility with Windows Vista / 7 / 8 / 8.1 / 10, reports Windows 10 in the About dialog.

The About dialog shows the date of the current build.

End of list.

162 post(s)
#14-Sep-21 14:26

Thanks for the new M8 version.

The shifts in rendered lines fails in my case as shown in those two map views exported at 400dpi both but for a restricted portion on the right and the full extend on the left. The left version shows strong horizontal patterns along the ridges.



10,239 post(s)
#14-Sep-21 16:25

Yes, we see. Thanks a lot. We'll try to do something else then.


10,239 post(s)
#11-Sep-21 15:43


For 8, we mostly would like to make sure that the fix for Make Image works. We might make a couple of other changes, but 8 is not the focus.

For 9, we are working on the following things:

  • Better labels - we are working on a big improvement which we thought would make it to this build, but it needs a little more work. It will appear in the next build.
  • Editing traverses - we have a working prototype, ironing out the details.
  • Better table window - we completed the design for a new way to access records which will be a big improvement to the current state where you are limited to 50,000 first records. Works start next week. This will take a couple of builds, but we want to get this done for 9.0.176.
  • Better error messages for queries - we started with better error messages for operations performed when the query is already working. We will continue there. But that's far from everything and there are also error messages for syntax errors that occur when the query is parsed. These error messages are much harder to make useful. We are far from alone here: in any modern database people complain that error messages thrown during parsing are bad and either don't suggest what the real issue is or suggest a wrong thing, even after tens of years of work. This stems from SQL dialects routinely having a much more complex grammar (think 10x or 50x) than a general-purpose programming language. Still, we think we can get a big improvement by walking through the most common cases. Currently running experiments. This will pretty certainly not get done in 9.0.176 and it remains to be seen just how much ground we will be able to cover for it.
  • Better raster math - being able to use per-pixel operations in queries and much easier means to combine different images in analysis. This is currently in design and it seems like we need new keywords and clauses. We want to get this done for 9.0.176. (Our technical writers that are currently writing the SQL doc will likely eventually kill us, we are stalling them with plans for changes and new features all the time.)

Last, we will try again to make the distance between the builds smaller. We had very limited success with this so far, trying to make it work. Ideally, we would be issuing a new build every 7-10 days.

Have fun with the builds. :-)


3,285 post(s)
#11-Sep-21 18:22

Thanks for the roadmap on 8 and 9. Anything new coming with SQL for ArcGIS Pro?


7,115 post(s)
#12-Sep-21 05:48

Anything new coming with SQL for ArcGIS Pro?

All the new items listed for 9 also apply to SQL for ArcGIS Pro as well, for example, the error messages. Some of the others were prompted specifically by sql4arc experience, such as the mobile geodatabases stuff / SQLite, caching in databases (lots of GDB limits, etc.). All that is good for 9 too, given the ubiquity of work with GPKG/SQLite and GDB of various forms encountered when working with 9.


10,239 post(s)
#12-Sep-21 10:01

Like Dimitri said, all of this applies to SQL for ArcGIS Pro. Better raster math, for example, is directly useful. The whole idea of the product is to allow doing useful things fast and easily with the help of SQL, making raster math easy to write plays right into this. Same for better error messages. We have more things planned, too, some were just too small and disjoint to mention in the list and others we are not prepared to talk about just yet. :-)


1,970 post(s)
#12-Sep-21 02:48

Many thanks for the detailed road map. For me the latter three items are very welcome and I look forward to seeing them develop.

Thanks also for the improvements in error messages in this build. I make a lot of mistakes while tinkering and any pointers towards the errors are most welcome as I often find it difficult to spot even simple errors without syntax highlighting.

Landsystems Ltd ... Know your land |

Mike Pelletier

2,053 post(s)
#13-Sep-21 15:15

Actually Dan the first two items are more welcome. Must be a southern hemisphere thing .

Thanks Adam for the detailed look ahead and of course all of it is very welcome!


1,970 post(s)
#14-Sep-21 03:05

Labelling improvements I am looking forward to but I likely wont make a map layout in M9 until scalebars and north arrows appear. Traverses are interesting and I will have a look and see where I might use them.

Anyway, nice to know that the immediate plans have a good balance between between hemispheres

Landsystems Ltd ... Know your land |

Manifold User Community Use Agreement Copyright (C) 2007-2021 Manifold Software Limited. All rights reserved.