Subscribe to this thread
Home - Cutting Edge / All posts - Manifold System 9.0.176.2
adamw


9,956 post(s)
#07-Feb-22 07:41

9.0.176.2

manifold-9.0.176.2-x64.zip

SHA256: abc0b985e31d08809c16b3e08e1fd3a893cfeedcdb7d93477dd06c4dd9c5e79c

manifold-viewer-9.0.176.2-x64.zip

SHA256: 947bb0eba1c6abe4fa5e0927f63580dd228d81b092d10c64881ed43c380e74ff

sql4arc-9.0.176.2-x64.zip

SHA256: 7ed34e265135729ed011d9be696cc485a114b9b6e5bc45c7e0e3c54496a95b1e

adamw


9,956 post(s)
#07-Feb-22 07:50

Changes

The query engine optimizes IN / NOT IN <table> on a single value, for cases where the table has no index on the searched field.

The query engine optimizes IN / NOT IN <table> on a tuple, both for cases where the table has no index on the searched fields and for cases where it has one, by making more efficient use of it.

(start note)

As is usual for optimizations that start building and using indexes instead of doing full scans, the speed ups are rather dramatic. For example, here are the results for a test query similar to the one that brought this performance issue to our attention in the first place:

--SQL9

VALUE @count INT32 = 1000;

DROP TABLE t;

CREATE TABLE t (mfd_id INT64INDEX mfd_id_x BTREE (mfd_id), v INT32);

INSERT INTO t (v) TABLE CALL ValueSequence(1, @count, 1);

INSERT INTO t (v) TABLE CALL ValueSequence(1, @count, 40);

DELETE FROM t WHERE mfd_id NOT IN (SELECT First(mfd_id) FROM t GROUP BY v);

With @count set to 1000 records: old 1.110 sec -> new 0.029 sec.

With @count set to 5000 records: old 27.411 sec -> new 0.130 sec.

With @count set to 20000 records: old 482.044 sec -> new 0.544 sec.

(end note)

New query function: StringStart. Reduces a string to the first N characters. If the specified number of characters is negative, the entire string is returned, consistent with the behavior of StringSubstringLen.

New query function: StringEnd. Reduces a string to the last N characters. If the specified number of characters is negative, the entire string is returned, consistent with the behavior of StringSubstringLen.

New query functions: TileSieve / TileSievePar. Perform a sieve operation on an image channel: pixels with the same value form areas, areas smaller than the specified threshold are merged into their neighbors = get assigned the value of the neighbor they are merging into, the result is a new image. Parameters:

  • source image,
  • source channel (0 ... total number of channels - 1),
  • quantization level (setting this to 1 tells the system that pixels with the values of 1.1, 1.2 and 1.7 are similar enough to each other and should be put into the same area if they are neighbors, but a pixel with the value of 2.0 should be put into a different area),
  • area threshold in pixels (areas smaller than this value will try to merge into their larger neighbors, areas equal to or larger than this value will accept their smaller neighbors),
  • method used to determine which neighbor to merge into when there are multiple candidates: 1 (default) = merge into the neighbor with the biggest area, 2 = merge into the neighbor with the biggest common boundary,
  • thread configuration (TileSievePar only).

New query functions: TileSieveStep / TileSieveStepPar. Perform a single step of a sieve operation on an image channel. (Performing a single step instead of running the operation until there are no more areas to merge is sometimes useful as an intermediate step in bigger computations.)

New tile transform: Sieve. Performs a sieve operation on an image channel (runs until there are no more areas to merge).

(A note on performance: the Sieve transform could be slower on some images than an analogous operation in QGIS / GDAL, however the operation in GDAL frequently leaves areas that are smaller than the specified threshold not merged into their bigger neighbors. It seems to us that the operation implemented in GDAL might be somewhat similar to our single-step sieve, and perhaps even smaller in scope in that the merges seem to proceed in a subset of directions instead of in all directions.)

New numeric select: Unique. Selects values based on how many times they occur in the table:

  • duplicate - for each value, selects all records except one,
  • non-unique (occurs more than once) - selects records for values that occur more than once,
  • unique (occurs once) - selects records for values that occur only once.

New numeric vector select: Unique. Similar to numeric select.

New binary select: Unique. Similar to numeric select.

New date select: Unique. Similar to numeric select with an extra option: Ignore time (default: on).

New text select: Unique. Similar to numeric select with extra options: Ignore case (default: on), Ignore whitespace at start and end (default: on).

New uuid select: Unique. Similar to numeric select.

New numeric transform: Random. Produces random values between specified minimum and maximum, with option: Whole numbers (default: on).

New uuid transform: Random. Produces random values.

New query aggregate function: Percentile. Returns the value for the specified percentile. Percentile 0 returns the minimum value, percentile 100 returns the maximum value, percentile 50 returns the median value. Works for values of any type.

New numeric select: Statistic. Selects values based on how large or small they are relative to other values in the table:

  • maximum - selects records with the maximum value,
  • minimum - selects records with the minimum value,
  • median - selects records with the median value,
  • top - selects the specified number of records with the largest values,
  • bottom - selects the specified number of records with the smallest values,
  • top percent - selects records with the values larger than the specified percentile, the bound can be included or excluded,
  • bottom percent - selects records with the values smaller than the specified percentile, the bound can be included or excluded.

New numeric vector select: Statistic. Similar to numeric select.

New date select: Statistic. Similar to numeric select with an extra option: Ignore time (default: on).

New text select: Statistic. Similar to numeric select with extra options: Ignore case (default: on), Ignore whitespace at start and end (default: on).

Layouts support min / max scale for map layers.

Switching the source window in the Register pane maps already entered control points to the source control points by name, to preserve as much of the already entered data as possible.

End of list.

Mike Pelletier

2,004 post(s)
#07-Feb-22 10:36

Although I'm really hoping to see label improvements :-), its great to see more tools for exploring data.

The Select Pane is getting better at selecting records but we often still need to write a group by query on the selected records. Also, the method of displaying just selected records (on/off) would be better if it was one click like in Mfd8.

I wonder if the Select Pane could be given a group by option to make it work more like a query and provide a subset of the data. Sure we can write a query, but we want to take advantage of the Select Pane's fast process for selecting the records of interest.

Mfd8 provided the option to select a single unique value (randomly chosen) from a field. Then we hit the button to show selected records and you could see all the unique values for the table. Very nice.

Better yet is the following query, because it shows you how many records are associated with each unique value.

SELECT [TaxParcelType], Count(*) AS count

FROM [Taxparcelassessor table] GROUP BY [TaxParcelType];

It would be great if the Select Pane could be given more check boxes to show this info.

adamw


9,956 post(s)
#08-Feb-22 06:28

Thanks.

We are going to continue adding features to the labels soon. We will also work on the show all / show selection only filter, it will be as easy to use as it was in 8. We thought about packaging something like the query you suggested into the UI as well -- the reason we did not do this right away is that it feels weird to have something that generates a new table be part of the Select pane, this should perhaps be a transform.

Mike Pelletier

2,004 post(s)
#08-Feb-22 07:32

Thanks for the good news here and I hesitated to suggest that the Select Pane be able to make a new table for the same reason. However, perhaps it is more reasonable since exploring the different selections of the table is the main brain focus of the effort. The stat calculation is just a click or two. By exploring, I'm thinking it would be good if the Select Pane could store selections and/or some sort of selection history a user can revisit.

On the other hand, maybe it is quick enough to switch to the Transform Pane to do this once the selection is made. Either way it would be nice to first see the table and then decide whether to actually add the table to the Project Pane. As always I appreciate that I see/understand the very tip of the iceberg. Thanks for all the effort!

Mike Pelletier

2,004 post(s)
#08-Feb-22 18:26

Actually it would be nice if the solution could also apply to a table resulting from a query.

danb


1,931 post(s)
#07-Feb-22 12:20

What a great way to get into the new year proper!

This is a great new build, thanks so much for sieve which is a vast improvement over an already very useful GDAL function. This will be so useful for generalising grid datasets. It will allow us to effectively set a minimum mapping unit for our new LiDAR data and will greatly improves our ability to produce datasets that can be pragmatically implemented at a paddock scale using this data.

I am also so pleased to see the select duplicates function back in addition to the other statistic functions. I have so missed this and it was probably the principal thing (other than mapping) that drew me back to M8.

The optimisation resulting from the recent thread by Marcus is also a great step forward. I realised that I use this sort of query quite often, so many of my existing processes will be greatly expedited.

Great work! I can't wait for the next installment now


Landsystems Ltd ... Know your land | www.landsystems.co.nz

adamw


9,956 post(s)
#08-Feb-22 06:31

Thank you!

The performance improvements to IN / NOT IN, by the way, are key to the new select templates. :-)

HMS
161 post(s)
#08-Feb-22 07:59
  • area threshold in pixels (areas smaller than this value will try to merge into their larger neighbors, areas equal to or larger than this value will accept their smaller neighbors),

Hi Adam, like danb stated, indeed the sieve function seems great for generalizing grid datasets. Does the mentioned "area treshold" refers to the number of pixels?

adamw


9,956 post(s)
#08-Feb-22 08:07

Yes, it's the number of pixels. We might convert it to a physical area in the future, in the units of the coordinate system. (And then the picker for that parameter will show the used unit on the right side and allow editing it.)

HMS
161 post(s)
#08-Feb-22 08:49

Thanks for the clarification. Since the manual does not refer to the sieve transformation (yet), while working on the translation file I thought that the "minimum area" (without the pixel reference) might induce some kind of error.

Manifold User Community Use Agreement Copyright (C) 2007-2021 Manifold Software Limited. All rights reserved.