Subscribe to this thread
Home - Cutting Edge / All posts - Manifold System 9.0.172.4
adamw


10,447 post(s)
#11-Aug-20 15:08

9.0.172.4

Here is a new build.

manifold-9.0.172.4-x64.zip

SHA256: 3e6107a3d16cc0570ac869be9ea8ce2220d8d805b080b9f39934215fa85923b5

manifold-viewer-9.0.172.4-x64.zip

SHA256: 716922fa89fa1980ddf0d90a9f88d2cc4247b9be2089a7efeac42c879b011130

We apologize for going so long without a new build. We will work on our planning.

The new build contains numerous breaking changes to queries, mostly due to extensions. The full list of changes is in the Queries section.

adamw


10,447 post(s)
#11-Aug-20 15:13

Select and Transform

The Select and Transform panes have been redesigned. Both panes use a two-step workflow:

In the first step, we pick a layer and a field. The pane then shows a list of templates available for the field. The list can be filtered using the filter box. Double-clicking a template (or repeat-clicking or focusing and pressing Enter or focusing and clicking the Edit Parameters button at the bottom of the pane) picks the template and proceeds to the second step.

In the second step, we specify parameter values for the picked template and then click the Select or Transform button at the bottom of the pane to perform the operation. The name of the picked template is shown at the top. To go back to the list of templates, click the button with an "up" arrow in the right top corner of the pane. The layer and field used to pick the template can be changed without going back to the list of templates with the picked template remaining the same and the parameter values automatically adjusting to the new choices.

The Select and Transform panes automatically update themselves after changes to components. For example, if we open a table, then open the Select pane and pick a template, then add a new field to the table, the pane will adjust and start offering the new field in lists for parameter values. If we pick the field for use as a parameter value and then delete it from the table, the pane will adjust again and switch the parameter value referencing the now deleted field to use a constant value or a different field, depending on the parameter. Similarly, if we open a map and start adding or removing layers, the panes will automatically adjust to all changes. Unlike before, changing the active layer has no effect on the panes, adding a new layer has a minimal effect (the new layer becomes available for use but the picked layer does not switch to the new layer), deleting a layer only has an effect if that layer was the one picked.

Switching between component windows automatically saves and restores the state of the Select and Transform panes for each window. For example, we can open a drawing, start preparing a transform, then decide to take a look at the records in a different table, opening that table in a new window, and when we return to the drawing, the Transform pane will be in the exact same state as we left it. We can have several alternative transforms or selects being prepared for different windows and switch between them freely.

Performing a select or transform does not clear the relevant pane and leaves it in the state where the operation can be repeated with or without modifications. Since many operations complete nearly immediately, after you click Select or Transform and the operation completes, the pane shows the time it took for the operation to complete next to the clicked button. The time readout indicates that you did indeed click the button and that the operation has already finished. After 10 seconds, the time readout disappears.

The templates have been made much bulkier with each new template frequently combining many of the formerly available templates. This, together with not resetting the panes after performing a select or transform, encourages staying on the screen with the parameter values for longer, by reducing the number of times you have to go back to the list of templates to pick a different one.

Each performed select or transform is saved for the duration of the Manifold session and last saved operations are shown in the list of templates in the relevant pane. Picking a saved operation picks the template used for the operation and sets the values of the non-field parameters to the ones used during the operation. (We are considering saving and restoring the values of all parameters including those that can be set to fields, this is a design choice.) The name of a saved operation shown in the list includes the values of key parameters. The total number of saved operations is not limited (the limit is not realistically reachable), however the panes only list the last ten saved operations from those that can be applied to the picked field. Performing the same select or transform multiple times does not create multiple saved operations flooding the list, and instead updates a single saved operation putting it as last every time.

A saved select or transform can be pinned for persistent use between Manifold sessions by clicking the pin icon in the right column of the list. Pinned saved operations are listed first, with the non-pinned saved operations following, with the templates listed last, each group is sorted alphabetically.

The list of templates shows a tooltip for each template. The text of the tooltip shortly describes the details of the template. The filter box searches the name of the template, the tooltip, and the available values for key parameters. This allows typing in '>' in order to quickly find the select template that selects values greater than specified, or typing 'aspect' to quickly find the transform template that computes aspect even if the names of the relevant templates are more general.

Performing a select or transform logs operation time in the log window (previously, this was only done for transforms). The log message includes the names of the used layer and field, the name of the template, and the values of key parameters.

Queries composed for select and transform templates list used parameter values in the opening comment section. Queries for transforms that create new fields or components separate the part of the query that performs one-time setup from the rest of the query that can be run repeatedly.

Expressions used for parameter values are edited in a separate dialog with the query builder that helps compose the expression. The dialog automatically checks the syntax of the entered expression and fails if there is an error. The dialog also automatically checks the type of the entered expression and fails if the type is incompatible with the expected type. The expected type for the expression is shown in the left bottom corner of the dialog. Expression code shown in the parameter control and used in the template is automatically compressed: comments and unnecessary whitespace are removed, multiple lines are squeezed into a single line, etc. Expressions that consist of multiple terms are also enclosed in parens to enforce computation order in the query text.

Result options for a select are now listed in a separate Action parameter: replace selection (the default), add to selection, intersect with selection, invert with selection, subtract from selection. Result options for a transform can be either:

  • same field - perform the transform on the picked field and put the result back into it.
  • a specific field - perform the transform on the picked field and put the result into a different field of a compatible type. Tile fields are considered to be compatible when they have the same tile size and the same number of channels.
  • a new field - perform the transform on the picked field and put the result into a new field of a compatible type, the name and type of the new field are specified in the pane. For a tile field, the type is that of the channel values.
  • a new table - perform the transform on the picked field and put the result back into it, copying data into a different table, the name of the new table is specified in the pane.

If the pane is set to create a new field or a new table and the resulting field is geometry or tile, a new drawing or image is also created with the name specified in the pane.

Attempting to perform a transform on a computed field or on a built-in identity field (MFD_ID) disables the 'same field' result option. Attempting to perform a transform on a table that cannot alter its schema (common for result tables of queries) disables the 'new field' result option. The default name for a new field is set to a blank string to force entering it before running the transform. After the transform is run, the field name is left unaltered so that repeating the transform without any changes quickly fails with the 'name already in use' error and does nothing. Similarly, the default names for new components are set to blank strings to force entering them, and after the transform is run the component names are left unaltered so that repeating the transform without any changes quickly fails.

Creating a new table for a transform automatically creates an autogenerated identity field and a unique index on that field. If the target database does not support autogenerated fields, the transform fails. (We support autogenerated fields on the vast majority of databases that we can write to. The only major databases left out are those based on Microsoft Jet: Access and Excel files. But for Access and Excel files, it is better to copy the data into a MAP file, perform the transform in the MAP, then copy the result back into the Access or Excel file anyway. Performing the transform through the MAP instead of directly on the data stored in the Access or Excel file limits the time the Jet database is being written to, which helps ensure its integrity, particularly when the database is being open from a network share, as happens often.)

If the resulting field of a transform is geometry, the new table excludes all other geometry fields and creates a spatial index on the kept geometry field. If the resulting field is tile, the new table excludes all other tile fields and creates a spatial index on the kept tile field. (We exclude all geometry and tile fields except the resulting field because we assume that when a table contains multiple such fields, they are intermediate results created by prior transforms. When you start a new table, this starts a new chain of transforms from the desired point and the new table is set to include all non-spatial attributes, but not the intermediate results of prior transforms. If you want to keep all or some of the intermediate results, you can first make a copy of the table and then run the transform on the new table updating an existing field.) If the target database does not support spatial indexes required for the resulting field, the transform fails. (This limits transforms creating new tables with tiles to MAP files. This may change in the future. Transforms creating new tables with geometry can run on MAP files and on most databases.)

Both selects and transforms require that the original table has an identity index. Transforms on tile fields that depend on tile placement (have to place tiles relative to each other, eg, to read the values of neighbor pixels) also require that the table has a spatial index on the picked field.

Operations on drawings no longer specify the tolerance value except for a few special cases where this value materially defines the operation (normalize). The tolerance value passed to the query functions is set to 0 = auto. In the future, we will likely remove this parameter from most functions altogether, the user should not have to worry about it.

Operations on images have been significantly extended. Operations that work with a single channel allow specifying the channel number and can be used with multi-channel images. New images created by transforms have their bounds automatically set to the extents of visible pixels (not tiles). New images created by transforms use the automatic style computed from pyramid statistics, with the smallest available pixel values being set to black, the biggest to white and the rest to a shade between black and white. The automatic style applies to all images which have a spatial index and do not have an explicit style set, which is also helpful for images created by user queries and scripts. The automatic style for images with UINT8 channel values is special-cased to always map 0 to black and 255 to white for compatibility reasons and because this is the desired behavior in the majority of cases.

The autocontrast options in the Style pane for images limit the computed contrast range to the actual range of pixel values within the image for better image quality. The options are reordered from smallest to biggest adjustments so that options most likely to be used appear first.

All operations in queries can be performed directly on tables without any other components involved. For the technical details, see the Components and Queries post below.

For transforms, there is a new Resources parameter that specifies computation resources available for the transform. The available options are: 'all CPU cores, all GPU cores' (the default) / 'all CPU cores' (GPU cannot be used) / 'one CPU core, all GPU cores' (only one thread is allowed, but GPU can be used) / 'one CPU core' (only one thread is allowed and GPU cannot be used). Each transform decides which of the allowed resources to use based on the nature of computations, including the specified parameter values. For example, if there are multiple CPUs available, but the computations are too small to benefit from that, a transform may elect to use a single thread to make better use of the resources.

Most transforms (specifically, transforms on non-tile fields and transforms on tile fields which do not depend on tile placement) allow restricting the transform to the selected records. Transforms that involve secondary components allow restricting data used from those components to the selected records as well.

Many transforms protect from NULL values. For example, attempting to append a text value to a text field using Concatenate treats NULL values as empty strings to keep the field value unchanged instead of turning it into a NULL. (Sometimes such a protection is undesirable and whether to have it or not depends on the typical use of the transform. We added current protections based on our own judgement. If you find some of the protections that we added undesirable - or find cases where we do not have such protections and it would make sense to have them - tell us.)

Geometry transforms that only make sense for a particular geometry type keep geometry values of other types unchanged whenever this makes sense. For example, reversing lines will keep areas and points unchanged instead of turning them into NULLs. Geometry transforms that operate on distances also automatically compensate for uneven X and Y scales. (Previously, if a coordinate system of a geometry field had different scales by X and Y, creating a buffer would create a circle in the coordinate system of the drawing which would become an ellipse if the scales for X and Y were made the same. Same for other distance computations. Now the transforms make the X and Y scales even prior to computing the buffer and then force the computed buffer back to the scales used in the coordinate system. This makes the results of computations independent of the scales used in the coordinate system, which is much more reasonable.)

Creating a new component using a transform automatically selects that component in the Project pane.

gjsa100 post(s)
#18-Aug-20 08:04

Would like to just register my early, initial experience with this redesign.

I don't like it - there seems to be a creeping tendency to group operations and menus where it was previously open and clear as to what was possible.

I know don't know where to find transform templates - what is in arithmetic ? what is in trignometry ? I need to think about or search through menus to find where stuff is instead of just instantly eyeballing a single manageable list as it was before.

And we've lost the autopopulation of the transform layer drawing and table name - without ANY facility to copy and paste the name of the layer we are transforming to use as a basis for the name.

This is bad. Just from a workflow perspective this is bad, even if you disagree with my preferences mentioned above - this is undeniable: every transformation query takes LONGER to set up.

It's gotta be optional. Really. The original interface offers ultimate flexibility. Please don't take that away!

Or if this is inevitable, release 9.0.162.3 as a commemorative version without an expiry.

gjsa100 post(s)
#18-Aug-20 08:18

* 9.0.172.3

Dimitri


7,413 post(s)
#18-Aug-20 08:41

And we've lost the autopopulation of the transform layer drawing and table name

That's coming.

I know don't know where to find transform templates

Use the tool tips to remind you. Very quickly you know the name of options within templates, so entering a few characters in the filter box lets you find it if you don't remember, say, which template to use for "=".

You also very quickly get a feel for what is where. It helps a lot that templates group together options that do similar things, or which involve similar options or workflow. The names are helpful too. If you look inside what is in "arithmetic" and what is in "trigonometry" you'll never again wonder what is in those. Sines and Cosines are "trigonometry", right? :-)

Recently used and pinned templates let you jump directly to what is frequently used. Those populate very quickly with a bit of use.

Just from a workflow perspective this is bad

I felt that way too for the first few hours. But then as I started using it, after a day there's no way I'd go back. The new interface provides way faster and smoother workflow, once you get to know it. Really, once you get used to it, it provides dramatically faster and easier workflow.

Make a commitment to learning it and using it, and then after a few full days of active use in real life projects see how you feel.

I found that after a couple of weeks of using it, I liked the interface a lot but I wanted a few small tweaks, like auto population of names. I sent those in as suggestions. But now after a month or so of using it, I can't think of further suggestions for now (no doubt there will be more over time), other than filling in the transforms that are underway and providing some new ones.

tjhb
10,094 post(s)
#18-Aug-20 10:02

This is a bit silly?

The "creeping tendency" you mention is a plan.

Some things take some getting used to.

Needing to think freshly is normal with progress.

this is undeniable: every transformation query takes LONGER to set up.

Queries don't, but transforms can, yes. But there are fewer of them, and that is a benefit.

Why? Because the Transform list was already becoming too long to navigate in real life. Now we have fewer transforms, with a range of closely-knit options. One more click in many cases... but a whole lot less scrolling.

That is one thing. The next thing, is that there is now room for the next big thing.

There wasn't room before. Now there is.

And more than either of these things, transforms can now do more. They now have two hands, previously only one. A very big deal.

gjsa100 post(s)
#19-Aug-20 00:01

It's a radical redesign and I do need to give the concept the benefit of the doubt. I have confidence the M9 team know what they are doing. But I was frustrated and shocked when I opened this version - I couldn't figure out something as simple as transforming a float field with the "area" transform.

Dimitri


7,413 post(s)
#19-Aug-20 07:52

figure out something as simple

Watch the two videos on the new UI... they help get started quickly.

(switching gears) Replying to your post and Tim's on this:

this is undeniable: every transformation query takes LONGER to set up.

Strongly disagree. In fact, in real-life workflow it is exactly the opposite with transforms taking dramatically less time to set up. It's just amazing how much faster and easier it is.

That's for two reasons:

First, because real-life workflow tends to be repetitive and iterative, and the dramatically better persistence of the new UI makes repetitive and iterative operations way faster and easier.

Second, because settings in the new UI persist and survive through context switching. In the old UI you could set up a transform and then just touch a different window and what you set up was gone. Now, you can do a transform for a window, go off and do some other things in other windows - even involving transforms in those windows - to make ready for the next iteration, and when you go back to the original window the transform pane will automatically switch back to the context and settings as you left them.

In real life workflow, the above two effects are huge. The new UI is way faster for most real life work with select and transform.

tjhb
10,094 post(s)
#19-Aug-20 08:15

Cache is King.

Absolutely right Dimitri.

adamw


10,447 post(s)
#18-Aug-20 10:04

Thanks for your post.

We agree we need to allow filling in the names of new components easier. We don't want the names to be filled completely automatically like before, because then it becomes too easy to accidentally perform the same transform twice producing two sets of results. There was no risk of this happening in the old panes, but only because the old panes were resetting controls after each operation, which we absolutely want to avoid, because that's very destructive to the workflow. Maybe it would be a good idea to add a button to autofill the names whenever desired.

On grouping of templates, we believe the new system is better than the old, including in discoverability. In the old system, Contour Areas and Contour Lines were separate transforms. Does it make sense to merge them into the same transform with an option? We think so. Same for different types of curvatures. Same for Lower Case and Lower Case, Intl. Etc, etc, etc. We agree that lists are good when they are manageable, but one of the major problems was exactly that our lists were becoming unmanageable. The list of transforms for a text field in 9.0.172 could only show ~60% of the items on the 1920x1080 screen with the pane using all available vertical space. The list of transforms for a tile field was only showing ~30% of the items, you had to page down three times to see everything. Realistically, one frequently had to use a filter box to limit the lists - very experienced users in our testing sessions did so routinely despite knowing which template they were looking for, simply because scrolling to the item was slower than filtering the list.

Keep in mind that we wanted to add many, many new transforms and selects. We already added around 100 new operations simply during the rework, these would have been new list items before. We are going to add many more. The filter box would be the only thing keeping this together in the old system. Grouping similar templates allows us to dramatically reduce the number of items in lists which makes it easier to remember which item does what in the long run. There's a re-learning curve, we agree, but we try to make it as painless as possible with things like tooltips (don't have to dig into the template to see what it does) and saved operations (once you found something, you don't have to remember where it was again). And the filter box still works and can search the values of key parameters, more or less allowing you to filter using names of the 'old' transforms the same way you'd filter in the old system without grouping but with a big list that has to be scrolled instead.

Give the new system a try. If there are specific pain points, let's fix them. For example, if the number of clicks for a particular operation increased, let's fix that. The autofilling of names for new components is a good example of something that can be improved, absolutely, thanks for bringing it up.

vincent

1,972 post(s)
#19-Aug-20 19:54

Once again, I'm lost with a new version.

I'm used to work on a surface, and using the SUM transform with a specific radius and a circle shaped filter. How do I do that with this so-called improved version ?

Thank you.

rk
621 post(s)
#19-Aug-20 20:30

search for 'sum'

choose Filter.

Set

Filter: sum

Shape: circle

Attachments:
p1.png
p2.png
p3.png

adamw


10,447 post(s)
#20-Aug-20 09:28

A radius of 77 is pretty daring. :-)

On topic: many of the former templates have been merged together and some have been renamed. To find an old template in the new system, use the filter box and the tooltips.

Dimitri


7,413 post(s)
#20-Aug-20 10:21

I'm lost with a new version.

The changes are in the Select pane and Transform pane, and not in other panes, so virtually everything you already know still applies.

Some tips to help you get your bearings:

Check out the two new videos to see how to work it. Once you see it done, it falls into place quickly.

For finding templates you've used before, as rk and adamw suggest, use the filter box.

Although templates and the operations within them have become more flexible, they're still in general the same transforms you know from before, so the existing transform template topics and example topics (all of which are being re-written to show the new UI) are still very helpful.

Forest
625 post(s)
#21-Aug-20 01:08

I am appreciating the new version. I only use Manifold sometimes and struggled to get everything working so the added behaviour is generally quite helpful.

Forest
625 post(s)
#21-Aug-20 06:37

removed - was looking for a feature that has not been done yet.

adamw


10,447 post(s)
#11-Aug-20 15:13

Select Templates

(All field types) Expression - allows specifying a boolean expression and selects all records for which the expression is TRUE.

(All field types) Null - select NULL or non-NULL values.

(Binary fields) Search - select values with the number of bytes less / less or equal / equal / not equal / greater or equal / greater / between the specified values.

(Boolean fields) Search - select values equal / not equal to the specified value.

(Date fields) Search - select values using date and time / date without time / year / year day / month / week / week day / day / hour / minute / second / millisecond, comparing to the specified value or values (for between). For week numbers and week days, there is an option to select the week start.

(Geometry fields) Search - select values using type / number of branches / number of coordinates / number of curves / area / bearing / length / minimum or maximum x or y or z, comparing to the specified value or values. Area / bearing / length for lat/lon data are computed using geodetic formulae. For metric data, there is an option to compute in projected coordinates (Measure = auto) or transform to lat/lon and compute using geodetic formulae (Measure = geodetic). Area / length allow specifying a metric unit, bearing allows specifying an angular unit. (Example search: select all areas between 10 and 20 square miles. Or: select all objects with curvilinear segments.)

(Numeric fields) Search - select values comparing to the specified value or values. There is an option to round the field value before comparisons, useful for selecting all values nearly equal to the specified.

(Numeric vector fields) Search - select values using vector value / value N (0 to 3 depending on the type), comparing to the specified value or values. For individual values, there is an option to round the value before comparisons.

(Text fields) Search - select values using text / number of characters, comparing to the specified value or values. Text searches also support conditions for: contains / starts with / ends with / contains regular expression / matches regular expression / matches pattern (like) / sounds like (soundex). Text searches allow specifying the collation, the default is 'neutral, nocase'. Text searches with regular expression allow ignoring case. All text searches support an option to trim the field value before the search.

(Tile fields) Search - select values using number of pixels / number of missing pixels / number of channels / x or y size / channel statistic (average, minimum, maximum, etc), comparing to the specified value or values. For channel statistics, specify the channel number. (Example search: select all tiles with pixels in channel 0 greater than 1000.)

(Uuid fields) Search - select values equal / not equal to the specified value. The search value can be copied and pasted from a table cell.

adamw


10,447 post(s)
#11-Aug-20 15:15

Transform Templates

(All field types) Expression - allows specifying an expression of the type compatible with the field and filling the field with the result.

(Binary fields) Copy - copies binary data / number of bytes.

(Boolean fields) Copy - copies boolean value.

(Boolean fields) Logic - performs a logical operation: not / and / or / xor.

(Date fields) Copy - copies date and time / date without time / year / year day / month / week / week day / day / hour / minute / second / millisecond. For week numbers and week days, there is an option to select the week start.

(Geometry fields) Buffer - computes a buffer with the specified distance (positive = outer buffer, negative = inner buffer). The distance is specified in units compatible with the coordinate system of the field. (If the field has no coordinate system attached, the default pseudo-Mercator coordinate system is assumed, as always.)

(Geometry fields) Center - computes a center: circle / inner (area) / weight (area).

(Geometry fields) Clean - performs maintenance operations on the metric: normalize metric / remove curves / convert curves to lines / remove z / fill z / fill missing z. Fill missing Z has no effect on geometry values which already have Z. Convert curves to lines allows specifying the maximum number of coordinates generated per curve (curve limit, the default is 50) and the tolerance value (the default is 0 = auto). Normalize metric allows specifying the tolerance value. The tolerance value is specified in units compatible with the coordinate system of the field.

(Geometry fields) Clip - clips geometry with areas from a secondary drawing. If there is no secondary drawing available (eg, we are working from a table window or from a map that contains only one drawing layer), the clip with parameter is empty and the transform will fail. There is an option to limit the clip drawing to only the selected records. There is an option to keep either the inner (checked, the default) or the outer part of the clip.

(Geometry fields) Convert - converts geometry to a different type: area / line / point.

(Geometry fields) Copy - copies geometry / branch / coordinate x or y or z or xy or xyz / type / number of branches / number of coordinates / number of curves / area / bearing / length / minimum or maximum x or y or z. There is an option to control computations of area / bearing / length (Measure, see the description of the Search select template for geometry values). Area / length allow specifying a metric unit, bearing allows specifying an angular unit. Branch allows specifying the branch number, zero-based. Coordinate allows specifying the coordinate number, zero-based, and the counting direction (count from start or end).

(Geometry fields) Enclose - computes enclosing geometry: rectangle / rectangle with rotation / circle / convex hull.

(Geometry fields) Interpolate - interpolates geometry values into an image. Available interpolation options: triangulation / triangulation with segments / gravity / Kriging / Kriging with medial polish / Kriging with regression. Z values can be taken from a numeric field or directly from geometry values (the default). There is an option to specify resolution of the produced image, in units compatible with the coordinate system of the geometry field. For triangulation with segments, there is an option to remove flat areas using DEST (on by default). For gravity and Kriging, there are options to specify margins (the default is 0), radius (the default is 0 = auto), number of neighbors (the default is 0 = use Voronoi neighbors), step (the default is 0 = auto), model (the default is auto) and regression (the default is auto). The result is always a new table and a new image.

(Geometry fields) Merge - merges geometry values together into: area (dissolve) / center / center inner (area) / center weight (area) / circle / convex hull / line / points / rectangle / rectangle with rotation. There is an option to specify a group field (none by default = group all geometry into a single value). The result is always a new table and a new drawing. Values in numeric fields are aggregated using Sum, values in non-numeric fields are aggregated using FirstNonNull (see the descriptions of added query functions). It is always possible to use different or more aggregates by using a Join dialog after the transform.

(Geometry fields) Rehsape - applies a simple transform to geometry values: shift / scale / unscale / rotate / reverse / flip horizontal / flip vertical / smooth / snap to grid. Rotation angle is specified in angular units. Shift amounts and grid steps are specified in units compatible with the coordinate system of the field. Smooth uses a tolerance value specified in units compatible with the coordinate system of the field.

(Geometry fields) Split - splits geometry values into: shapes / shapes (ESRI non-OGC) / branches / convex parts / coordinates / coordinates with step / segments / triangles. The two shapes options differ in normalization rules for areas: the 'shapes' option uses OGC rules and the 'shapes (ESRI non-OGC)' option uses rules required by some ESRI products that are different from those mandated by OGC. The coordinates with step option is used to put equally spaced coordinates onto lines, and allows specifying the starting, ending and step distances in units compatible with the coordinate system of the drawing. To cover whole lines, set the end distance to an unreachable value, eg, 1000000 meters, computations handle this efficiently. The result is always a new table and a new drawing.

(Geometry fields) Topology - perform maintenance operations on topology. The only available option is currently: clean (generalize). We will likely add more options in the future for correcting specific types of issues. The result is always a new table and a new drawing. (The number of returned records currently coincides with the number of original records so the current transform could technically be made to update an existing table. We don't offer that option because we will likely adjust the transform to split records as necessary which would make putting the result into the new table mandatory.)

(Geometry fields) Trace - traces bounded areas from existing geometry values. (This was previously called Bounded Areas. We changed the name because we will likely add other options in the future.) The result is always a new table and a new drawing.

(Geometry fields) Triangulate - triangulates points and produces: areas / lines. The result is always a new table and a new drawing.

(Geometry fields) Voronoi - computes Voronoi diagram and produces: areas / lines / points. There is an option to specify margin in units compatible with the coordinate system of the drawing.

(Numeric fields) Arithmetic - performs an arithmetic operation: add / subtract / subtract from / multiply / divide / divide integral / divide remainder / square / square root / cube / cube root / power / reciprocal / absolute value / sign / sign invert / exponent / exponent binary / exponent decimal / logarithm / logarithm binary / logarithm decimal / error / error complement / gamma / gamma logarithm.

(Numeric fields) Bit Logic - performs a bit logic operation: not / and / or / xor. Bit logic operations are 32-bit, all bits higher than that are lost.

(Numeric fields) Copy - copies numeric values.

(Numeric fields) Hyperbolic - computes a hyperbolic function: hyperbolic sine / cosine / tangent / arc sine / arc cosine / arc tangent. There is an option to specify an angular unit.

(Numeric fields) Round - rounds numeric values: to nearest / to zero / down / up. There is an option to round to values other than 1: 10 / 100 / etc, or 0.1 / 0.01 / etc.

(Numeric fields) Special - computes a special function: Bessel, 1st or 2nd kind, with the specified order.

(Numeric fields) Trigonometric - computes a trigonometric function: sine / cosine / tangent / arc sine / arc cosine / arc tangent. There is an option to specify an angular unit.

(Numeric vector fields) Copy - copies vector value / value N (0 to 3 depending on the type).

(Text fields) Case - converts text values to: lower / title / upper case. There is an option to specify the collation, the default is 'neutral, nocase'.

(Text fields) Concatenate - concatenates text values adding a new value either at end or start. There is an option to specify the separator (an empty string by default), which is used if both concatenated values are not empty. (Example: add a value to a list of values, adding a ',' if the list is not empty.)

(Text fields) Copy - copies text / number of characters.

(Text fields) Encrypt - encrypts or decrypts text values using specified password.

(Text fields) Pad - pads text values adding characters at end or start until the text value has the specified length. There is an option to specify the repeat pattern (eg, '.' or '- ').

(Text fields) Reduce - reduces text values to: start / end / cut / cut to end / regular expression occurrence. For start / end, specify the number of characters to keep. For cut to end, specify the starting character. For cut, specify the starting character as well as the number of characters to keep. For regular expression occurrence, specify the regular expression and the occurrence to take, zero-based. For regular expression there is also an option to ignore case.

(Text fields) Replace - replaces: text / text occurrence / regular expression / regular expression occurrence. For text replaces, there is an option to specify the collation, the default is 'neutral, nocase'. For regular expression replaces, there is an option to ignore case.

(Text fields) Reverse - reverses text values.

(Text fields) Trim - trims text values at: start and end / start / end. The trimmed characters can be: whitespace / whitespace and custom / custom, with custom characters to trim specified in the transform.

(Tile fields) Arithmetic - performs an arithmetic operation on the specified channel: add / subtract / subtract from / multiply / divide / divide integral / divide remainder / square / square root / cube / cube root / power / reciprocal / absolute value / sign / sign invert / exponent / exponent binary / exponent decimal / logarithm / logarithm binary / logarithm decimal / error / error complement / gamma / gamma logarithm. (If the tile field contains multiple channels, the result has to be written into a different field.)

(Tile fields) Contour - computes contours on the specified channel and outputs: area / line. The Full Range button sets the start and end values for the height range to the approximate range computed for the image. There is an option to round start and end values to the step (on by default). For start = 112, end = 390, step = 100, without rounding the contours will be produced for 112, 212, 312. With rounding, the contours will be produced for 200 and 300. There is an option to split produced geometry into shapes (on by default). The result is always a new table and a new drawing.

(Tile fields) Copy - copies pixels / channel / number of pixels / number of missing pixels / number of channels / x or y size / channel statistic (average, minimum, maximum, etc). When copying a single channel, the transform can put the result into a specific channel of any tile field of the same size.

(Tile fields) Filter - filters the specified channel using: average / blur / blur direction / blur gaussian / count / diversity / diversity index / edges / edges direction / major / maximum / median / minimum / sharpen / std deviation / std deviation pop / sum / variance / variance pop. All filters allow specifying the radius (the default is 1). Many filters also allow specifying shape: square / circle / diamond / cross. Some filters also allow specifying the center value, direction filters allow specifying the direction angle. (The result of the transform does not have to be put into a new table or even into a new field. If the tile field contains a single channel, the result can be put back into it. This applies to several other transforms which previously could only put the result into new components.)

(Tile fields) Hyperbolic - computes a hyperbolic function on the specified channel: hyperbolic sine / cosine / tangent / arc sine / arc cosine / arc tangent. There is an option to specify an angular unit.

(Tile fields) Path - computes paths on the specified channel from or to specified points using: Euclidean / cost / cost for slope. Euclidean paths allow using barriers (off by default). Cost paths allow forcing each pixel to have the same cost (otherwise the cost is taken from the channel value). Cost for slope paths compute costs based on slopes: the only allowed slopes are those between low slope and high slope, there are costs specified for the low slope / flat slope / high slope, costs for intermediate slopes are interpolated linearly. Since costs for slopes depend on which direction the path is traversed, there is an option to reverse path = trace paths not from the specified points, but rather towards them. The points to compute paths from are specified as a drawing, there is an option to use only the selected points. The output of the transform can be: value / direction / distance. For value, the pixels in the produced image contain a numeric identifier of the closest point. This identifier can be taken from a field or be assigned sequentially (the default).

(Tile fields) Round - rounds values in the specified channel: to nearest / to zero / down / up. There is an option to round to values other than 1: 10 / 100 / etc, or 0.1 / 0.01 / etc.

(Tile fields) Slope - computes a slope or a similar function on the specified channel: slope / aspect / gaussian curvature / mean curvature / plan curvature / profile curvature. All functions allow specifying the radius (the default is 1). Slope and aspect allow specifying the unit: angular for aspect, angular + grade for slope.

(Tile fields) Special - computes a special function on the specified channel: Bessel, 1st or 2nd kind, with the specified order.

(Tile fields) Trace - traces areas on all channels of the tile field. There is an option for the similarity level (the default is 10, the value heavily depends on the data used). There is an option to split the output into shapes (on by default). The result is always a new table and a new drawing.

(Tile fields) Trigonometric - computes a trigonometric function on the specified channel: sine / cosine / tangent / arc sine / arc cosine / arc tangent. There is an option to specify an angular unit.

(Tile fields) Watershed - computes watersheds on the specified channel: lines / downstream lines / upstream lines / areas / areas for sinks / upstream areas. There is an option to specify the channel with the precipitation amounts (the default is none = all pixels are assumed to have equal precipitation). Downstream and upstream objects require specifying starting points as a drawing. There is an option to use only the selected points. There is an option to keep overlaps for downstream and upstream objects. Some options allow specifying the value for the minimum flow to limit the number of created objects (the default is 100). The result is always a new table and a new drawing.

(Tile fields) Watershed Prepare - prepares the specified channel for computing watersheds by filling sinks. There are options to specify fill height (the default is 0 = no limit) and fill flow (the default is 0 = no limit).

(Uuid fields) Copy - copies uuid values.

dyalsjas
157 post(s)
#18-Aug-20 02:22

The update of templates to allow a user to select units of measure is a very welcome change. The new output options of degrees, and grade when creating a slope image from an elevation are very welcome. This scratches a long time workflow "itch" for me. Thanks very much!!

adamw


10,447 post(s)
#11-Aug-20 15:15

(Technical details, can be skipped on first read.)

Components in Queries

Previously, we sometimes relied on drawings and images to describe the structure of spatial data in tables. Starting with this build, we instead rely on indexes and treat spatial data in tables as self-describing, with drawings and images used only for visual purposes.

Operations on geometry use a table with a geometry field. The geometry field does not have to be part of a spatial index, although having such an index will frequently help performance. A query function that operates on geometry can accept either a physical drawing component stored in a database, or a virtual drawing created on a geometry field of a table or query component (using the ComponentFieldDrawing function). A virtual drawing supports all functions available for regular components, eg, it can report its coordinate system or the name of the underlying component. Both physical and virtual drawings can be limited to only the selected records with the result being accepted as a drawing by the query functions.

Operations on tiles use a table with a tile field, a pair of X and Y fields, and a spatial index connecting these fields together. Images on some data sources, eg, web images, use a slightly different structure with a non-spatial index on X and Y fields, sometimes also including a level field, instead of the spatial index. Such images have to be converted to images with a spatial index prior to the operations. A query function that operates on tiles can accept either a physical image component stored in a database, or a virtual image created on a tile field of a table or query component (using the ComponentFieldImage function). A virtual image supports all functions available for regular components. Both physical and virtual images can be limited to only the selected records but the result will NOT be accepted as an image by the query functions and will only work as a plain table with no spatial context. We are planning to extend selections in images to be per-pixel and after we do this, the result of limiting an image to only the selected pixels will work as an image.

The rectangle stored in an image component is used solely for display purposes and is ignored for both selects and transforms. Previously, some selects and transforms ignored the rectangle and others used it. Starting with this build, we treat the rectangle as a strictly visual thing that limits what part of the data is shown on the screen, similar to how a table window can limit which records are shown in the list by filtering field values, and ignore it for selects and transforms. Operations that do not depend on tile placement and treat pixels independently from each other may operate either on all or selected records. Operations that depend on tile placement always operate on all records. After we allow per-pixel selections, all operations on images will be able to operate on only the selected pixels.

(There is a potential issue with ignoring image rectangles for data analysis. Since some of the transforms previously used the rectangles to limit queried data, it might have been possible for an image to contain garbage outside of the rectangle without this being noticed. It is also possible that some of our older dataports created such images during the import. Our preliminary checks on test files that we have found no such images existing de-facto, but it is possible that they exist. If you have such an image, running a transform on it will read data outside of the image rectangle which it might not have been reading in prior builds, and the results of the transform will differ from the results you were getting before, likely to the worse. To understand whether an image contains data in pixels outside of the image rect, you can force the image rectangle to the extent of visible pixels. This currently requires using a query, see the changes to the ComponentBounds function. We will allow changing the image rectangle to cover all visible pixels via the UI in the future. We will also likely allow cropping the image to the current rectangle, throwing away all data outside of it. If you find that a particular dataport or a UI tool may create an image with data in pixels outside of the image rectangle, let us know and we will fix this, this is a bug.)

adamw


10,447 post(s)
#11-Aug-20 15:17

Storage

Spatial indexes for tiles have been extended to support a number of operations which help the UI: computing bounds limited to visible pixels, computing approximate statistics, etc. Spatial indexes for tiles are supported in MAP files and in file dataports that store data in MAPCACHE files.

Queries

BETWEEN has been changed to return FALSE if the lower bound is bigger than the higher bound, similarly to other databases.

(Fix) Tuple comparisons no longer sometimes incorrectly handle non-constant NULL values. (Example 1: (1/0, 1) BETWEEN (1, 1) AND (2, 2) is now NULL, was FALSE. Example 2: (1/0, 1) < (1, 1) is now NULL, was FALSE.)

(Fix) Tuple equal and not equal comparisons no longer sometimes incorrectly stop on constant NULL value. (Example 1: (NULL, 2) = (2, 1) is now FALSE, was NULL. Example 2: (NULL, 2) <> (2, 1) is now TRUE, was NULL.)

New function: Bound - bounds a numeric value to the specified range. If the strict parameter is TRUE, the minimum value has to be lower than or equal to the maximum value, otherwise the function returns NULL. If the strict parameter is FALSE, the minimum value can be greater than the maximum value.

New function: CoordUnitScale - takes a definition of the coordinate system unit, parses it and returns unit scale as a number.

New function: CoordSystemScaleXYZ - composes XYZ scales for a coordinate system. If the coordinate system is metric, the values are in meters. If the coordinate system is lat/lon and the forceMeters parameter is FALSE, the XY values are in degrees and the Z value is in meters. If the coordinate system is lat/lon and the forceMeters parameter is TRUE, the XY values are converted to meters based on the scale at the equator.

New function: CoordSystemScaleXYZBounds - composes XYZ scales for a coordinate system and the specified bounds. The function behaves similarly to CoordSystemScaleXYZ except the lat/lon scale is converted to meters based on the scale at the center of the bounds.

ComponentCoordSystem always forces the coordinate system for an image to XY axes.

ComponentCoordSystemScaleXY is removed, use ComponentCoordSystem and then CoordSystemScaleXY,

ComponentCoordSystemScaleXYZ is removed, use ComponentCoordSystem and then CoordSystemScaleXYZ / Bounds.

ComponentCoordSystemAutoXY is removed, use ComponentCoordSystem which now does what ComponentCoordSystemAutoXY was doing before.

New function: ComponentFieldDrawing - creates a virtual drawing for a geometry field in the specified component.

New function: ComponentFieldImage - creates a virtual image for a tile field in the specified component. The tile field has to be part of a spatial index.

ComponentBounds is extended to compute bounds for virtual components and to limit bounds for an image to visible pixels. If the forceCompute parameter is FALSE, the function returns bounds stored in a spatial index or the rectangle stored in the metadata for a physical image (all select and transform operations use rectangles from virtual images that include data from all records, but retrieving the rectangle stored in the metadata is still useful, eg, for creating a copy of an image). If the forceCompute parameter is TRUE, the function computes bounds from the field values, and for a tile field it also restricts bounds to the visible pixels. If the forceCompute parameter is TRUE and the tile field belongs to a web image known to be very big, the function ignores the parameter (this protects from trying to download all tiles from Bing or Google).

New function: ComponentFieldBounds - computes bounds of a geometry or tile field in the specified component.

New function: ComponentFieldCoordSystem - returns the coordinate system for a geometry or tile field in the specified component.

Selection / SelectionWindow ignore attempts to create a selection on a selection (were prevously stacking).

New function: GeocodeSearchMatches - searches for geocoding matches for an address with a filter (eg, 'hotel') near a lat/lon location with or without radius in meters.

New function: GeocodeSearchRectMatches - searches for geocoding matches for an address with a filter in a lat/lon rectangle.

New function: GeocodeSearchSupported - checks whether a geocoding data source supports GeocodeSearchXxx functions above.

New functions: GeomBoundsX / Y / Z - compute minuimum and maximum X / Y / Z for a geometry value as an x2 value.

New function: GeomCurveCount - returns the number of curve segments in a geometry value. Takes a curve type: negative value or 0 = count any curvilinear segment, positive value = count curvilinear segments of a specific type.

GeomSnapToGrid collapses equal coordinates together.

TileContourXxx functions take an additional parameter for the channel to use.

TileInterpolateXxx functions take an additional parameter for the Z field to use. If the name of the Z field is a blank string, the functions use Z data stored in the geometry values.

TileInterpolateXxx functions take an additional parameter for the XY scales to use for the produced pixels, expressed in degrees or meters depending on the coordinate system of the image. The XY shifts are assumed to be 0.

TileDistanceXxxMake functions take an additional parameter for the channel to use.

TileDistanceTilesClosestXxx functions take an additional parameter for the source field to use. If the name of the source field is a blank string, the functions use sequentiel numbers, starting from zero.

TileDistanceTilesXxx functions return an indexed table to allow joins.

TileFillSinksXxx functions take an additional parameter for the channel to use.

TileFillSinksXxx functions return an indexed table to allow joins.

TileWatershedXxxMake functions take additional parameters for the height and water channels to use. If the water channel is a negative value, all pixels are assumed to have equal precipitation.

New function: TileFilterDefDiamond - creates a diamond filter shape (mid-way between cross and circle).

New function: TileSize - returns the dimensions of a tile as an x2 value.

New function: TileValueCountMissing - returns the number of missing pixels in a tile.

New function: TileUpdateFieldPyramids - updates pyramids for a tile field in the specified component. If the tile field is not part of a spatial index, the call does nothing.

New function: TileUpdatePyramidsAll - updates pyramids for all tile fields in the specified component.

StringPadEnd and StringPadStart treat negative length as no padding.

StringConcat takes a separator parameter and concatenates two strings through the separator as long as both are not empty.

StringSubstring and StringSubstringLen treat negative start as start of the string and negative length as full length of the string.

New function: VectorCross - computes a cross product of x3 values.

New function: VectorDot - computes a dot product of x2 / x3 / x4 values.

New aggregate function: FirstNonNull - returns the first non-NULL value. (First may return a NULL even if the field contains non-NULL values.)

New aggregate function: LastNonNull - returns the last non-NULL value. (Last may return a NULL even if the field contains non-NULL values.)

adamw


10,447 post(s)
#11-Aug-20 15:17

Other

The MySQL dataport wraps batch operations into transactions for performance and safe rollback in case an operation fails mid-way.

The Join dialog hides the 'merge areas' transfer to avoid the confusing choice between 'merge areas' and 'union areas'.

The Join dialog uses the FirstNonNull aggregate function for 'copy' / 'convert' transfers, to return a non-NULL value whenever possible.

Computing Kriging parameters can be canceled. (Previously, attempting to cancel had to wait until the parameters are computed in full.)

Export to GDB no longer splits geoms with and without Z and instead adds zero Z to geom that do not have it.

The GDB dataport better handles renames and deletes of components and changes to table schemas. (Various operations could sometimes fail.)

The GDB dataport supports refreshing data from the Project pane. This is useful for picking up changes to the geodatabase made outside of Manifold.

The GDB dataport supports autogenerated fields (OBJECTID).

The GDB dataport supports non-NULL constraints.

The GDB dataport translates systems with WKID values matching the correspondent EPSG code to EPSG:xxx to let Manifold enable coordinate transforms. Small differences between the definition of the coordinate system used by the database and the definition of the EPSG code used by Manifold are put into the coordinate system overrides.

End. of. list.

oeaulong

521 post(s)
#11-Aug-20 16:04

Amazing list of changes.

Applause to the dev team.

Thank you.

tjhb
10,094 post(s)
#11-Aug-20 17:05

This is great reading, everything very well explained.

Now everything really really is a table. This is very welcome, a big simplification both conceptually and in practice (when writing code). (First thought.)

Lots of thoughtful minor enhancements amongst the big stuff.

I am looking forward to a big celebratory breakfast. (The dogs are too.)

Thanks!

adamw


10,447 post(s)
#12-Aug-20 12:01

Thanks. :-)

Regarding code, it absolutely becomes simpler and not just with being able to work solely with tables. One more thing we did is changed the way we generate code for the templates, making the process smarter. Many templates now significantly alter their code depending on the parameter values. For example, text searches use plain StringXxx calls or plain comparison operators instead of StringXxxCollate calls if the collation parameter is set to just 'neutral'. There were also multiple simplifications due to extensions to query functions. For example, interpolations no longer have to create a temporary database with temporary components to attach Z data to the geometry - they just pass the name of the Z field to use to the relevant function.

Mike Pelletier

2,122 post(s)
#11-Aug-20 20:13

Great to have the new build and appreciate all the effort and new agility of the UI. Couple of thoughts that might be useful.

The Search option in the Select pane is a tad confusing in that you first pick a field, then choose Search, then you can choose the field again but it is limited to same field types of your first field choice. Unwary users, like me, going back to pane after doing something else see the limited list of fields and wonder why.

Seems like it would be best when revisiting an automatically saved select/transform, that the user typed in parameters were also saved. Maybe allow user to name the saved select/transform.

Dimitri


7,413 post(s)
#12-Aug-20 06:22

see the limited list of fields and wonder why.

Only until you get used to it. I've been using the new UI for over a month and all that clicks into place very quickly. So why is it that way?

When you use the Select and Transform panes, they often are used in workflow consisting of many steps that iteratively do similar or related things. The UI is designed to maximize what can be re-cycled in repetitive / iterative tasks, with a minimum of clicks and other moves.

The operations that work on different data types often are profoundly different. For example, selecting in text has very different commands than selecting in numbers. If you're going to switch between the two you'll have to make a new choice in any event. But it's often the case doing Select / Transform workflow that you're repetitively working with the same types of fields, for example, extracting text from JSON fields, or re-working text scraped from the web into usable fields, where there's a select on a text field, a transform, another select on a text field, and so on. In such cases, it's a really big win to get the benefit of persistence from working with the same data type.

Like I say, if you switch data types you'll switch between different menus of possible operations in any case. However you design it there will be an extra click or two. It's the same with this new UI, except that you also pick up the big benefit of much better persistence with iterative operations using the same data types.

On top of that, there are plenty of opportunities to tune the new UI. For example, we could have a shortcut for "back" and "forward" between contexts, so if you're alternating between doing something to numbers and doing something to text you could rapidly jump back and forth between the two. In tests during the 15 internal builds (!) that helped refine various details of the 172.4 UI there was no obvious way of doing that which didn't loose more smoothness than it gained, but you never know... that could get worked out after we all get more experience with the new UI to help cook up refinements.

adamw


10,447 post(s)
#12-Aug-20 11:40

Seems like it would be best when revisiting an automatically saved select/transform, that the user typed in parameters were also saved. Maybe allow user to name the saved select/transform.

We do save parameters, just not all of them. Specifically, we do not save parameters that can be set to fields. We did it this way because a saved template can be used with any component and if you originally used a field named Population, that field might not exist in a component you are trying the saved template on, so restoring the parameter set to Population would fail and you would have to check which parameters were restored vs which were not prior to repeating the operation. But after some experience with the workflow we see that having to check which parameters are set to what values prior to repeating the operation of a saved template is a good idea anyway, and that this is not a terribly big burden, so we are leaning towards restoring the values of all parameters, yes. We'll see.

We'll perhaps allow renaming saved templates, yes, that's a good idea.

tgazzard
146 post(s)
#15-Aug-20 10:28

Just watching the select and transform part 2 video. This gives a good insight into how things have changed/improved.

It struck me that it would be good if your are entering the drawing name that the table name could be automatically created based on the name you type for the drawing. For example if you name your drawing "Test" then the transform could/should automatically create the table name as "Test Table". Or the other way around. If you name your table then the drawing could be automatically named.

Also, I wondered if you could have a tick box that lets you delete the existing drawing and table in the transform (assuming they exist) and maybe another tick box to automatically refresh the map. Rather then having to go over to the project pane to delete and then hit Shift F5 to refresh. This would allow for rapidly exploring changes to your data. I guess a bit like preview does.

tg

adamw


10,447 post(s)
#18-Aug-20 10:17

On adjusting the name of the new table as you edit the name of the new drawing and vice versa - yes, we'll do that.

We'll also add keyboard navigation to the panes - it is tempting to use Tab and Shift-Tab to move between the controls but you cannot do that currently (to be clear, you couldn't do it before this build either). This will also help type the names quicker.

On deleting the existing drawing and table - do you mean deleting the components that were just added by the transform because you want to repeat the operation with different parameters? If so, let's wait for the preview, it should reduce the number of times you want to throw away previous results significantly. On deleting of transform results from the Project pane, we automatically select a newly created drawing / image, you can use Delete Related to delete it together with the table. (There are also talks of making a variant of Delete Related the default for the Delete key in the future.)

Thanks a lot for your notes!

adamw


10,447 post(s)
#11-Aug-20 15:18

What's Next

First, we have a couple of missing things: previews / saved selections / some transforms (most notably, topology overlays and viewsheds, plus several smaller transforms). We do not have these things in the build because they are being reworked. We should have most of them in the next build - which we would like to release a week or so from now. For localizations, it might make sense to wait for this next build because some of the existing strings will no longer be used.

Then we are going to plug in the registration.

After that we are going to issue a public build.

Please tell us what you think about the changes!

Also, we already have a raw version of Help describing the new UI. It is going to be extended massively plus we are going to have new videos.

RAR111 post(s)
#11-Aug-20 18:38

Very clean and great changes. Thank you!!!

I noticed that the former build 9.0.172.3 can read correctly the projection from a gdb file (GCS_North_American_1983) but the new one 9.0.172.4 is not (NAD83 (EPSG:4269)YX).

Thank you again.

adamw


10,447 post(s)
#12-Aug-20 11:44

Thanks!

This is automatic detection of GDB systems that are really EPSG. Is the issue limited to the axes being YX instead of XY? Can you copy and paste the same GDB drawing into a MAP file using 172.3 and 172.4 and post it here or send it to tech support, so we could take a look?

RAR111 post(s)
#12-Aug-20 16:48

Adam,

I think the issue is the axes being YX instead of XY.

How can I change that to be the default please?

Thank you,

adamw


10,447 post(s)
#13-Aug-20 13:19

You should not need to change anything. We'll fix our code to force the axes to XY for GDB.

But until we do this, you should be able to change the axes from the Info pane: click the button next to the coordinate system, invoke Repair Initial Coordinate System - Force XY Axes.

Dimitri


7,413 post(s)
#13-Aug-20 14:18

I think the issue is the axes being YX instead of XY.

That can always be a possible problem given that there is so much data out there which says it is in YX ordering but really is in XY ordering, and sometimes you encounter data which says it is in YX ordering that really is in YX ordering. There's no sure-fire solution to second-guessing data that says it uses axis ordering that is different from what it actually uses. See the That XY Thing essay.

Informed guessing about what's really intended can help, of course, as Manifold already does in many situations. But such guesses can be complicated by ambiguous systems like GDB. GDB basically reports the projection in use using PRJ style citation of projections (heck of a way to design a so-called "database"... recycle ambiguous shapefile hacks...) but in addition cites a number for those projections. Those numbers usually correlate to EPSG codes. PRJ style citation of projections is silent about axis ordering, so whether the data is YX or XY can't be determined from just that, but the logic of using the numeric code as a tip would help choose YX or XY ordering.

If the GDB uses a numeric code that says "YX" ordering in the EPSG system, that's what 172.4 uses. What adamw is talking about is nudging guessing back to be more similar to 172.3, and to force using XY ordering for GDB.

I think that's a good approach because genuinely YX axis ordering in GDB appears to be extremely rare, so guessing on the side of XY will more likely be correct than guessing the other way around.

RAR111 post(s)
#13-Aug-20 17:16

Forcing XY Axes fixed the problem.

Thank you Adam

RAR111 post(s)
#13-Aug-20 17:17

Good remarks; I will keep that in mind.

Thank you Dimitri,

RAR111 post(s)
#11-Aug-20 19:13

I was looking for any of the Compose Template (Compose Point, Circle, Rectangle,...) but could not find them. Are they available please?

Thank you,

Dimitri


7,413 post(s)
#12-Aug-20 07:39

No, those are some of the smaller transforms still in process. You can check the complete list of transforms for geometry in the Transform - Geometry topic. The preliminary intro to the new Select and Transform panes is in the Select Reference and Transform Reference topics.

RAR111 post(s)
#12-Aug-20 12:58

I will check these.

I love the new UI.

Thank you Dimitri!

adamw


10,447 post(s)
#12-Aug-20 11:41

Not yet, but we'll re-instate them. You can use Expression in the meantime.

RAR111 post(s)
#12-Aug-20 13:36

Great.

Thank you Adam!

Mike Pelletier

2,122 post(s)
#13-Aug-20 23:39

With the new Transform pane, how do we fill Lat and Long fields with values from a point geom?

oeaulong

521 post(s)
#14-Aug-20 00:52

[didn't read well enough] deleted.

tjhb
10,094 post(s)
#14-Aug-20 04:17

I wanted to say: use Expression, targeting a New Field, and Edit the expression to something like

GeomCoordXY(Geom, 0)

for the X coordinate (assuming the point is already unprojected, the simplest case).

But the dialog does not allow the target field to have a type other than Geom, giving error

float64x2: Expression type mismatch.

I imagine you see exactly the same Mike.

So it seems for now we must use SQL for this task.

Dimitri


7,413 post(s)
#14-Aug-20 08:11

Start by choosing a field of the type you want for the result. If you want to fill a float64 lat or lon field, choose one of them and then write the expression as you like.

tjhb
10,094 post(s)
#14-Aug-20 08:41

Thanks!

So in such cases, the new New Field option does not apply.

It is only available to produce a new field of the same type as the chosen context field.

I was holding it upside-down.

Dimitri


7,413 post(s)
#14-Aug-20 14:31

Even better... (sorry for missing this) there's now no need to use an expression as you can use the Copy template, with a Use choice of coordinate x. That pulls the x coordinate out of the point and puts it directly into a float64 field.

pslinder1
228 post(s)
#14-Aug-20 16:57

A suggestion: Create a template solely and specifically to extract data from a Geom.

"Copy" is very powerful but as a name it is not intuitively obvious that this is where you go to extract coordinates or other information contained in the geom.

Dimitri


7,413 post(s)
#14-Aug-20 18:25

What would you call that template? If you do that for a geom, why not for a tile?

It's basically the same function for different data types: it takes data from the source field, data which could be expressed in different forms, and puts it into some other field.

Having one name for the same thing is convenient and reduces clutter. I'd be curious to hear other names for that thing. Ultimately though, no single word is intuitively obvious. It's just too much to ask of one word, or even a few words.

I think what happens is that at first whatever word is picked feels artificial in some situation or another. And then you get used to it, it becomes familiar, and then it seems totally correct. But even so, if anybody can come up with better suggestions for terminology, now is the time! :-)

LeRepère
153 post(s)
#14-Aug-20 20:16

What do you think of: "Extracting" or "Extract from"

Dimitri


7,413 post(s)
#16-Aug-20 14:56

I like Extract or similar, but then is computing bearing, length, or area in square units, or reporting the type an "extract?" I suppose one could say those are no more or less "extract" than they are "copy," but if you did an "extract" you'd also need a separate Copy command. I guess for a general purpose command I prefer a more general name.

tjhb
10,094 post(s)
#16-Aug-20 15:53

I think Copy is OK, just something to get used to. (For Mike's question above, I did look under Copy, but entirely missed the relevant choice, coordinate x, which you helpfully pointed out Dimitri.)

My gut feeling though is that Copy can be improved upon, mainly since, after all, it is already freighted with so many other uses, not directly relevant for this template. Also because these template functions are not simply copying; they are first making a simple manipulation of data, a simple computation or an enquiry.

I wonder if you have instinctively hit on a better choice just now Dimitri: Report.

Semantically and functionally, Report is closer to ideal, I think. Does it miss anything, now or perhaps in future?

pslinder1
228 post(s)
#15-Aug-20 15:53

What would you call that template? If you do that for a geom, why not for a tile?

Why not have one for the geoms and one for tiles?

Having one name for the same thing is convenient and reduces clutter.

Not really because as soon as you pick the a field the irrelevant template(s) disappear.

I would call them "Extract from Tile" and "Extract from Geom". I would also make them different from the standard "Copy" template by allowing them to extract multiple fields in one template. In other words pull out coordinates and area and bounding area at once for example instead of running the template three separate times.

Dimitri


7,413 post(s)
#16-Aug-20 15:02

Why not have one for the geoms and one for tiles?

Because then you need another one for datetime, another one for text and so on. It adds up.

Not really because as soon as you pick the a field the irrelevant template(s) disappear.

I meant at the top level. Keep in mind that even bundling 20 or so similar/related capabilities into the same template, the list of templates is going to grow much larger over time.

In other words pull out coordinates and area and bounding area at once for example

OK. But then what sort of controls would you have to specify where to put all those? Copy three characteristics to somewhere else and you need to specify three different Result destinations. That could get very busy, especially considering that those multiple different results might be different types, for example, numbers or geometry, with different numbers of Result parameters appearing for each. It could get very busy very fast.

pslinder1
228 post(s)
#16-Aug-20 18:28

Because then you need another one for datetime, another one for text and so on. It adds up.

In essence you are already doing this. Once you have chosen the "Copy" template you can only change the field if it is the same type of field. You have to go back up a level to pick a new field type in the same component. So the Copy template is functionally a different template for each field type.

Taxinomically (probably not a real word) I think that Tiles and Geoms are unique from the other field types in that you are extracting information from within them that is not viewable normally in their tables.

I meant at the top level.

There is effectively no top level that is unfiltered. When no component is selected the Transform pane seems to be completely blank. As soon as you choose a component a field is automatically selected and the Templates are filtered by that field. So I don't think this is much a problem. At most you would add one template to the view for any field type.

OK. But then what sort of controls would you have to specify where to put all those? Copy three characteristics to somewhere else and you need to specify three different Result destinations. That could get very busy, especially considering that those multiple different results might be different types, for example, numbers or geometry, with different numbers of Result parameters appearing for each. It could get very busy very fast.

Not really, you could add a small grid to the pane where you specify one output per row with columns for the field, the new field name, the field type, and if needed the new drawing output. Quite simple visually I think and quite a familiar interface.

adamw


10,447 post(s)
#17-Aug-20 14:43

Regarding Copy vs Extract:

We could have both Copy and Extract for each data type, sure. We decided to have just one template doing both things because there are cases of Copy slowly morphing into Extract.

Here's an illustration:

  • If you want to copy a geom value, that's a classic case of Copy.
  • If you want to copy the number of coordinates in a geom value, that's a classic case of Extract.
  • But if you want to copy a single branch of a geom value, what to call that? We still create a geom, just from partial data. It's like copying a substring. It's even more similar for copying a single channel of a tile. It is tempting to be calling such operations Copy.

Now, if we agree to call mixed cases in the last item Copy, if we have Copy and Extract as separate templates, it would perhaps be hard for a new user to tell which operations are in Copy vs which ones are in Extract. And if we do not agree to call mixed cases in the last item Copy and decide to call them Extract, declaring that Copy has to be exact and everything that does not Copy a full value should be called Extract, well, then if we have Copy and Extract, the separation is clear, but Copy is an awfully short template which does pretty little. And if you start with Copy, then decide that you want just some part of the value, switching to copying that part requires that you go back to the list of templates and pick a different template, which is slower than it is now.

Would it help if we altered the tooltip for Copy to include the word 'extract'? That would perhaps make it clearer that extracting is done by Copy (and allow finding Copy by typing 'extract' in the filter box as well).

pslinder1
228 post(s)
#17-Aug-20 17:03

if you start with Copy, then decide that you want just some part of the value, switching to copying that part requires that you go back to the list of templates and pick a different template, which is slower than it is now.

An easy solution to this problem: just allow users to select an extract all (i.e "Copy") from the extract template. Manifold has many functional overlaps or multiple workflows for the same ends (e.g. Joins and Overlays). I do not see it as a problem if the Extract Template ends up containing the ability to do a copy.

but Copy is an awfully short template which does pretty little.

Again not sure I see that as a major issue. So it is short, but it does what the title promises it will do. Good enough in those cases where you think you need to copy one column to a new one.

BTW now that you have broken the conceptual barrier of adding a column to a table outside of the Schema Pane (which I always found annoying) why not take it a step further? Why not allow the user to select a field (column heading) directly in a table and do a copy and paste? That would be the more natural and intuitive workflow for a copy rather than using the template in my opinion.

adamw


10,447 post(s)
#17-Aug-20 17:16

Copy and paste a field using context menu commands in a table window - sure, why not. We have been planning to add a couple of similar commands as well, eg, rename or delete a field. We'll try to add it. (To be clear, copying and pasting a field would be limited to the same table. No copying a field from table A and pasting into table B - that's the job of the Join dialog.)

On Copy vs Extract though - you are proposing to basically rename the current Copy to Extract and add a new Copy which would just be Copy of a full value, correct? Not sure about that.

pslinder1
228 post(s)
#17-Aug-20 17:42

Copy and paste a field using context menu commands in a table window - sure, why not. We have been planning to add a couple of similar commands as well, eg, rename or delete a field. We'll try to add it. (To be clear, copying and pasting a field would be limited to the same table. No copying a field from table A and pasting into table B - that's the job of the Join dialog.)

Love the above!

On Copy vs Extract though - you are proposing to basically rename the current Copy to Extract and add a new Copy which would just be Copy of a full value, correct? Not sure about that.

That is basically what I am proposing. I get your hesitation in that it seems duplicative. But I think the semantics are important here. The word "Copy" just gives me no idea of the power of that template in relation to Tiles and Geoms. So a little functional duplication would be worth it for a large gain in semiotic clarity.

I would add one refinement to the Extract template. I suggest allowing the user to extract multiple fields at once instead of having to run the template multiple times on the same Geom or Tile. I see the interface using a small grid inside the template pane where each row is a separate extract field in the Geom and the columns are the new table field name, the new table field type, and optionally the name of the new drawing.

adamw


10,447 post(s)
#18-Aug-20 12:55

We'll think about it.

We thought about extracting multiple values at once during our design sessions. The idea has a lot of appeal, the reason we didn't implement it yet is that it works for some cases and doesn't work for others, and we didn't yet determine how to split these cases cleanly enough.

Here's an illustration of what I am talking about: if we take extracting a coordinate from a geometry value, it is easy to imagine offering the user controls for two result fields: X, Y. If we allow setting some of the controls to '(ignore)' or something similar to ignore the relevant result, this might even be three fields: X, Y, Z, with the user ignoring whatever he doesn't need (which would frequently be Z). This works for several other things as well. When we try to apply this to extracting something like branches, however, it breaks. Since the number of branches is not limited (or, rather, the limit is very high and rarely reached), we cannot provide a separate control for branch 0 / branch 1 / branch 2 / etc. We have to go back to extracting a single branch and ask for its number. And if we want to extract multiple branches, we would only be able to do so in multiple steps.

So for now we decided that since we are going to have scenarios where extracting multiple things would take multiple steps in any case, let's extract everything one thing at a time. We might revisit this in the future. Eg, if the number of things to extract is unlimited or the limit is very high, we can allow extracting up to X of those things citing their numbers - similarly how this forum allows attaching up to four new files to a post in one go.

Mike Pelletier

2,122 post(s)
#14-Aug-20 17:00

Thanks! It would be nice to have a lat long option. The results are in units of the projection, so I quickly changed the projection to lat long and the results are roughly lat 126, long -350 when they should be 38, -106. Any ideas why?

Dimitri


7,413 post(s)
#14-Aug-20 18:27

Not without seeing the data, no. Could you post a sample, with before and after?

PS: I just did something like that today. I was working with a map with about a dozen similarly-named layers and was wondering why the projection wasn't working. Turns out I was insistently, repeatedly, reprojecting a different layer than I had intended, so the intended layer, of course, was not changed.

Mike Pelletier

2,122 post(s)
#14-Aug-20 18:42

Here's a map with a single point that I think shows the issue.

Attachments:
T.map

Dimitri


7,413 post(s)
#15-Aug-20 06:59

Thanks! I see what's going on. If you take a look at the Colorado Central projection used for the local projection, it is a Lambert Conformal Conic projection that uses a local scale of 0.3048... for X and Y. Projecting into Lat/Lon retains that local scale, which is automatically reckoned so that everything continues to line up correctly. That also ensures images will line up, where the local scale is critical to preserve.

When you extract the x and y coordinates, the template extracts literally what those coordinate values are, which, indeed are values like -350 and 126. They have to be or otherwise the spot would be in the wrong location given the local scale that the coordinate system says must be used. Multiply them by the local scale and you get around -106 and 38, which are the lat/lon coordinates for that spot on the Gunnison-Crested Butte airport in Colorado.

There are two ways to get the coords you want:

1. Ask that the Copy template be extended with an option to generate local scale adjusted coords. Might call that "coordinate x scaled" and "coordinate y scaled". Note that this is a different thing than asking for a lat/lon version, that might be called "coordinate latitude" and "coordinate longitude" since applying local scales is something that isn't just about latitude and longitude but applies to all coordinate systems where such scaling is used.

2. Open any drawing layer and then in the Info pane click the coordinate picker and chose Favorites. Switch the setting for the Latitude / Longitude favorite from "keep metrics" to "override metrics". Click OK. Now, that favorite will appear as "Latitude / Longitude #" to indicate it overrides metrics. If you use it to reproject your local Colorado Central projectioninto that "Latitude / Longitude #" projection, it will override the metrics, setting local scales to 1. Do that and the coordinate x and coordinate y operations will report what you expected to see, around -106, 38.

To anticipate follow on discussion, it's an open question whether to set the default Latitude / Longitude projection to keep metrics or to override metrics. Whichever way you choose will cause unexpected effects in some cases. Keeping metrics has the merit of not blowing up images when they are reprojected, which is really confusing to beginners, so I suppose that is why it is the default.

To provide an easy mechanism for beginners who are using templates to do things like creating a geocoded table from a table that just has encoded geometry (whether that geometry is in geom, geomWKB, WKT, etc.), I think the best solution is to add "coordinate latitude" and "coordinate longitude" options to Copy. May as well also add the "scaled" options as well, for more advanced users.

Mike Pelletier

2,122 post(s)
#17-Aug-20 14:46

Thanks Dimitri and Adam for the explanations! Not sure what is the best method for not tripping up beginners or having to remember this nuance. Maybe the Lat long name could include something like "(save metrics)" to help remind people of the issue. However, probably best to just add the additional options to Copy.

adamw


10,447 post(s)
#17-Aug-20 14:12

I guess you reprojected original data to lat/lon by invoking Reproject Component, then clicking the picker button for the new coordinate system and picking the default Latitude / Longitude item. The default Latitude / Longitude item does NOT override coordinate system metrics, you can see that in the Favorites dialog. We allow favorite coordinate systems to either keep original coordinate system metrics (useful when you are fixing a partial coordinate system - eg, after importing a data set that had the world file but no other coordinate system data), or override them (useful for other cases). Maybe the default favorite coordinate systems should be overriding metrics. In any case, you can alter them to override metrics in the Favorites dialog right now as well and this will fix the issue.

RAR111 post(s)
#16-Aug-20 15:18

Is there a possibility to have mfd_id starting from 0 instead of 1 for Manifold please?

Thank you,

tjhb
10,094 post(s)
#16-Aug-20 15:34

Can you say why?

RAR111 post(s)
#16-Aug-20 16:15

Manifold identity mfd_id and mfd_id_x are already there for any features and instead of creating new calculated field like [mfd_id-1] and index, I would like to take advantage of exisitng ones.

Thank you,

tjhb
10,094 post(s)
#16-Aug-20 18:44

OK but why?

You haven't mentioned why you need/want a value starting at 0.

In other words, take advantage how, for what purposes?

RAR111 post(s)
#16-Aug-20 21:29

It's OK Tim.

Thank you,

tjhb
10,094 post(s)
#17-Aug-20 08:09

I don't know quite what to say to that.

You must have had a reason for suggesting the change!

I have been trying to find a conversation I know had with adamw where I, like you perhaps, asked the reason for the switch from zero-based to one-based indexes.

I haven't managed to find it today. Maybe tomorrow.

But I would still love to know why you believe it's important.

(And let's say you were right. Well, there is zero chance of Manifold investing time in making such a change unless you could suggest or show why.)

RAR111 post(s)
#17-Aug-20 12:57

Tim,

Please do not be sad. I was just wondering if there is a way in manifold configuration that we could have achieved that. I did not mean Manifold should change it to zero-based indexes while that is an alternative.

It was not that critical, that's why I did not insist giving you the reason.

Thank you for your effort to help Tim. I truly appreciate it.

adamw


10,447 post(s)
#17-Aug-20 14:27

I am a little late to the discussion, it seems. :-)

There's currently no way to make Manifold accept zeros for MFD_ID. The reasons are technical - some code deep inside the core treats zero IDs as special and unavailable to the user. We agree allowing zero IDs would be desirable - primarily because there are many cases of real-world tables which start their IDs from zero rather than from one or even higher (we saw negative IDs too, but much less often) and it would be good to allow copying such tables into MAP, putting IDs into MFD_ID without making any adjustments to the data values. We will likely allow MFD_ID to accept zeros in the future.

RAR111 post(s)
#17-Aug-20 15:07

Thank you Adam!

geozap
264 post(s)
#21-Aug-20 19:12

I can't find the Compose Point transformation, so I could create points in a Geom field from XY columns. Is it still here?

tjhb
10,094 post(s)
#21-Aug-20 20:05

See above.

Just use SQL.

adamw


10,447 post(s)
#22-Aug-20 16:18

Status update:

We are currently planning to issue the next build somewhere next week. We are re-instating transforms that didn't make it into 172.4, with improvements, and are adding several other useful features, both related and unrelated to transforms. The feedback and discussions in this and other threads have been (and continue to be) very useful!

dchall8
1,008 post(s)
#26-Aug-20 18:15

Something I noticed in the new transforms format: when you are filling a field and hit the Tab key, the focus does not move to the next field. Is that permanent or something that was set aside for expediency?

Also would it be possible to make the transform categories into drop down selection lists?

Attachments:
Dropdowns Question.jpg

tjhb
10,094 post(s)
#27-Aug-20 02:15

I would second the first request, for enabling the Tab key to switch between input fields. This is something that has been largely deferred in the devlopment of 9. There are many such instances.

I would disagree with the second request though. Drop-downs hide (until clicked and scrolled). The current interface shows. Showing what is possible is really useful. (It's a menu.)

Also, making lists/menus fully visible may help prevent their becoming overpopulated.

Dimitri


7,413 post(s)
#27-Aug-20 07:49

Regarding tabs: that's planned, but deferred for now. Tabbing between combo boxes is tied up with the Windows interface in a way that requires a lot of work to resolve within "always on" panes, that stay active when you move the focus between other windows. It will get done, but given that there are other, higher priorities it's one of those details that will appear later on in panes.

Regarding hierarchical "dropdown" navigation in the template list. That's an appealing idea but it doesn't handle as well as the current system the richness of capabilities in templates. It ends up getting in the way (been tried). Some of the nuances:

The biggest factor: Templates vary greatly in their dimensionality of operational options. It's not that always there is a top level grouping, like a folder, and then within that folder there is a list of choices. Sometimes that's the case, but sometimes there are two or three different dimensions of options where (to take the two dimension case) if you choose one option there are a variety of operations and for a different option a different list. If you want to represent that in a hierarchical list where there are three dimensions of combinations you really need a 3D cube list and then for transforms that might have four degrees of freedom a hypercube list. Trying to force that into a hierarchical list ends up being very messy, long, repetitive, and unclear.

The saved ("most recent") list and the pinned list are both very effective for typical, repetitive work, but how do you blend that with a hierarchical navigation system? Have one item to click for that particular command combination or two? If it's just one, does it appear in the hierarchy in standard form, or in "customized" (as in, how it was actually pinned) form?

Hierarchies are not an efficient way to find something if you don't know where it is, especially if there are two, three, or four different ways ("dimension") to specify the configuration of operations. What's compact within a picked template as a drop down list of configuration options for a specific operation based on a specific extraction of data from a given data type very rapidly through combinatorial arithmetic explodes into a hierarchy with thousands of end point leaves, each leaf of which is often a distinct transform that in the old system would have been at its own level in a one level list (Right now there are about 500 transforms with hundreds more planned). Trying to find something in a very large system like that is like trying to find a file within the Windows file system by manually popping open folders and subfolders to search for the file. Much easier is to use the filter box, which searches through at least two and could be more configuration dimensions (the filter box can always be adjusted to look deeper and broader, if that seems necessary).

Choosing the field / data type you want to work with is the first step, since people reaching for a Select or a Transform know what they want to work with. That right away reduces down templates to only those that make sense given the context.

The tool tips that pop open as you hover the mouse on a template should help beginners with a reminder to get into the right template for a task. Beyond that the filter box works great to find a particular operation / option if you don't quite remember where it is. Frequently used operations can get pinned, or just picked off the "recent" list.

On those rare occasions where you want to browse what's in a template (fewer and fewer as you get to know the system), just pick it and look. You'll be able to see in context all the dimensions of configuration that you can apply. If you're working with a hierarchy, you'll have to click anyway to drill down, so may as well make that click count to show you full context when there are many degrees of freedom to what you can do with a transform.

dchall8
1,008 post(s)
#27-Aug-20 20:13

Good answers. Thanks. I have been too impatient to hover over the transforms. Those are adequate for now. The really best news on the new transforms is that they are limited to only those that pertain to the selected feature and field.

Manifold User Community Use Agreement Copyright (C) 2007-2021 Manifold Software Limited. All rights reserved.