Subscribe to this thread
Home - Cutting Edge / All posts - Manifold System 9.0.169.5
adamw


10,447 post(s)
#18-Jul-19 15:43

9.0.169.5

Here is a new build.

manifold-9.0.169.5-x64.zip

SHA256: 5906719bc30f397051c0222a3cbef420a5efbdeb81881f5a2f1515a9ccbac275

manifold-viewer-9.0.169.5-x64.zip

SHA256: 96e4ac0a73e9684ac6f5d4a303cc9999bccfd0088af4d50fe228dfa8c443cfea

adamw


10,447 post(s)
#18-Jul-19 15:44

Changes

Watershed Areas and Watershed Lines transform templates automatically create a unique index on stream ID and a non-unique index on target ID, for further analysis.

Computing directions on plateaus for watersheds performs significantly faster.

Tables returned by TileWatershedDownstreamLines, TileWatershedDownstreamLinesPar, TileWatershedUpstreamAreas, TileWatershedUpstreamAreasPar, TileWatershedUpstreamLines, TileWatershedUpstreamLinesPar query functions include starting / ending coordinates used to extract downstream / upstream geometry.

TileWatershedAreasUpstream, TileWatershedAreasUpstreamPar, TileWatershedLinesDownstream, TileWatershedLinesDownstreamPar, TileWatershedLinesUpstream, TileWatershedLinesUpstreamPar query functions take an additional parameter for keeping or removing overlaps.

(Fix) TileFillSinks, TileFillSinksPar functions no longer fail to convert produced tiles to the original tile type (altered tiles were being returned as FLOAT64).

Reading a CSV file allows date fields to use multiple different languages (one field US English, another field German, etc).

(Fix) Reading coordinate system from a SID file no longer shifts the origin half-pixel by X and Y.

(Fix) Template parameter pickers indicate current choice in the dropdown menu using radio icon instead of check icon.

Number keys on the numeric keypad are treated as number keys on the number row for shortcuts.

Double-clicking an expand or toggle icon in the Layers pane no longer starts editing layer alpha / field size.

The note on image shading requiring a style based on a single channel is moved from the shading checkbox to a separate readout below shading controls.

Field headers and record handles in grids are made slightly flatter.

Table fields, layout frames, map layers with invalid or missing Z order info are put last rather than first (to the bottom of the display stack rather than the top).

Layer groups in tables, layouts, maps are reworked to use folders instead of leading layers. Group and Ungroup commands are removed. The new New Folder command that creates a new folder asking for its name. The new folder is created in the same place in the folder hierarchy as the current layer. The new Move to Folder command allows moving selected layers to one of the unselected folders, the target folder is specified using a dropdown menu. Moved layers are placed after all existing layers in the target folder. Folder names can be edited. (Groups with leading layers created by prior builds are automatically dissolved. We are going to allow drag-and-drop of layers between folders using the cursor in the next builds.)

The Move to Folder dropdown menu in the Layers pane allows moving selected layers out of all folders, into the root folder, using the Move to Root command.

Moving layers into a folder in the Layers pane automatically expands the folder if it is collapsed.

Pressing Insert in the layer list in the Layers pane invokes the New Folder command.

The map window no longer allows multiple layers to reference the same component. When multiple layers reference the same component, all but the first one are rendered invalid and do nothing. Invalid layers will still show up in the Layers pane and in the layer tabs so that they can be deleted.

The map window better distinguishes between changes to the underlying data and no longer repaints after changes that cannot modify the screen: changes to data in hidden layers no longer force a repaint, changes in Z order of a mix of hidden and visible layers that do not change the Z order of visible layers no longer force a repaint, changes to alpha of hidden layers no longer force a repaint, etc.

The map window better tracks changes to metadata of layer components and automatically reloads layers after changes to referenced components (eg, table) or fields (eg, geom). (Previously the window was only tracking the reference from layer component to its table, and was not doing so in all cases.)

Dropping new components into a map window creates new layers in the same folder as the active layer.

Moving layers using layer tabs in a map window changes the folder of the moved layer to that of the target layer.

(We are going to have layer tabs use folders more, we tried multiple variants on paper and a couple in code, the results are coming in the next builds. We are also likely going to take the Layers pane out of the Contents pane set and allow using it simultaneously with any other pane under Contents.)

The LAS / LAZ dataport creates a specialized spatial index for points, which allows quick display of big point sets as drawings. The specialized index is only created for a linked file. Importing a file will create a standard RTREE index, which will perform worse than the specialized one. The specialized index creates a hierarchical data structure, stored as a .MAPINDEXP file, that is then used to service spatial queries. During a spatial query performed for display, the data structure thins the points in the query area and returns a limited number of the best representatives with the limit depending on the display window size.

The size of .MAPINDEXP file is significantly smaller than the size of uncompressed LAS, but usually bigger than the size of compressed LAZ. The file is versioned so that future builds can choose to reuse files created by this build or overwrite them with new versions.

(We are going to add this specialized index for point sets into the MAP file as well, as soon as it stabilizes. We already did a number of iterations on the structure and it evolved quite a bit.)

The LAS / LAZ dataport reports the time to create the index (could be long, but this is only done once per file).

End of list.

BerndD

162 post(s)
#18-Jul-19 19:22

I like the watershed improvements. Will take a look at it ASAP.

In regards to the folders I wasn't able to put them to work.

The Move to Folder dropdown menu stays inactive. Maybe I am missing how to select a layer prior to moving it into a folder!

Also, I could create the same folder name twice.

Attachments:
manifold_folder.png


Organizations that want to adapt to CHANGE are using products that can adapt.

www.yeymaps.io

Dimitri


7,413 post(s)
#18-Jul-19 20:13

Maybe I am missing how to select a layer prior to moving it into a folder!

It's the same as in all the other table/grid displays: Ctrl-click it. The Layers Pane topic has some examples (toolbar has changed, but the selection stuff is the same).

BerndD

162 post(s)
#18-Jul-19 21:46

Thanks. Now it's fun to play with.

Ctrl Click on the folder and one can switch all layers on/off with just a single click. Love it.

Attachments:
Manifold_Group_Layers2.png


Organizations that want to adapt to CHANGE are using products that can adapt.

www.yeymaps.io

tonyw
736 post(s)
#18-Jul-19 23:10

Groups in Layers is nice. The process to rename a group folder is a departure from renaming folders in say the Project tab. In Project, it's a slow double click. Left click, wait half-a-second, left click again and the name becomes editable. Or you can use F2. In layers only F2 works to rename a group folder.

adamw


10,447 post(s)
#19-Jul-19 07:14

In addition to F2, you can also use Enter, that's standard to all grids. The other standard way to start editing is double-clicking the cell, but this currently does not work for folder names in the Layers pane due to an oversight, we will fix it to work.

danb

2,064 post(s)
#19-Jul-19 00:06

Plenty to work through here. I am particularly pleased to see the additions to the hydrology toolkit and even more so the point cloud index. A couple of quick observations about both.

With regards hydrology, we often want to be able to weight a flow accumulation grid to model lags such as vegetation interception etc. Is there a way that a weighted flow accumulation can be fed into the final derivation of streams and watersheds under the current model?

Most of our point cloud data holding in las/laz format is accessed through a read only connection. When linking to read only las files, this fact is reported in the log pane as expected. If however I exit and don't save when linked to a read only las file, Manifold 9 crashes (most times if not all).

If I copy a las file to a local SSD and link the index takes ~4 seconds to build. If I then copy this same las file to a read write network drive and link, it takes ~104 seconds for the index to be built.

Though the network drive is definitely much slower than the SSD, it doesn't take 104 seconds to copy the entire las file to this drive which is about twice the size of the index. Therefore ~104 seconds could perhaps be seen to be abnormally long. Is this likely to be a function of the way the index is written to the drive during creation (incremental writes or all in one go) or perhaps just a glitch on the network?

I am happy to test further and report back if it would be beneficial.

Finally I think the drawing of the point cloud with the new index will greatly benefit from the introduction of zoom rendering when this is added.

All in all great additions and great work


Landsystems Ltd ... Know your land | www.landsystems.co.nz

adamw


10,447 post(s)
#19-Jul-19 07:55

With feeding watersheds weighted flow accumulation do you mean giving the functions two rasters: heights + precipitation, and then computing streams / watersheds like we do currently, but computing total flow values from the precipitation data? If so, we could surely do it, the issue is organizing the two rasters so that it seems natural - the best way would perhaps be to pass a two-channel image, but preparing such an image would require tinkering with SQL.

We'll look into the crash in LAS / LAZ when exiting without saving changes (wow, apologies and thanks a lot for reporting this).

Your guess regarding why building an index for LAS / LAZ on a "slow" drive is even "slower" than the drive seems to be for file operations is correct. Building an index is done in-place with the index file created in the final location and being read from / written to incrementally. In the case of a slow drive, it would be faster to build an index on a fast drive and then copy it over in its entirety, that way the slow drive does the bare minimum of work. But we cannot generally do that because we don't know how large the index is going to get (the fast drive might not have enough space, or the slow drive might not have enough space - both things fail worse than if we didn't involve the fast drive), same for permissions. We do have some improvements planned to the process of building the index, however, they will help by just minimizing the amount of data we have to exchange with the drive directly.

tjhb
10,094 post(s)
#19-Jul-19 02:22

Why does a table have layers?

To reproduce:

  • New project
  • Open [mfd_meta]
  • Contents > Layers (huh?)
  • Layers panel > Create folder
  • Layers panel > select all fields (everything except Folder)
  • Layers panel > Move to Folder

Is this useful or just nuts?

adamw


10,447 post(s)
#19-Jul-19 07:08

Some tables have hundreds of fields, we thought we'd try using folders to help deal with them. Frequently, with a huge table you want to show just a couple of fields that you want and hide all others, because showing all fields fills multiple screens horizontally and it is pretty hard to work with records that are this wide, you end up scrolling left and right all the time. Then you usually want to show some more fields you missed at first, maybe hide a couple of fields you thought would be useful to have but it turned out they are empty or do not contain the data you thought they had. The Layers pane helps with all that. It's not just folders that the Layers pane is useful for either - you can use it to quickly search for fields by part of their name (eg, in a table of 300 fields, quickly show all fields with '2019' or 'POP'), etc.

Unlike with layouts / maps, we do not allow deleting items from the Layers pane for tables - that is better done in the schema dialog. All you can delete in the Layers pane for tables is folders.

tjhb
10,094 post(s)
#19-Jul-19 08:50

Thanks. I had kind of got halfway there in my head.

Tables with so many fields that they need filtering are a data design failure--normal form barely a memory--but they do exist. Some people have to use (or abuse) tables in this way.

But in my opinion, it strains the language to cast fields as "layers". Are they analogous to layers? In a way. We can think of plywood. Fields are arranged sideways instead of vertically--that is not a major problem. The thing is that they are not similar to each other (except formally)--instead their main characteristic is to be different. That may seem a nuance; perhaps this is just something to get used to. Perhaps a prominent explanation in the manual would make the analogy seem less strange. I don't know.

Maybe the "Layers" panel could be renamed "Items" or "Members" as a concession to the alternative use.

This might be more important: if fields qua "layers" can be visualized, elsewhere in the UI (that is, other than in the Layers panel), then they are probably a real thing. Can a table be shown with "folders" across its fields? Yes it can, or could, just as for the layers in a map. Is that helpful? Maybe. (At least for badly managed data.)

If it is helpful, then perhaps only the name needs questioning. Or again, perhaps it is just something to get used to. I don't know.

Dimitri


7,413 post(s)
#19-Jul-19 09:54

Tables with so many fields that they need filtering are a data design failure--normal form barely a memory--but they do exist. Some people have to use (or abuse) tables in this way.

Yes. Unfortunately, tables with an absurd number of fields are very frequently met. Natural Earth is a good example, used in the Layers Pane topic as an example. There are numerous tables from national census and statistics bureaus, many with dozens or even over 100 fields. The ability to group fields to make it easier to manipulate groups, on/off, move left/right, is very useful when you have many fields.

perhaps it is just something to get used to

I think it's just something to get used to. If you called it anything but a "layers" pane that would confuse people in most case where they are used to seeing a way to manipulate layers.

tjhb
10,094 post(s)
#19-Jul-19 10:18

Thanks Dimitri.

Thinking of fields as "layers" might stir my porridge brain into a new pattern and maybe bring some new SQL.

KlausDE

6,410 post(s)
#19-Jul-19 14:56

I love table fields in folders as this allows to toggle visibiliy of groups and just have a representation of a group of data.

I would like this feature to cover display of data in the record pane as well and in particular remember it permanently for queries and derived drawings where this feature could help to visualize the fields from different sources of a join.


Do you really want to ruin economy only to save the planet?

tonyw
736 post(s)
#19-Jul-19 17:17

The ability to turn off visibility of whole groups of fields is handy. I regularly use forest vegetation information from our provincial government, the Table has 186 fields. Field names are 10 characters names, probably a holdback from mainframe days to conserve storage space. The field names are abbreviations and finding a field related to tree "species" is not easy, there are 54 fields with the abbreviation "sp". Now I can group and turn off visibility of the majority of fields. Very handy.

Now, next wish for a feature is being able to go from a field name in the Contents > Layers pane to that field in the table. Sliding left and right across 186 fields to find my field of interest is a slow, non-reproducible process. So the desired feature would be much like clicking on a record in M8 then using the magnifying glass icon to centre that area, line, or point in the view of the drawing or map. Something similar when viewing a Table: I would use filters to find the name of the field then click something to go to that field/column in the table.

Dimitri


7,413 post(s)
#19-Jul-19 19:28

Now, next wish for a feature is being able to go from a field name in the Contents > Layers pane to that field in the table.

Great idea. I'd like to see "selectback" from any pane to whatever else corresponds. Right click onto a record in a table and center/zoom that in the drawing. Or like you describe from the Layers pane to bring a column into view in a table window.

tonyw
736 post(s)
#19-Jul-19 20:19

...from any pane to whatever else corresponds...

Yes, another application of this idea. When applying styles to layers in a map I need an easier way to find the tab corresponding to that layer. With my many layers, the layer names get truncated on the tabs at the bottom of the map. As well, often the modifiers are at the right hand end after a transformation and it's hard to distinguish between layers with similar names. However, in the Contents > Layers pane the layers are easier to identify and find because more of the layer name is visible. What I would like to be able to do is click on a layer in the Contents > Layer pane and be able to apply Styles to that layer. This requires being able to change the focus on layers in Contents > Layers. Changing focus could happen in both the Contents > Layers pane and as is currently, by clicking on layer tabs at the bottom of the map.

tjhb
10,094 post(s)
#19-Jul-19 22:40

Right click onto a record in a table and center/zoom that in the drawing.

That one is tricky. It would need to be clear from the context, both which geom field is in question--the table might have more than one--and also which drawing should be centred/zoomed--more than one drawing may be drawn from the table.

The first thing can be managed, for example if the requirement is to right-click on a specific geom field within the record.

The second thing is less easy. Must the drawing(s) in question already be open (as well as drawn from the current table)? Must it be in the same datasource?

This is more naturally done from the Record pane, since that pane is invoked from the drawing (or image) window itself, and from a specific geom (or tile)--that of course also indicates which geom field to use. When the link is made that way around, there is no ambiguity. We already have Zoom to object from the Record pane.

tjhb
10,094 post(s)
#20-Jul-19 05:43

I think often I try too hard to be clear, and wind up sounding peremptory. It's a fault. I apologise.

Dimitri


7,413 post(s)
#20-Jul-19 06:54

No apology necessary. First, not a hint of peremptory, and second, even if there was, sharing opinions straight up without sugar coating shows you trust your colleagues here in the Manifold "locker room" not to be sissies. :-)

KlausDE

6,410 post(s)
#20-Jul-19 20:51

Here is the update from version 9.0.169.5 of the german UI file to the real version 9.0.169.5 with only very few changes.

Attachments:
ui.de.txt


Do you really want to ruin economy only to save the planet?

KlausDE

6,410 post(s)
#21-Jul-19 13:27

The reason why there are so few changes at last in this build is not my anticipation of 169.5 in the previous build but a problem I don't jet know how to deal with systematically:

I found that the number of parameters for some BuilderFunction... strings and TemplateTileWatershed... strings have changed but not the name of the string. This results in missing parameters in the query builder and in missing captions in transforms - see attached screenshot as an example. The parameter 'keep overlaps' is missing or not labeled.

I detect new strings that need to be added to the ui file by comparison of names of the strings, that in the alphabetic order of the files are simple to find in Kdiff3 searching for the start of line to the equal sign. kdiff uses sed to restrict the comparison to a pattern and the corrosponding preprocessor command in the kdiff3 configuarion is sed "s/=.*//"

I'm out of my depth in this gnu environment and I guess I will have to find a way to create a more flexible way to analyse tokens that need translation in manifold9 without external tools and do it not line by line manually (which BTW let me follow the development in digestible chunks).

Attachments:
screenshot.png


Do you really want to ruin economy only to save the planet?

adamw


10,447 post(s)
#22-Jul-19 08:36

Instead of trying to find all strings with mismatching numbers of parameters, etc, automatically, it might be easier to do the translation like this:

Suppose we have UI.TXT and a fully translated UI.DE.TXT for version X.

Version X+1 comes.

We take UI.TXT from version X and UI.TXT from version X+1 and check what changed.

We then take UI.DE.TXT from version X and update / insert / delete strings to update it to version X+1. Inserts for new strings and deletes for former strings that got deleted could start automatic, and then you can provide translation for each inserts and update translation for each update.

KlausDE

6,410 post(s)
#22-Jul-19 14:01

Problem is I do not know when and who missed parameters May not be the last UI version


Do you really want to ruin economy only to save the planet?

adamw


10,447 post(s)
#23-Jul-19 11:48

OK, then I guess you could try getting everything imported into tables, then separating string names from string values at '=', then choosing all TemplateXxx strings and comparing the number of parameter separators '|' in values for the EN values vs the DE values.

KlausDE

6,410 post(s)
#23-Jul-19 12:07

I hope I can come with a solution that lets you translate <value>, <table>, ... only once and overcomes one's inhibitions to start a transtlation now.


Do you really want to ruin economy only to save the planet?

volker

1,086 post(s)
#24-Jul-19 13:17

I have download a "*.nc"-File from

https://land.copernicus.eu/global/products/ndvi

It`s a 1GB great NDVI layer. I am able to create a Data Source to this file

but after this EDGE 9.0.169.5 says "Can't open file".

What's the problem with this ?


http://www.thegisservicesector.de

adamw


10,447 post(s)
#24-Jul-19 15:44

It might be a later version of NetCDF that we do not support.

I don't see how to download the file from a cursory look into the link, it would help if you posted a direct link to the file or uploaded it somewhere or sent it to tech support, that way we'd be able to check for sure.

We have plans to update support for NetCDF (and HDF, too).

If the file turns out to be indeed not of a supported version, you could still try importing / linking it using the GDAL dataport - we allow working with everything GDAL supports.

volker

1,086 post(s)
#30-Jul-19 17:42

Thank you Adam. I installed GDAL and all it needs, GDAL works, if i go to cmd

and type gdalinfo --version i get the "GDAL 3.0.0, released 2019/05/05".

But if i try importing via GDAL in EDGE nothing happens no .nc, no h5, no gdb.


http://www.thegisservicesector.de

adamw


10,447 post(s)
#30-Jul-19 18:01

Check the log window, it probably complains about not being to load GDAL DLLs and if so, that should be fixed (by including the path to the correct DLLs into the PATH variable - either system or user).

rk
621 post(s)
#30-Jul-19 18:54

Only gdal 2.0.x, 2.1.x, or 2.2.x is loaded. Not newer.

Dimitri


7,413 post(s)
#31-Jul-19 07:44

version i get the "GDAL 3.0.0, released 2019/05/05".

Please read the GDAL / OGR topic carefully. Make sure you follow the installation procedure it provides (well illustrated, every step) for things like PATH and so on.

Note also this:

To use GDAL or OGR we mustinstall GDAL in Windows, as described below, and then we can use it from Manifold. If we do notinstall GDAL, we will notbe able to use the capabilities described in this topic. The Manifold GDAL dataport allows working with GDAL 2.0.x, 2.1.x or 2.2.x and automatically selects the latest version available with automatic adjustments for the call interface.

So... you should not expect that 3.0.0 will work. It might, or might not. GDAL versions might change function names, etc., so each new version has to be tested.

adamw


10,447 post(s)
#30-Jul-19 12:25

We are planning to issue 9.0.169.6 in a couple of days (likely on Friday) and we noticed that 9.0.169.5 will stop working earlier than that (will stop working after Wednesday). So, here is an updated build of 9.0.169.5 that will keep working past that.

manifold-9.0.169.5a-x64.zip

SHA256: 55e74618037f3d845d2642c11d7296c89bd1b18b913dcc231beaa5aa43ed3b53

manifold-viewer-9.0.169.5a-x64.zip

SHA256: dcb48f7ad50e89298bacbb3c928ae51b622235ac98dd3b277bb9231c410bb590

Manifold User Community Use Agreement Copyright (C) 2007-2021 Manifold Software Limited. All rights reserved.