9.0.172.3

Here is a new build.

This is an out-of-band build that fixes a fairly sizeable regression in the TIFF dataport introduced in the previous build. The regression appeared as a result of our work on a new feature. The regression was quite unpleasant, with some images refusing to import at all. We decided that this cannot stand and did the following: (a) identified and fixed the issue, (b) finished the feature that we were working on and thoroughly tested it, (c) took the opportunity to stress-test the dataport on all variants of TIFF files that we have and fixed more or less everything we could find.

We do have a couple of issues still outstanding. One is related to interpreting pixel values (if a TIFF file is structured in a particular way, which is kind of strange and isn't used often, importing it makes missing pixels black rather than missing, while linking works correctly) and the rest are all cases of old-style projection data (pre-2000) not being converted into the Manifold projection or being only converted partially. Given the complexity and the age of the code, there might also be issues that we don't know about. If you are using TIFF files, please try working with them using the new build and let us know if you run into any problems. In the future, we are planning to rewrite the dataport, both in order to equip it with new features and to modernize and strengthen the code.

manifold-9.0.172.3-x64.zip

SHA256: 42c2c2dfbc95c55fb947bd8996df174c59bc3810443ee3dadc95042b94884f58

manifold-viewer-9.0.172.3-x64.zip

SHA256: 8775364f6a74ab65b727a008cc89ebc0a0bcb01062cbab1cb39244f17ccb4321

Changes

(Fix) Reading a TIFF with tiled data no longer sometimes shows no images. (The regression that is the reason for this out-of-band build.)

(Fix) Reading a TIFF no longer sometimes fails to mask pixels with no data values for int8 / int8u / int16 data types.

(Fix) Reading a TIFF no longer sometimes mis-decodes CMYK data.

Linking a TIFF supports pyramids stored within the file. (This is the feature that we were working on. It allows rendering a big TIFF without spending any time on preparing data for intermediate levels, if the file already has that data prepared and stored internally.) There is a limitation in that the pyramids are only supported if both the main image and all sub-images within the file are tiled, but that's how these files are usually organized anyway.

Dataports for databases skip loading support for personal geodatabases for technologies that cannot have them, for faster starts.

Dataports for databases skip loading support for SDE geodatabases for technologies that cannot have them, for faster starts.

(Fix) The color picker tool no longer picks the color from a wrong pixel on screens with DPI scaling.

The limit on the amount of temporary data used in a Manifold session in 64-bit mode is increased to 1 TB.

The internal implementation of SQLITE is updated to 3.32.2, adding a couple of new SQL functions.

(Fix) Dataports for web servers no longer sometimes fail to connect via HTTPS for the first time (the first HTTPS web request no longer sometimes fails due to SSL / TLS policy being applied at the wrong time).

Exporting a map to GPKG runs progress and reports export speed.

Reading a GPKG exposes an additional virtual table for each image. The virtual table has a fixed structure and includes both a BTREE index on x-y and an RTREE index on x-y-tile. This allows both (a) rendering intermediate levels stored in the database and (b) Alt-clicking into image tiles to see pixel values. The virtual table is read-only. The physical table storing image data is still accessible and writable.

Importing data into a data source other than MAP no longer stops if some of the indexes fail to create.

Pasting an image table with an RTREE index on x-y-tile automatically generates data for intermediate levels.

End of list.

Cloud Optimized GeoTiff support confirmed! Linking works correctly, internal mask correctly handled, internal tiles and overviews handled correctly.

Thank you.

The limit on the amount of temporary data used in a Manifold session in 64-bit mode is increased to 1 TB.

I'm glad about this, but don't know why(!).

It would be interesting to know the motive behind it. And I suppose: what was the previous limit; why there must be any limit (just pointer data type?); what is the effect of hitting the limit.

Also a question:

(Fix) Reading a TIFF no longer sometimes mis-decodes CMYK data.

What colour space and intent do you use when "decoding" CMYK images to RGB?

(For that matter, does Manifold 9 RGB assume or suppose a defined RGB colour space? In the past I have ~assumed it is ±sRGB.)

We pretty much do need to know these details (colour space and conversions). They are crucial for output.

Any respectable client will ask, and we must have a respectable answer.

Temporary data:

The limit on the amount of temporary data applies to the data spilled to disk. There are different types of temporary data, some data only lives in memory, other data may spill to disk and is accessed through cache.

The first type is limited by the size of the system pagefile, we ask the system to allocate more and more memory, eventually it runs out and says 'no' and then the operation fails. The size of the pagefile is usually bigger than the amount of physical memory, but frequently not much bigger (something like 2x) and there's competition for it with other applications.

The second type of temporary data uses both memory and disk and can grow much bigger than the pagefile, without putting pressure on other applications. The growth, however, has to be limited in some way due to the overhead for record-keeping. Prior to this build, the limit in 64-bit mode was 256 GB. This was fairly high, but over the years people started hitting it sometimes when performing operations that generate big results, like building dense contours on big rasters. When you hit the limit, the operation fails - everything rolls back but you don't get the results that you wanted and the processing time is lost. So we decided to increase the limit to 1 TB, increasing the overhead slightly (most, but not all of it, only occurs when you actually use more than 256 GB of space). In the future, we will likely adjust the design of the cache so that we can have no limit other than the size of the disk.

Color space:

Yes, we are using / assuming sRGB. We convert CMYK to RGB mostly due to compatibility reasons. Eventually we'll leave data as CMYK and just show that directly converting to whatever we need on the fly. This will let the user specify how to convert as well. Same for other color spaces.

Thank you! Great explanations. Extra questions:

When converting CMYK to RGB (sRGB), what CMYK colour space, black point and rendering intent are assumed?

Is there any change in the recommended size for RAM cache under Tools > Options? Should we bump it up from 4GB, or will “auto” still give best results?

Regarding CMYK -> RGB, we use the simplest direct mapping possible with no adjustments for anything: R=(1-C)(1-K), G=(1-M)(1-K), B=(1-Y)(1-K), scaled to the number of bits in the channels. This is very crude, but perhaps least surprising.

Regarding the amount of memory reserved for the cache set in the options, the increased limit of up to 1 TB for temporary data per session does not change the guidance for the cache, because both the old and the new limit are much larger than the cache can realistically be. For memory-intensive operations, with a single Manifold instance running, we recommend using half of the physical memory you have installed and leaving the other half to 'very temporary' data and to other application. The default '(auto)' choice is only using 4 GB because we don't know how many instances of Manifold you are running. In the future, we might have different instances of Manifold coordinate with each other and then '(auto)' might sync with the rule of thumb above: half of the physical memory, but distributed among all instances of Manifold currently running.

Colour picking seems to be working reliably now. Thanks.

german ui file for version 9.0.172.3

BTW menu shortcut are not localizable right now but hard-wired to the en-US keybord layout. Thus for the shortcut to open a new SQL command window we have to type <strg><ö> on a german keyboard.

And I ignore the dynamic key shortcuts in dialogs (marked by ampersand in the original ui file) for now. This must wait until the development of the ui has settled and duplicates can be avoided.

Attachments:
ui.de.txt

Do you really want to ruin economy only to save the planet?

Status update:

We are planning to issue the next build in the beginning of the next week. The focus is on Select and Transform panes, although we are going have new features outside of that, too, as usual.

Another status update:

As we were trying to make incremental improvements to the Select and Transform panes, we came to realize that we need to make a fundamental change to how we approach templates in those panes. The change will allow us to make the panes significantly easier to use while adding a number of useful new features that we don't have now. The change, however, requires that the code generating queries for all existing templates is rewritten. That is a pretty big task.

We decided to bite the bullet and implement the change, rewriting all existing templates. We believe that the improvements are going to be well worth it. We expect to have the build ready sometime next week, possibly near the end of it. In return, one of the key parts of Manifold will become significantly better, forever.

We realize that it's been 12 days since the last cutting edge build and will issue the next build as soon as we can. Sorry for the delay.

Thanks for the update.

Can I ask whether the new build will introduce general breaking changes, in the sense that SQL written for previous builds will not function in the coming build or subsequently? That is, not just in respect of a few new or changed functions, but generally.

No, the changes are in how the UI is organized and what code is generated for the templates, the functions themselves stay the same - sans the usual incremental improvements and additions as you say, we might have a couple. Queries written for previous builds should continue working.

That all sounds great. One thought is that I would think by far the most common desired search is Text Contains, Intl, neutral, no case. That should be made super duper fast to access and/or be the default template if possible. Alternatively, maybe some sort of fast accessing Favorites section.

We will think about pinning templates as favorites, that's an interesting idea.

How about just MRU, at the top? Say 4.

How about custom styles as favorites? For DEMs I currently like the US Altitude palette with the following settings

• Channel 0
• Equal Intervals
• Breaks - 200
• Altitude, US Relief
• Lightened a few clicks
• Hillshading with Z scale - 20 and

Oooh yes, that would save so many clicks! Great idea.

Do you mean the static value -> color mapping or the dynamic one which adjusts to the range of values in the channel (within the bounds of the image)? The former is easy to save, but if it is the latter that you are after, that doesn't help much.

I'm not crystal clear what the dynamic one is, but if static is easy, start with that and we'll see if that is enough. Of course as we see new features our feature cravings seem to intensify.

Perhaps this might work analogously to favourite projections, which can be saved (or applied) with or without custom metrics.

The basic position might be to save the current style JSON text as a favourite; but perhaps with an option checkbox controlling whether to save image-specific parameters, or to strip them (as far as possible).

In any case in general I think users would tend to save relatively simple styles as favourites, since this makes better sense and because they are more likely to be reusable than complex styles.

That's an interesting thought. Perhaps we could allow for some flexibility in what gets saved or what gets applied on restore.

With dynamic vs static I meant whether saving the style should be saving the rules (please split the data set into 5 parts using natural breaks = dynamic) or the breaks (please split the data set using breaks at 10, 20, 50 and 100 = static). I guess it might make sense to start with saving the breaks and go from there.

We'll see. Currently we are neck-deep in Select and Transform, but we might have some room for improvements to Style until the public build as well. We are already planning some improvements for it, maybe we could fit this idea with the favorites as well.

Thanks Adam.

One more status update:

We brought the planned design for the Select and Transform panes to the point where it could be coherently tested, and tested it on several real-life scenarios. The results were a mixed bag. Many things became notably better, but some became worse. The generated queries were OK, but the user interaction was not as straightforward as we wanted. We weren't happy with this outcome.

We then spent considerable effort figuring out how to improve the design. After trying many different ideas, we think we have the solution that fixes all of the low points of what we did, and is a huge improvement over what we all have in the last published cutting edge build. Switching to this second design, however, is, unfortunately, once again a serious effort. The changes that we need to implement are ideological rather than cosmetic. It would have been much better if we came to the second design without going through the first one, but regretfully we took the long road.

There is a good side in this. As we are making the Select and Transform panes better, many of the templates also get new options. In addition to improving the user interface, we are also frequently increasing how much a particular template can do. We have already added multiple new query functions and are going to add more. The combined effect from all improvements taken together is pretty big.

In terms of the schedule, the optimistic picture is us fully implementing the second design in 9-10 days from now. This places the next build at Jul-11 (Sat) or Jul-13 (Mon). This is, again, optimistic, things might take slightly longer. We'll do all we can to finish the works and issue the build as early as possible to make the new features available to you and to hear what you think about them. If there are going to be any serious deviations from this plan, we'll post about it.

Thanks a lot for your patience.

I have a question, pretty obvious I suppose.

You have a massive resource here, all sorts of people who can tell you what they think. (Mostly wrong--just kidding.)

When you're planning massive changes, why don't you ask us, before dedicating the equally massive time and resources?

Rather than after.

Is there a reason why you don't?

We ask before doing about smaller things all the time. For bigger things we ask by doing cutting edge builds.

There are limits to which you can discuss the merits of an implementation of a big feature that does not yet exist on the forum. There could be insights from such discussions, absolutely, but we think we get most of them anyway from discussions of other, related, questions in threads concerned with practical solutions to practical problems.

We could write a couple of pages of how something like the Transform pane could potentially work and then ask for opinions. But in our practice, without an actual implementation that people could try, it would be difficult to assess what it is we are proposing, how that would actually change the experience and what potential problems could arise. The topic is just too big.

I can sympathize with the situation. My inclination is to let the programmers work.

Back in the 90s I picked up a project to predict jet engine maintenance based on onboard sensors. We started with a fairly primitive engine and found that there were huge maintenance cost advantages to the software, but the users had to understand some basic logistics in addition to simply using the program. We had annual users conferences to train new users and retrain the rusty. Those turned into great feedback meetings, too. Then we added more and more modern engines with more sophisticated sensors. Initially the programmers made the mistake of duplicating the entire original package for each new motor and then customizing for the new sensors. Too soon the software was 10x bigger than it needed to be. We took a year off from training to make one base program with branches for the various engines. The newest engine came in about that time with the most vocal user base and, essentially, sucked up our management resources simply defending what we were doing. The entire program suffered and nearly died due to misplaced priorities of the few. Users lost track of how to make the best decisions, and maintenance costs increased. That's not the end of the story, but it's enough to remind me to let the programmers do what they do, especially when "it's complicated."

Great lesson, well written.

(Sounds very like Plato!)

At the end of that story, at least where I bowed out, the vocal minority were both not following the program, at all, and denying that they were not following the program. We found one lone-wolf base where they were flying double the average flight hours per month with 25% of the normal, per engine, maintenance budget. We audited them and found 31 things they were doing that nobody else was doing...and it was all in the FM. If you could extrapolate that efficiency to all the bases, we could have expected a $3 billion per year savings. That number was so ridiculous for such a small program that it was laughed off - even though it was documented with senior leaders signing off on the audit. Sadly, after I left, the program landed in the lap of the ignorant savages who nearly killed it, not that I'm a little bitter about any of that.

But this part of the story does not apply to the Manifold situation.

I think you underestimate both yourselves and us.

I would never suggest a wholesale change, everything is basically right.

But there may be opportunities to ask a high-level question of the dedicated community here, and get a range of useful answers, that could (optimism glasses on) potentially save work and time. Maybe.

You are really good at this high-level stuff, Manifold communication is ~the best English on the planet.

(I doubt there is any need for a new closed restricted forum.)

I doubt there is any need for a new closed restricted forum.

I agree. This forum is a great place to discuss high level questions, and lower level details as well.

But speaking of high level questions, that's not the hard part: those are easy to ask and get answers, especially given that people naturally express themselves. High level issues, like whether or not somebody prioritizes georegistration, tend to be pretty clear, both in this forum and in suggestions.

Details like the specific sequence of clicks or keystrokes or whatever are the commands to denote or edit a control point, or where that falls in the workflow, are more difficult to determine. I think the forum is great at that as well, if the community is given something specific to evaluate. That's where the work and time savings come in, because it makes both immediate tuning/correction as well as the next iteration more efficient.

In many complex issues the devil is in the details, which often you don't even see until the thing is done to the point it can be used in real life work. What sounds great at the high level may be super or may be awful based on the implementation of details. Just a few small differences, like when and how you pick a field that's the target of an operation, can determine whether the workflow feels easy or seems clumsy.

In some cases, no matter how good you are at anticipating wide ranges of workflow and how they intersect with a very wide range of user skills and preferences, it's hard enough to construct specific implementations, in dozens or hundreds of small details, that are worth trying by a wider community. And even then, what you sometimes find is that what seems OK for the first dozen hours of use gets on your nerves after the next 20 hours of use.

That's the point of having cutting edge builds, because they provide an opportunity for worldwide assessment in situations where no amount of mockups, focus groups, discussions about one way or another, or other advance work tells you whether some change is good or bad.

Or, as is more often the case, what parts of a generally good idea need to be improved so the whole thing isn't just "good enough" but is felt to be an especially good step forward. Manifold puts a lot of effort into real life testing of variations on new approaches before they come out in a cutting edge build, so what gets presented for worldwide assessment by the community isn't going to be truly awful and waste everybody's time. The company is also willing, even towards the tail end of an iteration, to say, "no, not good enough, must be improved here and here," even if that means a significant amount of additional work.

It's true that such an approach, spending a lot of effort so something real can be presented for assessment by the whole community in a cutting edge build, is costly both in engineering resources and calendar time. But it does provide a reality check and real world guidance that cannot be beat.

Considering where the original beta builds started, we've already had 3 or 4 significant iterations of interfaces for select and transform functionality. All were based on high level guidance from users, and all have benefitted from community guidance and from new capabilities made possible by improved internal infrastructure. This next iteration will also be a step forward, based on high level guidance from the community, and it too will benefit from tuning based on real life assessment by the community. It too won't be the last iteration. All that is a lot of work, both for Manifold and for the community, but that's what you have to do when you want ever faster, smoother, easier, more efficient and more powerful workflow in complex settings.

Best possible answer, thank you Dimitri.

And even then, what you sometimes find is that what seems OK for the first dozen hours of use gets on your nerves after the next 20 hours of use.

The current outline of work is really great and I'm looking forward to when 9 gets used like the latter above. Once labels and vector editing catch up, that will likely happen. I suppose my thought here is that it would be great if 9 was complete enough to use day to day for map making rather than just specific mapping functions before other significant changes occur. However, you guys have the full scoop on what makes best sense with the underlying programming.

Time for hitting base:

We have been working on the new design for the Select and Transform panes all the time since the last update. The panes have been progressing quite well. We are pleased with our second take on the design. The changes we implemented since the first take were pretty painful in terms of the amount of work, but they did fix everything that felt wrong. The UI became more intuitive and faster to navigate.

We did hit a couple of problems we couldn't have foreseen before. This was pretty much unavoidable given how big the changes are (the bigger the changes, the more chance of hitting roadblocks you couldn't have anticipated before you started). As just one example, we found a number of errors in the query engine which had to be fixed. Thankfully, none of the problems we hit so far were show-stoppers.

We will need additional time to finish the build on top of the earlier optimistic estimate, but not terribly much, 3-4 days or so. Then we are eagerly looking forward to your feedback.

Thanks again for your patience and we think the next status update is going to be the build itself!

Thanks for the update.

Can't wait to see the new Select and Transform panes.

Organizations that want to adapt to CHANGE are using products that can adapt.

www.yeymaps.io

I don't know why it seems longer, but it's only been a month since this release. I had just started working with GeoTIFF images from USGS at the time, so I've been pretty happy after only a day of TIFF frustration.

Quick update: Engineering expects to issue a new cutting edge build late this week. They really want to get it out in July, and it looks like that will happen.

I've been involved in alpha testing, with the ninth internal build just having come out today. Besides steady progress implementing more and more transforms using the adjusted UI with each such build, there has been considerable extension of options for what various transforms can do, plus the addition of more transforms.

I know it's been a long time since the last cutting edge build, but there are a lot of moving parts to this new build that all have to get done together for it to be worth presenting to the community. From what I've seen so far it's definitely worth the wait.

One more status update.

We are still not done with the build despite having worked on it for such a long time. We think we are very close to finishing, however.

The amount of changes has been staggering.

To give you a glimpse into what we have been doing:

A big part of work as of late was in the way we handle components of different types in the query engine. There were three main reasons for changes in this area. First, the new UI for the Select and Transform panes is much more flexible than the old one and allows doing things that the old UI could not. This requires new functionality in the query engine. Second, we wanted to extend and simplify the use of query functions which work with tables representing higher-level components like drawings and images. We also wanted to standardize our requirements for queries used as a base for drawings and images and guarantee that all query functions working with drawings and images will work with a component based on a query as long as the query follows the standard requirements. Third, we wanted to prepare for the new architecture for images that is going to come in the not too distant future and will allow things like per-pixel selections, to make sure we don't have to rework things all over again when this happens. This affected many internal decisions.

This all is merely the last big batch of work seemingly only barely related to the original goal of redoing the UI. There were a couple of other big batches before it, now finished.

Most of the above batch has been implemented. The only part left to complete is making all query functions working with drawings and images accept all types of such components: based on a query with the now-standardized requirements, restricted to a selection, etc - some of the functions currently reject a type or two or work incorrectly with it. This is what we are currently working on.

Overall, the total number of select / transform templates that we had in the old system was ~420. Out of those, ~300 are now plugged into the new system and need no further changes, many had been extended. About 60 of the remaining templates are using query functions mentioned above and will be finished shortly after we finish changes to these functions. About 30 of the rest need minor additions in other areas. All others we are not going to plug into the new system because they are no longer needed as separate templates after the extensions that we made to other templates.

We think we are close to finishing the build. We are currently planning to issue it on Friday, 7. Maybe slightly earlier.

It is bad that the build is taking so long. We certainly have to eat quite a bit of crow on the build date predictions in particular. But we think the new features will be a very good addition to 9 and the quality of the product will improve a lot.

Thanks a lot for waiting patiently!

Back in the mid 70s we had a saying in software development: take the contractor's estimate, double it, and change the units to the next higher unit. So if the contractor told us it would take 20 days to make a change, we changed it to 40 weeks. We were right more than we were wrong. Of course we were insisting on full documentation, which the contractors objected to strenuously.

But it does sound like we have a lot to (re)learn. And sounds like a major rewrite of the user docs and revamp of the videos is in order.

I hope Manifold doesn't beat itself up about missed predictions.

It's not as if we are waiting for a product that works fully--we have that already. These are improvements on an already great thing.

The communication is great (and intriguing).

I don't think it matters whether the Friday prediction is extended again. Don't miss your families too much!

I suspect you will have done this in the present process, as an investment: to make it easier to rewrite existing templates again in future as required.

I've said before that I would like you to put conceptual questions earlier to this community, before planning the engineering investment, but I am probably wrong I know. It may be impractical. Cutting edge builds ensure focussed questions.

Quick update: 9.0.172.4 was built today and it all looks good, but the release notes are way behind.

Normally it only takes a few hours to write up release notes. With a build this large (a new record, I think), the process of writing up notes has pushed the release of the build out into tomorrow. It's a staggering amount of info. :-)

Too bad nobody predicted something like that back on Aug 2.

Oh, no, that would be way too boring. Only small, unambitious builds are predictable. If you can predict your builds you're not trying hard enough. :-)