Subscribe to this thread
Home - General / All posts - M9 Performance: "Inserting records" during Overlay topology operations
gjsa100 post(s)
#15-Mar-18 06:44

Overlay topology operations can run for hours without a result (meaning I give up waiting) when at least one of the two layers is large. When I say large, I mean an imported polygon shapefile around 2 Gb.

My system has 24 Gb RAM, 4 cores, Win 10, CUDA/GPU enabled. Latest Edge (.165) version M9. But, none of the CPU, GPU, RAM or disk access are anywhere near being fully utilised.

"Inserting Records" is the time hog - and the progress indication does not advance giving no idea how much time is required to complete the task.

I guess I'm looking for anyone with similar experiences and what, if anything, might be done to identify the bottleneck before seeking a possible solution.

adamw


10,447 post(s)
#15-Mar-18 07:34

We are planning to the rewrite the part of the algorithm for topology overlays that is causing this issue. It works, but it is sequential which immediately makes it the bottleneck, and it does not track progress / allow canceling all that well.

Until we do this, consider splitting the task into smaller portions or simplifying geometry (Normalize Topology with the value of tolerance high enough to reduce the number of coordinates).

gjsa100 post(s)
#24-Jun-18 07:54

Ok thanks.

I am trying to 'dissolve' a layer with approximately 9 million features created after running the "Enclosing rect " tool. There are many shared and overlapping boundaries (as expected) that I want to simplify.

"Union Areas" is the approach I have in mind to dissolve this layer.

Similar to the "Overlay - Union" operation on two layers, the "Union Areas" tool has been running for 15 hours with no result.

So my question is - if I split this layer into four parts and run four instances of Manifold (I have four cores), will the run time be 1/4 of the (unknown) total time? Or less than 1/4?

(I am also trying an alternate approach to "Normalize Topology" with a tolerance of 1 meter to reduce the vertices before the "Union Areas" tool, but it has also been running for hours (at 26% CPU) without finishing.)

...Btw, none of this secondary processing would be possible if Manifold 9 wasn't so blazingly fast at converting a raster to polygons! I'm impressed with the speed of that process and many others.

tjhb
10,094 post(s)
#24-Jun-18 08:28

3 1/2 months -> “OK thanks”. Sheeesh.

For the next question, which BTW needs a new thread, how about some data?

You are clearly doing something unwise.

gjsa100 post(s)
#24-Jun-18 09:23

I have spent 3 1/2 months in the course of my normal job testing different approaches and having a lot of success and I'm hugely grateful for the price-performance ratio of M9, but there are still some cases where M9 clearly does not (or cannot) take a parallel approach to processing a large dataset.

Advice noted: I will start a new thread and pose the question more openly with some data.

gjsa100 post(s)
#24-Jun-18 14:37

This thread covers most of the issues I am experiencing and some possible approaches to speed things up until single threaded tools like "Normalize Topology" and "Union Areas" can be made multi-threaded in future versions.

Manifold User Community Use Agreement Copyright (C) 2007-2021 Manifold Software Limited. All rights reserved.