Subscribe to this thread
Home - General / All posts - I ♥ Manifold ♥ COPC
rk
518 post(s)
#08-Jun-22 14:21

I just want to say that the way Manifold can work with copc is wonderful

No problem of working with laslibrary of 3.3Bn points. At no point is disk or memory usage excessive. I do not have to think it is composed of 60+ laz files, 18GB total. I have 1 whole drawing zoomable and pannable at once.

rk
518 post(s)
#10-Jun-22 07:37

And I was little bit too optimistic.

I'm currently stuck for 30+ minutes at 'Canceling operation...'

I wanted to make a small spatial selection of my copc library.

I have actually had great experience before, filtering points by areas from single copc.

code

-- 129_260_444 points

VALUE @drawing TABLE = [copc]::[copc Drawing];

-- 6264 small areas

VALUE @overlay TABLE = [Areas D];

DELETE FROM [filtered_points];

INSERT INTO [filtered_points] (

  [mfd_id], [X], [Y], [Z], [intensity], [gps_time], 

  [Geom]

SELECT

  [mfd_id], [X], [Y], [Z], [intensity], [gps_time], 

  [Geom]

FROM CALL GeomOverlayTouchingFilterPar(@drawing, @overlay, 0, ThreadConfig(SystemCpuCount()));

-- (1208.334 sec) 12_252_053 points

This is very good, I think.

Maybe the current query with implicit 'SELECT *' has some bad effect, beside pointcloud being 30x bigger, but only 1 simple area to overlay?

Attachments:
copc-lib.png

adamw

10,011 post(s)
#10-Jun-22 07:52

The only problem is a slow cancel for the query above, right? We will look into it.

rk
518 post(s)
#10-Jun-22 07:57

Mostly, yes. I'd like this CALL to finish quickly also, but not being able to cancel is worse.

rk
518 post(s)
#10-Jun-22 08:06

My %temp%\*.tmp files related to this M9 session, has grown to 172 GB. and keeps growing.

Performance monitor shows many processes reading C:\$BitMap (NTFS Free Space Map), that I didn't see before.

There is still 100GB free on my C:\ (NVMe SSD 500GB for system, pagefile, %temp% - basically all except *.map, *.laz, etc. )

rk
518 post(s)
#15-Jun-22 12:19

I shoud probably clarify what is and what isn't excessive disk usage in this particular case. Because one can call this wonderfully excessive when M9 first reads copc-library and creates %temp%/*.tmp files for each .laz. The speed at what those files are created is astonishing 2+GB/s. Temp size grew to about 40GB for library of 909M points.

On the other hand, when performing spatial filter over copc-library, then the growth of %temp% size was as big in size but much slower. 2h from the beginning of GeomOverlayTouchingFilterPar 1 .tmp file had grown to 59GB.

Attachments:
manifold-copc-library-read.png
temp_59gb.png

adamw

10,011 post(s)
#15-Jun-22 15:32

I will just say that we are currently working on several improvements relevant to this thread. They will appear in the next build, which we will likely issue this week. Let's revisit the thread after this next build.

rk
518 post(s)
#20-Jun-22 10:06

I did not experience speedup but slowdown with latest build. I also see display artefacts [attached image].

--SQL9

-- 909_504_205

VALUE @drawing TABLE = [copc]::[copc Drawing];

-- 10401

VALUE @overlay TABLE = [Sections Buf];

DELETE FROM [filtered_points];

INSERT INTO [filtered_points] (

  [mfd_id][X][Y][Z][intensity][gps_time][Path],

  [Geom]

SELECT

  [mfd_id][X][Y][Z][intensity][gps_time][Path],

  [Geom]

FROM 

 CALL GeomOverlayTouchingFilterPar(@drawing, @overlay, 0, ThreadConfig(SystemCpuCount()))

;

--  (11478.219 sec) filter 56_448_352 points with 10401 small areas from COPC library of 1324 files, 909_504_205 points  9.0.176.5  40GB + 60GB of %TEMP%

--  (32858.475 sec) filter 56_448_352 points with 10401 small areas from COPC library of 1324 files, 909_504_205 points  9.0.176.6  60GB of %TEMP%

~1.5h to build 1 temp file of 60GB and rest to insert filtered records

Attachments:
copc-display-tiling-1.png

adamw

10,011 post(s)
#20-Jun-22 16:31

Thanks.

That's likely because you are filtering with so many areas. Previously, we were unpacking each LAZ and this was pretty terrible in terms of disk pressure / straight time to unpack, because we had to unpack everything including parts of the data that the query was not interested in, even with COPC. But this was only done once per file. Now we unpack on the fly and just the parts that we need. But since you have 10401 areas, this means unpacking small parts of the file 10401 times, and for areas that are close together, this unpacking is repeated work. Are the areas disjoint or are they adjacent to each other? If it is the latter, what happens if you join 10401 small areas into one? (Maybe try with like 20 files in the library to get a sense of whether this was a right guess without waiting for hours.) If they are disjoint, what happens if you filter with just 10 areas instead of 10401? (Same suggestion to try small.)

Overall, unpacking parts that are needed on the fly vs unpacking everything more or less on first access is a conscious trade-off. Because if you want to work with unpacked data, you can just import data into a MAP. If we find out that unpacking everything on first access is a better strategy in your case, that would mean we have to expedite adding a specialized index for point cloud data to MAP files.

rk
518 post(s)
#20-Jun-22 17:21

Thanks.

Areas are mostly disjoint, less than 10% have some overlap.

I got some ideas how to optimise. With little work I can group them roughly by individual LAZ files.

I did't want to copy into MAP more points than I needed. Maybe I should try. Temp/disk usage is anyway high.

adamw

10,011 post(s)
#21-Jun-22 14:43

In general, if you take an example LAZ and all areas that touch it, how big is the bounding box of these areas? Because that's what is being optimized for. If the areas are small, but their common bounding box covers, say, 80% of the LAZ, 80% of the LAZ will get uncompressed. I figure that with a library, many files get covered 100% simply by being in the middle. Maybe you have a scenario which calls for a different optimization.

Manifold User Community Use Agreement Copyright (C) 2007-2021 Manifold Software Limited. All rights reserved.