What I'm trying to accomplish:

As the title suggests, I have a large DEM raster that is approximately 100 GB and am trying to clip it to fit a large, complex vector. I have been using QGIS, which can accomplish this using GDAL_warp. But it's very slow, and generally takes 14 - 20 hrs using the command line. It also has an annoying tendency to fail if there are too many features (lots of island contours, like 50k+). When searching for ways to improve this I found a lot of people on reddit saying "Use manifold". So I bought it and am trying to get this to work, which is where I need some help.

What I've done so far:

I followed the steps outlined in this tutorial youtube video here: https://www.youtube.com/watch?v=LZdeQW3nkiI&list=PLC3vkyPx3gUuUGjO5hbd-Fks9r3Y6IIYQ&index=96. Works just fine at small scale.

Problem I'm having:

At small file sizes, this works fine, but it's multiple steps while using GDAL is only a single step. I've tried using the exact same steps for a large, 100 GB raster and a large vector with 53,364 features. The step where you join the whited-out image with the "Mask = 0" field from the vector is not working at this scale. It has processed 53,362 features (records) in about 10 minutes. But it's hanging on the last 2 features. It has been stuck on record # 53,363 for the last 13 hrs. All features/records are geometrically valid and non-overlapping. This has been exhaustively tested.

What I'm looking for:

Is there a more streamlined way to clip a large image to a large, complex drawing? I got this program essentially for this sole operation. It's clearly very powerful but I have zero SQL knowledge and the documentation is a little dense. So I apologize if there is an easy answer I've missed. Thanks in advance.

unfortunately, Manifold 9 does not perform a classic mask routine like you would see in QGIS or ArcGIS. Manifold 8 can do it, but I don't know if it would be any faster/slower than using QGIS. If you want to contact me offline, I can try it out using 8 and see if it would work. I could also give it a shot in ArcGIS Pro just to see how fast that goes.

with GDAL, presume you're using gdalwarp and "-cutline", make sure to set "-crop_to_cutline", "-multi", and "-wo NUM_THREADS=[ncpus/ALL_CPUS]" , also "-wm" might help, and depending on output format use "-co" to set GeoTIFF creation options.

https://github.com/mdsumner

This is a fast operation in Manifold 9, but yes, you do need to learn Manifold SQL9.

(Take the time. It is an investment. If you or your work are not worth it, then don't.)

The workflow:

Make a new image from the mask drawing. The resulting image should contain two values, one meaning "blank", the other meaning "preserved".

It should have exactly the same projection and dimensions as the image you want to mask. (That can be tricky, but is the most crucial requirement.)

Then you apply one image to the other, not using a conditional expression (slow) but using multiplication (very fast), to make a new filtered image.