Subscribe to this thread
Home - General / All posts - Problem merging two surfaces
Sloots

678 post(s)
#14-Jan-15 15:29

Hello,

I have two surfaces (actually a lot more, but I try to merge the bottom left and the top right of the whole bunch). If I copy the first and paste it into the second Manifold asks me if I want to extend the surface to fit the combined surface. I have done this before, but this time the size of the surface after expanding should be something like 60000 x 70000 pixels. But Manifold seems to allow only surfaces of 32760 x 32760 (see image). [Since this is almost the same as 2^15, the max of a signed int I'm always a bit suspicious!]

I'm sure it's not the maximum size of a surface, because I can easily create one that is much bigger.

Any suggestions are welcome.

Chris

Attachments:
maxsurface.png


http://www.mppng.nl/manifold/pointlabeler

cartomatic

905 post(s)
#14-Jan-15 15:55

some ideas:

create a surface with needed size and coordsys first and then paste into that surface

or get the x,y,z off all surfaces into a table component, copy it and paste as a surface


maps made easy - www.cartomatic.pl || www.cartoninjas.net

oeaulong

521 post(s)
#14-Jan-15 16:15

I am sure we need more detailed information on your attempt, however in my experience 60000x70000 in a surface is Far too big to be thought workable by any machine we may be using. Surfaces of any size require muuuuch more processing and temp storage than images of the same size.

Like you mentioned, the numbers look familiar of 15 bit (16 actually considering the leftmost sign bit.) 32760^2 = 1073217600 'pixels',cells, whathaveyou. Each of these are multiplied by the storage size of the cell datatype, say a single float of 32bits, or a 32-bit integer. The result, Just To Store The Surface, is 34.343 Gigabytes.... huge. The addressability of that much space, even at manifold's limit is toooo much to easily handle. This is without the notion, of say, the copy/paste, duplication (even temporarily) of portions of the surface, the arithmetic manipulations needed either in the cpu part or the apu part (int ops or float ops) of the chip become a big issue. The temporary files needed to be generated by manifold to process this will take forever to create even if you had plenty of space. This is the current limit of manifold's storage and processing of such a raster. All this is ignoring the size of your .map file, the other apps & services running on your OS, as well as the capabilities of your hardware.

I believe you will need to re-tune your thinking on processing surfaces at a tiled/subtiled level rather than the 'convenience' of a single source layer.

please see: http://www.georeference.org/doc/performance_tips.htm

and http://www.georeference.org/doc/problems_with_performance.htm

I know the numbers/references in these two manual pages will look out of date considering current hardware configurations, and much and more processing has been able to be achieved with CUDA enhancements since ver.8 was released in 2007. However, these are still what we need to deal with. I may have been incorrect on some of my numbers above, if so, i expect to be corrected. You may get better help from other community members if you can show what you are doing, why you feel the need to use surfaces of such large sizes, and what your goal is. It *may* be helpful to get a sense of the level and specs of your hardware in addition.

Oeaulong

Gustavo Palminha

1,010 post(s)
#14-Jan-15 17:58

I believe that a surface with those dimensions will not be "workable" but I would follow cartomatic tip or try to involve a DBMS like PostgreSQL (with postgis) on such task.

I believe that you can then export your surface to postgres as a component and link it back to manifold but I do not have a manifold enabled workstation to try it out.

Good luck.


Stay connected:

Linkedin >> http://www.linkedin.com/in/gustavopalminha

Twitter >> http://twitter.com/gustavopalminha

Other >> http://www.spatialmentor.com

tjhb
10,094 post(s)
#14-Jan-15 21:10

32760^2 = 1073217600 'pixels',cells, whathaveyou. Each of these are multiplied by the storage size of the cell datatype, say a single float of 32bits, or a 32-bit integer. The result, Just To Store The Surface, is 34.343 Gigabytes.... huge.

Overstated by roughly an order of magnitude. 32760^2 = 1,073,217,600. 1,073,217,600 * 32 bits = 34,342,963,200 bits. 34,342,963,200 bits / 8 bits per byte = 4,292,870,400 bytes. 4,292,870,400 bytes / 1024^3 = 3.998 GB.

Manifold can happily cope with that.

60000 x 70000 pixels (of type single) -> 15.64 GB. That's quite big (for now).

There are some hard limits in Manifold (for now). The main one is the limit on the size of a virtual table for an image or a surface, which is currently 4 GB. That means we can address a surface of 32760 x 32760 pixels in Manifold SQL if it uses single-precision values, but not if it uses double-precision. And for a 60000 x 70000 pixel surface, SQL will be available only if the surface uses 8-bit integer values.

(Manifold has allowed table components to exceed 4 GB since version 8.0.19, but if I'm not mistaken the relaxation did not extend to virtual tables. There are apparently some similar limits on the size of query tables per se, which danb and I looked at last year, without getting to the bottom of the matter.)

If you need to manipulate really large surfaces, use a Surface Transform rather than SQL. Surface Transforms can cope with very large sizes (I don't know if there is an effective limit).

Is there a link between the 4 GB limit on virtual tables, and the hard 32760 x 32760 pixel limit Chris has discovered for surface enlargement via the clipboard? Maybe Manifold uses the same data structures for abstracting a surface in both cases?

tjhb
10,094 post(s)
#14-Jan-15 22:33

A clarification to this paragraph:

There are some hard limits in Manifold (for now). The main one is the limit on the size of a virtual table for an image or a surface, which is currently 4 GB. That means we can address a surface of 32760 x 32760 pixels in Manifold SQL if it uses single-precision values, but not if it uses double-precision. And for a 60000 x 70000 pixel surface, SQL will be available only if the surface uses 8-bit integer values.

SQL is available for very large surfaces--for some purposes. In particular, you can usually write an UPDATE query to change surface values. What you can't do (successfully) with SQL is address the virtual table for a surface (or image) if the virtual table would be over 4 GB in size.

For example, for a single-precision surface 32760 x 32760 pixels [added: or 32767 x 32767 pixels], a query like this will succeed:

SELECT [S].[Height (I)][P].[N]

INTO [Test]

FROM

    [Surface] AS [S]

    CROSS JOIN

    (VALUES (1) NAMES (N)) AS [P]

;

But for a single-precision surface 32768 x 32768 pixels, the same query will always fail (after processing for 10 minutes or so), with an "unknown error".

(I know, it's a useless query. But it forces Manifold to build a virtual table for the whole surface.)

Manifold User Community Use Agreement Copyright (C) 2007-2021 Manifold Software Limited. All rights reserved.