I am trying to open a large Tif image file - 6GB. When I try to open it I get the error message - Can't open file. i assume its due to the limitations of my system but I thought I would have enough space etc. I have 32GB RAM 1.5TB space on one hard drive and 148GB free on my SSD where manifold is run from. Is there anything I can do to be able to open this ? Any suggestions? Thanks

I don't think Mfd8 supports bigtiff format only normal tiffs which are limited to 4GB I think.

Landsystems Ltd ... Know your land | www.landsystems.co.nz

Can you run gdalinfo on it to get a summary? It's the dimensions that matter, more than 2^31 - 1 pixels in one band and it has to be BigTIFF, which M8 does not support. The file size is not necessarily relevant, because of possible compression, and the actual data type in the pixels.

But at any rate, Manifold on a bigger machine *might* open it. If you get GDAL utilities on your system you can do just about anything you need with it so that Manifold can read it, perhaps in parts.

https://github.com/mdsumner

Post a link to where it can be downloaded so people in the beta can take a look at it with Radian. 6 GB is trivial for that.

About the beta. I would like to join it but am not sure if it is closed. I sent an email early on but was told that perhaps later.

How soon?

So, it's later! You could try again.

Sorry, whats GDAL? And what is beta?

GDAL is an open source image translator utility (amongst many other things) Use your usual search to find more.

Info on the beta is here

What is this problem image Ian?

Is it publically available? Do you want help converting it?

Hi Tim

No its not publicly available. I need to convert it to a ECW but I can't import it - I think because it to big for manifold? I have also been given a copy as an ige but manifold doesn't use that format that I can see. It will import as a img but is just black rectangles to imagery. For some reason the council that are supplying the image can't export it from ArcGIS as an ECW.

Well, what do you want to do?

Manifold 8 can't import it in its current format. Future versions are currently no use to you. As Mike has mentioned, the GDAL utilities would allow you to convert it, but you might not be comfortable using them. So would Global Mapper, if you have/had a licence.

What about the current versions of QGIS? They use GDAL internally. Are you keen to give that a go?

If not, are you able to post the BigTIFF file somewhere online (privately) for someone else to convert?

We're not going to get far if you don't reply.

I've dealt with similar issues recently by opening in QGIS and saving as two smaller tifs that Manifold could open. Presumably I was using GDAL without knowing it.

I can easily import the image into QGIS - have just done it and zoom in and out with the picture barely blinking and that on my old computer that struggled with much smaller images in manifold. I can see how to convert it to the ECW but I don't know all the details or script that are needed - can anyone help me here?? The image I'm working with is over 5GB and as I have said this program handles it with no problem at all. I am currently working with a linked 1.5GB ECW image on my new computer in manifold and its pretty slow. Why the big difference?

When I try to convert the image in QGIS I get this message

RROR 6: GDALDriver::Create() ... no create method implemented for this format.

I haven't changed any of the boxes only the output format

I think because it to big for manifold?

Hard to say without knowing details about both the image and the computer system. You don't say, for example, what Windows you are running and whether it is 32-bit or 64-bit, a pretty big difference, nor do you say what version and build of Manifold you are running and whether it is 32-bit or 64-bit either.

Release 8 does not read big tiff but we do not know if the image is big tiff. We are just guessing it might be without knowing important details about the image. Besides big tiff, another possibility is it could be something as unexpected (but fatal) as a limitation on TEMP space. Or it could be some special peculiarity in the TIF otherwise not supported in 8 (TIF is a very wide format and very few packages, if any, support all possible TIF content). Or you could be running release 7 or a have not updated to the current Release 8 build but are running with some prior build where some TIF problem was not fixed. I assume you are using 8 but that is not an assumption tech support would make.

I don't think it is the above but you never know, which is why when tech support gets such questions the first thing they do is insist on getting a list of basic info.

My own experience with 8 is that if you are running 64-bit with plenty of resources and nothing artificially limiting (like a hard limit on TEMP size despite having plenty of free space...) it is amazingly durable. With patience you can work with some pretty big images. I personally have created raster images over 40GB in 8, for example. I'm not sure how long those took because I started the task at the end of the day and when I looked at the computer the next morning it was done.

But, if your image is in a format not supported by 8, well, then it is not supported. Could the provider tell you if it is a big tiff or not? Could you tell us more about the image, such as the dimensions, how many bits per pixel and stuff like that?

It will import as a img but is just black rectangles to imagery.

I suppose that is a good sign regarding 8's ability to handle the size of the image. Was that ENVI IMG orERDAS IMG?

It's a shame we can't try the image. With the data in hand an instant solution would probably be possible (read it in Radian, write out an ECW...). :-)

I am running windows 10 which is up-to-date, manifold 8 which is also up to date, I have put my computer specs in the first post. Not sure how to check my temp space I thought with 148GB space on my SSD that would give it plenty of room?? How do I check how many bits per pixel? I can roughly work out the number of pixels by the resolution of the image and the area it covers (approx 25,000 x 18,000) then divide that into the size of the image (3.4GB) which comes out at just over 8 bits per pixel. The file format from what I can see is just a Tif. The other format I have the image in is an IGE. There is no option in manifold to import this format so I tried the IMG option and that's when I got the black rectangle. Am quite happy to leave the computer running to get the file into an EWC format but I can't get past the Can't open file error message. Is this something I should go to tech support with?

I am running windows 10 which is up-to-date, manifold 8 which is also up to date, I have put my computer specs in the first post.

No, not a word about 32 bit / 64 bit in that first post. Let's try again:

You don't say, for example, what Windows you are running and whether it is 32-bit or 64-bit, a pretty big difference, nor do you say what version and build of Manifold you are running and whether it is 32-bit or 64-bit either.

The specs you posted don't say if you are running 32-bit or 64-bit nor does saying windows and 8 are up to date tell us if either windows or 8 are running 64 bit. That's a very big deal. Also, if you don't tell us the build of 8 we can't double check that it is up to date.

Sorry missed that bit, windows and manifold are both 64 bit. I have manifold personal addition Build 8.0.29.0

So to recap...

You have Windows 10 64-bit, Manifold 8.0.29 64-bit, 32 GB RAM, an SSD with 148 GB free (which Manifold is run from; I think this is the OS drive) and an HDD with 1.5 TB free (I think this is a data drive). So far we don't know which drive the TEMP folder is on, but let's guess it's the same SSD as the OS, with 148 GB free. 148 GB is not a lot of free TEMP space, but it's probably enough to open one moderately-sized Manifold project.

(To check where your TEMP folder is in Windows 10, right-click on the Start button and select System. Then select Advanced system settings on the left to bring up System Properties, Advanced tab. Then click the Environment Variables button at the bottom, and look at the value of the TEMP user variable in the upper pane. There other ways.)

It doesn't look as if resources are a pressing issue--subject to TEMP space, when Manifold is actually in use.

The image is in at least two formats: (a) a TIFF image, described as having a file size of 6 GB or over 5 GB, or an image size of 3.4 GB (see below), and (b) an IGE image (no file size given).

To me, (a) looks definitely like a BigTIFF file (so long as it is over 2 GB). Manifold 8 can't open that.

Format (b) looks like an ERDAS IMG file (there should be an IMG file beside the IGE, along with a header file and metadata), with overflow into an IGE auxiliary file for a large image. Manifold 8 can read ordinary ERDAS IMG files, but it can't read IGE auxiliary files. For this file(set), Manifold attempts the import, but the result is apparently all blank. (That would make sense.)

QGIS can open the image. In which format(s)? TIFF or IGE, or both?

Attempting to export from QGIS to ECW fails with a GDAL error.

You've said the image is approximately 25,000 x 18,000 pixels, but not how you've measured this. E.g. in Manifold (the blank rectangle imported from IMG/IGE), or in QGIS, or from metadata.

You've said the "image size" is 3.4 GB, but this is much smaller than the "file size" of 6 GB or over 5 GB. Where are you getting the image size measurement from? I suspect it is taken from the blank image imported into Manifold from IMG/IGE format, read from the Project pane. That would be just the size of the failed import.

There are some holes above but perhaps the main things we don't know are: what the image contains (a picture or elevation), who supplied it, how you took delivery, what you want to do with it. These basic things should come first!

And then: how many channels the image has, how many bits per channel, and what is the native projection of the image.

The image is an aerial photo that is georeferenced and ortho corrected. It has been supplied by the regional council gis department. i need to use it for making a farm paddock map. I have three versions of the image. the largest is a tif of around 6gb - have taken this from the file details, the second is also a tif and is a clipped version of the first one and is approximately 3.5gb the third is the ige. The imagery was downloaded frm Dropbox or copied on to a hard drive. It is in nztm projection. am not familiar with channels. I will check the temp file when im back in the office on Monday. Qgis can open and use the tifs but not the ige

Excellent, thanks. That makes things much clearer.

Option 1: convert the BigTIFF image to PNG format, then import the PNG image to Manifold. (Manifold can read very large PNG files.) Then assign the correct projection, including local offsets and scales. (From memory, Manifold does not read .pgw world files, so you'd have to read these manually.) Then (if you like) export to ECW and link back in.

Option 2: convert the BigTIFF image to ECW format, then link into Manifold 8.

For either PNG or ECW, I would use Global Mapper. Purchase costs $US499, but it pays for itself very quickly (and over and over). You can request a 2-week trial installation, which allows 4 exports (only). That would do it.

For PNG, you can the GDAL interface in QGIS. (Or GDAL itself if you want to go there.)

You've already tried QGIS for conversion to ECW. The error message you got is because writing ECW files (above 500 MB) requires a licence. (Manifold and Global Mapper have built-in ECW licences, QGIS and GDAL do not.)

Perhaps the simplest thing is to try exactly the same thing (QGIS: Raster > Conversion > Translate), but using PNG as the output format (again with default options).

It's possible you'll be blocked by an annoying libpng error, but it's worth a try.

Purchase costs $US499,

Ouch. Another plan is to ask if someone in the Radian beta or with access to GM could convert it for you, posting your email address in your profile to enable an off-forum contact. The usual deal with even restricted rights images is that you are allowed to have your employees or contractors or volunteers work on them.

Yes ouch, moreso since the last two annual releases (17 and 18) have dropped the former 1+1 licensing (similar to Manifold 8), where the same user could install a second instance of a "fixed licence" on a portable computer for his/her use. That's gone now. Annual version upgrades cost $US199.

It's not nothing, but certainly an investment that pays for itself in work time.

Global Mapper is complementary to Manifold 8. Will that still be true when Radian Studio is released? They may find they have to think harder about their pricing (and licensing). Even moreso when Manifold 9 is available--Global Mapper has no ambition to be a full GIS, and if Manifold 9 can do all of the "Swiss Army Knife" stuff that Global Mapper can do (and visualization?) then... a no brainer.

They have put a lot of work into their LiDAR analysis add-on module--and into making Global Mapper access and display unlimited point clouds extremely fast (in 2D and 3D). The add-on module is an additional $US499 (also subject to annual upgrades), cheap in the general scheme of LiDAR software. I've had no reason to purchase it so far, though that could change.

Two more things to consider for those who haven't tried out Global Mapper for a while. GM version 18 has had substantial rewrites under the hood to improve multiple threading and parallelism (previously patchy). Not on the same level as Manifold parallelism by any means, but useful.

Equally important, version 18 finally sees GM showing a decent user interface to the world, also a lot of new work. Previous versions felt very Heath Robinson--you had to know exactly where to look in order to find anything (the answer was usually "right-click here"). Now it's reasonably intuitive. (Though the colours are louder than a tropical parrot.)

Will give your above suggestion a go. Alternatively do you have access to another program that could do the job and we could arrange it that way as we have discussed previously?

Yes I could convert it, using either Radian Studio or Global Mapper (or both). Easy peasy no trouble.

Will send you an email

I found my temp folder and it is empty. Is that normal? From what I read on manifold I thought you have to manually clear it out from time to time?

The user TEMP folder is not usually empty, but it can be. That's not necessarily a problem.

If you used the procedure I mentioned above to find the user TEMP folder then, well, it's empty.

Ian has sent me a link to his original TIFF files.

Opening them in Manifold 8, Global Mapper 18, or Photoshop CS6 results in a blank (black) image.

Opening them in QGIS 2.18 shows a normal image. Immediately, with no problem at all.

A PNG exported from QGIS to PNG then shows in Manifold 8, Global Mapper 18, Photoshop CS6 as a blank (black) greyscale image, RGB in Photoshop.

Windows says the exported PNG has bit depth 64.

For now, baffled.

Can you run gdalinfo and list the output?

https://github.com/mdsumner

Thanks Mike.

gdalinfo.exe "****" >

Driver: GTiff/GeoTIFF

Files: D:\Downloads\****\****.tif

       D:\Downloads\****\****.tif.ovr

       D:\Downloads\****\****.tif.aux.xml

Size is 26059, 22370

Coordinate System is:

PROJCS["NZTM",

    GEOGCS["GCS_NEW_ZEALAND_2000",

        DATUM["NZGD2000",

            SPHEROID["GRS80",6378137,298.257222101]],

        PRIMEM["Greenwich",0],

        UNIT["degree",0.0174532925199433]],

    PROJECTION["Transverse_Mercator"],

    PARAMETER["latitude_of_origin",0],

    PARAMETER["central_meridian",173],

    PARAMETER["scale_factor",0.9996],

    PARAMETER["false_easting",1600000],

    PARAMETER["false_northing",10000000],

    UNIT["metre",1,

        AUTHORITY["EPSG","9001"]]]

Origin = (1695848.124999617000000,5986994.687499603300000)

Pixel Size = (0.312500000000000,-0.312500000000000)

Metadata:

  AREA_OR_POINT=Area

Image Structure Metadata:

  INTERLEAVE=PIXEL

Corner Coordinates:

Upper Left  ( 1695848.125, 5986994.687) (174d 4' 0.94"E, 36d15'25.96"S)

Lower Left  ( 1695848.125, 5980004.062) (174d 4' 4.03"E, 36d19'12.81"S)

Upper Right ( 1703991.562, 5986994.687) (174d 9'27.22"E, 36d15'22.92"S)

Lower Right ( 1703991.562, 5980004.062) (174d 9'30.57"E, 36d19' 9.76"S)

Center      ( 1699919.844, 5983499.375) (174d 6'45.69"E, 36d17'17.89"S)

Band 1 Block=128x128 Type=UInt16, ColorInterp=Gray

  NoData Value=256

  Overviews: 13030x11185, 6515x5593, 3258x2797, 1629x1399, 815x700, 408x350, 204x175

Band 2 Block=128x128 Type=UInt16, ColorInterp=Undefined

  NoData Value=256

  Overviews: 13030x11185, 6515x5593, 3258x2797, 1629x1399, 815x700, 408x350, 204x175

Band 3 Block=128x128 Type=UInt16, ColorInterp=Undefined

  NoData Value=256

  Overviews: 13030x11185, 6515x5593, 3258x2797, 1629x1399, 815x700, 408x350, 204x175

(GDAL 2.4.2, using GISInternals x64 release 1800 dated 20161119 104348.)

Looks like ArcGIS has written incorrect ColorInterp values here? And QGIS doesn't care?

I think the ColorInterp is "fine", that in particular is a limitation of GeoTIFF:

https://lists.osgeo.org/pipermail/gdal-dev/2011-April/028451.html

You can use -stats to get summaries of the actual band values.

https://github.com/mdsumner

Opening them in Manifold 8, Global Mapper 18, or Photoshop CS6 results in a blank (black) image

Very interesting. What happens when you try to import it into Radian?

It's telling that PhotoShop CS6 opens it as a blank image, since Adobe is the publisher of the TIFF 6.0 format, with which GeoTIFF is said to be fully compliant. As Wikipedia puts it:

The GeoTIFF format is fully compliant with TIFF 6.0, so software incapable of reading and interpreting the specialized metadata will still be able to open a GeoTIFF format file.

Reading the standards it seems ColorInterp should not be the problem.

Searching theTIFF6.pdf standard published by Adobe (the makers of Photoshop CS6) there is no mention of ColorInterp. If what Wikipedia says is true, then if the file in question is a correctly written GeoTIFF file it still should be compatible with TIFF 6.0 with no problems caused by anything not part of TIFF 6.0.

But then ColorInterp is not part of GeoTIFF either: Visiting the official GeoTIFF standard site at https://trac.osgeo.org/geotiff/ if you search the entire site for ColorInterp (and also do the same for the "official released standard site" at http://web.archive.org/web/20160403164508/http://www.remotesensing.org/geotiff/spec/geotiffhome.html )

... there is no mention of ColorInterp in the GeoTIFF standard.

It sounds like the file is something that GDAL wrote which now GDAL can read (QGIS uses GDAL) but which may not be fully correct TIFF or GeoTIFF.

Adobe, in particular, is usually pretty good about honoring their own standards so if CS6 cannot read it that is a strong indicator there may be something not quite standard about the file.

Thanks Dimitri, that's ringing some bells. I'll have to check this tomorrow, but my hunch is that UInt16 is out of TIFF spec, Should be Int16 (signed).

(And yes, Radian Studio also shows blank images.)

Why would these be signed? Why would unsigned be "out of spec"? I feel there's a key point here that Nick used to make, that I could never fathom. If the spec explains, sure let's read it but reasoning about information and code can be enough. If these data are right for their purpose, should a different format have been used so that Manifold's slavish adherence to principles would have made it all work perfectly without any user input?

There are no formats that suit my work, so we do what is needed, hence my wondering here.

https://github.com/mdsumner

I'm still checking the specs, but here is a better report and a practical result.

Some of what I reported earlier was wrong. Fixed below.

To recap: the original GeoTIFF imagery is RGB (no alpha), with each channel encoded as UInt16. One image is 3.26 GB (26059 x 22370 px), the other 5.54 GB (40192 x 24591 px). Both are in BigTIFF format (though the smaller image does not need to be). Neither file has compression.

Manifold 8 can't read either image ("Can't open file"). I think the blockage here is BigTIFF format (see below).

QGIS reads both files correctly with no intervention. It applies sensible rendering and colour options in the Style dialog automatically.

Global Mapper 18 sees a 3-band image, 48 bits in total. With sensible manual settings in the Color/Constrast adjustment options dialog (linear min-max stretch looks best) the image looks good.

Photoshop sees 16-bit greyscale images, a Gray channel plus two alpha channels. All 16-bit values are between 0 and 128 (so the image appears black; this also shows that there was no need or value in encoding the images in 16 bits).

This isn't the ideal place, but since Dimitri has asked, Radian Studio beta reads both images, with type uint16x3, but computing statistics seems to show a value range of (0, 0) for all three channels. More testing and discussion elsewhere perhaps.

For correct display in Manifold 8, I needed to convert to 8 bits per channel. In QGIS this was easy: invoke Raster > Conversion > Translate (interface to GDAL), specify a new GeoTIFF output file, and add "-ot Byte" to the gdal_translate parameters.

Manifold 8 could also read a version converted in QGIS to (signed) Int16, which it imported as both an RGB image and as 3 separate 16-bit surfaces. It could also read a direct rewrite of the smaller image from Global Mapper, with no changes (still UInt16), except that GM did not use BigTIFF for the rewrite (since unnecessary). In both these cases, image colour in Manifold showed a strong Cyan cast.

Why would these be signed? Why would unsigned be "out of spec"?

I was clearly wrong about this, roughly 180° wrong. Well, it was the middle of the night.

The TIFF6 baseline specification states that unsigned integers are the norm, though it does allow for signed integer data, floating-point data, and undefined data formats (section 19, Data Sample Format), and warns

Beware of new pixel types. Some TIFF files may have pixel data that consists of

something other than unsigned integers. If the SampleFormat field is present and

the value is not 1 [unsigned integers], a Baseline TIFF reader that cannot handle the SampleFormat value must terminate the import process gracefully.

The standard even allows for different bitness in different channels, even for RGB images:

BitsPerSample

Note that this field allows a different number of bits per component for each

component corresponding to a pixel. For example, RGB color data could use a

different number of bits per component for each of the three color planes. Most RGB

files will have the same number of BitsPerSample for each component. Even in this

case, the writer must write all three values.

For what it's worth, here is what I thinking of (and confused about). This is from the Photoshop CS6 Plug-ins SDK.

3.3 Plug-ins and 16 bit data 3.3.1 Why does my 16 bit plug-in only see values of 0 - 32768 and not 0 - 65535?

Photoshop uses an internal representation of 16 bit data of 0 - 32768. When acquiring, importing, reading, or writing files check or set the maxValue in the parameter record. Filters will also see this range for 16 bit data.

3.3.2 Why does Photoshop use such an odd range?

Speed, speed, and more speed. First of all this range keeps more of the math in 32 bits, avoiding heavy use of 64 bit math, which is still slow on todays machines. Second of all you can use shifts instead of costly divide operations, speeding up the 16 bit version of our routines by quite a bit. Another feature of using this range is that there is a true center, unlike 2^N-1 representations, which gives us significant improvements in many of the blending routines.

(My italics. Showing Photoshop's age.)

I was somehow thinking that allowing negative values was a necessary corollary of this range-halving. It isn't. Anyway, I've posted the extract because it is interesting.

Thanks Mike.

If these data are right for their purpose, should a different format have been used so that Manifold's slavish adherence to principles would have made it all work perfectly without any user input

I could be I misunderstand the above, as I doubt you intended to suggest that good data becomes more useful if standards are disregarded in the publication or use of that data. Likewise, it seems wrong to think you intend that Manifold, Adobe and others (note I used the example of Adobe and their publication of TIFF 6.0...) should ignore technical standards because paying attention to standards would be "slavish adherence to principles."

To get back to basics:

The whole point of having standards is the common sense utility they provide. Take a standard like a country deciding to drive on the left side of the road or the right side of the road. Do you want drivers in that country to have a "slavish adherence to principle" in respecting that standard? I don't know about you, but I sure do. The last thing I want is to be driving around in Calais and have to worry about some Brit running into me head on because he wants to prove that maybe others, but not him, are too slavishly adhering to principles. :-)

Nobody is forced to use standards. Whether or not somebody chooses to gain the convenience and common sense utility that a standard brings is entirely up to him. Nobody is holding a gun to the heads of the folks at Adobe, GDAL, Global Mapper or anybody else to say they support either TIFF or GeoTIFF, and nobody is forcing any user to seek out TIFF files with the expectation that doing so would be a wise move to gain the benefit of software packages which say they support TIFF.

But if a package says it supports TIFF or GeoTIFF it is fair game to trust the authors are not lying to you and that when they say they support TIFF they mean that in the way grown ups of good faith mean it: they support the standard as written. It is fair game to expect that and to *not* think that when they say they support TIFF they really mean "well, I don't exactly know what I mean but in case I get it wrong I have this lawyer here who will come up with a hundred excuses why it is OK for me to say something that sounds cool for me but which in technical reality is total nonsense..."

Users have to be able to count on a package telling the truth when says it supports a standard. If they use Global Mapper to write what Global Mapper claims to be a TIFF by dang that file better conform to what the TIFF standard says. If it doesn't, it ain't no TIFF. Global Mapper is a high integrity group so they 100% agree with that. Part of their integrity is that if they say they do something they're going to attack and fix any discrepancies to ensure they do what they say.

The reason they do that is not because of any sort of slavish devotion to "how many angels can dance on the head of a pin" abstraction nonsense not of interest to anyone, it is because they know perfectly well that if they fail to honor a standard that costs users who trusted them to honor that standard lots of wasted time. That's a mark of their integrity and good for them. Adobe is the same way and so is Manifold.

The file in question is said to be a genuine TIFF. If that is true, than any software which honors the TIFF standard should be able to read it. If it is a TIFF and some package cannot read it, then the correct solution is not to ask the publisher of the file to write a deformed something which is not quite a TIFF to get around a bug in a software package, it is for the package that fails in its obligation to read TIFF correctly to fix whatever bug is causing the problem.

Conversely, if the file in question is not really a TIFF but departs from the standard in some way, the correct solution is not for all the packages which correctly read and write TIFF to start ignoring the standard, it is to fix the file so that it is a genuine TIFF.

The point is a purely technical one driven by the common sense utility of standards. The standard is the touch stone that helps grown ups of good faith coordinate their actions to elevate the common good of all. People who cheat the standard, whether out of laziness, inattentiveness or technical inability, and then insist everyone else accommodate them are not acting in good faith and are acting against the interests of the common good. It's like the guy who flies in from the US to England and then starts complaining it is everybody else's fault he can't drive on the side of the road to which he is accustomed.

So far we as yet do not know what the deal is with this particular file. TIFF is a very broad standard that allows for many variations, some of which are rarely met. This particular file could be a perfectly legal TIFF file that Global Mapper and Adobe and Radian for whatever reason, possibly different reasons for all three, cannot handle (I assume, perhaps wrongly, it is BigTIFF and thus a format which 8 does not claim to read...).

Or, it could be malformed in some way or another and thus the focus on repair should be on the file and whatever package incorrectly handled the TIFF standard and not on those packages which read and write TIFF correctly.

As for just throwing hands up and saying, "well, I know it is malformed but the package which wrote it is widely used and so there will be many other files that are in error like this one... perhaps it would be useful to ignore the standard in this case." ... that is risky but it may be an unpleasant reality that is easier to adopt. It's like people in England who live near car rental outlets at international airports: they know that like it or not they have to be on their guard against foreign visitors who will be driving on the wrong side of the road.

In fact, there is precedent at Manifold: consider the widespread blunder within GPKG that fails to handle EPSG specs for XY or YX ordering in the case of codes such as EPSG:4326, a GPKG error that Radian dodges when reading GPKG by assuming wrong ordering, knowing full well that almost all GPKG files which claim to use EPSG:4326 do so wrongly. That's a terrible precedent but it is a useful, real-world reaction to the widespread failure by GPKG to honor the EPSG and OGC requirement to follow axis ordering when specified.

There is also precedent in how 8 routinely reads seriously malformed shapefiles (as common as breathing in GIS...) in order to extract whatever data can be saved.

So sure, if something as widespread as ArcGIS or GDAL routinely writes malformed TIFF files one could make an argument that an exception could be made for reading such malformed TIFF files. But to do that you first have to identify exactly what the issue is: is it a failure in the various software packages to read a valid TIFF or is the problem a malformed TIFF? If the problem is a malformed TIFF, how widespread is that problem and is it always the same deformity, or are there variations in ugliness that should be considered? If the problem is a malformed TIFF, is that a problem which will be fixed soon in whatever package is generating the malformed TIFF?

So far, this thread has not arrived at any conclusion on this particular file. It is absolutely possible that the TIFF in question is perfectly legal but that the various packages which do not open it as we would like are in error. Or, it could be the file, or it could be both an error in the file as well as errors in the various packages. If it is the file then we'd have to find out what package wrote the malformed file and do further research to see if that's something that's going to continue or if there is a bug fix in progress for that package.

I guess in closing I'd like to emphasize this is a totally routine deal: with hundreds of formats and hundreds of packages nothing could be more routine than dealing with situations where standards are not being honored. But the way to deal with that is not to throw out the standard, it is to patiently do the routine detective work and then bug fixing to ensure that everyone can continue to benefit from the common sense usefulness of standards.

It was meant to be complimentary in a backhanded way :0. I just think it's totally misguided to have rules about both how the data is stored and how it is to be presented, without one or the other being optional. I can't see how that makes sense ever. Like palettes, it would be great to have an *absolute palette* stored on a numeric raster, so that there's a nice default guide to how those data can be displayed, mapping intervals to colours discrete or continuous. But that almost never happens, it's data or it's colours, an obvious fact of life for efficient transmission.

TIFF just fundamentally cannot do both those jobs, but it's a brilliant container for raster values or for integers encoding some colour model. . That should be the end of it, and if there's a convention that says how software should interpret it, very nice but it also shouldn't get in the way. You can shoehorn this kind of stuff in a TIFF with subdatasets, i.e. as 1D arrays with palettes encoded and all kinds of trickery, but it's not very discoverable, or useable in software like Manifold

Are you saying the TIFF standard is supposed to tell you how to display the data? And in this case it doesn't? I only see you saying "should be able to read it", but I see nothing on "display it". We are drowning in software that can "read it" with no problem. We don't have many options for total control over displaying it, and so I wonder why you discuss it here at all. What exactly would the TIFF container be telling you in this case, if it was perfectly adhering to the standard? Or are you just talking generally to remind us how compliant Manifold is? "Malformed" is a big charge to throw around and you are merely suggesting it as a possibility as far as I can see, and so why do you do that, especially when no Manifold product can be used to confirm it?

https://github.com/mdsumner

Just to recap what I found out, in fewer words.

Everything except Manifold 8 could access these TIFF files. Manifold 8 only failed because of BigTIFF format (changing to standard TIFF where possible gave it access).

Only QGIS knew how to interpret the data for display, without intervention. With sensible manual settings, Global Mapper could also interpret it, Radian Studio should have been able to (but currently couldn't, which I'll follow up), Manifold 8 couldn't (it made a sensible try, but really needed an 8-bit RGB format instead), and Photoshop would need quite a lot of help (probably also amounting to a change of format).

I'm not sure what the remaining disagreement is about, if there is any.

I'm not sure what the remaining disagreement is about, if there is any

We do not yet have the technical info necessary to agree or disagree. It's a pity that for such an easily-solved, technical question we do not have the required info.

Precise info is always required to debug technical issues involving standards like TIFF, which is full of legacy hair. To show the sprawling nature of TIFF I've uploaded a zip file with the classic collection of ancient TIFF examples. It is pretty rare to find a package that can consume all of these, but all are perfectly legal and none are malformed. Get it at http://www.manifold.net/downloads/tiff_examples.zip

Packages which claim to read TIFF should be able to consume all of those files. Let's avoid any possible disagreement over the meaning of "to read" by using instead the verb "to consume" to mean "to read and usefully to display without intense additional user engineering".

What we know about the file that kicked off this thread is that a) it claims to be a TIFF, b) three respected packages, including that of the current publisher of the TIFF standard, Adobe, fail to consume it while only one package consumes it right off the top, and finally c) we do not know the actual content of the file, that is, we do not know whether it is malformed.

Point c) is a big deal: No professional tech support engineer would fail to begin the debugging process by determining if the file in question was malformed.

"Malformed," by the way, is a purely neutral technical word, not a political or religious word. It just means that the file does not conform to the standard it claims to honor. That's all. Malformed files are like malformed SQL or malformed expressions in whatever language you use. They are not valid inputs for processes that use the claimed standard.

For valid inputs I refuse to dumb down quality expectations for software packages: When an image is published in a standard image format, then a package which claims to consume that standard format should produce, by default, something that looks like an image, especially in the case of a file that apparently is just a visual image and not exotic remote-sensing data not intended for visual comprehension.

If that doesn't happen with such a popular format as TIFF and with programs at the level of Adobe, Global Mapper and Radian it is important to determine why there is a failure.

The three most likely situations are:

1. A malformed file.

2. A bug or documented limitation in the program.

3. Both 1 and 2 above.

It all starts with point 1 above. Until we know whether the file is malformed there is no way to determine whether the problems seen are caused by 1, 2 or 3 above.

Once you know then the usual non-political, purely-technical process can proceed.

For example, if it is 2) above then the engineering teams for the programs which fail to consume can swing into action and perform either a bug fix or document an explicit limitation for this particular form of TIFF.

If it is 1) above then likewise the engineering team for whatever program created the malformed file can produce a bug fix as well.

Either way, a problem that causes wasted time for users can be easily eliminated. Progress!

Irfanview64 displays all of those tiffs as does RS9; Irfvanview64 also identifies the type of compression. M8 displays all except smalliz (all black) and zackthecat (all green). Photoshop CC 2015.5 can't display quadlzw (problem parsing the tiff) and smalliz, text, ycbcreat and zackthecat (not the right kind of document). Global Mapper 17 ybcreat (unknown pseudotag 65538) and zackthecat (error rendering read access violation).

Aussie Nature Shots

That's an interesting report. Thanks!

I'll add two more data points: QGIS (which, of course, means GDAL since QGIS depends upon GDAL) consumes all but text.tif, which displays as noise unless you chance upon the right zoom (about a one in ten chance, given random tries at various zoom levels).

GIMP consumes all but smallliz.tif and zackthecat, which cause the tiff import module to crash. Others are read but often with all sorts of error messages and warnings. GIMP also gives the ability to consume page 1 of text.tif or page 2 or both. The GIMP strategy is interesting, as it is one way to deal with strange files without taking ownership for defects: advertise the heck out of anomalies found in the files via warning dialogs. If the user persists, then that is their choice.

I'm surprised that PhotoShop did not consume five of the files. Given my respect for Adobe that causes me to walk back what I wrote earlier about all of these files being legal. Let me change that to write that at the time they were written they all were perfectly legal and not malformed given the TIFF standard that prevailed in their day. That is a different statement than saying they are not malformed under TIFF 6.0. It is true that TIFF 6.0 goes way back to the 90's but still, in the case of very old files like some of these they may predate TIFF 6.0.

The GIMP example is also instructive, as TIFF files can use in turn methods such as JPEG compression that have been deprecated in their own standards. What was legal back in the day may be malformed under today's standard.

I suppose there is a reasonable expectation that every new iteration of the TIFF standard is backward compatible. But that may not be the case and only a careful, expert review of the various standards going back to the previous century (TIFF is that old...) would show. In a standard with so much hair as TIFF it would be easy, I suppose, to find a file that was legal in the past century but not legal decades later today.

If such ancient TIFFs do not reflect either current GIS practice or any reasonable presence within archived data sets, even those going back a couple of decades, I don't think it is any big deal that some package fails to read a legal TIFF from thirty years ago if that particular form of TIFF is so ancient and so rare that never in a thousand years of GIS work would you ever encounter it. That's certainly fair game to document as a rare subform that is not supported.

But for all the unlikelihood one would ever encounter in GIS practice an ancient TIFF form (really, who interchanges spatial data in a version two format designed for facsimile machines that became obsolete thirty years ago?), the presence of such ancient TIFFs is a useful counter-example to any temptation to think of TIFF as a simple format. The complexity of TIFF lends itself to all sorts of, ahem, colorful constructions that are both exotic and legal. Even today people create packages that will write TIFFs in all sorts of unexpected and atypical ways, some of which might be perfectly legal even though they are as unusual in their internal structure as a version 2 CCITT fax file from thirty years ago.

But that's part of the challenge of writing software. If it is legal under the standard and you have a package that claims to consume files written to that standard, you either have to consume it or document that you don't. :-) Conversely, if you have written a package that writes malformed files under the current TIFF standard it is up to you to either fix your package or to call the malformed files it creates something other than TIFF.

Say, any chance someone would send this particular file to Manifold tech support so we can have some engineers dissect it? If it is a legal TIFF, would it not be a great idea for Manifold products to be able to consume it? Conversely, if it is malformed, wouldn't it be a great idea to know that so whoever contributes it could file a bug report with the source of the file?

I'm not sure if this will help, hinder or just be tangential to the problem, but I've run into similar issues with TIFFs generated in (likely) similar ways. This should at least answer some of the questions around Manifold 8's failure to read the file after some of the conversions that have been done.

My TIFFS are NOT BigTIFFs but Manifold 8 (x64 on Win 10 x64, dedicated 128 GB swap SSD, lots and lots of room on the project and system drives, 32 GB RAM. etc) will not read them - Unknown Error dialogs. It turns out the tiles are using tiled ordering instead of the older strip ordering. The solution has been to use IrfanView 64 batch conversion to change the tiles to PackBits compression. Manifold 8 then reads the tiles without issue and I can easily generate the merged ECW that is the purpose of all this effort.

Here are two tiles as reported by IrfanView, the first being produced by Pix4DMapper version 2.1.58 (strip ordering) and the second by Pix4DMapper version 3.0.13 (tile ordering)

Created with 2.1.58

ImageWidth (1 Short): 5000

ImageLength (1 Short): 5000

BitsPerSample (4 Short): 8, 8, 8, 8

Compression (1 Short): LZW

Photometric (1 Short): RGB

StripOffsets (5000 Long): 40475, 48138, 55788, 63424, 71088, 78718,...

SamplesPerPixel (1 Short): 4

RowsPerStrip (1 Short): 1

StripByteCounts (5000 Long): 7663, 7650, 7636, 7664, 7630, 7660, 7672,...

PlanarConfig (1 Short): Contig

Software (12 ASCII): pix4dmapper

DateTime (20 ASCII): 2016:05:11 10:41:00

Predictor (1 Short): 2

ExtraSamples (1 Short): 2

SampleFormat (4 Short): 1, 1, 1, 1

33550 (3 Double):

33922 (6 Double):

34735 (32 Short): 1, 1, 0, 7, 1024, 0, 1, 1, 1025, 0, 1, 1,...

34737 (29 ASCII): WGS84 / UTM zone 17N|WGS 84|

42113 (7 ASCII): -10000

Created with 3.0.13

ImageWidth (1 Short): 5000

ImageLength (1 Short): 5000

BitsPerSample (4 Short): 8, 8, 8, 8

Compression (1 Short): LZW

Photometric (1 Short): RGB

SamplesPerPixel (1 Short): 4

PlanarConfig (1 Short): Contig

Software (12 ASCII): pix4dmapper

DateTime (20 ASCII): 2016:05:03 10:30:36

Predictor (1 Short): 2

TileWidth (1 Short): 256

TileLength (1 Short): 256

TileOffsets (400 Long): 3687, 100163, 196891, 291519, 384382,...

TileByteCounts (400 Long): 96476, 96728, 94628, 92863, 88260, 89582,...

ExtraSamples (1 Short): 2

SampleFormat (4 Short): 1, 1, 1, 1

33550 (3 Double):

33922 (6 Double):

34735 (32 Short): 1, 1, 0, 7, 1024, 0, 1, 1, 1025, 0, 1, 1,...

34737 (29 ASCII): WGS84 / UTM zone 17N|WGS 84|

42113 (7 ASCII): -10000

I can supply the community and/or tech with as many before and after samples as you can possible need or want, but as expected, they are large files.

Don't expect, suggest!

Thanks for the offer of files! Could you upload a sample to the tech support FTP site *before* any manipulation with IfranView?

If they are tiled TIFF then Radian should be able to consume them fine but that is worth testing... the more samples, the merrier! :-) At this point it makes sense to focus on that and not 8.

I will do two sets - stripped and tiled

Don't expect, suggest!

All make me think of iSTQB .

i cover in osm documentation subject relative to ecw tiff "file" use with irfanview oziexplorer "tools"( search in google openstreetmap fr oziexplorer).

I understand now why in ozi map merge documentation , i explicit find ecw limitation size !! after read tjhb post ... and wikipedia about ecw file extension name .

ECW is under licece like h265 /cineform own respectively by intergraph, mpegLA ,go Pro . Hope google ll create a new file format for ecw/jpeg2000 like they do against hevc/h265 with vp9 ( youtube context ads) .In gis we need lossless format even the codec turn around dct/Dwt or Cosinus /ondelette ! . compression are use mainly for transfert speed and reduce space size storage but so need higher computing to decompress data before beable to use them !!!

ecw an openess format to not loss the market against jpeg2000 (see manifold forum t102286). The post is about open raster format and font ( projection content) inside pdf(jpx) file or outside .

if vp9 purpose is multi images so the codec for one image is already implemented and available but need to be test to confirm if this kind of algo can be use in the context of raster image and not video !!! webP perhaps use vp9 algo for one image !! hope intel cpu ll support not only h265 intruction set ( quick sync on sse ,avx ) but also vp9 instruction set ( now kabylake ll be only supported by windows 10 and unix like ( mac linux) ). Kabylake (intel tock model from skylabe) don't support 4K stream bandwith output connector except display port .....but gis are image not video ...

regard's

Book about Science , cosmological model , Interstellar travels

Boyle surface fr ,en