Import Image - SID, MrSID

Multiresolution Seamless Image Database (MrSID) is a closed, proprietary format for storing images originally developed at the US Government's Los Alamos National Laboratory in the early and mid 1990's and then exclusively licensed in a somewhat aberrant transaction to a small private company. Despite the general replacement of MrSID by more modern formats such as ECW and JPEG2000 (both of which are true open formats), there are still legacy collections of images that utilize MrSID format.

 

There are three ways of getting MrSID files into Manifold:

 

·      In Windows, open a command prompt window and use the LizardTech mrsiddecode.exe utility to convert MrSID files into GeoTIFF and then import the GeoTIFF. This utility can be used in a background command prompt batch file so that the long time required can be invested during off hours.

·      In Manifold, use the Import - Image dialog for MrSID files. This Manifold capability depends upon having the mrsiddecode.exe utility or equivalent installed on the computer and is equivalent to the above. This is a good idea only for smaller MrSID files since the process of conversion and import takes so long.

·      Use a third party utility or script that converts MrSID into ECW or some other modern format. The ECW files can then be instantly opened in Manifold.

 

Using the Manifold Import - Image dialog, the MrSID files can take nearly forever to import into Manifold because they are first converted by a LizardTech or other vendor's utility into GeoTIFF and then the GeoTIFF file is imported into Manifold. Depending on the speed of the hardware used, the version of windows, the amount of RAM available and other system configuration issues, the conversion and import process takes tens of minutes for small files, hours for medium or large images, many hours for larger images and days for very large images.

 

Even using the mrsiddecode.exe converter as a standalone program in a command prompt window the conversion process is very slow because the conversion process can go no further than the speed of the converter provided by LizardTech, which is not known for speedy performance. Users should be forewarned that a MrSID image occupying 100MB on hard disk as a .sid format file can expand into tens of gigabytes as a GeoTIFF file and take days to convert. Recent editions of the LizardTech decoder are said to be faster, but users should nonetheless brace themselves for a trial by endurance.

 

Comments published on various web sites have expressed the opinion that the LizardTech converter has artificially been made slow to discourage users from abandoning MrSID format in favor of more modern alternatives. Whether that is true or not, it is certainly true that there are converters available that have been written by third parties which are said to operate much faster than the LizardTech converter and which use the same command line interface. Manifold can work with any such compatible converter as well.

 

Whatever the speed of the decoder used, the wisest strategy for dealing with MrSID images is usually to acquire or to create a small Windows command prompt batch file to use the LizardTech or other decoder to convert all MrSID images in background to a more modern format like ECW, running the decoder at night or during the weekend. If this process is accomplished before the images need to be used there will be no inconveniences from delays in importing when using the images in Manifold. See the comments at the end of this topic on how to accomplish this strategy.

 

Until that strategy is accomplished, the Manifold Import - Image dialog used for MrSID import as discussed in this topic should be used only for relatively small MrSID files when sufficient time is available, such as when taking a break or going to lunch, for the process to be completed.

 

Requirements for Importing MrSID Files

 

Manifold can import images from MrSID format if a compatible MrSID decoder has been installed on the same computer. Manifold can work with any MrSID decoder that is compatible with the mrsiddecode.exe decoder distributed by the LizardTech company, the current licensees of MrSID. At this writing, the mrsiddecode.exe decoder may be downloaded at no charge from the www.lizardtech.com website. Drill down through the site to find the download page for this utility.

 

A possibly better idea than using the LizardTech tools (which run unnecessarily slow) is to use MrSidDecodeFast or MrSidExtract or similar third party tools that run much faster.

 

Since websites change frequently, if you cannot find the decoder on the LizardTech website, use Google or some other search engine to find the current location for downloads of that utility. For example use MSN search or Google to search for "MrSidDecodeFast.zip" or "MrSidDecodeFast" to locate these utilities.

 

Since company policies frequently change, if the necessary decoder is not available for free at some time in the future from the LizardTech site, you may use any other compatible MrSID decoder by changing the name of the .exe file to use in the Tools - Options - File Locations dialog to the name of the decoder .exe you use. For example, you could change it to MrSidDecodeFast.exe if you have downloaded that free third party utility.

 

To prepare your system for MrSID use:

 

1. Login as Administrator if running Windows 2000, XP or 2003.

2. Visit the www.lizardtech.com site and find and download the mrsiddecode.exe and mrsidinfo.exe utilities. These may be packaged in zip files that may be downloaded and then unzipped to get the .exe files they contain.

3. Place both .exe files somewhere in the PATH defined for executable programs on your Windows system, such as in the C:\WINDOWS or C:\WINNT folder.

4. Verify the programs have been installed and are functional by opening a command prompt window in Windows and doing a mrsiddecode.exe -help command. If it runs OK you know your .exe has been installed correctly.

5. Launch Manifold and in the Tools - Options - File Locations dialog verify that the correct names for the mrsiddecode.exe and mrsidinfo.exe files are entered.

 

Once the mrsiddecode.exe and mrsidinfo.exe utilities have been installed on your system and their names have been correctly entered into the Tools - Options - File Locations dialog, you may use File - Import - Image to import MrSID format images into Manifold.

 

To import a MrSID image:

 

1. Choose File - Import - Image

2. In the Import Image dialog's Files of type box choose SID Files (*.sid)

3. Browse to the file desired and double-click on the file to be imported.

 

Import Image Dialog Options

 

Scale

Scale reduction factor ranging from x1 (native) (full resolution) to x64 (one sixty-fourth the resolution). Use this when importing large MrSID images to automatically create a smaller, albeit lower resolution, image.

Use password

Check this box to supply a password if the MrSID file requires a password.

 

MrSID images are very large images and even small MrSID files can take a substantial amount of time (tens of minutes or even longer) to import into Manifold. The process is lengthy because first the decoder utility program is used to decode the MrSID image and then the decoded image must be imported into Manifold. The entire process of using an external utility is hidden from the user with Manifold showing only the standard import and progress bar dialogs.

 

By default, Manifold will automatically set the Scale factor so the resulting image will contain less than 4 million pixels. For large MrSID images, this will result in an image with substantially less resolution than is possible. To import a full resolution image, use the x1 (native) setting for Scale. Caution: large MrSID images can be very slow to import due to the incredibly slow functioning of the LizardTech MrSidDecode utility. As a safety measure, Manifold will raise a confirmation dialog when commanded to import a MrSID image with more than 16 million pixels.

 

Since MrSID has largely been replaced in modern usage by open formats such as ECW and JPEG2000, Manifold will import MrSID images but it will not export images to MrSID nor utilize MrSID as a native format for linking images as compressed images as is possible with ECW and JPEG2000.

 

The assumption is that if you are working with Manifold and must deal with an image in MrSID format, your first objective will be to immediately convert the image into a compressed image and to save it out as an ECW or JPEG2000 format image. Once the image has been converted into and saved as a compressed image, loading it and viewing it will be instantaneous and it no longer will be imprisoned in MrSID format.

 

Because many images are trapped in MrSID format and most GIS users would prefer to use a more open format, some software developers who have licenses to work with ECW or JPEG2000 have created scripts or utilities such as MrSidExtract.exe that can automatically traverse a hard disk and find and convert all MrSID files into ECW files. That is the most efficient way of freeing images trapped in MrSID format since the lengthy process of converting a MrSID image to a more modern format can be accomplished at night or over the weekend.

 

Although at the present writing the mrsiddecode.exe and mrsidinfo.exe utilities may be freely downloaded from the www.lizardtech.com website, it is not clear exactly what their licensing agreement says about who may or may not use those utilities. If these utilities disappear or if you disagree with their licensing terms, you may be able to find alternative decoders on the web that perform the same function.

 

Manifold.net has not tested any alternative decoders, but just as there was once only a single MP3 encoder/decoder and now there are many including MrSidDecodeFast. Manifold includes the ability to work with alternative decoders should such emerge in the future so long as they are compatible with the mrsiddecode.exe and mrsidinfo.exe utilities.

 

Projections and MrSID Files

 

MrSID format can optionally store projection information for the contents; however, many MrSID files do not contain such information. Such files may or may not be accompanied by .sdw "world" files. Unfortunately, the LizardTech code for decoding MrSID files cannot be relied upon to capture either any projection information within the MrSID file or from the accompanying "world" file, if any. Manifold has facilities to deal with these issues should the need arise.

 

Although in theory the mrsiddecode.exe decoder created by LizardTech should capture any coordinate system (projection) information available for the MrSID file when converting to GeoTIFF, as a practical matter there is no guarantee that it will do this correctly in all cases, since it might not know the GeoTIFF equivalents for certain coordinate systems that can occur in a MrSID file. It would be unwise to expect the LizardTech code to accurately capture projection information in all cases: instead, the experienced user is always ready to manually assign the desired projection to the imported image.

 

When relying upon third party decoders, such as MrSidDecodeFast or MrSidExtract there is no guarantee there will be any projection information inside the MrSID or that the third party decoder will be able to do anything with it. In such cases it is wise always to assume the worst and to manually assign the desired projection to the resultant ECW or image.

 

When manually assigning projections, we can consult any metadata or readme files that accompany the imagery, as well as manually opening and reading any "world" files that might accompany the images. See the discussion on importing shapefiles and in the Importing Images topic for a forewarning of the hassles that may be encountered using "world" files and strategies for dealing with those hassles. Users of MrSID files should be ready to assign a correct projection manually after importing such images using the Manifold facilities provided for assigning projections.

 

Users of MrSID files should also be forewarned that MrSID compilations that are published by various government bodies historically have had a much higher percentage of erroneous metadata descriptions than any other format known to Manifold Technical Support. Users routinely report problems of misaligned MrSID images that were obtained from some government web site, imported into Manifold, the projection described by the metadata carefully assigned and then when the image was found to be inaccurately registered the problem was ultimately traced to inaccurate metadata published on that web site.

 

The Strange History of MrSID

 

According to the Los Alamos National Laboratory website, MrSID technology is based upon two Los Alamos inventions: MCICR and MrSID.

 

Monte Carlo Image Conversion and Representation (MCICR) technology was developed at Los Alamos by Vance Faber, James White, and Jeffrey Saltzman, MCICR consists of two patented techniques for compressing 24-bit digital images into smaller 8-bit images without loss of resolution.

 

Multiresolution Seamless Image Database (MrSID) was developed by Jonathan Bradley through the Los Alamos laboratory's Sunrise Project funding. The combination of MCICR with MrSID produced image technology that became known as MrSID.

 

On January 9, 1992, LizardTech, formerly "Paradigm Concepts, Inc.", exclusively licensed MCICR from the Los Alamos National Laboratory. On October 25, 1995, LizardTech also exclusively licensed MrSID from the laboratory.

 

As described on the Laboratory's web site, "Before licensing Los Alamos' technologies, Paradigm Concepts, Inc. was a small service-based New Mexico company consisting of 4 employees. Since licensing MCICR and MrSID, the company has grown to 55 employees, including Bradley, Faber, and White - three of the original developers."

 

Also quoted on the Laboratory's web site was the then-President of LizardTech, John R. "Grizz" Deal, stating, "The creation of the MrSID technology could have only happened at the world's premier applied research institution, Los Alamos National Laboratory. Without MrSID, LizardTech would not be in business today."

 

According to news reports, LizardTech grew considerably after the above was written, and at its peak employed approximately 150 people. However, the company apparently never achieved a solid financial footing and spent considerable effort in an obnoxious legal action against Earth Resource Mapping, the company that created ECW, a competing image compression technology.

 

LizardTech was apparently trying to win in court what it could not accomplish in a competitive marketplace. The lawsuits failed, in some cases so miserably that courts ruled against LizardTech in summary judgments. Nonetheless, the legal actions consumed large sums in legal fees. A further negative outcome was that LizardTech clouded the status of open standards like ECW and JPEG2000 with potential patent claims, however unfounded, and so caused unnecessary expense within the open source community as LizardTech's claims in court were examined, refuted and ultimately rejected.

 

Although LizardTech was reported to have raised over $44 million in venture capital money, by 2001 the company was already making massive layoffs, firing about half of its employees, with more layoffs in 2002 as well. It appeared to be well on its way to wasting a staggering amount of venture capital.

 

By 2003 LizardTech had managed to burn through tens of millions of dollars of venture capital and was forced to fire almost all of its employees and sell the remaining assets to a Japanese company, Celartem Technology Inc. As of this writing, it remains to be seen if Celartem can revive MrSID despite the market preference for open technologies such as ECW and JPEG2000.

 

Why is MrSID Conversion and Import so Slow?

 

The incredibly slow functioning of the LizardTech mrsiddecode.exe program when converting MrSID images to an open standard like GeoTIFF appears to be yet another negative result of having to work with a seriously "closed" format like MrSID. Because the read/write technology inside the decoder is also closed, users are kept hostage by whatever inclination LizardTech may or may not have to improve the functioning of the decoder. Some critics have expressed the opinion that LizardTech has provided a deliberately slowed-down conversion utility to make it difficult to convert images from MrSID format to modern, open formats like ECW or JPEG2000.

 

Considering that LizardTech's historical strategy has been to maintain MrSID as a deeply closed format, it is obvious that LizardTech has every incentive not to try to speed up a decoder that allows people to free themselves from MrSID captivity. If anything, a cynic would point out that LizardTech would want to offer a decoder that works well enough for the company to say that MrSID can be converted to open formats but which works so poorly that as a practical matter few people will be able to use the decoder to free themselves from MrSID.

 

Contrast that situation to the much more open world of ECW or JPEG2000, where source code is easily available and as a result anyone who wants to contribute to improved performance can do so. As a result, both ECW and JPEG2000 provide at least four times faster performance in "native" compressed format applications and allow conversion to other formats using dramatically faster processes than made available by LizardTech.

 

The recommended Manifold strategy for dealing with MrSID is to convert MrSID format images into ECW format. Although LizardTech may not provide any tools for that purpose, there are third party tools such as MrSidExtract.exe that can do an automatic conversion of all MrSID files in a given folder or folders into ECW.

 

Many people have written scripts or programs like MrSidExtract.exe that can search through a folder or even a hard disk full of MrSID files and in background accomplish the above two-step process to convert those images into ECW. If accomplished in background, such as at night or during weekends, the tremendously slow process of getting images out of MrSID and into fast, modern formats like ECW can be done without the interactive pain of sitting and watching a conversion process taking forever. The resulting ECW images can be loaded into Manifold and viewed in seconds.

 

Such scripts or programs are not a part of Manifold and so are not covered by this documentation. Use a search engine like MSN search or Google or search Manifold forums such as the Georeference forum on the manifold.net home page and the archives of the Manifold-L discussion list to find links to pages that provide such scripts.

 

Tech Support

 

Although Manifold works with the LizardTech tools and other third party tools, such tools are not a part of Manifold System and are not supported by Manifold Technical Support. As noted above, tools vary greatly in performance, memory requirements, ability to extract projection information and other characteristics. These characteristics are not under the control of manifold.net, nor does manifold.net have any inside ability to either analyze or repair shortcomings in non-Manifold products. If you experience any unexpected results when using a third party tool, contact the source of that tool for assistance.

 

See Also

 

Tools - Options - File Locations

Compressed Images