Subscribe to this thread
Home - Cutting Edge / All posts - Manifold System 9.0.176.4
adamw


10,447 post(s)
#06-May-22 17:52

9.0.176.4

We are finally done with the build.

The big feature that we have been working on is the Manifold database server which allows serving Manifold data over the network. The database server is currently read-only. We are planning to make it read-write in the future, but this is not a big priority right now, because even a read-only server can do a lot of useful things: sharing data that is too inconvenient to copy around or needs to be updated regularly, offloading computations from an underpowered machine such as a laptop to a bigger server, distributing computations onto multiple machines, etc. There are also many new features which now become possible.

manifold-9.0.176.4-x64.zip

SHA256: e3f8549de5ea796790d370700abc324d310eac1a020996dfb02907d3d2872e43

manifold-viewer-9.0.176.4-x64.zip

SHA256: 00594be15df3efb0cb7738be87d2fdbbbe969b3caeaa8404c22bb7dad01ad775

sql4arc-9.0.176.4-x64.zip

SHA256: 8f23a445f6bb46fd4b212a8cdef46e324f4cfb2f9f1dfa1c32ecf61c43d92da8

adamw


10,447 post(s)
#06-May-22 17:55

Read-only database server

Installation packages for Manifold 9 include two new executables for the Manifold database server, MANIFOLDSRV: ~\BIN\MANIFOLDSRV.EXE (32-bit) and ~\BIN64\MANIFOLDSRV.EXE (64-bit). Manifold Viewer and SQL for ArcGIS Pro do not include MANIFOLDSRV. The Manifold database server is currently read-only.

MANIFOLDSRV is a console application. The command line syntax is:

MANIFOLDSRV [<options>] <file>

Supported options:

  • -addr:xxx -- network interface to listen on, the default is the primary network interface.
  • -port:xxx -- port to listen on, the default is 9099, currently unassigned by IANA.
  • -user:xxx -- user required to connect, the default is none.
  • -password:xxx -- password required to connect, the default is none, ignored if user required to connect is not specified.
  • -connections:xxx -- maximum number of connections, the default is 20, use 0 to remove the limit.
  • -workers:xxx -- number of worker threads to process requests, the default is 4, the number can vary between 1 and 128.

The file is required and can be of any type supported by 9. If no file is provided, MANIFOLDSRV prints the command line syntax and exits. If the file is provided, MANIFOLDSRV opens it and starts serving connections to it.

If the system does not have a license for 9 installed, MANIFOLDSRV exits without prompting to install the license. Install the license using the main application, MANIFOLD.

To stop MANIFOLDSRV, press Ctrl-C or Ctrl-Break. MANIFOLDSRV automatically stops if the console window attached to it is closed, if the current user starts logging off, or if the system starts shutting down.

MANIFOLDSRV writes log files, similarly to MANIFOLD. The MANIFOLDSRV log files are named xxx-srv.log, extended to xxx-srv-yyy.log for multiple instances of MANIFOLDSRV running at the same time, to distinguish them from the log files for MANIFOLD. MANIFOLDSRV logs each accepted and closed connection as well as messages generated by worker threads that process requests.

To connect to a MANIFOLDSRV database, open MANIFOLD and add a new data source of type: 'Database: manifoldserver'. Specify the server to connect to as a network address plus an optional port number after a colon. Specify the user to connect as, if MANIFOLDSRV has been launched with the -user:xxx option. Specify the password to connect with, if MANIFOLDSRV has been launched with both -user:xxx and -password:xxx options. Specify the path to the data source to connect to, if you want to connect not to the root file used by MANIFOLDSRV, but rather to a nested data source within that file.

The connection string for a MANIFOLDSRV database consists of key=value pairs separated by semicolons. Special characters ('=', ';', '\') in keys and values are escaped. Unprintable characters in keys and values are encoded as '\uXXXX'.

Supported keys:

  • server=xxx -- the server to connect to, either an IP address or a fully qualified domain name, optionally followed by a port number after a colon.
  • database=xxx -- optional path to a data source within the root file used by MANIFOLDSRV to connect to (eg, '[usa_east]::[new_york]'), if omitted, the connection is to the root file itself.
  • user=xxx -- user to connect as.
  • password=xxx -- password to connect with, ignored if user to connect as is not specified.

Unknown keys are ignored.

The data source for a MANIFOLDSRV database attempts to connect to the database in background. If the connection cannot be established for some time, the data source logs an error and stops attempts to connect. After the connection is established, the data source monitors it, and if the database breaks the connection unexpectedly, eg, because MANIFOLDSRV is being shut down, the data source logs an error and becomes disconnected. (In both cases above, the data source will not attempt to reconnect until you close and reopen the parent data source -- usually the MAP file. In the future we will allow reconnecting via the Refresh command.)

The data source for a MANIFOLDSRV database sends multiple requests to the server in parallel, a request does not have to wait until all previous requests have been processed by the server or even have been received by the server in full. The server processes multiple requests from the same connection in parallel, if there are worker threads available (not busy processing requests from other connections). Both sides split big data packets into parts and communicate with each other to avoid overwhelming the network with too much data sent at once. Both sides prioritize transferring small data packets (like 'get schema of a table X') before big ones (like 'fetch next N records of a big table Y'), to minimize the number of requests waiting in queues. Both sides put no limit onto the number of requests waiting in queues outside of available memory.

The data source for a MANIFOLDSRV database supports all features of spatial indexes, including those that aid rendering: access to intermediate levels for raster data, thinning for vector data and thinning for point cloud data used with LAS / LAZ files. (What this means is that if MANIFOLDSRV operates on a MAP file, rendering data from the MANIFOLDSRV database will use all smarts that rendering from a MAP file on a local system would use. This generally makes rendering perform much faster than rendering from a general-purpose database, such as PostgreSQL.)

Queries in a command window for a MANIFOLDSRV database run on the server. Queries in query components stored on the server run on the server. Script functions called from queries that run on the server also run on the server. Scripts in script components stored on the server currently run on the client. (In the future, scripts in script components stored on the server will also run on the server. Also, because scripts can access the file system, we are going to provide means to disable them on the server completely or partially, for security reasons.)

Connecting to a nested data source in a MANIFOLDSRV database creates a new connection to the server and tunnels data through that connection. For example, if MANIFOLDSRV operates on a MAP file which has a link to a JPEG2K file, attempting to read the contents of a JPEG2K file from a network client will ask the server to open the JPEG2K file and read its contents on behalf of the client. Web data sources work slightly differently: (a) connecting to a nested web data source in a MANIFOLDSRV database explicitly by specifying its name in the connection string will tunnel data through MANIFOLDSRV, but (b) connecting to a nested web data source in a MANIFOLDSRV database implicitly using the UI (by opening the data source in the Project pane, by opening a map which contains a layer from that data source, etc) will connect to the web data source directly. Direct connections to web data sources reduce the load on the server and are usually faster for clients as well.

Tunneling to nested data sources applies recursively from the root database operated by MANIFOLDSRV to all child data sources at all levels. Since each nested data source may potentially create a separate connection, if the data served by MANIFOLDSRV includes many nested data sources, consider increasing or removing the connection limit from the default 20.

Tunneling to a nested web data source in a MANIFOLDSRV database automatically turns off the option to save cached data between sessions, because the parent database used to store cached data might be read-only. If the web data source has been configured to cache data at all, the cache will still be used, but it will be cleared when MANIFOLDSRV exits.

danb

2,064 post(s)
#07-May-22 20:31

This is excellent and a huge addition to the package. Well worth the wait and I cant wait to give it a go


Landsystems Ltd ... Know your land | www.landsystems.co.nz

danb

2,064 post(s)
#07-May-22 23:37

GDB dataport formats a numeric / text field with a coded values domain using a choice style. (We are going to do the same for date fields.)

I've just seen this change too. Thanks very much for implementing this it will be very useful.


Landsystems Ltd ... Know your land | www.landsystems.co.nz

marcus52 post(s)
#08-May-22 18:50

If the system does not have a license for 9 installed, MANIFOLDSRV exits without prompting to install the license. Install the license using the main application, MANIFOLD.

Can you please clarify the licensing arrangement. So from above quote it appears the server hosting the database requires a license. Is that correct? And then any client accessing the database from another system requires an additional license?

Dimitri


7,413 post(s)
#09-May-22 05:17

So from above quote it appears the server hosting the database requires a license. Is that correct? And then any client accessing the database from another system requires an additional license?

Yes. It's the same as before: To run Manifold on two different machines you need two licenses.

Once you have a license installed an activated on a machine you can launch as many desktop sessions of Manifold as you want on that machine, and you can also launch as many Server instances as you want on that machine.

jsperr
143 post(s)
#08-May-22 22:00

Very cool. Loaded manifoldsrv.exe on my workstation running a 4.6 GB map file with 14.5 million points -- the "Big Yellow Taxi Project".

Loaded the latest Viewer on my old machine -- a 3 GHZ Core 2 Pentium Duo -- created a data source to the server, and it sprung to life with quick scrolling, zooming, and pulling in of the Bing Street Map background.

It was simple and fast to get it up and running. I'm looking forward to getting it set up where I can expose maps with dynamic content to other users on the web.

Graeme

990 post(s)
#09-May-22 01:31

Have a project with a linked Kobo data source (great, thanks for that feature support!), works fine in 176.3 but seems to be broken if opening the file in 176.4. Screenshots attached.

Refresh doesn't seem to resolve it.

Attachments:
Kobo link broken 9.176.4.jpg
Kobo link OK 9.176.3.jpg

adamw


10,447 post(s)
#09-May-22 08:54

We'll check what is going on. Thanks for the note.

adamw


10,447 post(s)
#12-May-22 13:54

An update: we found and fixed the issue, this was a bug on our side. The fix will be available in the next build.

jockeryl
178 post(s)
#27-May-22 10:37

Thank you for this! Great start on server features!

Great design, that the queries are run on the server and only the results are returned over the network.

Here are some feedback and questions (will send suggestions to support also):

  • The server console app title bar show username and password in clear text, there should be an option to start the server without exposing these kind of credentials and secrets for everyone to see.
  • Can we secure the MAP file on the server with a password? To avoid someone copying the file and getting access to the content and internal connection strings and credentials.
  • Can you add option to enable SSL/TLS and run this TCP server on port 443?
  • I also look forward to enable serving of vector tiles via REST API, for use in MapBox/Leaflet/OpenLayer/MapLibre etc.
  • Serving normal rendered raster tiles with optional caching would be great also.
  • REST API for querying the map DB by sending the SQL queries as part of the request would also be great. Returning the results as Geojson would also be a good feature.
  • Will we be able to run this in a Docker/Container? Any recommendations running in a static VM in a cloud, GPC/Azure/AWS? - Licensing concerns?
  • Does it take advantage on local GPU if it's available in the PC/Server?

  • Are there any recommendations on Memory and CPU cores? Same as Manifold desktop? Do we consider concurrent server users the same as normal Manifold Desktop users, assuming same PC hardware?
  • Can we redirect the logs to another location?
    • Can we run the server as a service? without the console window. (I see that you are planning that in the Server manual)

  • Dimitri


    7,413 post(s)
    #27-May-22 15:15

    No doubt adamw will contribute, but just a few notes from what I've heard...

    • The server console app title bar show username and password in clear text, there should be an option to start the server without exposing these kind of credentials and secrets for everyone to see.
    • Can we secure the MAP file on the server with a password? To avoid someone copying the file and getting access to the content and internal connection strings and credentials.

    Both are in process

    Any recommendations running in a static VM in a cloud, GPC/Azure/AWS? - Licensing concerns?

    That's the same as regular Manifold: if the VM changes then it is the same as a move to a different machine in terms of activation.

    • Does it take advantage on local GPU if it's available in the PC/Server?

    Yes, it does.

    Can we redirect the logs to another location?

    AFAIK, no. Sounds like that could be an option (so long as people remember to specify a location to which they can write...)

    I think the other points are about a possible future web server product. Never hesitate to suggest what you want for things like that. :-)

    The technology within Server is clearly just a very short step away from being a web server, but for now it is a product that serves Manifold clients.

    adamw


    10,447 post(s)
    #31-May-22 14:14

    Point by point:

    • Hide user / password in console title - we will add an option to do that. Sometimes showing that info might be desirable, but sometimes not, we agree.
    • Secure MAP file with a password - we are planning to add this in the future, yes.
    • Enable SSL/TLS and run on port 443 - we will allow using HTTP instead of TCP sometime later, adding a REST API as you mention. That will let you use HTTPS immediately, by wrapping HTTP into HTTPS using lightweight servers like NGINX. Then we might look into supporting HTTPS natively if needed (but in general adding HTTPS on top of HTTP using NGINX would perhaps be more than enough, this lets you handle all certificates in a single place, etc).
    • REST API producing vector tiles - no commitment right now. We will allow using REST API to render maps with control for layers, etc, producing raster tiles. Undecided on vector tiles.
    • REST API producing raster tiles - yes, we will be doing that, as above.
    • REST API running queries - sure, obviously. We will provide an option to return geometry values as GeoJSON as well.
    • Running in a container, licensing for cloud - not right now, but we are thinking about how to do that, yes.
    • Does the server use local GPGPU - yes, it does. We will add an option to configure that (you might want to disable that or let some instances use GPGPU and others not).
    • Recommendations on memory / CPU - right now the server uses the settings from the Options dialog. We will change that. There will be different defaults (more conservative ones) and there will be options to override them. Interactive users cost more memory / CPU than server ones even if server users are rendering data / reading tables / running queries as often as interactive users. How much more depends on activities, but the difference is usually significant: 2x, 3x, etc.
    • Redirect logs to another location - we will add an option to do that.
    • Run as a service - we will allow doing that, yes.

    Hope this helps. Happy to see questions like these!

    jockeryl
    178 post(s)
    #03-Jun-22 10:29

    Thanks Adam,

    All good, looking forward to these improvements!

    • Enable SSL/TLS and run on port 443 - we will allow using HTTP instead of TCP sometime later, adding a REST API as you mention. That will let you use HTTPS immediately, by wrapping HTTP into HTTPS using lightweight servers like NGINX. Then we might look into supporting HTTPS natively if needed (but in general adding HTTPS on top of HTTP using NGINX would perhaps be more than enough, this lets you handle all certificates in a single place, etc).

    Ok, so the current TCP connection from server to client is mainly intended for LAN intranet network.

    I'm considering the options for a government system, but TCP through the FW (they use various packet scannign and WAF tools) will be hard to justify.

    Good to know HTTP/HTTPS is in the pipepline.

    To be clear, by using NGINX or equivalent proxy to add SSL, we would need to install the Manifold Server on same instance as the NGINX. Otherwise there would be unencrypted traffic between VM/Server instances. It's a common requirement for the government agencies to enforce SSL everywhere, always encrypted. We have to pass 3rd party security reviews and scans.

    Hence, it would be great if Manifold Server could support SSL as an option. Your workaround is ok for now.

    Dimitri


    7,413 post(s)
    #03-Jun-22 11:34

    Ok, so the current TCP connection from server to client is mainly intended for LAN intranet network.

    No. It's intended both for LAN and WAN (Internet).

    TCP works great for connecting through Internet, as you might expect given that Internet itself depends on TCP. To quote O'Reilly: "HTTP connections really are nothing more than TCP connections, plus a few rules about how to use them."

    As Wikipedia puts it, "Major internet applications such as the World Wide Web, email, remote administration, and file transfer rely on TCP, which is part of the Transport Layer of the TCP/IP suite. SSL/TLS often runs on top of TCP."

    As far as HTTPS is concerned, I respect that your organization has specific requirements for HTTPS everywhere. But if you look what Server does, it connects to a Manifold client using internal communications that are known only to Manifold. Unless you're a source code customer of Manifold, even if you could intercept the connection you wouldn't know what's being exchanged between server and client.

    I understand that whether or not it is a security issue, there's no arguing with organizational policies, so no doubt sooner or later Manifold will provide an end to end encrypted solution. But for those people who are not forced into it now, there are sensible reasons to not use https if it doesn't really add anything.

    After all, https isn't free. It can reduce performance, and it increases the worldwide energy cost of Internet infrastructure. There's a lot of energy being spent to provide https encryption on things like TikTok videos, which probably could do without encryption given environmental concerns.

    adamw


    10,447 post(s)
    #07-Jun-22 14:51

    We completely agree support for HTTP / HTTPS is very desirable because the security implications of both are well-known, they are commonly allowed through firewalls, etc. Arguing to allow a new TCP protocol through firewalls as well could be pretty hard for a government org, as you say.

    But do orgs that you work for use, say, PostgreSQL or SQL Server? Those products also have plain TCP protocols. I ask because if the answer is -- yes, non-HTTP TCP protocols can also be used -- then it might make sense for us to find ways to add encryption on top of our own plain TCP protocol, without having to go through HTTP.

    jockeryl
    178 post(s)
    #05-Jul-22 10:33

    Thank you, to clarify:

    But do orgs that you work for use, say, PostgreSQL or SQL Server?

    Yes, but they are not allowing desktop users direct access to the DB servers.

    DB servers (MSSQL, Oracle, PostGres) are all accessed via secured App layers, which in turn can only be accessed through Web/Proxy layers (traditional 3-tier architectures).

    So if we place the Manifold Server services in the App layer, then we are good. The necessary TCP connection can be allowed through the FW between the App tier and the DB tier.

    This means that a Manifold Desktop Application could use HTTPS to connect via Proxy/Web tier to the Manifold App Server services, which then returns data from the DB Servers. Each stage should be encrypted to pass required security policies.

    This will also make it possible to host the DB services and Manifold Servers in the cloud, and the users access using HTTPS via internet or VPN.

    Direct DB access might be allowed in smaller organizations with simpler LANs, I have been working with those setups in the past where ArcGIS Desktops needed direct SQL Server access. I used Manifold to those same SQL servers.

    adamw


    10,447 post(s)
    #05-Jul-22 14:11

    Thanks, that helps.

    You want Manifold Server to cover not just the third tier, but also the second tier. Because that way you don't need to write an application for the second tier to keep the third tier secured. (I mean, there's frequently more to the application tier than HTTP / HTTPS, people are frequently putting there all kinds of logic. But in this particular case we are talking about bulking up the Server so that if we *don't want* any additional logic applied, we can let the Server cover both the data tier and the application tier for simplicity.)

    Sure, why not. :-)

    adamw


    10,447 post(s)
    #06-May-22 17:56

    Other

    The 'Use integrated security' / 'Use login and password' radio buttons in the Database Login dialog are replaced with a single 'Use login and password' checkbox, like in the Web Login dialog.

    (Fix) The CASE operator no longer sometimes fails to convert the result value to the required type and produces a NULL value, when used inside a function.

    ArcGIS REST dataport supports authentication via user name and password. (ArcGIS authentication is based on temporary security tokens. The dataport first checks if the server uses security tokens at all. If it does, the dataport uses the specified user name and password to generate these security tokens, refreshing them as needed. Federated servers that generate security tokens based on authentication data from SAML or OpenID services are not supported. Specifying a long-lived security token - API key - directly is not currently supported either, but will be supported after this feature of ArcGIS gets out of beta.)

    ArcGIS REST dataport supports linking to a specific layer (eg, http://.../FeatureServer/0).

    GDB dataport formats a numeric / text field with a coded values domain using a choice style. (We are going to do the same for date fields.)

    Reading an ECW with a valid datum and 'RAW' coordinate system preserves the datum info, even though this combination is disallowed by the spec.

    (Fix) Writing an ECW with an exportable datum (a datum that can be encoded in ECW) and an unexportable coordinate system, sets the coordinate system to 'LOCAL' instead of 'RAW'.

    Writing a TIFF always organizes raster data into tiles rather than strips. This produces files which can be read more efficiently.

    Writing a TIFF writes data for intermediate levels into the TIFF file, rather than into an accompanying AUX file. The levels are written from smallest to largest, producing a cloud-optimized TIFF. (In the future, we will allow switching between producing a plain file with no intermediate levels or a cloud-optimized file with all intermediate levels.)

    Writing a TIFF with floating-point pixels uses NaN for missing pixels. (Previously we were using a maximum value allowed for the pixel type.)

    Writing a TIFF with integer pixels uses a mask for missing pixels. (Previously we were using a maximum or minimum value allowed for the pixel type, and pixels that coincided with that value were incremented or decremented to be different from it.)

    (Fix) Reading a TIFF with an accompanying AUX file no longer sometimes misrenders data for intermediate levels by failing to shift it properly.

    Reading a TIFF with raster data organized into tiles but no intermediate levels no longer caches data for level 0 and renders it directly from the TIFF.

    (Fix) The ODBC driver correctly communicates to the caller that the number of the records in a result table of a query is unknown, when it is unknown. (This allows accessing Manifold data via PyODBC, eg, from a script in ArcPython.)

    (Fix) The ODBC driver correctly communicates all types of errors occurring in the process of fetching records. (Previously, some rare types of errors could escape to the caller in ways that could potentially lead to a crash.)

    The ODBC driver allows connecting to files of all supported types, not just MAP.

    (Fix) Reading data from a SHP file using an index no longer sometimes fails. (Observed when accessing SHP files via the Manifold ODBC driver.)

    (Fix) PostgreSQL dataport no longer fails when reading an empty non-NULL binary value and returns a NULL value instead. (There is a bigger issue here. Many databases make a distinction between NULL binary values and empty binary values. We do not make such a distinction and this may be a problem in some cases. For example, copying a table with empty binary values and a non-NULL constraint on the relevant field from PostgreSQL into a MAP file will currently fail on the first empty value encountered. We are going to address these issues in the future.)

    (Fix) Web dataports set to save cached data between sessions no longer sometimes fail to operate when parent database used to store cache is read-only and instead switch to work without cache.

    (Fix) The ODBC driver no longer sometimes crashes when used to connect to a nested data source which is set to save cached data between sessions.

    End of list.

    Dimitri


    7,413 post(s)
    #06-May-22 18:41

    See also the new Manifold Server user manual topic for illustrations and examples.

    artlembo


    3,400 post(s)
    #06-May-22 18:52

    quick typo: excepting web data sources such as image servers

    need to change to accepting. Otherwise, the read is pretty clear.

    any chance you guys will stand up an example server so that people (20 at least) can get a hands-on look?

    Dimitri


    7,413 post(s)
    #07-May-22 05:05

    quick typo

    If you mean in...

    When Server shares a Manifold project, it automatically provides connections throughServer to data sources within the project being shared, excepting web data sources such as image servers.

    ...then I think it's correct as written, using "excepting" as a preposition in the sense of "except for". The tunneling Server does for data sources is for all data sources except for web sources like image servers. But I think simply writing "except for" would be better.

    On trying out an example Server, it's trivially easy to run your own. If you have just one computer, launch a Command Prompt, cd to the bin64 folder for your portable build and enter

    manifoldsrv "c:\folder\my project.map"

    Use, of course, the path and name of the project you want to share with Server. If there are no spaces in the path name or project file name, you don't need the quotes.

    Note the IP address it says, which will be something like 10.2.0.2. That's what Windows told Server is the "loopback" IP address when you're not going through any external network interfaces, equivalent to "localhost".

    Don't close the Command Prompt window, just minimize it or move it out of the way. Launch Manifold and then create a manifoldserver data source using 10.2.0.2 as the IP address, and you're off and running.

    If you like, launch three or four more Manifold sessions. Copy the Server data source you created in the first project, and then paste it into each of those three or four more Manifold sessions. You now have four or five Manifold client sessions running that are connected to the same Server.

    If you have more than one machine, launch Server on one of your machines but do that with a -addr: option in the command line to specify the IP address that machine is using on your local network, and then use that IP address in your client Manifold sessions on the other machines to connect to Server.

    When I launch Server on the same machine as the clients I'm using I always specify the external IP address using -addr: instead of just using the default internal IP address. Why? Because then when I connect to that machine with a client I can save the project and if I move that project to a different computer on the local area network it will still work on any computer. Using an internal IP address like 10.2.0.2 only works on that same computer, but using the computer's IP address on the local network, like 192.168.2.12, will work on any computer on the local network. None of that is particular to Server by the way, it's just basic network stuff that's the same for any database, like connecting to Postgres or whatever. It's just easier to manage with Server.

    Running your own examples is better for testing than logging into somebody else's Server because when you log into somebody else's server you don't know how many other people are sharing that data or what they're doing with it, so you can't make repeatable, apples to apples comparison. You also don't know what's going on in the network between you and wherever the Server is located, which is going to be half-way around the world for many users. With your own Server you know what's going on and you know what the network is.

    You can do fun stuff like put an old hub between your Server and the network as a crude way of regulating the network speed. If your local area network is a gigabit network, many of us have old hubs lying around (old routers will work) that are limited to 100 mbit ethernet speeds. Put one of those in between the Server and the rest of the network and you just limited the connection to your Server machine to 1/10th the speed of the gigabit network. See how that goes, and then take out the old hub and see how it works at full network speed.

    You can also try out constellations of Server and clients: If you have two or three machines run a Server instance on each of the machines and then simultaneously on each of the machines also run Manifold as a client connecting to each of the other servers. Assuming you're not using an ancient two-core CPU, performance is amazingly good even when a machine is simultaneously running a few Server instances and also a few Manifold sessions at the same time. I've done that on laptops running low-power CPUs, on low end Core i3 and Ryzen 3 machines, and also on run of the mill desktops like Ryzen 5 and 7s. You expect it will run great on a Threadripper with dozens of cores, but it's nice to see it also runs great on inexpensive desktops as well.

    adamw


    10,447 post(s)
    #07-May-22 11:05

    A short clarification on addresses:

    The standard loopback interface is 127.0.0.1. If you want to limit the server to connections from the same machine, this is a good address to use.

    Both 192.168.x.x and 10.x.x.x (as well as 172.16-31.x.x) are private address ranges used on LANs. What Dimitri is saying above is that if you have multiple networks running on a system -- which happens all the time these days because many systems have virtual networks configured in addition to physical ones -- you should pay attention to which particular network interface the server is going to use. Because if you let it choose the network and it ends up choosing a virtual one that only exists on your system, other machines won't be able to connect to it. (In general, you can simply try running MANIFOLDSRV with no options besides the path to the file to serve, then check which network it will start listening on. If that's not the network you want it to use, stop and add an -addr:xxx option.)

    artlembo


    3,400 post(s)
    #07-May-22 11:57

    If that is the meaning then yes, you are correct. I thought those things were accepted, not excepted :-). Maybe with the exception of.

    tjhb
    10,094 post(s)
    #06-May-22 19:55

    Congratulations, I was hoping this was the "major new feature". ("Major" is of course an understatement.)

    Slightly confused by

    Supported options: ...

    • -connections:xxx -- maximum number of connections, the default is 20, use 0 to remove the limit.

    versus

    To stop... MANIFOLDSRV automatically stops... if the current user starts logging off, ...

    Haven't tested, but that doesn't seem right.

    Dimitri


    7,413 post(s)
    #07-May-22 05:18

    I think the latter phrase refers to one of the three ways you can stop the server. The server is launched with a command line in a Command Prompt window. You can stop it by either 1) with the focus on the Command Prompt window hit Ctrl-C, or 2) close the Command Prompt window, or 3) log off.

    Personally, I think a default connections limit of 20 is a bit on the low side. I think it's set that way so that default use of Server even on very weak machines for personal use is OK. But that's why there are Cutting Edge builds so that feedback can help adjust defaults, etc.

    tjhb
    10,094 post(s)
    #07-May-22 06:42

    (What I meant is that it I think it must be when the last logged-on user logs off that the server shuts down, rather than the current user.)

    Dimitri


    7,413 post(s)
    #07-May-22 09:32

    No worries. When Server is started, it runs and listens even though no clients are connected. If clients connect and then disconnect, when the last one disconnects the server will still be running, listening and ready to go for any new connections. The "current user" just meant the user who opened a command prompt and launched Server.

    A few comments on Server being a console application:

    For now, Server is a console application. Somebody has to log in, open a command prompt window and launch it. It runs in that command prompt window until it is stopped with a Ctrl-C or by closing the command prompt window. When a user logs off, of course any of his or her command prompt windows are automatically closed as well, thus stopping any Server instances that are running in those command prompt windows.

    If you want to run more than one Server instance on the same machine, you just open up an additional Command Prompt window and launch Server in that. Each server instance you fire up lives in its own command prompt window, so it's easy to keep track of which ones are which (the command line launching each instance is in the title bar of the command prompt window). You can stop and start Servers in each command prompt window independently of each other.

    Manifold will add the ability to run Server as a service, so you could start, pause, stop, etc., it as with any Windows service. In that case it wouldn't be tied to somebody logging in and staying logged in.

    For now, it's a console application because that is an extremely simple user interface: run it in a command line and it keeps running until you stop it.

    It will be a more sophisticated user interface when running it as a service, because then you'd have to have some way of noting which project files various Server instances are using when there is more than one Server instance running as a service. None of that is rocket science but it is not quite as totally simple as launching it as a console application.

    tjhb
    10,094 post(s)
    #07-May-22 10:15

    Got it. Silly me. Thanks for the extra context Dimitri.

    oeaulong

    521 post(s)
    #07-May-22 16:34

    Just finished my initial tests. An outstanding feature addition and fine server foundation for web presentation down the road. Excellent job and well worth the schedule slippage.

    I was able to load and launch a large Mapfile, hit it internally, through router front door and then from Mfd installation across town using full ver & viewer. It daisy-chained through datasources to pull down updates remotely from REST svc. etc.

    Very slick and should be worthy of many licenses itself. Looking forward to the integration for this type of backend service connections and the container windows app cloud plumbing possibilities.

    danb

    2,064 post(s)
    #22-May-22 21:13

    Server is fantastic. I have only tested it on the same machine, but it is incredibly easy to set up and deploy and very responsive.

    My test was to link some 22 map projects containing various LiDAR holdings (1.8TB of images, points, contours and supporting layers) into a new map file and to save and serve this. I would say that from saving to serving took less that a minute having never done it before and the result was to all intents and purposes as fast as an interactive session of Manifold 9.

    Some of the linked map files contained links back to las files on disk and I was able to tunnel back to these without issue.

    Straight away, I can use it to expedite my LiDAR QA workflows by serving the data to various QA map projects at the same time.

    Fantastic work. I can't wait to see what comes next.


    Landsystems Ltd ... Know your land | www.landsystems.co.nz

    lionel

    995 post(s)
    #07-May-22 16:47

    From now i can have acces to *.map(v9) manage by manifold 9 server using manifold 9 in read mode .

    The Manifold 9 menu items is block in original file when sever is active or not ( see follow pink arrows ) .

    Manifold 9 Server make me think about IIS service so many functionnalities ll soon be here !! ( authentification ,protocol ...... )

    NB gdam is GADM world map so manifold map is Big 2,5 Go !!

    Attachments:
    gadm_france_idahoprojection.png
    gadm_france_idahoprojection1.png
    IIS_behaveLike_IMSmapserver9.png
    Manifold_map1_and_map2-with-server-To-map1_reduce.png


    Book about Science , cosmological model , Interstellar travels

    Boyle surface fr ,en

    Mike Pelletier

    2,122 post(s)
    #07-May-22 18:07

    Congratulations on getting over this big hurdle, it is very much appreciated. Certainly exciting what doors it opens up. Definitely, curious what new features will initially be worked on that Server makes possible.

    I'm sure they will be useful, but thought it worth repeating the need for fixing the basic GIS tools. I really don't want to push Manifold on novice users when labels grossly overlap, there's no dynamic legend, and COGO tools default to atomic level of accuracy with no way to change it. Can't help think that better basic GIS tools will result in many new licenses, but on the other hand you guys surely have a handle on what is best for the big picture of Manifold. :-)

    adamw


    10,447 post(s)
    #08-May-22 09:43

    Yes, we will now get back to labels, etc. These things will go in parallel with advances for the Server and with the new features based on it.

    dchall8
    1,008 post(s)
    #08-May-22 18:32

    I hoped the update would be the long awaited implementation of textural overlays including DEMs blended with colored raster images. According to Dimitri, "It's simply an elementary use of fractional Laplacian operations (see Leland Brown's discussion at the end of http://www.mountaincartography.org/activities/workshops/banff_canada/papers/brown.pdf ) that Brown implements using interpolations simplified to convolution matrix operations, which, of course, is very easy to do in 9. It's even amenable to parallelization." To which Adam followed up with some suggestions for built-in tools.

    I'm not an ESRI user or enthusiast, but the publishable products I see coming from Arc users is eye watering.

    tjhb
    10,094 post(s)
    #08-May-22 20:27

    You can hope, but that is something you can easily do for yourself, given the tools Manifold has already given us, and your skills.

    If you needed your bottom wiped for you, you could use ESRI. But that might limit your imagination.

    We are best placed to do new things.

    dchall8
    1,008 post(s)
    #08-May-22 22:37

    Tim said,

    that is something you can easily do for yourself, given the tools Manifold has already given us, and your skills.

    Tim are you saying I can already make a DEM textured historical map like this using the built in Manifold tools?

    Attachments:
    California 3-d Map.jpg

    Dimitri


    7,413 post(s)
    #09-May-22 17:54

    are you saying I can already make a DEM textured historical map like this using the built in Manifold tools?

    If you mean the coloration of the map and look, sure, why not? Look at the various published examples to see the many different effects you could combine. For example, there's the "antique swiss look" but using a different, brighter colored palette. There are lots of layers involved and lots of steps and tinker time, for sure, but the effects can be totally striking. Some are very, very simple to do, like the four Florida illustrations in the Introduction topic or the Richat Structure ones, while others take more tinker time, like the "antique swiss look" one.

    I should note that much of what you can do in 9 is impossible in other packages, because they can't handle the processing of the large, detailed data required by the sparkling detail some of the really breathtaking images need. The 7 x 7 curvature enhancement used in the Richat Structure (load the image in your browser and then click on it to bring it to full resolution, so you can see such detail) would take what, a day?, in ArcGIS Pro if you could get it done without a crash. The ability to do such things in seconds is a really big deal when it comes to artistic creativity.

    marcus52 post(s)
    #09-May-22 18:36

    But the request is for something different isn't it? Unless I've misunderstood, the example image is a georeferenced scanned original paper map (in colour) draped over a DEM.

    dchall8
    1,008 post(s)
    #09-May-22 19:27

    Yes, marcus, that is correct. The process in M9 is to create a hillshade and place a georegistered image over top at some level of transparency. When you make the original image transparent, the colors and contrast wash out. Also you can see in the California map that the mountains cast a shadow onto the margin of the paper to increase the dramatic 3-d effect.

    The California map is two layers: The DEM and the original 1916 geology map. They were assembled in Blender. Once you know what you're doing and where you're going, the Blender part takes 5-10 minutes. Getting a good hillshade can take hours of downloading DEMs, merging them into one image, and clipping to shape. It's fun though!

    Also in Blender you can see the hillshaded image from an oblique viewpoint as you can do in Google Earth.

    Dimitri


    7,413 post(s)
    #10-May-22 06:23

    When you make the original image transparent, the colors and contrast wash out.

    That's where the tinker time comes in. Nothing prevents you from increasing colors and contrast by adjusting layers in 9 and layering multiple layers in different orders. Using many layers is easy in 9. I often use dozens to get exactly the effect I want (same as with PhotoShop).

    you can see the hillshaded image from an oblique viewpoint

    You can do that in Release 8, if you like.

    But the key questions in much of this are what, exactly, is the feature you want in 9, and what priority does that have over other things?

    Since you can do a vast amount of work on effects in 9 already, and if you can do what you want in Blender in 5 or 10 minutes using data prepped by 9, what does that mean for priority for creating 9 equivalents of specific capabilities in Blender, in terms of putting those ahead of other features that are not something you can do already in 9 or in a few minutes through off-the-shelf interoperability with a package you already know?

    Mike Pelletier

    2,122 post(s)
    #10-May-22 14:14

    Beauty speaks to the heart. Maybe that's Shakespeare? :-) Anyway, what I mean to say is that making beautiful maps are very captivating for people in this business and thus gets their attention. Making beauty requires creativity, which is best served by allowing the user to try different methods with speed and control. Like painting with a fine brush vs using a toothpick.

    My limited experience is that other blending (not overlaying with transparency) capable programs are slow with big images. Perhaps 9 has the control now for blending, but it is inaccessible to the average person. Would love to see blending tools after improved labeling, a few more vector tools, dynamic legends, and export arcgis rest service.

    Dimitri


    7,413 post(s)
    #10-May-22 15:48

    You're right when you say blending capable programs are slow, because they don't really have the infrastructure to handle bigger rasters with speed. That's something 9 could do more rapidly.

    You can do blending right now, but that takes queries that are harder for beginners than point and click tools. But when you say "blending tools" that has to be more specific to be sure you get what you want.

    dchall8
    1,008 post(s)
    #11-May-22 05:56

    Instead of naming tools which may or may not achieve the end result, what I would like to do is make the California map entirely inside Manifold 9. The Blender method is to create a DEM in a GIS program and georeference the image of a map. Then Blender magically combines the two with a hillshade effect extending beyond the original outline of the DEM.

    oeaulong

    521 post(s)
    #12-May-22 16:11

    I would extend the elevation of the state after clipping to bounds, out with a 0 elevation collar to show the exaggerated shadows. Thinking some more on this, it would also be my approach if doing it in Blender. I think most of this map can be done in Mfd9. Duplicating the legend and edge graticule/grid marks would be the most tedious challenge.

    Dimitri


    7,413 post(s)
    #13-May-22 12:37

    what I would like to do is make the California map entirely inside Manifold 9.

    Not trying to read your mind, but I suspect you have more in mind than the above, since you can do effects like that, or pretty darned close, in 9 today. I get the impression that you would like something different from the workflow that would be used today, perhaps something more point-and-click or dialog driven, right?

    dchall8
    1,008 post(s)
    #23-May-22 02:57

    Someone else summed it up well when they said words to the effect that when a process is complicated to remember, then it is nice to have a wizard or dialog box with check boxes or alternatives to remind the user of the steps involved. So yes, a Blend transform might be a way to go for mixing textures with images.

    danb

    2,064 post(s)
    #08-May-22 21:53

    Thanks for the link to the PDF dchall8, I will have a read. I certainly need my bottom wiped with this sort of stuff.

    The great thing about the Manifold infrastructure is that it gives you the ability to make almost anything it doesn't have built-in if you have the smarts, but you can also vote to have helper functions or tools built-in to the base package.

    My own model is mixed. Some I make, some I request.


    Landsystems Ltd ... Know your land | www.landsystems.co.nz

    tjhb
    10,094 post(s)
    #08-May-22 23:54

    Ye..es. You can. Easily? Not sure. YMMV.

    If you pointed at the ingredients, I could show how to produce it in 9.

    Possible yes. Obvious? Not so much.

    But was it obvious in 8? Of course not.

    You have done well to create that image, give yourself some credit.

    BTW Dimitri has given some similar examples, applying curvature shading as well as traditional (yes, in 9).

    [I think you have overdone both in your example image. It's always tempting. Depends on output scale obviously.]

    danb

    2,064 post(s)
    #09-May-22 00:13

    The examples come from this link Tim

    Scott Reinhard Maps


    Landsystems Ltd ... Know your land | www.landsystems.co.nz

    tjhb
    10,094 post(s)
    #09-May-22 00:31

    Oh, thanks.

    I will have to have a decent crack now then, won't I? Serves me right.

    (Unless Dimitri beats me. If so he will no doubt do it better.)

    dale

    630 post(s)
    #12-May-22 00:42

    following, and waiting.

    mdsumner


    4,260 post(s)
    #12-May-22 06:50

    (better done in R, takes about 10 lines of code for {rgl} with some off-CRAN mesh-code - or a bit more plus render time with {rayshader} for a presentation-ready result - Manifold may be the fastest with crazy big data, but you don't always have to read every pixel and they weren't exactly the first with lazy and efficient workflows) I'd happily put this together if there's a link to geology map as a geotiff, I can get the elevation - I'm not saying this to dismiss Manifold by any means, I'd love to see them nail this 3D mesh vis so maybe a bit of demo is hepful


    https://github.com/mdsumner

    Mike Pelletier

    2,122 post(s)
    #09-May-22 16:45

    Good to know. Thanks Adam and sorry for banging on the same drum about more basic GIS tools.

    I think it would be great if there was some single click means to push data out of a .map project into a Server project. Also something similar to push the data out to a relational database, file geodatabase, shapefiles, etc. Maybe it is just an example script. On the other hand, it is a very common need to export many layers that it might justify an internal tool that makes it easy to keep a list of commonly exported layers and the export format. Ideally it would also allow selecting which columns/rows to include in the export.

    In my case, I'll want/need to export layers to a Server for internal use, a Server for web maps, to a emergency dispatch center, to a regional GIS integration group, to an internal Assessor mapping application, and to a public data layer download site. I currently use queries to massage layers because different end users need slightly different data. This all gets rather tedious to perform, yet users want the latest data. I'd like to submit this as a suggestion, but thought it good to post first to see if there are other useful thoughts to add.

    Also, glad to see work being down on ArcGIS rest services. Is it possible/legal for Mfd to add the ability to share data out via this format or is it an ESRI proprietary format?

    adamw


    10,447 post(s)
    #12-May-22 14:01

    If you are asking whether we can implement a server that will talk to clients using a subset of the ArcGIS REST API, then there is simply a better option: do our own REST API. The number of clients that support ArcGIS REST is not big, and for web developers - which are the most important audience for such APIs - it doesn't matter much that our API is different from the ArcGIS one, what matters is whether the API is easy to use and powerful enough to support the functionality needed for the specific web application.

    Mike Pelletier

    2,122 post(s)
    #12-May-22 15:07

    In my case, our Assessor Dept. has two different custom mapping applications that support ArcGIS REST and I'm being pushed to use it rather than say uploading fgdb or shapefiles for them to use. Less hassle on their side is probably the motivator and perhaps blame speed issues on us.

    Sounds like a Mfd REST API would be best given the usual case of Arc vs Mfd, but hopefully the Mfd version would be easy enough for users accustomed to ArcGIS REST to make it work. In either event, I imagine this is quite a ways out in the list of to dos. Thanks!

    dchall8
    1,008 post(s)
    #12-May-22 15:30

    Isn't the current Manifold Imageserver more or less a REST type of architecture?

    adamw


    10,447 post(s)
    #12-May-22 15:54

    9 can be a client for REST data sources. The discussion is about letting 9 be a REST server that other clients will interact with (via HTTP). We do not currently have that, but we just added a TCP server and adding other types of servers is now simpler than it was before.

    joebocop
    514 post(s)
    #24-May-22 05:40

    The web developer crowd might really enjoy the ability to just use openlayers or leaflet with manifold as a "server". That could be achieved by server speaking ArcGIS REST or MVT, which those libraries can consume.

    Manifold User Community Use Agreement Copyright (C) 2007-2021 Manifold Software Limited. All rights reserved.