This is an announcement that starting with the next build, we are going to stop embedding V8. We will continue to be supporting scripts in Javascript via other engines.

Existing V8 scripts will be mapped to JScript.NET. They will have to be rewritten in places where they access Manifold objects, because our object model for .NET and COM scripts is different from our object model for V8 scripts. We don't expect this to be much of a problem as we don't expect the number of V8 scripts to be particularly high. The object model for V8 was being kept very basic, waiting for its turn to be co-developed together with some other features that we planned. As we moved closer to implementing these features we have revised our plans and one of the results was that we will not need to support a separate object model at all.

More details:

We started embedding V8 with two goals in mind: (a) allow running multitudes of short scripts, like those used by Manifold 8's active columns, with minimum overhead, to let active columns scale up to big tables, and (b) expose a dynamic object model for web applications that would be easier to write to than the traditional object model. We also used some of the V8's services internally for various tasks.

Over time, several things happened. First, the task of running multitudes of short scripts with minimum overhead was successfully solved by our query engine. (This is why our computed fields use SQL.) Second, new versions of .NET started supporting dynamic objects. Third, we developed our own replacements for V8's services that we used internally, which better suited our purposes.

Importantly, the costs of keeping V8 embedded kept increasing. V8 is a very active project that is updated frequently. Unfortunately, due to the way it is being developed, the main targeted platform for development is Linux, not Windows. While V8 nominally supports Windows as a development platform, the quality of that support is lacking. Windows builds break frequently (in the last 15+ major updates, Windows builds have been broken every single time) and have to be patched. This is a lot of manual work that comes at a significant cost.

Given the above, we decided to stop embedding V8 as the costs of doing so were outweighing the benefits.

In the future, our focus for scripts is going to be on .NET 5 and on CPython. .NET has evolved a lot, the new version is a big step up from .NET 4.8 that we are currently using, it has a lot to offer. CPython got to be the de-facto standard for writing scientific code, with thousands of mature libraries and plethora of experience and acceptance in the field. These two things are what we are going to be working on in this area.

We expect to have the next build ready later this week.

Will CPython be included in the next build to fill the gap? Will my SQL9 functions that call javascript scripts fail in the next build? Thank you.

Your javascript scripts--if they are V8, then you have done a lot of manual exploring? Its object model has never been published.

Otherwise, obviously JScript.NET will stay, also plain MS jscript via COM, both fully supported for now I take it.

My scripts leverage only the js language facilities and some 3rd party libraries (turf, for example); they do not access any part of the Manifold Object Model. As such, they can be rewritten, when I find the time and ability to profitably write them in another language. I had used all of a client's existing turf.js code from their node.js workflow to avoid having to re-write everything for them in SQL9, which was a pretty cool savings.

It will be great if CPython is included in subsequent builds. Part of what I enjoyed about V8 was that it was nice not to have to monkey with environments, additional language or runtime installations, which made for a comically low deployment effort. Contrast that with IronPython, which required some hunting for community-supported installers, and then modification of env vars or copying of DLLs or whatever. A little more friction there.

Will this mean that any "IMS" functionality will be ASP.net-based, rather than js?

Using Javascript language facilities not specific to V8 - as long as these facilities aren't too modern (JScript.NET supports a relatively early version of the standard, no latest features) - should be fine. Same for libraries. We'll check your example and some others.

Regarding using IronPython with 9, you can just unpack the ZIP file with IronPython DLLs referenced on the Downloads page (scroll down to: Python) into the ~\shared folder.

IMS functionality will use Javascript for client-side scripts, as always.

A follow-up: checked your example (turf), it is unfortunately a no-go, the library does use things that JScript.NET does not understand. We will keep V8 out, but will try to find a compromise solution in the near future.

Here's one method of running V8 scripts that can be used after we stop embedding it: post below.

Otherwise, obviously JScript.NET will stay, also plain MS jscript via COM, both fully supported for now I take it.

Yes. For people into javascript those options remain.

V8 may be a slightly better javascript, but breaking the Windows version every single time in the last 15+ major updates (wouldn't even compile without manual fixes) is not good enough quality to be embedded in 9. I think it's better to take the effort that went into repairing V8 and to put it into bringing CPython forward (CPython was always in the plan). CPython is not imminent, nor is it without its own evolutionary issues, but putting resources into it that were being spent on V8 will make it happen sooner.

One other gain of not embedding V8 is that the installation packages get smaller. I think you save almost 20MB in size for the portable Edge .zip.

Support for CPython requires a few features which we don't have right now. We are going to add them, but this will take time. You can run Python scripts using IronPython, differences from CPython are in the libraries you can call.

SQL9 functions originally written in V8 will be ran as JScript.NET. If a function does something like sin / cos, or, say, string manipulations, porting to JScript.NET should be straightforward, JScript.NET has all that available. If a function accesses Manifold objects, the places where it does so will have to be rewritten.

Yay CPython!

While IronPython development seems to be still healthy--an alpha release targetting Python 3 is imminent (this is fantastic)--the adoption of CPython would allow us to use NumPy, SciPy and so many other serious C-based libraries.

Almost as good as .NET in theory, but possibly significantly better in practice because, well, scientific consensus. The possibility to share much more actual code (both in and out).

(Even more so if you can model NumPy arrays onto Manifold images for GPGPU computation, but that is speculative.)

CPython will be an awesome addition. I wouldn't stake my paycheck on IronPython breaking free of v2.

Attachments:
Screenshot 2021-04-12 103219.png

Regarding the comment about IronPython 2 -> 3, here is how far they have got.

https://github.com/IronLanguages/ironpython3/blob/master/WhatsNewInPython30.md

I am very encouraged. If I could write good enough C# I would contribute.

Again, I would prefer CPython, although there is a downside...

Any Manifold code using CPython will, I think, have to be single threaded, at least for the duration of the CPython code. There will have to be locks, expensive synchronisation, and so on, because Python was not designed from the start to respect parallelism.

IronPython is better in this respect, because it has always had to respect .NET design principles.

So CPython may be a bit slower... or may not, given efficient use of NumPy for instance.

(At best what I wrote above will be roughly right. A bit out of my depth.)

But then, I don’t even know what PSA stands for, other than prostate-specific antibody. Adamw might not have quite got there yet, but for most of us (ideally about half) it is only a question of time!

[I joke but, fellow men, get a test if you are over 45. If you are younger, get a baseline you can use later. It matters quite a lot. Most people ignore or are unaware of the issue until it is a problem. Then options become limited, and while rarely fatal they can be crippling.]

Any Manifold code using CPython will, I think, have to be single threaded, at least for the duration of the CPython code. There will have to be locks, expensive synchronisation, and so on, because Python was not designed from the start to respect parallelism.

This is the main issue, yes. CPython has a global lock which it needs to protect its own structures, and the presence of this lock is a big problem for multi-threaded environments like Manifold. It's not just about the performance (although that degrades, too), it's that multi-threaded environments like ours also use locks (local ones), and there is a possibility of a deadlock.

Example: there are two threads, A and B, they call CPython. Thread A takes a local lock X and goes to prepare arguments for CPython. Thread B calls CPython, CPython takes its global lock G and starts running Python code. Thread A proceeds to call CPython and CPython waits on its global lock G. The Python code running in thread B calls a Manifold object to try and fetch some data from a table and that data is behind the local lock X taken by the thread A earlier. End result: thread A waits on lock G taken by thread B and cannot proceed, thread B waits on local lock X taken by thread A and cannot proceed, we have a deadlock, the process has to be terminated. All this happens because CPython has a global lock G which it takes for long periods of time, other scripting environments release all such locks when they are calling into the unknown (thread B should have released G prior to calling into Manifold), CPython cannot do this because it hasn't been written with threads in mind and needs G as a big protection blanket.

There are ways to make it work, though, at least for large scripts.

i hate javascript, and love python too . There was in the past a Web Client Front End side of python implementation that people can be use for free but javascript win !! ( i learn also zope that it a king of ASP but with python and a interesting OO like concept )

Book about Science , cosmological model , Interstellar travels

Boyle surface fr ,en

I will show how one can continue to use V8 even after we stop embedding it (can do this right now, too) using an example from this thread:

Accessing turf.js functions in queries

To recap, there is a Javascript library named turf.js, it contains various geometry functions, the task is to call the function that generates a hexagonal grid and output this into a drawing.

The original code ran turf.js using V8. JScript.NET cannot run it because the language version it supports is fairly old and turf.js uses features added in later versions. In the next build we are going to stop embedding V8. However, you can still use V8 to run turf.js and similar code, here's how.

There is a .NET project for running V8 (and some other) scripts, called ClearScript (github here). This is a Microsoft project, well-maintained and kept up to date.

The attached ZIP archive contains the latest ClearScript DLLs. Unpack the archive, then copy all DLLs except ClearScriptV8.win-x64.dll (leaving it will do no harm) into ~\bin and all DLLs except ClearScriptV8.win-x86.dll into ~\bin64.

Then launch a new instance of Manifold, open the attached MXB. Open the drawing, observe that it is empty. Now run the query, it should populate the drawing with a hexagonal grid created using turf.js.

Here's how it works:

The comments component, Turf_script, contains the text of turf.js. This is taken verbatim from the URL referenced in examples on turfjs.org: https://unpkg.com/@turf/turf@6.3.0/turf.min.js There are no modifications whatsoever, the text is simply copied over as is.

The script component, Script, contains this:

// C#

//

// $reference: Microsoft.Csharp.dll

// $reference: System.Core.dll

// $reference: ClearScript.Core.dll

// $reference: ClearScript.V8.dll

// $reference: ClearScript.Windows.Core.dll

// $reference: ClearScript.Windows.dll

//

using System;

using Microsoft.ClearScript;

using Microsoft.ClearScript.JavaScript;

using Microsoft.ClearScript.V8;

class Script

{

public static string HexGrid(

  double xmin, double ymin, double xmax, double ymax, double cellSide)

{

  Manifold.Application app = Manifold.Application;

  Manifold.Database db = app.GetDatabaseRoot();

  using (V8ScriptEngine engine = new V8ScriptEngine())

  {

    engine.Execute(db.GetProperty("Turf_script", "Text"));

    engine.Execute(string.Format(

      "var grid = " +

      "JSON.stringify(turf.hexGrid([{0}, {1}, {2}, {3}], {4}).features);",

      xmin, ymin, xmax, ymax, cellSide));

    return engine.Script.grid;

  }

}

static Manifold.Context Manifold;

static void Main()

{

}

}

The Main function does nothing. We are using the script for its HexGrid function which takes the parameters for the grid and returns it as a JSON. The script references ClearScript DLLs using $reference. HexGrid creates an instance of V8. Then it tells V8 to run turf.js, taking the source code for it from the comments component. Then it tells V8 to run the line: 'var grid = JSON.stringify(turf.hexGrid(...));' putting the parameters into the string. After this line finishes running, the generated geometry is in the variable named 'grid' inside the V8 script. The function then returns the value in that variable to the caller.

The query component, Query, contains this:

--SQL9

FUNCTION hexGrid(

  @xmin FLOAT64, @ymin FLOAT64, @xmax FLOAT64, @ymax FLOAT64,

  @cell FLOAT64) NVARCHAR AS SCRIPT [Script] ENTRY 'Script.HexGrid';

DELETE FROM [drawing];

INSERT INTO [drawing] ([geom])

SELECT StringJsonGeoGeom(StringJsonValue(value, 'geometry', false))

FROM CALL StringToJsonArrayValues(hexGrid(-95, 30 ,-85, 40, 50));

We first declare that we are going to use a function named Script.HexGrid from the component Script. Then we delete all records from Drawing. Then we call the function, split the results into individual strings for each returned polygon, convert each string into a GEOM, and insert that into Drawing.

We can also modify Script to download the contents of turf.js from the URL automatically (and put it into a corresponding comments component if it has not been created yet).

Hope this helps.

Attachments:
clearscript-dlls.zip
turfjs-clearscript.mxb

Well, I can't express my gratitude sufficiently, that effort is very very much appreciated. I'd never even heard of ClearScript, let alone possessed the knowledge of how to use it in c#. This example gives me lots to work with; thank you.

One really great thing ClearScript allows doing is exposing the entire Manifold object model for .NET for use from Javascript. They have what they call a "FAQtorial", item 22 in there shows how this could be done.

We might embed ClearScript in the future, eg, as an optional component. Need more experience with it, but it seems *much* friendlier to embed than V8.

[deleted]

Book about Science , cosmological model , Interstellar travels

Boyle surface fr ,en

I test the script in For 5 location of manifold.exe ( ?? / default path =official / Edge path and , 32/ 64 bit )

"C:\Program Files\Manifold\v9.0\Bin\Manifold.exe" 172........ default path 64bit=>run, no result

"C:\Program Files\Manifold\v9.0\Bin\Manifold.exe"172.........default path 32 bit =>run,no result

"C:\Program Files (x86)\Manifold System\Manifold.exe"..?? path 32 bit =>need Serial number

"C:\WORK\gis\edge\bin64\manifold.exe"173.5.......................edge 64 bit =>run SC introuvable

"C:\WORK\gis\edge\bin\manifold.exe"173.5..........................edge 32 bit => run no result

since mxb file can be convert to 64 or 32 map implicit , after open mxb file i save project to a new map file with a mix of 32/64bit and Default/edge name before extension ".map" and sometime ll download mxb file each time.

_________________________________________________________________________________

I am shure than i could open the project and run the script with result but after redo the test it seem that i don't achieve now the test whatever mix i use . I ll update to 174 to see if the new version let me run the script with success .

I think manifold to something implicetly in mxb or map or 32/64Bit or official or edge in the way it configure dll ( gac) ...in a way i add and remove ClearScriptxxxx dll ?

Where i have to put the ClearScript***.dll for each mix of manifold ?

Does open a mxb in 32 or 64 manifold don't change the mxb since manifold create a map file ( 32 or 54 bitss) ?

Does component install during offfical install can change the behaviour of edge zip install ?

I don't remenber that i have to reboot the OS after install manifold last version ? for 174 manifold ask a reboot OS.

regard's

Book about Science , cosmological model , Interstellar travels

Boyle surface fr ,en

even after update both official and edge version to 174, none give me result ( table still empty ) !!

(nothing related to ClearScript : Does Manifold icon change only for edge not official for each version ? )

Attachments:
icon.png

Book about Science , cosmological model , Interstellar travels

Boyle surface fr ,en

Yes. That's Donatello, by the way, from the famous painting of five Italian Renaissance artists in the Louvre.

Where i have to put the ClearScript***.dll for each mix of manifold ?

Into ~\bin and ~\bin64 of each install.

Does open a mxb in 32 or 64 manifold don't change the mxb since manifold create a map file ( 32 or 54 bitss) ?

Both MXB and MAP are bit-neutral, you use the same file in 32-bit and 64-bit mode.

Does component install during offfical install can change the behaviour of edge zip install ?

The only components shared between installs are system ones, like Visual C++ Runtime, .NET, etc. Everything else we load from ~\bin, ~\bin64, ~\shared folders specific to the install, and from PATH. Altering PATH will this have an effect on all installs. However, we probe ~\bin, ~\bin64, ~\shared first.

I don't remenber that i have to reboot the OS after install manifold last version ? for 174 manifold ask a reboot OS.

Our own code does not require a reboot after an update, but system components mentioned above (.NET, etc) might. If you are installing an MSI, the system will ask for a reboot if needed. If you are installing a portable ZIP, you don't need a reboot, unless you also upgrade the system components (which is something you should do regularly if you only ever install portable ZIPs), and then the installers for these components will ask you for a reboot if needed.

ask myself what mean features in this code above inside stringify

turf.hexGrid([{0}, {1}, {2}, {3}], {4}).features

I try to have a look in the API doc of Turf.js | Advanced geospatial analysis (turfjs.org) .

when read turf documentation AREA the term features is also use !!

What mean features in doc and in source code ?

discover FeaturePolicy.features() - Web APIs | MDN (mozilla.org) but not related !

Attachments:
turf_area.png

Book about Science , cosmological model , Interstellar travels

Boyle surface fr ,en

Those are not Manifold questions, so this forum is not the right place for them. Those are questions for the Turf community and whatever forum they use, or for general programming forums like stackexchange.

Yes , I don't think to go to the turf community , i don't know if i can post a question in github , i ll ask to a javascript forum then .

Book about Science , cosmological model , Interstellar travels

Boyle surface fr ,en

it was easy to find by read rfc7946 (ietf.org)

Book about Science , cosmological model , Interstellar travels

Boyle surface fr ,en