Subscribe to this thread
Home - General / All posts - In R, execution of an M9 query that calls a C# script fails
sga27 post(s)
#29-Nov-22 13:22

Hi all,

I'm working from R with an M9 project (link via ODBC driver), and am trying to execute a query that calls a C# script, but the query fails. However, the query runs without problem in M9. My R is in french, but the error message sounds like "Impossible to load the file or the assembly 'extnet' or one of its dependencies. File not found". I'm wondering if there is a technical reason such an error happens (maybe linked to the ODBC driver) or if I'm doing something wrong.

In more details, here is what I do

In the M9 project the query (called [Query]) that calls the C# script looks like this

-- $manifold$

FUNCTION FillRowNum(@t TABLETABLE AS SCRIPT [S ADD RANK]

  ENTRY 'Script.FillRowNum'

  ENTRYSCHEMA 'Script.FillRowNumSchema';

DROP TABLE test;

SELECT * 

INTO test

FROM

(

TABLE CALL FillRowNum(

      (

    SELECT 0 AS rownum, name, first(property), first(value)

    FROM mfd_meta

    GROUP BY name

    ORDER BY name

    )

)

);

The C# script in [S ADD RANK] is a script that was shared by one of the users of the forum. It adds a rank to a table. Here is its content

// C#

using M = Manifold;

class Script

{

// get names of all schema fields

static string[] GetSchemaFields(M.Schema schema)

{

  M.ValueSet values = Manifold.Application.CreateValueSet();

  foreach (M.Schema.Field field in schema.Fields)

    values.AddValue(field.Name);

  return values.GetValueNames();

}

// create a copy of source table and fill first field with row numbers

public static M.Table FillRowNum(M.Table src)

{

  // create empty copy of source table

  M.Table tgt = FillRowNumSchema(src);

  // copy all records from source table

  M.Sequence sequence = src.SearchAll(GetSchemaFields(src.GetSchema()));

  if (sequence == null)

    return tgt; // no records

  long counter = 0; // int64

  while (sequence.Fetch())

  {

    M.ValueSet values = sequence.GetValues();

    values[0].Data = counter++;

    tgt.Insert(values);

  }

  return tgt;

}

// tell the query engine the result of FillRowNum will be a table

//   with same schema as source table

public static M.Table FillRowNumSchema(M.Table src)

{

  // use all fields, ignore indexes and constraints

  M.Schema schema = Manifold.Application.CreateSchema();

  foreach (M.Schema.Field field in src.GetSchema().Fields)

    schema.AddField(field.Name, field.Type); // ignore expressions

  // create empty table with composed schema

  M.Table tgt = Manifold.Application.CreateTable();

  tgt.Design(schema);

  return tgt;

}

static M.Context Manifold;

static void Main()

{

}

}

And the R script contains

library(odbc)

library(DBI)

dstring1 <- "DRIVER={Manifold 9.0 Project Driver (*.map)};DBQ=D:\\manifold\\test_scriptr.map;Unicode=True;Ansi=False;OpenGIS=True;DSN=Default;"

con_m9 <- dbConnect(odbc::odbc(), .connection_string  = dstring1 )

dbSendQuery(con_m9, "EXECUTE [Query]")

Many thanks in advance for your inputs

rk
575 post(s)
#29-Nov-22 15:31

For errors like "Impossible to load the file or the assembly 'extnet' or one of its dependencies" I recommend Process Monitor. Filter by your process (probably the R's process), try running your script and look at Tools -> File Summary. You'll see which files were tried to open and which were successfully opened.

sga27 post(s)
#30-Nov-22 10:10

Thanks rk!

I tried to follow your advice. I was initially using the driver of m9 base version (not edge). Using the process monitor, I noticed that the file C:\Program Files\Manifold\v9.0\extnet.dll could not be opened. Yet in the base version, the dll is called ExtNet.dll and I thought that the capitals could have been messing things up here.

Noticing that it was called extnet.dll in the edge version (without capitals), I switched to the driver of the edge version: the error in R now sounds "Cannot compile script" (the extnet.dll has disappeared from the list in process monitor).

Any idea?

rk
575 post(s)
#30-Nov-22 10:39

s

C:\Program Files\Manifold\v9.0\extnet.dll could not be opened

On windows letter case should not matter for file names. If the file is there, why cannot it be opened, I wonder?

Have you checked Help->About in Manifold. Try re-installing the ODBC driver.

sga27 post(s)
#30-Nov-22 11:55

I reinstalled the ODBC driver. Bellow is the output from process monitor file summary, all files related to manifold.

I also reinstalled the edge driver. And now I have the same message with the base and the edge driver (same error as before, approximate translation in the first post):

Error in new_result(connection@ptr, statement, immediate) :

nanodbc/nanodbc.cpp:1412: 42000: S ADD RANK (Script.FillRowNum): Impossible de charger le fichier ou l'assembly 'extnet, Version=1.0.0.0, Culture=neutral, PublicKeyToken=058e4cb7a1c54ee9' ou une de ses dépendances. Le fichier spécifié est introuvable.

Both files Ext.dll and ExtNet.dll exist

sga27 post(s)
#30-Nov-22 12:12

Oops, header was missing in the picture from process monitor

rk
575 post(s)
#30-Nov-22 15:24

Fuslogvw.exe is another tool for debugging such problems. I have not used it before. Seems a bit difficult to get started.

It comes with Visual Studio. You must open Developer Command Prompt for VS as Administrator.

Here are some simple instructions (skip the Registry editor part).

https://techcommunity.microsoft.com/t5/iis-support-blog/fusion-log-viewer-fuslogvw-exe/ba-p/784396

I got it to log some errors.

adamw


10,281 post(s)
#01-Dec-22 13:02

So, we have R that calls the Manifold ODBC driver that runs SQL that calls a .NET script.

You said that Manifold itself runs SQL + script just fine. I take it that if you rework SQL to not use a script, R + Manifold ODBC + SQL will also work just fine. It's just the last + script that has issues.

As Riivo suggests, try using FUSLOGVW to generate a log to see what exactly happens when Manifold is trying to run EXTNET.DLL (the case does not matter, indeed). This could be either of two things: Manifold tries to locate EXTNET.DLL and fails (it should first try the same folder as EXT.DLL, though, so I'd be surprised if it fails here), or Manifold successfully locates EXTNET.DLL and it then fails to load for some reason (I think this is what is probably happening). The log might help determine what that reason is.

Just in case, what is the version of R? It is reasonably new, correct?

sga27 post(s)
#13-Dec-22 14:31

Hi rk and adamw

Thanks for the suggestions and sorry for the late reply.

To answer adamw's questions

I take it that if you rework SQL to not use a script, R + Manifold ODBC + SQL will also work just fine. It's just the last + script that has issues.

Yes indeed, it works if I rewrite the script using pure SQL. This is what I did for the moment, but I'd be glad if scripts could be used as well in the future

what is the version of R?

4.2.2, so the latest build.

Anyway, I tried to use fuslogvw. First it simply said that the file could not be found, but since the file I tried to open did exist, I went to the properties of the file and discovered a small message that said that the file was coming from another computer and therefore was locked. I checked the checkbox "unlock" next to it, did the same for all dlls and retried.

Now the error says

*** Entrée du journal Binder d'assembly (13.12.2022 @ 15:37:08) *** L'opération a échoué. Résultat de liaison : hr = 0x80070002. Le fichier spécifié est introuvable. Gestionnaire des assemblys chargé à partir de : C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll Exécution sous l'exécutable C:\Program Files\RStudio\bin\rsession-utf8.exe --- Un journal des erreurs détaillé suit. === Informations d'état de liaison préalable === JRN : DisplayName = extnet, Version=1.0.0.0, Culture=neutral, PublicKeyToken=058e4cb7a1c54ee9 (Fully-specified) JRN : Appbase = file:///C:/Program Files/RStudio/bin/ JRN : PrivatePath initial = NULL JRN : base dynamique = NULL JRN : base de cache = NULL JRN : AppName = rsession-utf8.exe Assembly appelant : (Unknown). === JRN : cette liaison démarre dans le contexte de chargement de default. JRN : tentative de téléchargement du fichier de configuration de l'application à partir de file:///C:/Program Files/RStudio/bin/rsession-utf8.exe.config. JRN : Le fichier de configuration C:\Program Files\RStudio\bin\rsession-utf8.exe.config n'existe pas. JRN : aucun fichier de configuration de l'application n'a été trouvé. JRN : utilisation du fichier de configuration d'hôte : JRN : utilisation du fichier de configuration de l'ordinateur à partir de C:\Windows\Microsoft.NET\Framework64\v4.0.30319\config\machine.config. JRN : référence post-stratégie : extnet, Version=1.0.0.0, Culture=neutral, PublicKeyToken=058e4cb7a1c54ee9 JRN : échec de la recherche dans le GAC. JRN : tentative de téléchargement de la nouvelle URL file:///C:/Program Files/RStudio/bin/extnet.DLL. JRN : tentative de téléchargement de la nouvelle URL file:///C:/Program Files/RStudio/bin/extnet/extnet.DLL. JRN : tentative de téléchargement de la nouvelle URL file:///C:/Program Files/RStudio/bin/extnet.EXE. JRN : tentative de téléchargement de la nouvelle URL file:///C:/Program Files/RStudio/bin/extnet/extnet.EXE. JRN : toutes les URL recherchées ont été tentées et ont échoué.

Sorry, the error logs are still in frech. But basically it seems that it tries to find the dlls in the RStudio folder. Whereas before, in the previous log (before I checked the unlock box), it was trying to locate the extnet dll in the manifold folder.

Any idea on what is going on or how I could try to fix it?

tjhb
10,027 post(s)
#13-Dec-22 17:25

Interesting.

It is normally necessary to “unblock” all source files (those that have, normally, been downloaded), in a folder location where you have full write permissions, before the files are installed, copied or moved to the location where they will be used.

This is to avoid creation of an alternative file stream for each such file in the destination folder, which is not always removed when the file is (ostensibly) “unblocked”.

It is possible to turn off this whole safety mechanism entirely either by group policy or by a registry patch, and it is also possible clean a set of folders of all of these alternate file streams if you really need to. Both of these things are covered elsewhere on the forum—just say if you want pointers to those threads.

But for now I would remove all the files your code needs to call from their installed locations, download new copies of the source files to (say) your desktop, “unblock” those files there, and only then install, move or copy them to where they need to go.

rk
575 post(s)
#13-Dec-22 17:39

I'm not sure these indicate a fatal situation:

LOG: All probing URLs attempted and failed.

JRN : toutes les URL recherchées ont été tentées et ont échoué.

Does your log contain any hr = 0x80070002?

Did it contain hr = 0x80070002 before unblocking?

Or lines starting with other than 'JRN:', like

ERR: Unrecoverable error occurred during pre-download check (hr = 0x80070002).

sga27 post(s)
#13-Dec-22 19:04

Thanks tjhb for explaining this, I was not aware of this.

I followed your advice and unblocked all dlls before copying them. But the error remains...

To answer rk's questions

Does your log contain any hr = 0x80070002?

yes, I sometimes get an hr = 0x8007002 error (in fact I don't always get the same error)

Did it contain hr = 0x80070002 before unblocking?

No, it was just saying the file could not be found.

Or lines starting with other than 'JRN:', like

Yes, the exact error you discribe with error during pre-download

sga27 post(s)
#14-Dec-22 11:17

To make it clearer, I investigated a bit why I had two different error messages.

In fact, I was initially using R version 4.0.2 with ODBC package version 1.3.0. To make sure that it was not related to my R's version I updated to R 4.2.2 with ODBC package version 1.3.3.

Now if I run my older R version, after unlocking my dlls, I get no assembly binding error, but I get a "Cannot compile script" error. There is no error in fuslogvw, and I have no other indications.

If I run my newer version of R, I get the error in fuslogvw

ERR: Unrecoverable error occurred during pre-download check (hr = 0x80070002).

So there seems to be two different problems: one related to the binding assembly in my new R and another vaguer error in my old R.

adamw


10,281 post(s)
#20-Dec-22 13:12

OK.

First, some observations. 0x80070002 is "The system cannot find the file specified", if it is relevant at all (it might not be), we are probably still talking about EXTNET.DLL not being able to load. The paths to EXTNET.DLL in the fusion log are all wrong, that's why they fail. That .NET is trying tons of wrong paths first is fine, that's by design, but what's puzzling is why it doesn't even try the right path near EXT.DLL.

That with the old version of R you are getting "Cannot compile script" (and with the new version something else) is pretty interesting. To clarify -- are you sure you are getting "Cannot compile script" and not "Error compiling script"? The latter comes from the inside of EXTNET.DLL and would mean that it managed to get loaded. The former comes from EXT.DLL and likely means that EXTNET.DLL failed to load. The error messages are very similar to each other, I just want to make sure we are looking at the correct one. The new version of R is 64-bit only, but the old one could be either 32-bit or 64-bit - are you getting the "Cannot compile script" error using the 32-bit or the 64-bit version of R? (Maybe the 32-bit version succeeds and the 64-bit one fails?)

Since we are talking about Manifold running as the ODBC driver, what path is the ODBC driver installed to? Could it be that you unlocked the wrong instance of EXTNET.DLL and the ODBC driver is pointing to EXT.DLL in some other folder?

sga27 post(s)
#21-Dec-22 10:36

Thanks adamw for your answer!

are you sure you are getting "Cannot compile script"

Yes, this exact error (error in english so no translation approximation). Yet, the difference in error has nothing to do with the version of R, as I initially thought. In fact, if I run the script twice, I get a cannot compile script error. So sorry for the confusion. If I close and reopen R, the "cannot load assembly" error comes back again.

are you getting the "Cannot compile script" error using the 32-bit or the 64-bit version of R?

I'm only using 64-bit version of R (both with old and new version).

what path is the ODBC driver installed to?

I'm currently using the edge (experimental) driver. My manifold is installed in C:\Program Files\Manifold\manifold-9.0.178.5. I opened the manifold.exe in there, and using the help>about I installed the driver. I guess the driver is then installed in C:\Program Files\Manifold\manifold-9.0.178.5, right?All dlls in that folder are unblocked.

Now, looking in more details at the logs, I have two entries about extnet.dll

One whose description in fuslog "WhereRefBind!Host=(LocalMachine)!FileName=(extnet.dll)"

L'opération a réussi. Résultat de liaison : hr = 0x0. L’opération a réussi. Gestionnaire des assemblys chargé à partir de : C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll Exécution sous l'exécutable C:\Program Files\RStudio\bin\rsession-utf8.exe --- Un journal des erreurs détaillé suit. JRN : liaison explicite IJW. Chemin d'accès au fichier : C:\Program Files\Manifold\manifold-9.0.178.5\extnet.dll. JRN : la liaison d'assembly IJW a retourné un fichier introuvable.

So in this log, it looks at the right location. Yet the log is a bit confusing. It first says operation successeful but the last thing says "returned a file that cannot be found".

The second (actually both are called at the exact same timestamp so I don't know if it is the first or second) entry's description is called "extnet, Version=1.0.0.0, Culture=neutral, PublicKeyToken=058e4cb7a1c54ee9"

L'opération a échoué. Résultat de liaison : hr = 0x80070002. Le fichier spécifié est introuvable. Gestionnaire des assemblys chargé à partir de : C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll Exécution sous l'exécutable C:\Program Files\RStudio\bin\rsession-utf8.exe --- Un journal des erreurs détaillé suit. === Informations d'état de liaison préalable === JRN : DisplayName = extnet, Version=1.0.0.0, Culture=neutral, PublicKeyToken=058e4cb7a1c54ee9 (Fully-specified) JRN : Appbase = file:///C:/Program Files/RStudio/bin/ JRN : PrivatePath initial = NULL JRN : base dynamique = NULL JRN : base de cache = NULL JRN : AppName = rsession-utf8.exe Assembly appelant : (Unknown). === JRN : cette liaison démarre dans le contexte de chargement de default. JRN : tentative de téléchargement du fichier de configuration de l'application à partir de file:///C:/Program Files/RStudio/bin/rsession-utf8.exe.config. JRN : Le fichier de configuration C:\Program Files\RStudio\bin\rsession-utf8.exe.config n'existe pas. JRN : aucun fichier de configuration de l'application n'a été trouvé. JRN : utilisation du fichier de configuration d'hôte : JRN : utilisation du fichier de configuration de l'ordinateur à partir de C:\Windows\Microsoft.NET\Framework64\v4.0.30319\config\machine.config. JRN : référence post-stratégie : extnet, Version=1.0.0.0, Culture=neutral, PublicKeyToken=058e4cb7a1c54ee9 JRN : échec de la recherche dans le GAC. JRN : tentative de téléchargement de la nouvelle URL file:///C:/Program Files/RStudio/bin/extnet.DLL. JRN : tentative de téléchargement de la nouvelle URL file:///C:/Program Files/RStudio/bin/extnet/extnet.DLL. JRN : tentative de téléchargement de la nouvelle URL file:///C:/Program Files/RStudio/bin/extnet.EXE. JRN : tentative de téléchargement de la nouvelle URL file:///C:/Program Files/RStudio/bin/extnet/extnet.EXE. JRN : toutes les URL recherchées ont été tentées et ont échoué.

And in there, the right file is not looked at.

I hope this clarification can help somebody to understand what's going on. I also tried on a different computer but it doesn't work there as well.

adamw


10,281 post(s)
#22-Dec-22 13:55

OK, it is evident that we are dealing with EXT.DLL trying to load EXTNET.DLL and failing. The reason could be managed (on the level of .NET) or unmanaged (below it).

To exclude the latter, since you are using a cutting edge build, hence portable, make sure to install the latest Visual C++ redistributable from:

https://docs.microsoft.com/en-US/cpp/windows/latest-supported-vc-redist

You need the 64-bit version only (x64) ...although it doesn't hurt to install the 32-bit one, too.

Other than that, we'll try to reproduce the issue in our lab.

tjhb
10,027 post(s)
#22-Dec-22 15:03

It probably is not this simple, but I am simple, so I will ask:

I'm currently using the edge (experimental) driver. My manifold is installed in C:\Program Files\Manifold\manifold-9.0.178.5. ... All dlls in that folder are unblocked.

When you copied the Manifold files for the Edge build to that folder, had the downloaded Manifold source .zip file already been unbloched somewhere else (somewherewhere you had full file and folder permissions)?

If not, extnet.dll (and other files) may only appear to be unblocked. (Because you don't have full file and folder permissions in C:\Program Files\... and its subfolders.) It happens.

sga27 post(s)
#23-Dec-22 09:20

Thanks tjhb for the suggestion. Yes the files were unblocked before copying them in the Program files folder

rk
575 post(s)
#22-Dec-22 15:32

I do not fully understand this, but it seems like (in the fuslogvw viewpoint at least) rsession-utf8.exe is the one that is trying to load extnet.dll. And rsession-utf8.exe has on knowledge where to search for extnet.dll. It only looks in its own folder, C:\Program Files\RStudio\bin\.

Like here, when before 9.0.177.2 ironpython.dll (3.4-beta) required System.Memory.dll and manifold.exe could not find System.Memory.dll unless it was copied beside manifold.exe.

adamw


10,281 post(s)
#22-Dec-22 16:13

The fusion log references paths relative to RSESSION-UTF8.EXE because it always tries paths relative to the running EXE, that's normal.

EXT.DLL is loading EXTNET.DLL not via a managed call (like, Assembly::Load), but via an unmanaged call (LoadLibraryXXX). The latter can fail early = before .NET gets involved, or late = after. Things actually get more complicated because with the way EXTNET.DLL is designed, there's also 'failing in the middle' -- of two different sorts -- but I am going to omit the details here before we have a page of text. :-) Since the fusion log does not contain a .NET failure, it seems that we are dealing with a failure that is happening earlier than that. If the latest CRT is installed, the earliest type of failure = failing to load the DLL because one of the DLLs it depends on is not installed or cannot load -- would be unlikely. This would then leave only failures in the middle, which are perhaps too laborious to diagnose remotely, so we'll try to diagnose them locally in our lab, after reproducing the issue.

(PS: Yes, maybe the root cause is the same as with IRONPYTHON.DLL. We'll see.)

sga27 post(s)
#23-Dec-22 09:19

Thanks for the hint

I installed the latest Visual C++ redistributable as suggested, restarted my computer. But the error remains...

I hope you will be able to reproduce the error in your lab

Manifold User Community Use Agreement Copyright (C) 2007-2021 Manifold Software Limited. All rights reserved.