I figured something out. I tried IronPython 3.4.0 and got error of not being able to load System.Memory v18.104.22.168
System.Memory.dll comes with and is placed right next to Ironpython.dll in manifold/shared.
Using Process Monitor (Tools -> File summary) I could see that manifold process looks for System.Memory from following locations but not where Ironpython.dll, which requires it, is.
Workaround is to copy System.Memory to bin64.
I know from previous experience that I can put gdal.dll in any subfolder to bin64, but plugins for gdal MUST be at bin64\gdalplugins. Otherwise manifold process does not find them. Usually gdalplugins folder is relative to gdal.dll.
This looks like "current directory" or "relative path" issue.
I could probably install System.Memory into GAC but I read there are some issues(*). I do not understand if I should avoid GAC or just use it and not worry.