Subscribe to this thread
Home - General / All posts - M9 does not show special characters as expected after importing .csv file // UTF-8 or ANSI ?
yves61
439 post(s)
#05-Mar-24 10:31

I have a .csv file that opens well in Notepad++.

It opens with UTF-8. I see all the special characters right as expected. (enclosed image Notepad_UTF-8.png).

Still with Notepad , when I convert this file to ANSI then the special characters change to unwanted characters. (enclosed image Notepad_ANSI.png)

After importing the .csv file in M9, M9 shows the special characters - as unwanted - as shown in the Notepad_ANSI image.

What should I do to have the special characters right in M9 ? ( enclosed image M9.png)

Attachments:
M9.PNG
Notepad_ANSI.PNG
Notepad_UTF-8.PNG

yves61
439 post(s)
#05-Mar-24 10:49

I have found a work-around for my case.

Open the .csv file with Notepad -> convert to ANSI -> save the file. Important : Save the file as normal txt but keep the .csv file extension ! Import the file in M9. Bang !!! All characters as expected. (enclosed M9_solved.png).

Or open the .csv file in Notepad -> Save As ; keep in mind to change the "code" setting from UTF-8 to ANSI.

Still, is there another way - within M9 - to achieve the same result ? If so, how ?

Attachments:
M9_solved.PNG
M9_solved_v2.PNG

rk
627 post(s)
#05-Mar-24 11:12

In Notepad++ Convert to UTF-8-BOM

Then Manifold 9 reads it as UTF-8

Attachments:
mfd_reads_UTF-8-BOM.png

yves61
439 post(s)
#05-Mar-24 11:28

Riivo, Thank you. Works well.

For my case, visually , there seems no difference between ANSI and UTF-8 BOM. But this is allright.

rk
627 post(s)
#05-Mar-24 13:09

Yes, if all your characters fit into 1 ANSI single-byte character set (your system default windows locale) then there is no visual difference.

As soon as 1 codepage is not enough you get ???

Converting from UTF-8 to ANSI changes bytes on disk for every character that are not basic ASCII.

Converting from UTF-8 to UTF-8-BOM just prepends 3 bytes to the file. Those 3 bytes help Windows (and Manifold) to detect that the file is in UTF-8 (and not UTF-16 nor any of the single-byte character sets).

Internally, Windows uses UTF-16.

David S
136 post(s)
#12-Apr-24 00:05

Many thanks for that tip off. Never heard of UTF-8-BOM. Just UTF-8 ..!

Also works for opening csv files in Excel.

Bit of a pain that it is necessary to go that extra step though.

Mike Pelletier


2,129 post(s)
#15-Apr-24 14:24

Manifold's importer can get tripped up with a comma in a number as well, when certain unknown conditions exist. For example, I've had a value of 3,860.86 get imported as 3. I shared the table that had this problem with Manifold Tech. Unfortunately, converting to UTF-8-BOM doesn't seem to work on this case. Weirdly deleting a few random records before or after the problem record in the table did.

Manifold User Community Use Agreement Copyright (C) 2007-2021 Manifold Software Limited. All rights reserved.