Author Topic: UTF-16 LE files don't open correctly  (Read 159 times)

bengle

  • Senior Community Member
  • Posts: 143
  • Hero Points: 4
UTF-16 LE files don't open correctly
« on: June 30, 2020, 06:34:31 pm »
I have my File Options --> Load --> Encoding set to "Auto Unicode".

When I open UTF-16 LE files, it opens them as Ansi, even though they have a valid byte-order-mark signature for UTF-16 LE.  You can reproduce the problem by saving a file with Notepad as UTF-16 LE,  and then open it with SlickEdit.   

Clark

  • SlickEdit Team Member
  • Senior Community Member
  • *
  • Posts: 5171
  • Hero Points: 427
Re: UTF-16 LE files don't open correctly
« Reply #1 on: June 30, 2020, 08:47:37 pm »
Try specifying the encoding (UTF-16)  in SlickEdit's Open dialog.

bengle

  • Senior Community Member
  • Posts: 143
  • Hero Points: 4
Re: UTF-16 LE files don't open correctly
« Reply #2 on: June 30, 2020, 09:11:35 pm »
It works if I open it with the Open dialog and manually specify the encoding.  It doesn't work if it is set to "Automatic".  I usually open files by simply double clicking on a file in the Open tool window.

I have just discovered that it doesn't happen on all files, but consistently does on some files.  Attached is a file where the problem happens.

Clark

  • SlickEdit Team Member
  • Senior Community Member
  • *
  • Posts: 5171
  • Hero Points: 427
Re: UTF-16 LE files don't open correctly
« Reply #3 on: June 30, 2020, 10:40:58 pm »
I suspect this is only happening for files which you have already opened using a different encoding.

Here's how you can force the problem:

Open the file as Utf-16 LE by specifying Utf-16 in the SlickEdit open dialog
Now use SlickEdit's File Save As and specify an encoding of Utf-8,no signature (other encodings would mess things up too).
Close the file
Copy the Utf-16 version of the file on top of the Utf-8,no signature version of the file.
Now reopen the newly encoded file.
SlickEdit will NOT open the file as Utf-16 because a specific override encoding was specified in the open dialog.

The reason the above sequence fails is that SlickEdit sometimes stores the encoding used to open a file and reuses it. SlickEdit is only supposed to do this when you manually specified an override in the Open dialog. Hope this makes sense.

If you exit SlickEdit and delete your perfile.xml settings file, SlickEdit will dynamically figure out all encodings.


bengle

  • Senior Community Member
  • Posts: 143
  • Hero Points: 4
Re: UTF-16 LE files don't open correctly
« Reply #4 on: June 30, 2020, 10:46:58 pm »
I deleted my perfile.xml file and it made no difference.  What happens if you try to open the file that I attached? 

Clark

  • SlickEdit Team Member
  • Senior Community Member
  • *
  • Posts: 5171
  • Hero Points: 427
Re: UTF-16 LE files don't open correctly
« Reply #5 on: June 30, 2020, 11:01:34 pm »
Make sure you exit SlickEdit before deleting perfile.xml

bengle

  • Senior Community Member
  • Posts: 143
  • Hero Points: 4
Re: UTF-16 LE files don't open correctly
« Reply #6 on: July 01, 2020, 01:28:26 am »
I exited SlickEdit, deleted the perfile.xml file, as well as an assortment of other status files.  When I start SlickEdit again, I have the same problem.  It is warning that the file has Unix line endings.  Could that have anything to do with it?  Notepad has no problem opening it, and neither does Visual Studio.

Clark

  • SlickEdit Team Member
  • Senior Community Member
  • *
  • Posts: 5171
  • Hero Points: 427
Re: UTF-16 LE files don't open correctly
« Reply #7 on: July 01, 2020, 01:58:55 am »
I have no problem opening your Utf-16s file with a clean config. I tested v24 and v23.

What happens if you invoke SlickEdit like this:

vs +new -sc c:\new-config-dir stdafx.cpp

bengle

  • Senior Community Member
  • Posts: 143
  • Hero Points: 4
Re: UTF-16 LE files don't open correctly
« Reply #8 on: July 06, 2020, 08:22:21 pm »
It opens correctly when I use your command.  I guess that maybe I have something corrupted in my configuration.  Do you have any suggestions on what files to delete to clean up my configuration, or should I just delete everything and reconfigure from scratch?

Clark

  • SlickEdit Team Member
  • Senior Community Member
  • *
  • Posts: 5171
  • Hero Points: 427
Re: UTF-16 LE files don't open correctly
« Reply #9 on: July 06, 2020, 08:52:13 pm »
If perfile.xml isn’t the problem, you may have some different encoding options. If you post your user.cfg.xml, I can check it out. Otherwise, you would need to just delete your entire config and start over. Doing an export/import could copy the same bad settings.

bengle

  • Senior Community Member
  • Posts: 143
  • Hero Points: 4
Re: UTF-16 LE files don't open correctly
« Reply #10 on: July 06, 2020, 09:17:47 pm »
Here is the user.cfg.xml file.  If you don't find anything, I will just delete everything and start from scratch.

Clark

  • SlickEdit Team Member
  • Senior Community Member
  • *
  • Posts: 5171
  • Hero Points: 427
Re: UTF-16 LE files don't open correctly
« Reply #11 on: July 06, 2020, 09:36:52 pm »
The problem is these lines:
                <p n="encoding-c++" v="+ftext"/>
                <p n="encoding-cpp" v="+ftext"/>
                <p n="encoding-h" v="+ftext"/>

They force the encoding for these extensions to be SBCS/DBCS (ACP). You can either delete these property lines or go to Tools>Options>Languages>File Extension Manager and change the encoding for c++, cpp, and h to be Automatic.