About Store Forum Documentation Contact



Post Reply 
[Solved] Character set converting (multibytes, widecharacters or so) problem!
Author Message
ralphthebear Offline
Source Licensee

Post: #1
[Solved] Character set converting (multibytes, widecharacters or so) problem!
Situation:
I have to use the exist server system and it use STL for packet send/receive. So I receive character set by multibytes encoding like UTF-8 or ANSI(EUC-KR, Korean). I need to convert it to render on Esenthel GUI system.

Symptom:
On PC, everything is OK with std::string, std::wstring, 'Multibyte to Unicode' and vice versa functions.

But on Android, there are very strange things. Only constant literal can be displayed correctly but variables can't be. Any manipulation encoded variables result in failure. Look at the below pictures. One is tested on the PC, another one on the Android with exact same code.

Test Environment:
Development: Windows 7, MSVS 2015, GCC and Ant with default options generated from EE Editor.
Tested Android: LG G4, 5.1 Kitkat

Testing on the PC
   

Testing on the Android
   
(This post was last modified: 03-18-2017 03:54 AM by ralphthebear.)
03-14-2017 03:43 PM
Find all posts by this user Quote this message in a reply
Esenthel Offline
Administrator

Post: #2
RE: Character set converting (multibytes, widecharacters or so) problem!
Hello,

If you would use UTF-8 or UTF-16, then there shouldn't be any problem at all, as they are platform universal, and independent of system settings.

But if you're using ANSI(EUC-KR, Korean) multi-byte encoding, then I think results depend on which codepage is currently set in the operating system (on both server and client).
I don't recommend using multi-byte encoding.

I recommend using UTF-8 or UTF-16.

So do I understand correctly that you're using a non-EE server appplication that sends text messages via sockets. And then try to receive those messages on EE client application, and read/decode the text messages?

The engine has functions for encoding/decoding UTF8 strings:
https://github.com/Esenthel/EsenthelEngi....cpp#L1775
https://github.com/Esenthel/EsenthelEngi....cpp#L1813

They are currently private (in the engine headers)
https://github.com/Esenthel/EsenthelEngi...ons.h#L277
They are inside "#ifdef EE_PRIVATE"
https://github.com/Esenthel/EsenthelEngi...ons.h#L269
So when you run "Esenthel Builder" and "Create Code Editor.dat" function, the function declarations don't get copied to Esenthel Code Editor.
But you can copy the functions manually to your project, if it solves your problem, then I can make the functions public, so they are always available in the Code Editor.

But if your server is also an EE based application, then you can just use Str8 or Str, and save them into File this way:
File f;
Str s;
..
f<<s;
03-14-2017 10:26 PM
Find all posts by this user Quote this message in a reply
ralphthebear Offline
Source Licensee

Post: #3
RE: Character set converting (multibytes, widecharacters or so) problem!
I've solved that problem. It was encoding issues as you advised me. So I set all encoding on the server to UTF-8.
At first, I would handle this only on the client side but it wasn't a wise one because there are lots of different locale, ANSI encoding is tightly bound to locale set of each device.

Anyway, my solution scheme is,

1. ANSI locale dependent encoding to UTF-8 encoding on server (use my own converting function like AnsiToUTF8(), it's important)
2. Use FromUTF8() of EE on client (receive from server as UTF-8, and convert it to EE:Str to render on EE:GUI system)

Hopefully, this would be a clue for people to find a solution like this similar situation.

It was a long days since I've adopted EE from 2009, from the very beginning time of EE. There were many opportunities to our company thanks for EE.

Thanks again EE and Greg.
03-18-2017 03:54 AM
Find all posts by this user Quote this message in a reply
Esenthel Offline
Administrator

Post: #4
RE: [Solved] Character set converting (multibytes, widecharacters or so) problem!
Thank you for letting me know and I'm glad you were able to solve this.

I see you've ended up using the 'FromUTF8' function, so I will make it public for the next release, along with 'UTF8' function.
03-18-2017 05:36 AM
Find all posts by this user Quote this message in a reply
Post Reply