Hi List.

I am reading some text of some .csv files, but the encoding of the files is not always the same. In fact it may vary greatly from a lot of european encodings, UTF8 and asian encoding.

I have tried using the function GuessEncoding(..) in LConvEncoding.pas, but from the source code I can see that it mostly focuses on BOM's and ASCII/UTF-8

Does Lazarus or FPC include other convinient tools that may detect the encoding of (the content of) a file?
What do others do in this situation?

My goal is to convert all the read text to UTF-8.

Kind regards,
Torsten.

What do others do in this situation?

I use
https://github.com/Alexey-T/ATSynEdit/blob/master/atsynedit/atstringproc_utf8detect.pas

Hello,

Some years ago I wrote a code that must be trained which guess encoding and language. The problems are that it must be trained with large texts and of course the result are only statistical and only quite good over quite large texts (like 1000 chars or more) so it is not good for single sentences.

If you are interested I can dive into old codes to catch it.

>What do others do in this situation? I use https://github.com/Alexey-T/ATSynEdit/blob/master/atsynedit/atstringproc_utf8detect.pas

That's checking if a zeroed string is valid UTF-8. GuessEncoding does that too, although it supports #0 chars in string.

Mattias

Torsten Bonde Christiansen wrote

Does Lazarus or FPC include other convinient tools that may detect the encoding of (the content of) a file? What do others do in this situation?

If you need cross-platform, this works with some trivial changes:

http://chsdet.sourceforge.net/

If you're Windows only, Microsoft provides
IMultiLanguage2.DetectInputCodepage():

https://msdn.microsoft.com/en-us/library/aa740986(v=vs.85).aspx

Thanks - this is just what I was looking for.

Regards,
Torsten.

ChsDet is in IDE OnlinePackageManager, "trivial changes" aren't needed.

Alexey

Alexey wrote

ChsDet is in IDE OnlinePackageManager, "trivial changes" aren't needed. Alexey

Oh super!

thanks,
Torsten.