Hi !

First time I am here. Hope I do well with posting my question here.

Working with a CSV file encoded with accents in French (?, ?, ?, etc...), how can I change the charset for the TSdfDataSet in order to see those accents and not the ? replacing char?

Thanks for your answer !
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20120906/3c6d8d95/attachment-0001.html>

Hi,

I have recently dealt with a CSV file and accents. I concluded that the TSdfDataSet had no problem with reading the file; I was using a TMemo to show the strings and to see them correctly I had to write something like this:

   Memo.Lines.Text := UTF8Encode(SdfDataSet.Fields[0].AsString)

I'm no expert at all, especially about string encoding, so maybe someone else could better explain what we should with the string format in the DataSet and in the LCL components, but that should make you work in the meantime.

R#

2012/9/6 GLBX

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20120906/7cad7655/attachment.html>

Thanks !

I try and let know about

GLBX

UTF8Encode converts a widestring to an UTF8 string. AsString returns an ansistring. So the above first converts from system codepage to widestring (UTF16) and then to UTF8.

I'm no expert at all, especially about string encoding, so maybe someone else could better explain what we should with the string format in the DataSet and in the LCL components, but that should make you work in the meantime.

I'm no sdf expert, so I will assume it does no conversion like most LCL code:

If the csv file is in UTF8 and sdf does no conversion, then you can simply load the csv file and don't need any the csv file is in the french windows code page 1252 then you can use the function ConvertEncoding of the LazUtils unit lconvencoding to convert its content to UTF8 before loading it into sdf.

Mattias

Hi,

I checked the encodings and got :

- GuessEncoding(dataset .fields[1].asString) and got UTF8
- [application]defaultTextEncoding = cp1252

Now in DBGrid I wrote in DrawColumnCell :
...
ExtTextOut( .... UTF8ToISO_8859_1(dataCSV.Fields[datacol].AsString......) and.....
Now accents are correctly displayed...

But I will get the problem back when validating changes in fields !!! (I Tried with OnSetText : but inefficient !)

I go on looking for solution....

GLBX

GLBX hat am 6. September 2012 um 11:36 geschrieben:

Keep in mind that it is impossible to reliably get the codepage of a text, unless the text begins with a BOM. GuessEncoding simply checks if the string validates some UTF-8 rules. If it is valid it returns UTF-8 otherwise the system code page.

It seems you are calling the windows API function ExtTextOut directly instead of the x-platform LCL one. The LCL one expects UTF-8, so you don't need the conversion.

But I will get the problem back when validating changes in fields !!! (I Tried with OnSetText : but inefficient !)

Mattias
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20120906/eb4e52bf/attachment.html>

Right !

with Textout, I get accents correctly

but... when I post the changes, now, I get the problem

Originally DAMMANS

Changed to D?MMANS

Display after editing
D?MMANS (coorect)

Saved in the CSV file as !!!!!
D??MMANS
and...

when I display after new loading :
D?MMANS (!!!!?!?!?!)

Oups !!

GLBX

2012/9/6 GLBX

when working on that project of mine, I was in a hurry and I couldn't go deeper in understanding things (like the double conversion through UTF16, thanks Mattias).
At that time I was also UNABLE to save back to csv through the sdf dataset, even with readonly = false, putting the db in edit/append and posting the changes.... it looked somehow a reader-only dataset.

A quick (and maybe unefficient or dirty) trick to get around it has been to use:
- a TSDFDataset to read the CSV file (my separtor was non-standard)
- copying it to a TMemDataset which can be manipulated really fast
- place a TCSVExporter component on the form (install in the palette from the dbexport package) connected to the TMemDataset When you're done with editing and posting, you can simply call TCSVExporter.Execute to save it to CSV.

An alternative (that I couldn't explore due to the limited amount of time I had) might be this one: http://wiki.freepascal.org/CsvDocument which supposedly reads and writes, either randomly (in memory) or sequentially, a CSV file.

R#

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20120906/1e14a75a/attachment.html>

Roberto P. wrote

2012/9/6 GLBX

[snip]

Yes, but if you read that page carefully, you will see that

"Both UTF-8 encoding and windows-xxx codepages can be used with CsvDocument library. The library uses string type for all string operations and will not do any encoding conversion for you. Keep in mind though that Excel does not support CSV files in UTF-8."

So you will experience the same problems.

Michael.

ok... looks fine...

Saved in the CSV file as !!!!! D??MMANS

that is UTF8 encoding of the "?" character...

and... when I display after new loading : D?MMANS (!!!!?!?!?!)

that is proper...

Oups !!

can't you convert the text back to CP1252 before writing it to the CSV?? it would seem that that's what you are wanting to do...

hat am 6. September 2012 um 14:32 geschrieben:

Excel supports CSV with UTF-8 BOM.

Mattias
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20120906/22835eac/attachment.html>