Sign in Join

Live chat for Free Pascal, Lazarus, Delphi and every Pascal programmer.

Mailing lists / fpc-pascal / 2012-09

Problems with Postgresql Character Encoding

6 messages · 3 people · –

Reply by email Subscribe to this list Your subscription

Read-only archive. Use j and k to move between messages.

Andrew Brunner

My cloud file/folder sync client desktop app builds and maintains a manifest of all files and folders from a specified starting point. I use this manifest to synchronize disk entries across devices. It uses HTTP requests and XML to carry out all transactions. Meaning there is no TSQLConnection. The problem is that the HTTP server receives the data as XML and refuses to post changes. If data (in updates or inserts) contains non-supported UTF8 characters postgresql rejects with character encoding notices.

I've tried with using AnsiToUTF8 and UTF8toAnsi. That works for email messages. However, since sync code has to scan tens of thousands of disk files/folders I don't think it's good for battery and memory consumption for a background app to consume cpu resources in this case going between 2 forms of the same file/folder name.

I need to be able to take advantage of Postgresql's client encoding method. I know postgresql allows me to tell the server that the connection is presently character set "LATIN1". And presumably postgresql server will convert what it needs to my database encoding of UTF8.

<snip>
SystemP^.Connection.Charset:='LATIN1';

This does nothing for me when Connection is a postgresql connection.

Anyone have any experience with postgresql character sets / encoding???

I can't hard-code any one character set because users from 1 region will have different encoding needs than another.

Any help, or comments is greatly needed.

Andrew Brunner Aurawin LLC 512.574.6298 http://aurawin.com/ Aurawin is a great new place to store, share, and enjoy your photos, videos, music and more.
Mark Morgan Lloyd
to Andrew Brunner
Andrew Brunner wrote 27 lines
My cloud file/folder sync client desktop app builds and maintains a manifest of all files and folders from a specified starting point. I use this manifest to synchronize disk entries across devices. It uses HTTP requests and XML to carry out all transactions. Meaning there is no TSQLConnection. The problem is that the HTTP server receives the data as XML and refuses to post changes. If data (in updates or inserts) contains non-supported UTF8 characters postgresql rejects with character encoding notices. I've tried with using AnsiToUTF8 and UTF8toAnsi. That works for email messages. However, since sync code has to scan tens of thousands of disk files/folders I don't think it's good for battery and memory consumption for a background app to consume cpu resources in this case going between 2 forms of the same file/folder name. I need to be able to take advantage of Postgresql's client encoding method. I know postgresql allows me to tell the server that the connection is presently character set "LATIN1". And presumably postgresql server will convert what it needs to my database encoding of UTF8. <snip> SystemP^.Connection.Charset:='LATIN1'; This does nothing for me when Connection is a postgresql connection. Anyone have any experience with postgresql character sets / encoding???

No, but I'm interested since this is something that I anticipate having to do.

You might need to issue an explicit instruction to the PostgreSQL server telling it to assume that the current session is encoded as /whatever/. If you have to do that then you might find yourself having to use the PQConnection's (or whatever the correct name is) low-level handle, since working from memory the .Exec method uses a temporary handle so any state change isn't retained. I had to delve into this when getting listen/notify running reliably.

Mark Morgan Lloyd markMLl .AT. telemetry.co .DOT. uk [Opinions above are the author's, not those of his employers or colleagues]
LacaK
to Mark Morgan Lloyd

Mark Morgan Lloyd wrote / napísal(a):

Show quoted text 40 lines
Andrew Brunner wrote: > My cloud file/folder sync client desktop app builds and maintains a > manifest of all files and folders from a specified starting point. I > use this manifest to synchronize disk entries across devices. It > uses HTTP requests and XML to carry out all transactions. Meaning > there is no TSQLConnection. The problem is that the HTTP server > receives the data as XML and refuses to post changes. If data (in > updates or inserts) contains non-supported UTF8 characters postgresql > rejects with character encoding notices. > > I've tried with using AnsiToUTF8 and UTF8toAnsi. That works for > email messages. However, since sync code has to scan tens of > thousands of disk files/folders I don't think it's good for battery > and memory consumption for a background app to consume cpu resources > in this case going between 2 forms of the same file/folder name. > > I need to be able to take advantage of Postgresql's client encoding > method. I know postgresql allows me to tell the server that the > connection is presently character set "LATIN1". And presumably > postgresql server will convert what it needs to my database encoding > of UTF8. > > <snip> > SystemP^.Connection.Charset:='LATIN1'; > > This does nothing for me when Connection is a postgresql connection. > > Anyone have any experience with postgresql character sets / encoding??? No, but I'm interested since this is something that I anticipate having to do. You might need to issue an explicit instruction to the PostgreSQL server telling it to assume that the current session is encoded as /whatever/. If you have to do that then you might find yourself having to use the PQConnection's (or whatever the correct name is) low-level handle, since working from memory the .Exec method uses a temporary handle so any state change isn't retained. I had to delve into this when getting listen/notify running reliably.

Do you have tried SQL: SET CLIENT_ENCODING TO 'LATIN1' ? May be that better would be add support for PQsetClientEncoding into TPQConnection ... I can look at it if you want.
-Laco.

Andrew Brunner
to LacaK: Mark Morgan Lloyd wrote / napísal(a): Do you have tried SQL: SET CLIENT_ENCODING TO…
LacaK wrote
Do you have tried SQL: SET CLIENT_ENCODING TO 'LATIN1' ? May be that better would be add support for PQsetClientEncoding into TPQConnection ... I can look at it if you want. -Laco.

Hi Laco,

Thanks for the help. Yes I did on connection I tried Connection.onConnected... ExecuteDirect ('SET CLIENT_ENCODING TO LATIN1). It didn't help.

Failed to insert UTF8 database with a string like ' the © character test.'

Postgres wants the data to be converted... ie.) AnsiToUTF8 would get the data to post.

Thanks for the patch - I'm updating my FPC /Lazarus and building now...

Andrew Brunner Aurawin LLC 512.574.6298 http://aurawin.com/ Aurawin is a great new place to store, share, and enjoy your photos, videos, music and more.
Andrew Brunner
to LacaK: Patch is here http://bugs.freepascal.org/view.php?id=22985 (I did only basic testing)

This did the trick!

I added a comment on reporting too. I say this one is good to go.

LacaK wrote 13 lines
> > May be that better would be add support for PQsetClientEncoding into > TPQConnection ... I can look at it if you want. Patch is here http://bugs.freepascal.org/view.php?id=22985 (I did only basic testing) > -Laco. _______________________________________________ fpc-pascal maillist - http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Andrew Brunner Aurawin LLC 512.574.6298 http://aurawin.com/ Aurawin is a great new place to store, share, and enjoy your photos, videos, music and more.