Hi,

The ThousandSeparator is "char" and supports only 1 byte characters. For example French and Russian need more.
Are there any plans to extend it?

Mattias

I am surprised,
for me, as French, I always believed that
the French thousand separator is simply a space,
what else should it be?

Pierre Muller
from Strasbourg, France

As a Russian, I'm equally surprised. We don't normally separate

In our previous episode, Pierre Free Pascal said:

I am surprised, for me, as French, I always believed that the French thousand separator is simply a space, what else should it be?

A shift space. In utf8 that takes two bytes.

2014-11-25 17:21 GMT+01:00 Pierre Free Pascal :

Not quite. It looks like a space, but:
- you wouldn't like to see 2 000 € printed as: 2 000 € would you?
- exact typography rules state that the exact space should be actually be a little thinner than a usual space. But I guess if it is as large, few people will complain. Search "expace insécable" on wikipedia.fr for more explanations.

Plans: yes. Time: no.

Maybe a widechar is sufficient ?
Making it a string is more invasive than making it a widechar.

Michael.

The range is sufficient. Until we support an alien language or people want emoticons as separators.

Making it a string is more invasive than making it a widechar.

Does concatenating a string and a WideChar create a UnicodeString? Can this become a problem?

Mattias

Are all possible separators members of the Unicode BMP? What when a properly decorated string has to be converted to a specific (AnsiChar) codepage?

I'd assume that national separators are part of the according codepage, but is that always true?

DoDi

In our previous episode, Michael Van Canneyt said:

If you start changing the encoding of formatsettings (to something else than whatever "string" is, you get a lot of conversions. Even to just print a number you get thousandseparator (multiple times), decimal separator etc.

I think biting the bullet and making it a string is better long term. If we ever target a different "string" it will also break less code.

Does concatenating a string and a WideChar create a UnicodeString? Can this become a problem?

Concatenation requires 2 strings, so everything depends on the concrete code. Regardless of eventual compiler magics, something like this will happen:

var c: WideChar; s, cs: string;
cs := c; //dunno if accepted by the compiler
s := s + cs;

The WideChar can be converted into an Unicode (UTF-8 or UTF-16) string. Afterwards this string may need another conversion, when the other string has a different encoding. In the worst case *both* strings are converted to the default Unicode representation (Delphi: UTF-16, Lazarus: UTF-8?), before they are concatenated. Another conversion may occur when the resulting string is assigned to a variable.

All this may become simpler when CP_ACP is used (at least in Delpi), and the separator is given in that encoding, as a single byte/AnsiChar in case of an SBCS CP_ACP. When Lazarus instead uses UTF-8 (MBCS) for CP_ACP, the character occupies more than one byte, so that this simplification is impossible. This suggests to store the delimiter as an string, instead of a WideChar, whereupon a concatenation of the strings may not require any further conversion.

Finally, when the expression (s+cs) is of type RawByteString (depending on the involved function declarations), the result will be stored in the target variable *without* another conversion. Then the static and dynamic encoding of s may be different afterwards.

DoDi

making it a widechar.

I wouldn't make it a string. The most sufficient one would be UCS4Char (4 Byte character) which would also support the non-BMP characters. But then there is currently no encoding/conversion support in the compiler for these... (UCS4String is simply declared as an array of UCS4Char in System)

Regards,
Sven

I found 12 thousand-separators and they are all in the BMP. It seems even Klingon's thousand separator is in the BMP, even though the digits are not.

Yes, because the thousand-separator comes from the OS in system code page.
The UTF-16 version of the RTL can query under Windows the GetLocalInfoW instead of GetLocaleInfoA.

When someone changes the DefaultSystemCodepage he has to set the format settings himself. If this is CP_UTF8, then GetLocalInfoW can be used.

Mattias

Hans-Peter Diettrich wrote

I'd assume that national separators are part of the according codepage, but is that always true?

Of the OS/window manager actually. You are of course right in that there are a certain set of separators that can be used, but the exact separator to use is dependent on the system.
Most OS's contain some form of configuration for this (`Country/Region & Language` in KDE, `Language & Text` in OS X, `Regional And Language Options` in Windows XP). So for visualization purposes and integration into the OS/WM, national seperators are to be fetched from the system, just as keyboard layout, to make an analogy.

Note that by stating the above I in no way mean to say that this `fetching` should be done automatically.

How does that look when hand-written on a cheque?

2014-11-26 1:54 GMT+01:00 Mark Morgan Lloyd <

:-) Not much more different than an em dash from an en dash or a baseline dot from a raised dot.

Mark Morgan Lloyd wrote

How does that look when hand-written on a cheque?

Supposedly this is a non-breaking space, that does not allow to be replaced by a line-break (&nbsp; in HTML)

-Michael

2014-11-26 9:47 GMT+01:00 Michael Schnell :

Precisely.

In our previous episode, Mark Morgan Lloyd said:

I haven't seen a cheque in a decade, but I assume that depends on the hand having an utf8 encoding or not.

Of the OS/window manager actually. You are of course right in that there are a certain set of separators that can be used, but the exact separator to use is dependent on the system.

Sounds easy, but just yesterday I ran into a bunch of related problems. Even if the following is somewhat OT, my observations may be helpful to somebody else:

I just encounterd an really problematic case, with the Ez-Builder IDE. That program aborts when the decimal separator is not a period '.', asking the user to adjust his national/language settings in the system. So what can/should a user do, in order to run this program on my German Windows with a comma ',' as the decimal separator?

A developer might ask the author to add proper handling of the system-wide national settings. But when that author spends time in presenting instructions, how to change the inconvenient setting, instead of correcting his code, I doubt that such a wish will be heard :-(

The dumb user might follow the instruction, causing problems in all other programs :-(
When the system does not notify all other (running) programs of such an global change, or when some other stupid program doesn't know how to deal with changed settings, the user better shuts down and restarts his system, before and after using that ill behaved program.

But exactly *what* should a clever program do, when it receives such a change notification? What should happen with the formatted numbers, shown in the forms of the program? Which code (app/OS?) puts the separators into formatted number strings?

I don't know if it's worth to discuss such problems in detail, so let me only present my preferred handling:

1) The actual settings are determined at program start, and remain unchanged until program termination.

2) Formatted numbers, as enterd by the user (maybe by copy&paste from other applications), can have various encodings. Before a conversion into binary values I'd remove all unexpected characters, except for the last (rightmost) '.' or ',', which then becomes the decimal separator as expected by the decoding function (RTL provided).

3) For all other (non-GUI) purposes a unique string format is used, according to the conversion functions used by the compiler. This means no thousands separator, and a '.' decimal separator.

But back to the original problem: I managed to create another user, whose number format settings match the expections of the Ez-Builder, while using my German keyboard. For Linux users this may sound like an easy job, but adding and configuring users in Win8 turned out as kind of a nightmare :-(
Win8 requires an eMail address for every new user, but entering a fake address only allows to create the account, without any chance to log in subsequently. Probably the requested password has to be established by mail, at least I found no way to disable or specify or reset the password for the new account.
Fortunately I had retained an Guest account, could log in and adjust the format settings as prescribed, and then could successfully start Ez-Builder.

After all I hope that these problems are due to the cheap (Premium?) version of my Win8, that is *intentionally* crippled in several ways.

Conclusion:
Proper handling of separators in formatted numbers is essential, or else users may run into so big trouble, that they will drop your program as unusable.

DoDi

(renamed subject due to change in topic)

I hear you, but this issue is so much wider than separators. I know one software package that will only successfully export data to excel if the system regional is one of the "English (xxx)" variations (Australian guaranteed to work, not really played with the rest...). In this case, the client (in Denmark) has one PC in a corner, set to Australian settings, just for exports...

As an Australian developer, this is just embarrassing...

Mike

At my old company our Delphi application handled runtime changes to these settings rather well. For display the normal XToY (e.g. DateToStr) functions are used which use the DefaultFormatSettings which are updated automatically (the VCL's message loop triggers a repaint when format settings were changed in the system). *This* is how I expect an application to behave (afterall Microsoft's own applications behave this way as well).

Regards,
Sven

My recollection is that there's a Windows message that indicates that a control panel applet has changed localisation etc. The lack of anything comparable on other OSes could be a problem, even if user apps and their associated RTLs were happy handling a change.

2014-11-26 16:54 GMT+01:00 Hans-Peter Diettrich :

You mean that the first string to be converted to binary would automatically set the decimal separator? That would seem dangerous to me. What if the first string to be converted contained something like "11,000", does this mean 11000 with thousand separator = comma (which would be true in at least USA), or 11 with decimal separator = comma (which would be true at least in France)? I can't think of any way to choose automatically. AFAICS, the code needs either to use the system settings or to be told explicitly by the developer. Even relying on the system settings may not be enough, because one may need to "import" data formatted with different national settings from the system's settings.

Am 27.11.2014 08:47 schrieb "Mark Morgan Lloyd" <

these settings rather well. For display the normal XToY (e.g. DateToStr) functions are used which use the DefaultFormatSettings which are updated automatically (the VCL's message loop triggers a repaint when format settings were changed in the system). *This* is how I expect an application to behave (afterall Microsoft's own applications behave this way as well).

My recollection is that there's a Windows message that indicates that a

control panel applet has changed localisation etc. The lack of anything comparable on other OSes could be a problem, even if user apps and their associated RTLs were happy handling a change.

Yes, there's a message for that and yes on non-Windows OSes this might be problematic...

Regards,
Sven

[...] Yes, there's a message for that and yes on non-Windows OSes this might be problematic...

AFAIK on Unix systems you can start each program with a different locale. The system wide locale is just the default on start. Therefore changing the system wide settings should not affect a running application - aka that would be a bug.

Mattias

You can also have completely different desktops etc. (Gnome on one display, KDE on another, a text session on a third), which would be unlikely to use the same notification mechanism.

+1

Linux uses strings too.
With LC_NUMERIC=ru_RU.utf8 the thousand separator is the two byte nbsp/160. French for some reason uses simple space.

BTW, the clocale unit, which sets the formatsettings, simply takes the first byte. It would be good if it would check if the character is ASCII. If not and codepage is utf8 then use the default. Should I create a patch for 2.8.0?

Hopefully in 2.8.1+ the type has changed and clocale can use the correct settings.

Mattias

A repaint by itself doesn't change the strings. How do the new strings come into all the edit boxes, of all open forms?

Similarly, when the user changes the system language, can he expect that every running application updates itself, with changed menus etc., up to eventually open help viewers? What if the program is not prepared for a different language, because e.g. a tax assistant is bound to a specific country?

DoDi

No, my code would make no assumption about the format of strings edited by the user.

Okay, that would require more knowledge about the value kept in a specific input field, for range checks or the like. As long as thousands separators occur in the string, different from '.' or ',', they are quite easy to identify.

AFAICS, the code needs either to use the system settings or to be told explicitly by the developer. Even relying on the system settings may not be enough, because one may need to "import" data formatted with different national settings from the system's settings.

Right. When e.g. a CAD program is fed with sizes from an external data sheet, it cannot be expected that the figures in that file change together with the system language, and are converted between inch and meter, temparatures between F and C ;-)

So it looks to me as a stupid idea, when the user changes such system settings *while* such a program is running. Furthermore the use of national formatting conventions for the exchange of values across applications looks to me like another stupid idea. Would somebody expect or even like it, when e.g. constant declarations are converted when copied into an Lazarus editor, and the compiler would require that all constants in source code conform to the current settings?

As mentioned in other contributions, the number formatting seems to be a Windows specific problem. How to deal with imported numbers on other platforms, with arbitrary settings per application?

After all a program could, when notified of such changes, ask the user whether to continue or restart, or force an restart. Restart should be safe, but when the user decides to continue, he must be aware of possible problems. When the restart takes considerable time, the user may learn that his behaviour is not very clever ;-)

DoDi

I don't see any reason why thousand seperators can't/shouldn't be to the right of the decimal seperator, nor any reason why a number with a thousand seperator also has to have a decimal seperator. So your algorithm would produce wrong answers in many cases.

Which brings us to the root of the problem, some numbers are simply ambiguous without knowing the conventions used by the people who wrote them. is 1,000 one or one thousand?

This isn't a problem limited to programming, when doing electronic design I often find myself dealing with datasheets where the authors have translated the text into english but have not translated the numbers to the english conventions. Ususally a human can make an educated guess based on the expected magnitude of the number but it still throws you at first.

3) For all other (non-GUI) purposes a unique string format is used, according to the conversion functions used by the compiler. This means no thousands separator, and a '.' decimal separator.

Which for an existing program means digging through your code to find every single decimal conversion and making the judgement call as to whether it's UI related or storage related and deal with the awkward situation that some scenarios (for example reading data from text files whose format is designed by others or which were written by previous versions of your program before you started taking this seriously) may require special handling or even asking the user which interpretion is correct.

I'm not surprised some programmers give up and tell users to set their systems to english mode and follow english conventions. Mainland europeans (which afaict are the main case which causes ambiguity) are a fairly significant market but losing them is probablly not a deal breaker for many developers

From what i've heard there is a way to create a local only user in win8 but it's fairly hidden.

Conclusion: Proper handling of separators in formatted numbers is essential, or else users may run into so big trouble, that they will drop your program as unusable.

The problem is proper handling is HARD.

Yes, you can run Windows 8(.1) with a local user only

Ralf
(typing this on a Windows 8.1/64bit laptop with a local user account (only))

---
This email has been checked for viruses by Avast antivirus software. http://www.avast.com

This may be a relict from the time, where Microsoft found it a good idea to nationalize VBA (VB, Word, Excel...). I appreciated that in so far, as no English macro virus could become active in my German Word (with only German keywords). The same language barrier may prevent proper data export, maybe starting with slightly different keyword spellings like for color/colour.

Similar problems exist(ed) in RTF export, so that MS had to ship another WinHelp compiler (HC) for every new WinWord version, that worked around the new errors in the RTF sources exported from Word, even after VBA was reverted to unique English-only keywords and function names.

As an Australian developer, this is just embarrassing...

I never felt a need or reason for considering Microsoft products as anything but buggy toys, hardly usable outside the USA :-(

To come back to the current discussions, the introduction of Unicode (as UCS-2 and UTF-16) was a similar (typically American) mistake, totally ignoring e.g. any Chinese character set (in favor of Klingon?). Apple, as another US company, invented the decomposed Unicode filenames - for lack of oversight, or to establish artificial platform barriers?

The step from strictly national Ansi applications to Unicode is a very tiny one, compared to the leaps that have to be taken afterwards, in order to make the program really work in foreign countries. I wonder how e.g. Belgian, Canadian or Swiss software has been written in such multi-lingual countries before, and how it is written nowadays.

DoDi

these settings rather well. For display the normal XToY (e.g. DateToStr) functions are used which use the DefaultFormatSettings which are updated automatically (the VCL's message loop triggers a repaint when format settings were changed in the system).

A repaint by itself doesn't change the strings. How do the new strings

come into all the edit boxes, of all open forms?

Of course it's not a mere simple repaint. Controls like TSpinEdit and TDateEdit (both controls are designed to display localized values) will convert their internal data value to a string using the current format settings upon a repaint. If you have a mere edit then you need to handle that yourself of course by also intercepting the settings change message. Then you can convert your internal data value (I don't know about you, but I store my values internally localization independent) to the correct display value.

Similarly, when the user changes the system language, can he expect that

every running application updates itself, with changed menus etc., up to eventually open help viewers? What if the program is not prepared for a different language, because e.g. a tax assistant is bound to a specific country?

Of course not every application does or even can handle this. Nevertheless an OS like Windows provides the possibility for an application to handle that situation to provide a more seamless experience for its users.

Regards,
Sven