The code below finds all caller adresses in a program to a known procedure/function adress.
With this I can get all caller adresses to the translate function.

Moreover - with another piece of code (not shown here) I got also the text snippets itself, because they are loaded immediately before executing the translate function with MOV textsnippetadr , EAX.

procedure i_realy_know_what_i_am_doing( p_opcode : pbyte ; count : longint ;
proc_adr : pointer );
const
 call_opcode = $e8;
 caller_adr_pc_offset = 5;
var
 p_reladress : pdword; // call works with a relative adress after the opcode
 x , caller_adress , jmp_adr_abs : dword;
begin
 for x := 1 to count do begin
  if p_opcode^ = call_opcode then begin
   p_reladress := pdword( p_opcode + 1 );                       // after
opcode = relative adress to procedure/function to call
   caller_adress := dword( p_opcode + caller_adr_pc_offset );
   jmp_adr_abs := caller_adress + p_reladress^;                 // absolute
adress = programcounter + 5 + relative adress
   if jmp_adr_abs = dword( proc_adr )
    then add_caller_adress_to_table( caller_adress );           // if call
proc_adr then found!
  end;
  inc( p_opcode );
 end;
end;

Rainer Stratmann wrote

procedure i_realy_know_what_i_am_doing( p_opcode : pbyte ; count : longint ; proc_adr : pointer );

Better: i_really_know_what_i_am_doing

Rainer Stratmann wrote

Am Saturday 11 August 2012 13:40:44 schrieb Rainer Stratmann: > procedure i_realy_know_what_i_am_doing( p_opcode : pbyte ; count : longint > ; proc_adr : pointer ); Better: i_really_know_what_i_am_doing

You have read the long discussion thread, about what happens to the cracker classes in the next fpc release, when the class-field-order optimization is introduced?

I do not read all discussions.
Which exactly headline has it?
Does it affect the possibility to get the information I want?

Re: [fpc-devel] Re: Class field reordering

No it does not help on your topic. It is about assuming that behaviour based on the current compiler implementation will never change....

You assume the compiler will always use the same asm-opcode for the call. Or at least you assume that if it changes, you can change it too. But future compilers (or maybe even the current, depending on optimization ?) may use different asm opcodes within the same exe, to make the call, and they may mix different ways of encoding the address.

If that changes, then your code fails.

---------
So if you do know what you do, hen you do know you are taking a risk....

The searched procedure/function is not a class procedure or function.

No it does not help on your topic. It is about assuming that behaviour based on the current compiler implementation will never change....

Where do I say that?

You assume the compiler will always use the same asm-opcode for the call.

No.

Or at least you assume that if it changes, you can change it too.

Yes, very quickly!

But future compilers (or maybe even the current, depending on optimization ?) may use different asm opcodes within the same exe, to make the call, and they may mix different ways of encoding the address.

I switched off optimizations.
I think I am able to handle more than one way to find a call to a function/procedure. But at this time I found only one way in my program so far. So there was no need to search for other ways...

If that changes, then your code fails.

I can easily adopt the code then.
There are not many different ways for a call.

http://css.csail.mit.edu/6.858/2011/readings/i386/CALL.htm

Yes, so the amount of effort may increase drastically. New CPU may bring new ways too.

And with each added way, the already existing risk of false positives (the byte sequence can occur as subset of another statement, and then it isn't a call at all) does increase.

But not the 80x86 CPU. It is still the same opcode for a call ($E8). By now since 36 Years.
http://en.wikipedia.org/wiki/Intel_8086

What do you mean with drastically?
At the moment there is no unsolved whish to this topic.

And with each added way, the already existing risk of false positives (the byte sequence can occur as subset of another statement, and then it isn't a call at all) does increase.

Note that it is only added to the table if also the adress is equal. I am aware of 2 byte opcodes, but I see no obstacle in this.

If there are very unlikely more results then expected it will not be a problem. The search through the whole program is mainly for me if adding new text snippets or a complete new language.

I will be able to solve most requirements, because I am an experienced programmer. On assembler language, too.

In our previous episode, Rainer Stratmann said:

Maybe not the call, but changing or adding address encoding is quite realistic.

I assume you tested your code for PIC cases (which are quite common on *nix?)

http://en.wikipedia.org/wiki/Position-independent_code

FPC supports PIC

And of course finding a perfect starting point so that you don't find false positives when you find $E8 inside the address (change 1 in 256 obviously)

Sorry for posting without bringing a solution, but i am curious, why do you need this ?

Doesnt sound like good programming practice...

2012/8/11 Marco van de Voort :

The call ($E8) itself operates with a relative adress as I found out and write already and documented in the piece of code shown in a previous mail.

If there is a $E8 then also the calculated absolute adress from the following 4 bytes must point to the searching text translate function. So errors are likely rare.

Jorge Aldo G. de F. Junior wrote

Sorry for posting without bringing a solution, but i am curious, why do you need this ? Doesnt sound like good programming practice...

See Thread:
"Get all caller adresses of a given function/procedure before executing" We did not find a solution there I feel comfortable with so I decide to check the code directly.

Why do you need to "Get all caller addresses of a given function procedure" ?

This can be solved with static code analysis - as long as there is no procedural variables involved - but its weird to have such a necessity... Sounds like the project started with the wrong assumptions...

Thats my two cents...

2012/8/11 Rainer Stratmann :

Jorge Aldo G. de F. Junior wrote

Why do you need to "Get all caller addresses of a given function procedure" ?

If you read the thread you will now.
Because I need an easy handling multi language interface.

This can be solved with static code analysis - as long as there is no procedural variables involved

How can it be done easily?
I think we discuss it already in the mentioned thread.

- but its weird to have such a necessity...

For the above mentioned purpose it works perfect since some days. I got all caller adresses and even more I got all text snippets (pchar to the snippet!). If you know something about assembler language then it is not a huge effort to get this information from the compiled code.

Sounds like the project started with the wrong assumptions...

The project has already started 20 years ago...
So there can not be very much wrong if it is still working.

Rainer Stratmann wrote

For the above mentioned purpose it works perfect since some days. I got all caller adresses and even more I got all text snippets (pchar to the snippet!). If you know something about assembler language then it is not a huge effort to get this information from the compiled code.

Hey, Jonas, when are you introducing the optimisation that a call at the end of a procedure can be replaced by a jmp? :-)

Florian already implemented that quite a while ago :) I suggest to drop the topic though, because Rainer does not appear to be open to taking another approach (and he has said he does not enable optimizations).

Jonas

Mark Morgan Lloyd wrote

Hey, Jonas, when are you introducing the optimisation that a call at the end of a procedure can be replaced by a jmp? :-)

Jonas only introduces something if it causes not much work :-)

Jmp will be opcode $E9.
http://css.csail.mit.edu/6.858/2012/readings/i386/JMP.htm

Jonas Maebe wrote

Florian already implemented that quite a while ago :) I suggest to drop the topic though, because Rainer does not appear to be open to taking another approach

It depends on the approach.
If you have a better solution would you take another approach?

I know there is a smiley, but: Only because Jonas sees no gain in implementing something in the compiler that would help you in your specific case does not mean that Jonas doesn't add anything that causes not much work. Besides his continous high quality work at the compiler one of his biggest works so to see was to enable FPC to generate ByteCode which can be executed by the JVM or even converted to Dalvik ByteCode.

Regards,
Sven

Sven Barth wrote

I know there is a smiley, but: Only because Jonas sees no gain in implementing something in the compiler that would help you in your specific case does not mean that Jonas doesn't add anything that causes not much work.

That my suggestions would help only me in my specific case is your subjective opinion.

It might be my suggestive opinion, but it's not only mine ;)

Regards,
Sven

What is wrong with known and explored ways e.g. gettext ? or TStringList descendant, if you need to change language on-the-run? Or resourcestrings like someone suggested ?

You were saying, that you want to know, which string has not been used / which string has been used and how many times ?

Well why not the T(Hashed)StringList descendant with custom counter field(s) ?

Curiously,
L.

It will be ok for those who are satisfied with the functionality. I dont know gettext in detail, but I assume there is more work than only to put a function ls('textsnippet') around.

or TStringList descendant, if you need to change language on-the-run? Or resourcestrings like someone suggested ?

See above.

You were saying, that you want to know, which string has not been used / which string has been used and how many times ?

I want to knof if the string was called the first time. But since I also get the pchar information of the textsnippets from the programmemory I can translate (make a list or whatever in the future) all snippets before they were called.

If I put a ls('snippet') around all texts I now get every caller adress (handle) and the text information 'snippet' itself. So before they are called I can make a list with all handles and all translations. When calling ls() I look in my handle list of caller adresses and find it in array nr x where all translations were already made.

Me neither - but IIRC using gettext (dxgettext) was something /incredibly/ easy : the translating function is called '_()' :) (yes, function _([const] input: string): string; )

(allegedly, in newer (dx)gettext, not even that- it's enough to have (dx)gettext unit included, but don't quote me on this one).

Then the dxgettext handling program creates the strings to be translated list out of that (saves to text .po file), together with programmatic context they are used in, as a help for the translator - and then compiles to .mo (pls sbdy correct me if I'm wrong) which are loaded at program start/on demand/as resourcestrings, (and IIRC may already support UTF-8/Unicode/multiple bytes per character strings too - do you)?

The gettext program is out there for (almost) all platforms, AND there are plethora of translation-handling programs compatible with it, out there too, but you could well enough handle it by editing the po files in any decent text editor.

So you don't want to translate it all at the same time, is that what it is? Only the strictly necessary portions/snippets as you call them?

Please forgive my attitude - but it looks like a very-low level and hence very inflexible and very un-Pascal solution to this problem. I think, this is what people were trying to suggest to you.

L.

In summary I need only the handle (caller adress) and the text snippets. I got both information now! I admit that it looks like an un-Pascal solution. But actually it works perfect for me and is also very flexible for me. Because all the other functionality comes step by step and I am free to add the functionality I want. I prefer to program solutions myself.

Those who want to use other solutions can do so and be happy with it.

In our previous episode, Lukasz Sokol said:

Afaik the dxgettext patched for D2009+

For resourcestrings yes, but using resourcestrings in structured constants and afaik also (D2010+) custom RTTI need updating manually. (while they don't with Delphi's own ITE)

Afaik it reads .po's directly, unicode only in D2009+

The gettext program is out there for (almost) all platforms, AND there are plethora of translation-handling programs compatible with it, out there too, but you could well enough handle it by editing the po files in any decent text editor.

The dxgettext team has an own editor in pure Delphi code called GORM. It has some features like multiline comments. In the past its main attraction was a google translate connection, but that has died.

Still if we want an own editor, converting gorm would be an option.

Since afaik with gettext, the gettext system is inbetween every lookup, you could simply maintain a database of looked up strings there.

Since basically there are three levels:

1. sourcecode inspection: in practice also harvest strings from uncompiled code.
2. assembler inspection: in practice also harvests strings from compiled code that is never executed.
3. runtime detection, mark when a string is accessed: risky since certain codepaths might only be rarely stressed.

Marco van de Voort wrote

Still if we want an own editor, converting gorm would be an option.

I do this online.

http://109.91.95.104/sprache

Not yet complete, but for demonstration ok.
From outside changes are not possible by now.

Rainer Stratmann wrote

Am Monday 13 August 2012 10:35:55 schrieb Lukasz Sokol:

[TRIM]

i've been following this whole thread with interest... one thing that i'm still not clear about, though, is why is this important? is it to see what areas of the program are actually used or something else? in my mind, a multi-language tool will have all strings translated into the supported languages... if there are some that are ambiguous or used in a different way, then they would get broken down further for proper language constructs... while i have numerous tools and utils out in the wild being used on a daily bases, none of them are multi-language but i do look forward to the day when i can produce such with minimal effort in coding... i have to leave it to others to do the actual translations, though...

Yes. It has to do with that.

in my mind, a multi-language tool will have all strings translated into the supported languages...

The translation is a different thing to that.

if there are some that are ambiguous or used in a different way, then they would get broken down further for proper language constructs...

I do not understand this...

while i have numerous tools and utils out in the wild being used on a daily bases, none of them are multi-language but i do look forward to the day when i can produce such with minimal effort in coding... i have to leave it to others to do the actual translations, though...

I put only ls('snippet1'), ls('snippet2'), ls('snippet3')... around the text everywhere I want a translation.

This is also better if you are searching for text in the program. You find then exactly the position you want.

The program then is - now before all ls() are executed - making a list of all caller adresses (handles) and of all snipets.

Caller adresses Table:
1 $0885BE80
2 $0886CE55
3 $088FF717
4 $0892B385

ID Table
1 snippet1
2 snippet2
3 snippet3
4 ID313

Snippet Table (original language)
1 snippet1
2 snippet2
3 snippet3
4 very long text snippet

Snippet Table (language 2)
1 schnipsel1
2 schnipsel2
3 schnipsel3
4 sehr langer text schnipsel

The ls procedure can get its (unique) caller adress and quickly search in the caller adress table and then knows the array nr of the translation.

You can make other tables (or fields in an array) for each translation for example. The translation is done somewhere else (for example online http://109.91.95.104/sprache).

There is a file with all snippets and translations (handles _not_ included, because they may change with each execution of the program). In this file you can store all information you want (date of creation, date of change, and so on).

If you have a very long snippet then for example you can give the snippet another identifyer by marking the identifyer with two '~' at the beginning of the text snippet, for example: ls ('~ID313~very long text snippet' ). With the shorter text snippets the snippet itself is the identifyer. If you work with aliases as identifyer then an additional field is required as shown above.

Again: all you have to do in the program is putting a ls() everywhere around you want a translation.

No need for tons od additional identyfiers and additional lines like:

var
 p_snippet1 : pchar;
 p_snippet2 : pchar;
 p_snippet3 : pchar;
const
 id_snippet1 = 'ids_snippet1';
 id_snippet2 = 'ids_snippet2';
 id_snippet3 = 'ids_snippet3';
p_snippet1 := trlt( id_snippet1 );
p_snippet2 := trlt( id_snippet2 );
p_snippet3 := trlt( id_snippet3 );
writeln( p_snippet1 );
writeln( p_snippet2 );
writeln( p_snippet3 );

My solution:

writeln( ls( 'snippet1' ) );
writeln( ls( 'snippet2' ) );
writeln( ls( 'snippet3' ) );

You are exaggerating the standard solution a bit... which would be:

resourcestring
   snippet1 = 'snippet1';
   snippet2 = 'snippet2';
   snippet3 = 'snippet3';
Writeln(snippet1);
Writeln(snippet2);
Writeln(snippet3);

Regards,
Sven

It was more or less exactly the way I did it before with little exaggerating as you notice :-)
Even now I save 50% codesize in comparison to resourcestring.

Rainer Stratmann wrote

It was more or less exactly the way I did it before with little exaggerating as you notice :-) Even now I save 50% codesize in comparison to resourcestring.

If you don't like resource string, inho there are 2 alternatives that are (IMHO) better:

1)
plain array with all text constants. You can break it into one array per unit, and have a register a search-handler via the unit's initialization.

const MySnippets: Array [1..xxx] of string = ('text', '...', ... ];

For more readability (but it adds source code (should not add more to exe)

const
    snip1: 'text';
   MySnippets: Array [1..xxx] of string = (snip1, '...', ... ];

(actually, I havent testet if the compiler takes the 2nd

2)
Since you seem to want something that is easier to type.

Keep using
   writeln(ls('text'));

Instead of analysing the exe, write your own pre-precessor. The pre-processor can scan the source, create a temporary copy in which it replaces the text in ls() with a reference to the "MySnippets[]" array. It also creates and includes a unit with that array. It then calls the compiler on the temporary created sources....

At least it will keep working even if fpc internals change. And it is cross platform (includes cross-CPU)

Then you have to put _all_ text snippets at once there (!) I didn't count my text snippets in this project, but they are spread over about more than 10 units.

I considered this, but the solution now is - in my view - unbeatably easy. And that is what counts for me.

I do not worry about this. The search procedure can easily adapted then.

And it is cross platform (includes cross-CPU)

Yes, that's true. But I am confident to find here also a solution once it will be necessary.

Why not scan the code with passrc, find the strings, build a table and then replace the strings with calls to a function that does the translation ?

your hack sounds like a maintenance nightmare... Thats what a C programmer would do and then boast about his programming skills, just to later discovar that his solution is nor portable nor safe...

2012/8/14 Rainer Stratmann :

Two informations:
1. All caller adresses of a function (lets say rs( s : pchar ) : shortstring )
2. All pchar pointers to the test snippets

That was all I need.

Your solution sounds not practical to me.
(scan the code with another tool, modify the source code or what do you mean? I don't want to have boast of tools for a simple solution).

I never talked about that my solution should be portable.

That was interjected here in this thread.

If you followed the discussion then I first asked for a (portable) solution. I asked if the compiler can provide the 2 informations above.

But that was not possible.

So I looked for an own solution.
Of course I am proud to find a solution and make some kind of marketing here. You call it "boast about programming skills".

You try now to make a bad atmosphere out of it.

ok... that's one part that is understandable from a statistics POV...

> in my mind, a multi-language tool will have all strings translated > into the supported languages... The translation is a different thing to that.

i don't understand... you have a list of the original strings and other lists of those strings translated into other languages... on startup, the program determines which language is chosen and it loads that translated language file... there is then no more translation needed... the program then simply uses those strings using the very same routines and no additional coding is needed... ???

> if there are some that are ambiguous or used in a different > way, then they would get broken down further for proper language > constructs... I do not understand this...

consider in english we might say "she opened the door" but in another language they might use the format of "door opened she"... basically what i'm trying to say is that the constructs of the sentences may be different... my comment above was looking that one might have to break down that sentence "she opened the door" into multiple parts so that the translation would be performed and assembled in the output of the program easier...

this would be no different than the program doing

   writeln(snippet1);

when all strings are translated and stored in some translated language file... snippet1 in english would be the same snippet1 in german, dutch, japanese, etc... it is my understanding that this is also the way the existing *.po files work... the code simply loads english.po or german.po or klingon.po at the start of the program and after that, everything is the same...

This is also better if you are searching for text in the program. You find then exactly the position you want.

i understand what you are saying but i don't understand why... since you mentioned an online translation service, are you looking at using live translations instead of static ones stored in a language file??

[trim]

You can make other tables (or fields in an array) for each translation for example. The translation is done somewhere else (for example online http://109.91.95.104/sprache).

so you are looking to perform live translations, then? what if that translation service disappears tomorrow? what about your users then?

There is a file with all snippets and translations (handles _not_ included, because they may change with each execution of the program).

i don't see how a language can change from one day to the next or even one execution to the next...

[trim]

Again: all you have to do in the program is putting a ls() everywhere around you want a translation. No need for tons od additional identyfiers and additional lines like:

but there's nothing like that with the existing po files if i'm understanding their purpose and method of use... the loading code simply chooses the proper po file and then loads the strings into an array or whatever using the same variables which are used everywhere no matter what language their contents are written in...

i must still be missing something :?

waldo kitty wrote

this would be no different than the program doing writeln(snippet1);

....

.i must still be missing something :?

If I understood him correct:

It is not
      writeln(snippet1);
it is
      writeln('snippet1');

With quotes.
He does want (or already has) the text snippets as in-lined constants.

He does not want to do the work of maintain them in another place (which an array, or resource string would require.)

yes, i caught that but thought that those were to be short examples of the strings...

He does want (or already has) the text snippets as in-lined constants.

ok...

He does not want to do the work of maintain them in another place (which an array, or resource string would require.)

what work? someone has to do it, somehow... it just seems better and even easier to have them in string variables and then just use them... i mean, ok... constants are one thing but they have their limits... if they are not intended to be constant then they should be in vars...

FWIW: i have, around here somewhere, code that does alter constants during execution... but it is TP6 based hackery code... in the same vein, i also have code that saves constants to the executable and then changes them based on settings... this instead of a configuration file... then there is also code to alter the parent environment's variables... all of these things are TP6 stuff but they won't work in FPC or many of today's setups... i'm really dreading to have to figure out some other way of handling these types of things... single file distributions are very nice even if they are actually multiple files all stored in the executable ;)

I am not this experienced with the english language, so sorry if I do noz understand everything 100%.
What means POV?

Yes this way.
It can happen that a new text snippet is used in the program. I use one file for all translations.

Yes, then the whole sentence has to be translated.

my comment above was looking that one might have to break down that sentence "she opened the door" into multiple parts so that the translation would be performed and assembled in the output of the program easier...

In my program there are 2 types:
1. Easy to translate words or little sentences.
2. User manual with longer Text.

But then you have to define snippet1 (pascal language identifyer in your example). I do not need to define snippet1 somewhere else.

writeln( ls( 'snippet1' ) );

snippet1 here is the text (in my case in germany) already. It is the written text and also the identifyer.

I do not know how the different *.po files work.

The users (in their countries) can translate may be better then I can so. For the english language I can do. Most people can understand english and from that language translate in their own language.

What do you mean with live translations exactly?
The web server and translation service I program myself. So it is available I think longer than until tomorrow.

One user uses english translation and the other dutch translation. It is far more easier for me to put all languages in one file, because of less administration overhead then.
If the languages are in one file I can also easily add a creation date, add date for changed entries (date of creation, date of change) and so on... Whatever I want and need for better administration and overview.

I have to write the code for that once and have the power for all snippets then.

I understand it that way that you have to define a pascal identifier for each snippet. I use the snippet itself as the identifyer. In combination with the caller address when calling the translate function ls(). If there are longer snippets (for example for the user manual) I will use other identifyers for that.

For example:
writeln( ls( '~ID555~longer user malual text ...' ); Then 'ID555' is the identifyer in the translation file.

If there is a new snippet a new entry with that snippet (and initially free entries for the other languages) is added automatically to the (one) translation file.

the loading code simply chooses the proper po file and then loads the strings into an array or whatever using the same variables which are used everywhere no matter what language their contents are written in...

Do you need a separate pascal identifier for each snippet?

i must still be missing something :?

It is writeln( ls( 'snippet1' ) );

With quotes. He does want (or already has) the text snippets as in-lined constants.

in-lined constants ???

He does not want to do the work of maintain them in another place (which an array, or resource string would require.)

Rainer Stratmann wrote

What means POV?

Ok. Point of view I assume.

No with (dx)gettext you don't need pascal identifiers for every string. And the translating function is _():string;
So all you do is writeln(_('snippet1'));, to follow your convention.

So it more or less uses the same idea as you are trying to replicate.

L.

Yes, that is possible as I understand it so far.
I did not know (dx)gettext before so replication is may not the right word. My solution will be more flexible I assume.
I put already some hours in it and I like to write my own code so I follow the already driven way.

Yes, it is. Replication is doing the same thing twice, regardless of the reasons/knowledge behind that fact.

My solution will be more flexible I assume. I put already some hours in it and I like to write my own code so I follow the already driven way.

I haven't used dxgettext, but would ask you to have a serious look at it. Presumably dxgettext users/developers had many of the same problems you face and have created solutions for those problems over the years. I understand you take pride in your own code and it feels like wasted effort to dump it, but using a set of proven code may be advantageous.

You can still submit your own improvements to dxgettext if you hit problems... and if those aren't accepted, in the worst case create your own fork.

You already have a very good grasp of the issues involved as you were writing your own solution. You could probably improve dxgettext with some things and get that in the main development tree, thereby improving things for others.
The other way round also works: other peoples' improvements can end up in your code for free (almost: you'll need some
implementation/verification time).

While this may all be very well known to you, I would ask you to step back and consider it: having a detached look at it may lead to a better choice.

Just my 2 cents,

regards,
Reinier

I you want call it replication.

My style of coding is normally in general writing an own solution. So I did it with a webserver also. Advantages are the flexibility. And a disadvantage is may the longer time to find a solution. But this fact relativizes the more specific the solution will be. I made the experience that the spent time will be the same (or even less if you exactly know what is needed) then.

If you are searching for an individual solution that is not done before than you have to ask the developers and hope that they support you. If not you have to find yourself a solution.

You already have a very good grasp of the issues involved as you were writing your own solution. You could probably improve dxgettext with some things and get that in the main development tree, thereby improving things for others.

As I read now here
http://dxgettext.po.dk/
gettext seems already to be a (very) good solution. I am wondering why there are some problems with as I read somewhere in this thread.

For me it would be interesting to know how dxgettext gets the snippet information from the program code...!

I'm not sure about that since replication isn't really a word, although it has crept into the English language. I look at it from a results POV rather from an action POV. Replication is about producing copies of the same result. You can do a similar action multiple times (duplication) and produce a dissimilar result. That wouldn't be replicating anything, it would just be a duplication of effort.

From a database POV (which is how I know what little I know about

replication since I was involved in a project to do this in the early 1990s before the whole db replication thing got so commonplace) the word replication means not that anything was done more than once, but that the one result was propagated to multiple places. So it seems to me replication is about multiple result instances, not multiple action instances.

"And a
disadvantage is may the longer time to find a solution."

Thats not the only disadvantage. Reinventing the wheel usually means you get something less correct and less researched. Its simple, 1000 eyes looking for a bug into a standard library will spot much more problems than your lone eye looking for a bug into your own implementation.

2012/8/15 :

My programs are bugfree :-)

Seriously: I feel ok with my style of programming. If you take a (standard) library it does not mean that everything is solved and that there are no questions as you can read in the many threads here in the mailinglist.

In our previous episode, Rainer Stratmann said:

Limitations, not problems. Compared to the Delphi internal translation systems. Yours has the same, by design.

Note that I think dxgettext is a netto plus compared to ITE.

If your source (main) language has a word with two meanings, how will you translate to a language where there is a word for each meaning?

IIRC dxgettext has only solutions for that when you recognize this early and add special code. This means that sometimes you can't simply ship a translation, but also have to

(dx)gettext also has support for stuff like multiples, textdomains. Some of it is not very well supported in the editor (gorm) though.

For me it would be interesting to know how dxgettext gets the snippet information from the program code...!

It comes with source scanner tools to harvest strings from the code.

The source for this + the editor are available. Most will readily work with FPC/Lazarus, except maybe unicode issues.

What's your unicode philosophy btw?

Which limitations?

Note that I think dxgettext is a netto plus compared to ITE.

I do not understand what you mean here.

If your source (main) language has a word with two meanings, how will you translate to a language where there is a word for each meaning?

You mean 2 words spread over the program?
Then I can put an additional identifier like already mentioned. ls( '~ID1~word' );
ls( '~ID2~word' );
In this case ID1 or ID2 is the identifier instead of 'word'. But in all cases 'word' is the result of the translation of the original language.

IIRC dxgettext has only solutions for that when you recognize this early and add special code. This means that sometimes you can't simply ship a translation, but also have to

In my case it can also be possible that there are not all words translated yet. Then the original text is shown or englich one. The translation can be made later. That is (also) an advantage.

(dx)gettext also has support for stuff like multiples, textdomains. Some of it is not very well supported in the editor (gorm) though.

I will use a browser as editor. So the utf8 (and unicode?) converting is made automatically by the browser. I have some code for converting ascii <-> utf8.

> For me it would be interesting to know how dxgettext gets the snippet > information from the program code...! It comes with source scanner tools to harvest strings from the code.

Another step more that means more work and less easiness (less userfriendly).

The source for this + the editor are available. Most will readily work with FPC/Lazarus, except maybe unicode issues. What's your unicode philosophy btw?

I need a converter from ascii to unicode. Then the browser can show this I think. With uft8 it works already.

Am 15.08.2012 10:48 schrieb "Rainer Stratmann" <

It scans the source beforehand and looks for resourcestrings and additionally for string constants embedded in a function call to "_()" (which is their translation function). It also allows to scan DFM files.

Regards,
Sven

If the compiler would simply store the 2 informations (caller adress, pointer to the pchar) in a list this scan beforehand would not be necessary anymore like in my solution.

I would call this function rs( s : pchar ) because it has to be short. if there is a rs() in the program the compiler stores the caller adress (unique handle) and the pointer to the pchar in a list.

That would be an advantage for freepascal if providing this information native and not by some kind of hack like I do now.
Then also the compatible/portable discussion would have an end. And the jokes about try to make it more difficult to search through the opcodes :-)

Rainer Stratmann wrote

I would call this function rs( s : pchar ) because it has to be short. if there is a rs() in the program the compiler stores the caller adress (unique handle) and the pointer to the pchar in a list.

If the pointer to pchar is unique then this one information will be already enough. That means the parameter s must be a constant text.

This question has long lingered in my brain just could not express it:

What happens in your program/translator, if the strings change their physical/relative address?
For example if you add/remove units and rebuild.

For example if FPC internals decide to add or remove some padding in front of the constants.

Will all your translation work go to waste ?

L.

Then the
mov pcharconst , EAX
command also had changed.
At every start of the program I see if it works here. If it works here ist works outside also.

For example if you add/remove units and rebuild.

All pchar adresses are searched at programstart.

For example if FPC internals decide to add or remove some padding in front of the constants.

Very unlikely.
For which reason should there be padding in front? I can not see any insolvable problem here.

Will all your translation work go to waste ?

Then I would have a look at the opcodes directly and see what has been changed and likely find a solution.

If the maintainers decide to build in the suggested function above then
everthing is solved. By now no one of the maintainers wants this.

I also can ask what happens if there are no more maintainers for fpc?

In our previous episode, Rainer Stratmann said:

> Limitations, not problems. Compared to the Delphi internal translation > systems. Yours has the same, by design. Which limitations?

see below

> If your source (main) language has a word with two meanings, how will you > translate to a language where there is a word for each meaning? You mean 2 words spread over the program?

yes.

No it can't, since if two the same texts translate to a different texts in another language, you need to add the identifiers and rebuild, distribute/migrate etc.

ITE doesn't have this limitation, since the fully qualified identifier is what uniquely identifies the text, not the content. The editor has some features so that you don't have to translate something as "settings" twice.

Moreover, ITE support widening and moving of components. (essentially anything published in forms is editable without recompilation)

What do you assume about the codepage of literals? Do you pass the right -Fc parameters for that?

> It comes with source scanner tools to harvest strings from the code. Another step more that means more work and less easiness (less userfriendly).

Explain. You need to scan some way too.

How do you handle combining translations btw? (e.g. general framework and customer specific part on top of that?)

Look at the top. The programmer must be aware of the two meanings and add a unique identifyer in front of the text. What you want to discuss here is specific and (more or less) deep. You want from me immediately a solution to all this kind of specific things.

ITE doesn't have this limitation, since the fully qualified identifier is what uniquely identifies the text, not the content.

See above. You can also add a unique identifier if you want.

Please explain it more.

Yes at programstart with searching the opcode for calls and loading the pchar const in EAX.

Post from 11.08.2012 which is searching the caller adresses of a given function (parameter proc_adr). With another piece of code I can get the pointer of all pchars if this proc has one pchar parameter.

It works!

procedure i_really_know_what_i_am_doing( p_opcode : pbyte ; count : longint ;
proc_adr : pointer );
const
 call_opcode = $e8;
 caller_adr_pc_offset = 5;
var
 p_reladress : pdword; // call works with a relative adress after the opcode
 x , caller_adress , jmp_adr_abs : dword;
begin
 for x := 1 to count do begin
  if p_opcode^ = call_opcode then begin
   p_reladress := pdword( p_opcode + 1 );                       // after
opcode = relative adress to procedure/function to call
   caller_adress := dword( p_opcode + caller_adr_pc_offset );
   jmp_adr_abs := caller_adress + p_reladress^;                 // absolute
adress = programcounter + 5 + relative adress
   if jmp_adr_abs = dword( proc_adr )
    then add_caller_adress_to_table( caller_adress );           // if call
proc_adr then found!
  end;
  inc( p_opcode );
 end;
end;

How do you handle combining translations btw? (e.g. general framework and customer specific part on top of that?)

Until now I did not come in contact with that.
I don't know if it is necessary for me.
So I don't need to spend the time now for that.
But I am confident that there will be a solution for that.

For the reason that they always say not to rely on 'internal workings of the FPC' :)

I can not see any insolvable problem here.

Yeah. But it is labour intensive.

See above.

If the maintainers decide to build in the suggested function above then everthing is solved. By now no one of the maintainers wants this.

I can understand why, more or less - this could be a security flaw if you can find the final procedure call address like that [and then inject/patch it from outside, while the program is running - see what I mean?]
Sort of the reason why Linux doesn't export System.map any more...

And the sort of reason why (dx)gettext scans the _source_ not the binary.

I also can ask what happens if there are no more maintainers for fpc?

Then you can fork it and maintain for you and for others for eternal glory :J

Lukasz

For me it was quite easy to see a system in how the compiler generates the opcodes. There aren't this much variants to do so.

> I can not see any insolvable problem here. Yeah. But it is labour intensive.

Initial search code about 3 hours work.
Add another search method (which should be very rarely) about 10-30 minutes work I guess.

Please explain.
I do not change the code. I am only searching some pointers.

Sort of the reason why Linux doesn't export System.map any more... And the sort of reason why (dx)gettext scans the _source_ not the binary.

If the pointers were provided natively then scaning the source (labour intensive) is no more necessary.

> I also can ask what happens if there are no more maintainers for fpc? Then you can fork it and maintain for you and for others for eternal glory :J

Very huge labour intensive I think!

In our previous episode, Rainer Stratmann said:

No. The route was this:

- In an earlier msg I mentioned this as dxgettext limitation (since I ran into it)
- In a later msg you said you got weary of dxgettext because of "problems" with it.
- I replied that it was not a problem, but a limitation of the concept, and that your system had it too.

I never asked you to have a solution for it. I don't see the particular advantage of your system over gettext anyway.

If there is a _('whatever') (or whatever your notation is)

what do you assume about 'whatever' 's codepage?

_always_ ? Scary.

Anyway, my point was that you scan either which way. Source or binary.

The source has a build requirement, is architecture dependent and generally more portable. Binary might be easier to only get the linkedin texts.

Actually proof of concept might be worthwhile for that purpose in combination with gettext. OTOH, gettext alos translates stuff from resources (like forms)

sorry... acronym... Point Of View...

Howsoever, I am not familiar with dxgettext.
By now I can not see a problem to use my concept.

For me it all mainly comes to the questions of providing the pointer to the text snippets and the caller adresses.
If the compiler provides this information then programming would be easier. The other things (double texts, coding of text, ...) are subordinated and could be solved when it is needed.

I never asked you to have a solution for it. I don't see the particular advantage of your system over gettext anyway.

For me the advantage is that I have already busy with that stuff some time and I had partly a solution except the pointers to the strings and the caller adresses which I asked to be provided.

ascii
In the main program I only need ascii code (german, english, some other european laguages which are compatible with ascii). 'Exotic' languages can only be accessed by the browser interface (utf8 which means also ascii).

By now yes. The searching routine gets all text information now. If for example push (to the stack) is also used later than I can adapt easily.

Anyway, my point was that you scan either which way. Source or binary.

Binary is much more easier for me and takes less effort.

The source has a build requirement, is architecture dependent and generally more portable. Binary might be easier to only get the linkedin texts.

The solution would be that the compiler stores the information in a table and then is available like already described.

Actually proof of concept might be worthwhile for that purpose in combination with gettext. OTOH, gettext alos translates stuff from resources (like forms)

Well, yeah, _you_ don't. What if somebody else could create a program that extracts private (unexported) function pointers from executables and be able to redirect entire code paths. Oh wait. this is called executable infection and great deal of people actually _do_ that.

> Sort of the reason why Linux doesn't export System.map any more...

(actually, they don't export syscalls table as r/w area, for the fear of somebody patching it with their own entries, they do it as r/o instead).

> And the sort of reason why (dx)gettext scans the _source_ not the binary. If the pointers were provided natively then scaning the source (labour intensive) is no more necessary.

So you think maintaining your own disassembler is easier, more stable and less maintenance than scanning the source?

L.

In fairness, he has said that this is for his own use. And insofar as FPC is a general-purpose tool we should acknowledge that "there's more than one way to do it".

If anybody is worrying, it should be his professional successors (i.e. whoever inherits the value or liability of his code should he fall under a tram). Or, for that matter, his customers or associates who by now can see everybody telling him that his technique is questionable (this is, after all, a public mailing list).

So you think maintaining your own disassembler is easier, more stable and less maintenance than scanning the source?

He apparently does. And writing as somebody who does disassemblers for various CPUs and platforms, I think he's deluded.

I wonder what would happen if somebody tried to run the code on an OS that protects executable code from being read?

I don't understand what you want to say here.
Am I responsible for things other people may do?

Yes, of course.
It runs automatically no additional step is needed.

Yes. Because your proposed changes open up avoidable security holes in the compiler, you are.
(If I were you my counterargument would rather be that "obviously this functionality would be driven by a compiler switch that is off by default"...)

ROFLMAO after reading this thread, especially Mark ML's post.

Customers are fully satisfied and know about my fast responce. You can show them, of course.
It is funny to see all the speculations and kind of FUD here.

Please note that it works perfect.
There can be more of those comments, making a bad mood. But that does not affect that this is working perfect. I further don't seen the need to explain everyone who is struggling with a tool for that and a tool for that how simple things work.

I wonder what would happen if somebody tried to run the code on an OS that protects executable code from being read?

Rainer Stratmann wrote

Customers are fully satisfied and know about my fast responce.

response

I still dont understand why dont you hack passrc and grab all the strings from the source ?

You can even replace them with identifiers and do all the resource string mumbo jumbo automatically if you write a app to do this based on passrc...

Actually, why theres no such tool included with freepascal already ? (there is ?)

2012/8/16 Rainer Stratmann :

Jorge Aldo G. de F. Junior wrote

I still dont understand why dont you hack passrc and grab all the strings from the source ?

Because I have an almost finished solution. I worked already for a while on it. Some integrated piece of code is easier to handle for me than using a tool which requires (an) additional step(s) and may can change its behaviour in the future. In the last case even more (demotivating) work has to spend on it.

You can even replace them with identifiers and do all the resource string mumbo jumbo automatically if you write a app to do this based on passrc...

This is more complex than own code that I know best and can change to my needs quickly if it there is a need for.

Actually, why theres no such tool included with freepascal already ? (there is ?)

If passrc changes, so can ABI and whatever else method you use change too...

If i was going to start a project now with multilanguage suport i would search for a existing solution or start a new project based on passrc.

I dont see how can searching for strings be so hard, you dont even need a full fledged parser to do that... having passrc is handy because its something standard and already done.

I believe such a tool written using passrc would limit itself to parse each source file, put each string found into a global string list and replace string instances with a call to a global function like GetMyString() with the string entry numer as parameter...

I would even export such strings to an external file to allow fast, on-the-fly, language selection...

IIRC this is called string pool.

but thats my two cents...

2012/8/16 Rainer Stratmann :

Rainer Stratmann wrote

Am Thursday 16 August 2012 10:50:25 schrieb Mark Morgan Lloyd:

[...]

Umm from what Rainer wrote elsewhere, I think, he's embedding the 'executable scanner' into the same executable in question, so that problem doesn't exist... more or less? (can't forbid the program from reading itself can't we?)

So from this POV, his process /is/ easier: he runs his program, and tells it to export strings to be translated, then loads 'snippets' on demand/on next execution.

To Rainer: Please accept apologies, I did not mean to offend you - I just tried to
find out more about your solution;

But still not something I would use if I had a choice, I must note that.

Reasons? Technically I don't think it's better to browse through executables. Requires knowledge /I/ don't have/find useful in plain everyday use of FPC - of its internals, assembler etc, processor opcodes... much too low-level for my comfortable liking. Every core change in FPC, may break it. You won't know until it happens. It won't work on any other processor/FPC target than x86 (Windows on ARM, Android, JAVA VM, to name just a few) without getting deep-level working knowledge of that first.

All I mean, your solution won't do for /me/ that's it :)

Regards,
Lukasz

I did not count the textsnippets in my project.
My philosophy is to have a good ground to make the following things as easy as possible.

I dont see how can searching for strings be so hard, you dont even need a full fledged parser to do that... having passrc is handy because its something standard and already done.

I did not say that it is difficult to search for strings.

I have an external file also.
All possible with my solution.

If there is a new unused string in the code then _automatically_ a new entry is made in this external file. Ready for translation by/over the integrated (from scratch written) webserver :-)

Lukasz Sokol wrote

So from this POV, his process /is/ easier: he runs his program, and tells it to export strings to be translated, then loads 'snippets' on demand/on next execution.

Yes.

To Rainer: Please accept apologies, I did not mean to offend you - I just tried to find out more about your solution;

No need for an apology :-)

It may seems difficult. Sure there is time needed to check the code, but once you found the right opcodes it is easy.

And if the compiler provides the pointer information than it is "legal" and also works on/with other platforms/processors.

All I mean, your solution won't do for /me/ that's it :)

:-)