I'm really surprised about the amount of short strings used on the FPC compiler source, such as short strings being used even to pass to Exec() when the linker is called on windows. This is obviously why for the past few years I've had all sorts of chopped off issues with FPC and got weird linker messages when using long directory paths. Especially when I am using the -o option in combination with a bunch of other linker options (since -o is passed to the linker, along with -K options and other stuff, such as --include, AFAIK). I'm wondering why short strings were used so dominantly - was it because FPC 2.0.x is based on old source code base which didn't have ansistrings available yet? Was it for speed? I don't see how using short strings could significantly improve the speed on for example an Exec call - I could see it greatly improving speed in some cases - but when it introduces hard to find bugs, not worth a few milliseconds IMO. And the Pascal language is supposed to have less bugs in it because of strong typing, but in fact strong short string typing has really caused me a lot of bizarre bug headache :-)

I've made some efforts to change some stuff to ansistrings to try and solve my bug with -k option that I mentioned recently - and although I am making progress, I realize that I am changing TONS of code since tons of the code uses short strings - is there any plan to get rid of the short string reliance in lots of the code? It will require a HUGE patch on multiple units. I'm just trying to find out some of your plans on the future FPC versions. The short string reliance kind of scares me, actually.

Op Tue, 16 May 2006, schreef L505:

It is partially historic, but mostly for for performance reasons. Shortstrings are really a lot faster than ansistrings, and therefore they will continue to be used.

While ansistrings are not really forbidden in the compiler source, there has never be a good reason to use ansistrings in a lot of code.

Daniël

how about adding a "shortstring length error" option and using it when compiling the compiler. The compiler must be doing length checking anyway to give clean truncation so i can't imagine this would bring a signifiant performance penalty.

This would make bugs related to shortstring overflow much easier to trace!

Op Tue, 16 May 2006, schreef peter green:

No, shortstring are completely safe types to use and therefore truncation is automatic i.e.:

s[0]=#255;
fillchar(s[1],255,'a');
s:=s+'a';

...will result in a string with 255 a's.

To display an error we would have to add manual checks in a lot of places, and strings are often used in inner loops in the compiler. (The compiler is very, very string sensitive, it spends an important parts of the compile time manipulating strings).

Usually, if a shortstring is not enough, you are trying to squeeze a list of something into a string. Convert the data structure into a data type suitable for a list, and the problem is solved in most cases. This is the case with linker options.

Daniël

thats what it does at the moment, i don't see how this precludes adding a compiler option to make doing this a run time error though (obviously disabled by default for compatibility with existing code) and then compiling the compiler with that option on.

L505 wrote

I've made some efforts to change some stuff to ansistrings to try and solve my bug with -k option that I mentioned recently - and although I am making progress, I realize that I am changing TONS of code since tons of the code uses short strings -

Are you getting rid of shortstring in general or just short strings that are utilized for paths?

I think it's only the shortstrings for paths that cause problems.

Op Tue, 16 May 2006, schreef Michael Van Canneyt:

A third reason: Most strings in the compiler are very short. I once did some statistics, allthough I can't remember the actual values, the percentage of strings < 8 bytes was well in the double digits.

The reason is that things like "var s,t:string" give very short identifier strings inside the compiler. Ansistrings are an expensive overhead here, for example even the 12 byte header counts heavily here.

Daniël

There is no problem with short strings, except for external paths.

What IS a problem is that changing it to ansistrings will significantly slow down the compiler. For 2 reasons:

1. It's heap based. There is a lot of memory manager overhead.
2. The implicit try...finally in each procedure that uses them introduces a memory penalty and a speed penalty.

So I would really advise against this change. There is no gain, except maybe for some external path settings. identifiers etc: there short strings are more than enough.

Michael.

Op Tue, 16 May 2006, schreef peter green:

Yes, it would be possible to build such a feature.

Daniël

Of course such a change would be only for paths and for calling external tools. That's what USE_SYSUTILS is all about. But the last time Peter benchmarked it, it made the compiler nevertheless almost 20% slower. There is after all a lot of unit searching etc going on behind the scenes. Still, it may be better now with all the ansistring optimizations which have been added recently.

Jonas

Are you getting rid of shortstring in general or just short strings that are utilized for paths? I think it's only the shortstrings for paths that cause problems.

It was a local copy experiment of my own compiler so don't worry I'm not committing to SVN :-)

Well the problem I saw was that one procedure requires shortstrings and it is linked to another procedure that requires shortstrings - it's not just one piece of code that runs the whole Exec() procedure but a bunch of procedures. For example the Exec on Windows is located in Dos unit, and Dos unit uses shortstrings - not just the compiler core uses shortstrings, but the RTL which the compiler core relies on uses them. Wouldn't want to change the Dos unit itself if that was causing problems, since Dos unit must remain compatible with old Turbopascal Dos unit - if Dos unit was part of the problem a custom exec procedure would have to be written and Dos unit would not be used for Exec, rather the custom unit would..

But I still have to verify I've solved exactly all spots where my shortstrings are getting chopped - whether it is only one place or several combinations of places. I suspect like Daniel M. says that the problem may be solved simply by increasing the number of strings being passed (in a list or record structure) instead of cramming them all into one string.

By the way - was it ever possible in Delphi or FPC to define a very special short string to be of the length longer than 255, let's say

string[1000]

I remember that coming up once before on the mailing list but I forget the end result of the conversation.
Not that this is the answer to all our the problems in this thread, though - I'm just curious.

well it would have to have a different in memory format. Maybe a -1 based array with the first two locations used for length would do it.

well it would have to have a different in memory format. Maybe a -1 based array with the first two locations used for length would do it.

Add new types that extend the 255 limit, using more bytes for storage to double and even triple our current 255 limit.

QShortString: 255 X 4 with 4 bytes used to store the length TShortString: 255 X 3 with 3 bytes used to store the length DShortString: 255 X 2 with 2 bytes used to store the length

var
  qss: QuadShortString[1020];
  tss: TripleShortString[765];
  dss: DoubleShortString[510];
  ss: ShortString[255];

I think once there existed something longstring with max len 2^16-1.

QShortString: 255 X 4 with 4 bytes used to store the length

maxlen: 2^32-1

TShortString: 255 X 3 with 3 bytes used to store the length

maxlen: 2^24-1;

DShortString: 255 X 2 with 2 bytes used to store the length

maxlen: 2^16-1

Vincent.

Because my math is horrible and I need to take a Computer Science course :-)

I think once there existed something longstring with max len 2^16-1. From usenet:

"..DEC Pascal has a VARYING OF CHAR type (for those of you with PL/I background) as well as a STRING type which can go upto 65535 characters (it has a leading length WORD, not a BYTE)."

The SysUtils unit misses some performance tweaks that the Dos unit has. Especially for the FindFirst/FindNext part. But this has low prio for the current developpers so it hasn't been analysed and fixed yet.

> The SysUtils unit misses some performance tweaks that the Dos unit has. > Especially for the FindFirst/FindNext part. But this has low prio for the > current developpers so it hasn't been analysed and fixed yet.

What do you guys thing about the idea to implement what DEC Pascal and Extended Pascal have - a 2 byte length ShortString (MediumString?), uprdade *some* of the path related ShortStrings to be MediumString[1000] instead of ShortString[255]. For some situations it would probably fix some problems immediately just by upgrading to string[1000] or even string[500].

I wondered why Borland/FPC didnt' implement a 2 byte length storage short string in addition to the 1 byte length storage short string.. maybe just haven't gotten around to it? Maybe there are other reasons? I think it is a good idea to have a really short string which has a maximum element count of 255, and to also have another larger shortstring which can go past 500 if you want it to. You'd rarely use 60,000 you'd mainly use figures within the 255-2000 range.

This would keep speed up without affecting way we think much - they are still short strings, just longer ones. You'd still have the option of using a 255 shortstring. Many changes would be needed if you turned items into an ansistring or a list of small 255 strings - but if all you had to do was upgrade the string to string[1000] it may serve the compiler well for 10 years until people start having hard drives with REALLY long paths that combine to exceed 1000 - then maybe you upgrade to 1500, or start separating the strings into a list of strings.

There is the basis of a "longstring" type in the compiler which does that more or less, but it was never fully implemented, because nobody cares/cared about it. And you'd still have to convert these longstrings to ansistrings and back every time you interface with sysutils, which would introduce extra overhead.

Jonas

We wouldn't have to use sysutils yet.. we could make a custom Dos unit which used "longstrings" instead of short strings, but keep the old Dos unit for compatibility..

i.e. go in to the dos unit and simply change all the short strings to a "longstring" but back up the original copy of dos unit.. then we would still not have to use Sysutils for the compiler, just a longstring based Dos unit..
In fact, not all the shortstrings would have to be changed to long strings. Just some path related ones.

IMHO, longstring are wasteful for memory, having a fixed length string of 1000 for paths, which on my computer paths are less 100 chars.

Vincent.

My paths are less than 100 chars. That is not the problem. Otherwise I wouldn't be noticing the bugs I've noticed for the last few years. I've never had a path beyond about 50 characters. The problem seems to be that several OPTIONS and PATHS at some point get combined into one short string at the end of the "funnel".

L505 wrote

We wouldn't have to use sysutils yet.. we could make a custom Dos unit which used "longstrings" instead of short strings, but keep the old Dos unit for compatibility..

This still means that someone has to finish and test longstring support in the compiler, and create this longstring Dos unit for all platforms. That's more work than it may seem, and partly wasteful since Dos is only a legacy compatibility unit.

In fact, not all the shortstrings would have to be changed to long strings. Just some path related ones.

As I've said already many times in this thread: of course only path- related stuff has to be changed in the compiler.

Jonas

I'm happy with current compiler because all I have to do is change several of my paths to a bit of a shorter path - it's not a big deal if you know this is an issue that you have to deal with. So now that the bug is known, I could live with it for really another 10 years. The important part is knowing this bug so I can work around it - if I hadn't figured this out I would have blamed all my problems on myself. Maybe it was GNU's fault, or someone elses.. but now I know what causes it.

Longstring is not a feature that needs to be implemented this minute - just like templates/generics don't really need to be implemented this minute.. it's just a cool feature that has some advantages. What I wanted to clarify was that a longstring is not utter useless and does have some advantages, unless my reasoning is flawed.

Op Wed, 17 May 2006, schreef L505:

Well, dos.exec of course requires the entire command line as a shortstring. This is no problem for Dos because Dos has a maximum path length of 128 chars. For some operating system you could implement a dos.exec which accepts an array of shortstrings, which could solve a lot of length problems.

Daniël

Well, dos.exec of course requires the entire command line as a shortstring. This is no problem for Dos because Dos has a maximum path length of 128 chars.

so with 2 max length paths plus some options in the command line your screwed with shortstrings?

yep silent string truncation was not one of borlands smarter ideas.

Having to check and guard each and every string operation is also not really productive. K&R tried that :-)

Op Wed, 17 May 2006, schreef peter green:

Under Dos: Yes, the operating system didn't allow more. But remeber, filenames were in 8:3 format, so 128 bytes was enough for a directory tree of 10 levels deep. Even today, few people use directory trees that deep.

Considering the 640k limit, the 128 bytes reserved in the PSP for the command line wasn't a that stupid tradeoff.

Daniël

Could happen, but also it might require a dynamic array of shortstrings which then might end up being slower than just one longstring using shortstring + shortstring + shortstring concatenation into a longstring (instead of a dynamic array of shortstrings). But Daniel's array method would probably be the easiest thing to implement right now, since we don't have longstrings available at this time.

Marco van de Voort wrote

Having to check and guard each and every string operation is also not really productive. K&R tried that :-)

And I thought they did it the other way around. Leaving the check as an exercise to the coder. ;-)

Vinzent.

That's stupid. Better use ansistrings then.

Hello,

Also assembler symbols/labels should get extended to strings > 255 in the future because there is already a bug open in which it is demonstrated that it is possible to create too long labels which makes a program uncompilable.
Or some scheme derived which makes sure that labels never get larger than 255 chars.
(Typed constant within 2 or 3 nested functions)

Additionally even the ppc64 compiler isn't able to cycle when compiled with -Cg because of the shortstring limitation, a few symbols get truncated, which makes the assembler fail.
This is because the assembler syntax for declaring a symbol in the GOT on this platform requires the compiler to add the symbol name twice to the directive. This effectively limits symbol names to around 120 chars already. Which is, as mentioned before, already not the case for the compiler as it is.

Regards,
Thomas

Op Thu, 18 May 2006, schreef Thomas Schatzl:

This is exactly a thing that will not happen. Longer assembler labels have no usefull advantage for the end user, they are only used hidden from the user. Long names only cost memory. For mangled names, the compiler already switches to hashes instead of type lists. If there is any other reason a symbol gets to long, it can be handled in a similar way.

The solution is to generate shorter symbols. Hash the symbol name if it gets too long.

Daniël

That's one solution, that's not the only solution.

I can see people arguing that a 50 element limited short string is enough, I seriously can.

I think you guys may be living in a 255 cave, simply because that's all we have to deal with at this time. Some say that ansistrings might be the way to go using sysutils - personally I think sysutils has no place in the compiler core and the compiler core should have tight custom units with no end user units like sysutils. One way to accomplish this, like I've already mentioned, is to use shortstring/longstring/array of string/ based Dos unit, using shortstrings where necessary, arrays of strings where necessary, and arrays of chars or longstrings where necessary. An array of char is just a dumb longstring, that's all. Upgraded Dos unit could contain some functions pulled in from sysutils, but not actual sysutils in the uses clause - just some optimized systutils pulled in and put into the upgraded dos unit. Still keeping the old Dos unit for compatibility for users, name the new upgraded dos unit anything - newdos.pp, whatever.

I'd be willing to help on this one and do some work, but unfortunately since we're all disagreeing it means we can't do any work until we come to an agreement. Once again, it's not just about having a team of programmers doing the grunt work - but also about having some sort of consensus or agreement before doing the work. Otherwise one of us will waste our time submitting a patch which won't be committed because some other folks don't like the way it was done.

If you use an array of strings you eventually have to combine these array of strings together into one common buffer to send to exec(), so you are reinventing the longstring or the ansistrings, or an array of char if it is one big piece being sent in the end anyway. The longstring is faster. It's perfectly okay that you don't want to implement longstring because it is hard work - but at least admit that it is useful, whether it is implemented or not. It's like saying ansistrings are useless garbage because we haven't implemented them yet. No, they are useful - but maybe they are hard to implement. I doubt a longstring is hard to implement compared to something like templates/generics, though.

But don't take my message as an offense - I'm sure you all know it is normal for this to happen among programmers - discussing topics and arguing their brains out.

Can someone tell me how slow/fast a dynamic array is compared to a fixed one? Say you used a dynamic array of chars or dynamic array of shortstrings - would the dynamic array be slow on a general basis? Maybe we will have to resort to benchmarks using the cpu timer. And then there is also a fixed array of shortstrings or a fixed array of chars too.

Op Thu, 18 May 2006, schreef L505:

That's one solution, that's not the only solution.

Very right. It is a trade-off. Do you fix the shortstring issue and continue to get their benefits, or do you abandon them, rewrite large parts of the compiler and pay the performance/memory usage price?

I don't agree here. The compiler is no special application regarding interfacing to the operating system. We should eat our own dogfood in other to provide the users the best runtime library. I agree there are technical issues against sysutils, which might warrant a new unit. On the other hand, you have to consider the cost/benefit. Sysutils is pretty usefull, and despite the issues, it is possible write very good software with it. Separate units increase the amount of work to port the compiler.

Remember there are more issues with Dos than just the string length. For example findfirst/findnext is not really modern either. A modern unit would be a design from scratch. Actually, I think it would be nice to have one.

If you use an array of strings you eventually have to combine these array of strings together into one common buffer to send to exec(), so you are reinventing the longstring or the ansistrings, or an array of char if it is one big piece being sent in the end anyway. The longstring is faster.

Not entirely true. On Dos-like operating systems the combining happens indeed in a buffer that is allocated using getmem. On Unix like systems, the paramaters are currently combined to pass them to dos.exec, then split again to pass them to execve, which is a strange situation.

Don't worry :)

Can someone tell me how slow/fast a dynamic array is compared to a fixed one? Say you used a dynamic array of chars or dynamic array of shortstrings - would the dynamic array be slow on a general basis? Maybe we will have to resort to benchmarks using the cpu timer. And then there is also a fixed array of shortstrings or a fixed array of chars too.

A dynamic array is like an ansistring, they are garbage collected and therefore your get an implicit try/finally. Generally, we use reallocmem inside the compiler, again for performance.

Daniël

L> Can someone tell me how slow/fast a dynamic array is compared to a fixed one? Say you used L> a dynamic array of chars or dynamic array of shortstrings - would the dynamic array be L> slow on a general basis? Maybe we will have to resort to benchmarks using the cpu timer. L> And then there is also a fixed array of shortstrings or a fixed array of chars too.

Dynamic arrays are reference counted, but it does not slow a program much if you mostly read/modify them, but do not assign. With optimizations on it doesn't differ from common arrays in inner cycles.

Some benchmarking some time ago (see maillist) didn't show anything clearly: dynamic arrays even appeared faster sometimes.

But they were introduced for Delphi compatibility, so I don't find them very good: they always start from zero (a[0] in one-dimensional case), if you used static array as formal parameter (use array name) (e.g. for blockwrite), you have to use first element with dynamic arrays. As I wrote (today?) you may have to check manually that length of array is not zero (to avoid range check error or something worse), even if nothing would be tried to write to the array of zero length.

So, generally developers don't like dynamic arrays, and I don't think that thet want to use them in compiler sources.

Ansistrings are even worse. A simple program which did ONLY modify elements of ansistring, not modifying its length, improved 20% in speed (it did some things besides toying with that string) by turning ansistrings off.

But they are said to be improved in recent versions (recent snapshots?).

If you talk of parameters etc, you should think of unicode strings too. Tonow they are implemented so poorly, that I use static array of words instead of widestring. The only advantage is ability to typecast to pwidechar. And it is indeed array of words: no unicode specific tasks may be performed.

Don't forget, that some Windows claim to support pathnames-filenames in unicode up to 32000 characters long! (However, it does not work both on user interface level and on api level. At least not as expected. Maximum under winxpsp2 is ability to create files/directories unreadable by user.) But it's only a matter of time: probably Windows will become totally utf16 (not really unicode, but at least utf16) really soon (at least in newer versions in a way incompatible with current ones).

So, longstrings (shortintegerstrings?) sound more likely to be used by FPK.

But it's only a matter of time: probably Windows will become totally utf16 (not really unicode, but at least utf16) really soon (at least in newer versions in a way incompatible with current ones).

A small correction, utf16 is a type of unicode.

thanks,

Dynamic arrays can be very handy and I never knew anyone who avoids them. Of course if your array has fixed length there's no reason to use a dynamic array either.
Fortunately it's no very often that one falls in Borland's trap that dynamic arrays aren't copy-on-write like AnsiStrings... BTW, is this the behaviour in FPC, too?

Ansistrings are even worse. A simple program which did ONLY modify elements of ansistring, not modifying its length, improved 20% in speed (it did some things besides toying with that string) by turning ansistrings off.

It's simply because the code has to check there's only one reference to the string on each change. If you know there's no concurrent access to the string (e.g. you app is single-threaded, or you have a local copy of the string) you should access it as a PChar.

But they are said to be improved in recent versions (recent snapshots?).

I find it strange that the cost of copying a ShortString (maybe because they are at most 255 bytes? Maybe cache locality usually is fine in this case? 8-| ) is lower(better) than the locked-count-reference and the exception trapping...

Anyway, isn't it just the case to correctly optimize string parameters as 'const' and 'var', and maybe using PChar in some few places, or can you think of any other reason for AnsiStrings to be slower than ShortStrings?

Cheers,
Flávio

Op Thu, 18 May 2006, schreef Flávio Etrusco:

Free Pascal is Delphi compatible.

A lot of people use getmem combined with possibly reallocmem if the array size should change after initial allocation. It is low level programming and therefore ugly, but dynamic arrays are being considered ugly as well by many people because they differ a lot from standard Pascal semantics.

> L> It's simply because the code has to check there's only one reference > L> to the string on each change. If you know there's no concurrent > L> access to the string (e.g. you app is single-threaded, or you have a > L> local copy of the string) you should access it as a PChar.

This code:

procedure z;
var a:string;
begin
  a:='abc';
end;

... generates this monster with $H+:

P$TESTASTRING_Z:
push ebp
mov ebp,esp
sub esp,52
mov dword [ebp-4],0
lea eax,[ebp-24]
mov ecx,eax
lea eax,[ebp-48]
mov edx,eax
mov eax,1
call NEAR FPC_PUSHEXCEPTADDR
call NEAR FPC_SETJMP
push eax
test eax,eax
jne NEAR
lea edx,[ebp-4]
mov eax,edx
call NEAR FPC_ANSISTR_DECR_REF
mov eax,dword [_$PROGRAM$_L12]
mov dword [ebp-4],eax

call NEAR FPC_POPADDRSTACK
mov edx,dword INIT__SYSTEM_ANSISTRING
lea eax,[ebp-4]
call NEAR fpc_finalize
pop eax
test eax,eax
je NEAR
call NEAR FPC_RERAISE

leave
ret

With $H- the result is:

P$TESTASTRING_Z:
push ebp
mov ebp,esp
sub esp,256
lea ecx,[ebp-256]
mov edx,dword _$PROGRAM$_L9
mov eax,255
call NEAR fpc_shortstr_to_shortstr
leave
ret

It is therefore not surprising that the shortstring version is faster. Other reasons why they are faster are that temporary strings are allocated on the stack, a "sub esp,xxxx" is a lot faster than a getmem. Shortstrings also do not need reallocmem if they grow.

A shortstring copy is really fast. They are copied with 4 bytes at a time in assembler code, so you need at most 64 steps to copy a string of maximum length. Most strings are shorter, like the example above.

However, you are right that copying is a limiting factor in shortstring performance.

> L> Anyway, isn't it just the case to correctly optimize string > L> parameters as 'const' and 'var', and maybe using PChar in some few places, or > L> can you think of any other reason for AnsiStrings to be slower than > L> ShortStrings?

A lot of them, see above.

Daniël

Sorry, these two letters were accidentally sent personally.

To Felipe Monteiro de Carvalho:

Uh, Unicode is a big standard with lotsa features.

UTF.. is "Unicode transformation format".

So, utf16 is a type of representing unicode characters.

And, as I think, basic utf16 support does not include support for representing symbols by several utf16 units (unicode is to be about 1000000 symbols, neither much more, nor significantly less).

Also, support for ligatures and other features is weird to non specialized programmers too.

I don't think that Windows will support all unicode features in their implementation of utf16 in filenames.

But Windows, MS Office, MS Internet Explorer and some more M$ products are listed on unicode page as programs supporting unicode.

To FlАvio Etrusco:

PChar type is no less ugly for pascal than dynamic array.

I have null characters in my strings sometimes. If a char may have zero code, string should be able to contain it at any place.

There are already some complains about the memory usage. Increasing the string size adds a lot more overhead. Especially for all the small labels like .L1 etc. Already using longstrings will add 3 bytes for the length alsomost doubling the size from 4 to 7 bytes.

Don't shortstrings always use 256 bytes of memory ?

Micha

Op Fri, 19 May 2006, schreef Micha Nelissen:

It depends on wether they are stored in a variable of type "string", or allocated on the heap with a "Pstring". In all data structures where memory usage is an issue, the memory friendly Pstring is used.

Still, because the compiler uses a lot of short strings, the internal heap data structures are an important factor in the memory usage of these strings.

Daniël

Forgive me if I'm saying BS but it's been about 10 years since I programmed in TurboPascal (and used ShortStrings), but don't ShortStrings use the size they are declared with? And 255 is just default size (if no size is specified)?

-Flávio

I know that FPC aims to be Delphi-compatible, but it's not always the case, as e.g. the WideStrings were reference-counted until a couple of months ago.
So you are saying that in this is specific case FPC is already (unfortunately, as for the WideStrings case) compatible with Delphi?

Wow, thanks really a lot for all the info :-)
What is the disassembler you use? Is there any nice free one? I'll try to digest that assembly since I'm not a "close friend" to it ;-) Also, that case is IMHO the bad case of AnsiString (i.e. we have to add a reference). I'm more interested if there's any overhead when reading from a 'const' string parameter...

Cheers,
Flávio

This is something you really don't want unless you want to make the compiler 4x slower.

There is already special code in the compiler to truncate long manglednames Peter

Op Fri, 19 May 2006, schreef Flávio Etrusco:

Yes, that is what I meant of course :)

Yes, there is a really nice one and it is called Free Pascal :)

I used "fpc program.pas -al -Anasmelf -s -O2" to get the assembler outputs.

Daniël

Op Fri, 19 May 2006, schreef Flávio Etrusco:

Forgive me if I'm saying BS but it's been about 10 years since I programmed in TurboPascal (and used ShortStrings), but don't ShortStrings use the size they are declared with? And 255 is just default size (if no size is specified)?

Of course. But unless there is a good reason to limit the size, most people use string without maximum length specification. And with a Pstring, only the amount of memory that is needed to fit the string is allocated, which is what we are talking about here. An assembler label like ".L1" is stored as a Pstring internally, and requires 4 bytes plus heap internal data structures.

Daniël

As the name implies, UTF is a format to allow to represeting every and all Unicode symbols using diferent block/character size. UCS-2 and UCS-4 are subsets of the Unicode which can be represented by single 16bit and 32bit - respectively - characters. UTF-8 and UTF-16 (is there UTF-32?) can represent all of Unicode using encondig some ranges of symbols with multible "characters".

Why dynamic arrays are ugly? So TList (and TStringList, for that matter) is ugly, too?
PChar is as ugly as any other Pointer, but in low-level programming it can be necessary.
Also, using some compiler trickery can be done to optimize usage of AnsiStrings so we can avoid use of PChars, but of course this will have to wait a bit.

I have null characters in my strings sometimes. If a char may have zero code, string should be able to contain it at any place.

Just get a PChar to the end to the string and compare your runner to it, instead of searching #0 ;-)

Cheers,
Flávio

Longstrings are not complex, but simply a lot of work to implement. You need to implement a lot of converters and overloads. Non of the core developpers see the usefullness so it isn't implemented.

Peter

There are already some complains about the memory usage. Increasing the string size adds a lot more overhead. Especially for all the small labels like .L1 etc. Already using longstrings will add 3 bytes for the length alsomost doubling the size from 4 to 7 bytes.

Peter

On 18 May 06, at 13:50, L505 wrote:

.
.

.
.

It would be much more than just unit Dos - you'd
need changes in at least System (string
manipulation routines), strings (StrPCopy) and
many parts of the compiler itself.

Regarding unit Dos - providing PChar versions of
some routines might be of general use, probably
more useful than longstrings.

Tomas

Forgive me if I'm saying BS but it's been about 10 years since I programmed in TurboPascal (and used ShortStrings), but don't ShortStrings use the size they are declared with? And 255 is just default size (if no size is specified)?

The shortstring type does, but you can use a pointer to a shortstring which is sort of like a pointer to a character array. You can allocate your own memory.

Also, using some compiler trickery can be done to optimize usage of AnsiStrings so we can avoid use of PChars, but of course this will have to wait a bit.

Or programmer trickery - the fpc souce already contains lots of programmer trickery around ansistrings such as using uniquestring, setlength, and string[i] indexing instead of concatenating with plus sign - but my case for LongString/WordString is that they are stack based while ansistring is heap based no matter what. And lots of people use stack based arrays of chars and arrays of bytes, so might as well use a smart array of chars instead of a stupid one. The smart one is either a wordstring or a longstring.

. . > have tight custom units with no end user units like sysutils. One way to accomplish

this,

> like I've already mentioned, is to use shortstring/longstring/array of string/ based

Dos

> unit, using shortstrings where necessary, arrays of strings where necessary, and

arrays of

> chars or longstrings where necessary. An array of char is just a dumb longstring,

that's

> all. Upgraded Dos unit could contain some functions pulled in from sysutils, but not > actual sysutils in the uses clause - just some optimized systutils pulled in and put

into

> the upgraded dos unit. Still keeping the old Dos unit for compatibility for users,

name

Admitting that C-language style programming (*char) is more practical and real world than Pascal programming?

;-) Well, that depends. I certainly don't think
that *char is more practical for string
manipulation in general. However, when talking
about unit DOS, the underlying OS API calls
usually expect null terminated strings anyway.
Anybody could then decide whether he prefers the
easy to use Pascal string, because he only starts
executables with few parameters which could never
cross the limit, or whether he needs PChar for
whatever reason.

Tomas

The DOS unit is a legacy compatibility unit; It's interface is fixed. It should not be tampered with, also not something as seemingly innocent as adding pchar overloaded calls.

Michael.

Of course, you will have "#0 not allowed" in the interface then. Most API's tackle this by returning size also.

IMHO this whole discussion is nonsense. For basic path related stuff, ansistring is good enough, and if findfirst/findnext are still limiting with the current memmgr (afaik we changed to Micha's one since those original tests).

Moreover, if the weak point of ansistrings is the heap copying and implicit exceptions, the weak point of shortstrings iirc is unnecessary copying. Something that might really start to hurt more if strings get longer.

So before deciding to introduce a new "long" shortstring based on ansistring vs shortstring performance, a more thorough assesment should be done:

For the use of ansistrings for paths inside the compiler:

- does the new heapmgr improve ansistring peformance relatively?
- how much of the ansistring slowliness really depends on ansistring, and not some system routine like findfirst/findnext.

For the use of ansistrings in other parts of the compiler (since maybe in time, UTF8 could make 255 byte limitations for non ISO8859-1 languages more pressing):

- Can turning implict excepctions off in core parts improve ansistring performance?
- Find an expression for the difference in string copying in say a compiler recompile for both ansistring and shortstring.
- What else can be done to improve ansistring performance ?

Marco van de Voort wrote

- Can turning implict excepctions off in core parts improve ansistring performance?

Implicit exceptions are already turned of for the entire compiler.

Jonas

I agree that playing around with some benchmarks might be wise as all the assumptions about stack based versus heap based could be proven false from doing some tests..

Well there is a way to emulate a wordstring and play around to see what it's performance is like, I haven't done any testing yet verus some heap based situation. Just FYI, what I've come up with in order to emulate my so called "wordstring".. as I don't think a longstring is as needed as just a wordstring, because how are you going to allocate 2gb of stack for that longstring which can go to 2 gigs..??

But then again, in the future, a wordstring might seem puny .. and we might end up calling longstrings shortstrings, if memory gets humungous.

program Proj1;
{$MODE OBJFPC} {$H+}
const MaxWord = High(word);
type
  TWordStr = array [-1..MaxWord-1] of char;
{ turn two bytes into a word }
function MakeWord(a, b: byte): word;
begin
  Result := (b shl 8) or a;
end;
{ set length of word string }
procedure SetWordStrLen(var ws: twordstr; size: word);
begin
  ws[-1]:= char(lo(size)); // low byte of storage word
  ws[0]:= char(hi(size)); // high byte of storage word
end;
{ get length of word string }
function WordStrLen(const ws: twordstr): word;
begin
  result:= makeword(byte(ws[-1]), byte(ws[0]))
end;
{ concatenate (combine) 2 wordstrings }
procedure WordStrConcat(const ws1: TWordStr; var dest: TWordStr);
begin
  move(pchar(@ws1[1])^, pchar(@dest[WordStrLen(dest)+1])^, WordStrLen(ws1));
end;
var
  wordstr1: twordstr;
  wordstr2: twordstr;
  i: integer; // loop
  sstring: shortstring;
begin
  writeln('The program will prompt you for several tests (hit enter)');
  readln;
  writeln;
  SetWordStrLen(wordstr1, 5); // no heap memory allocation occurs
  wordstr1[1]:= 'h';
  wordstr1[2]:= 'e';
  wordstr1[3]:= 'l';
  wordstr1[4]:= 'l';
  wordstr1[5]:= 'o';
  writeln('Length of WordString1: ', WordStrLen(wordstr1));
  writeln('Actual Memory Size of Word String: ', sizeof(wordstr1));
  writeln('WordString 1 Contents: ', pchar(@WordStr1[1]));
  writeln;
  SetWordStrLen(wordstr2, 7); // no heap memory allocation occurs
  wordstr2[1]:= ' ';
  wordstr2[2]:= 'w';
  wordstr2[3]:= 'o';
  wordstr2[4]:= 'r';
  wordstr2[5]:= 'l';
  wordstr2[6]:= 'd';
  wordstr2[7]:= '!';
  writeln('Length of Word String: ', WordStrLen(wordstr2));
  writeln('Actual Memory Size of Word String: ', sizeof(wordstr2));
  writeln('WordString 2 Contents: ', pchar(@WordStr2[1]));
  WordStrConcat(wordstr2, wordstr1); // no memory allocation occurs
  write('After concatenation of two wordstrings: ');
  writeln(pchar(@wordstr1[1]));
  writeln;
  writeln('Want to see over 65,000 ''a'' characters? (press enter)');
  readln;
  SetWordStrLen(wordstr2, MaxWord - 1); // no memory allocation occurs
  for i:= 1 to MaxWord - 1 do
  begin
    wordstr2[i]:= 'a';
  end;
  // write over 65,000 characters, the maximum capacity of a WordString
  writeln(pchar(@wordstr2[1]));
  { now let's combine a short string with a wordstring }
  writeln;
  writeln('Let''s combine a short string with a wordstring.. (press enter)');
  readln;
  sstring:= ', interesting world.';
  WordStrConcat(sstring, wordstr1); // no memory allocated
  // now write the contents of the new wordstring
  writeln(pchar(@WordStr1[1]));
  writeln;
  writeln('Done (enter to exit)');
  readln;
end.

writeln('Let''s combine a short string with a wordstring.. (press enter)'); readln; sstring:= ', interesting world.'; WordStrConcat(sstring, wordstr1); // no memory allocated

sorry, forgot to include the actual shortstring overloaded function - my copy and paste skills are getting poor.

{ concatenates a short string with a wordstring .. overloaded}
procedure WordStrConcat(const s1: shortstring; var dest: TWordStr);
begin
  move(pchar(@s1[1])^, pchar(@dest[WordStrLen(dest)+1])^, Length(s1));
end;