Hi All,

I encountered a strange problem with lua integration. ?Following is the problematic part (full source attached):

type
? TluaLReg = array of luaL_Reg;
var
? LUA: Plua_State;
? lib : array of luaL_Reg;
? //lib : PluaL_Reg;
? i: Integer;
implementation
//omitted, see attached source...
initialization
LUA := luaL_newstate();
luaL_openlibs(LUA);
SetLength(lib, 2);
//SetLength(TluaLReg(lib), 2);
lib[0].name := 'update';
lib[0].func := @l_dat_upd;
lib[1].name := 'clear';
lib[1].func := @l_dat_clr;
luaL_newlib(LUA, lib); ? ? ?<--Access Violation Here
lua_setglobal(LUA, 'data');
finalization
lua_close(LUA);
end.
? ? ? ? ? ? ? ? ? ? ? ? ? ? ??
The above code caused access violation, unless I use the commented blue code to replace the red code. ?I wonder what's the difference between PluaL_Reg and array of luaL_Reg?? ?What's more strange is that the same code is OK after I copied it to a separate console mode application. ?In the attached archive, there are both the problematic project and the OK project.

In LUA manual about luaL_newlib(), it said, the 2nd parameter (lib) must be an array not pointer to an array. ? I am confused what happened here and further more, what's the underlying structure of a normal array in FP (If I am correct, dynamic array is same as pointer, at least that's what I learned in Delphi).

My environment is 64 bit Linux 3.5.0 (Mint 14).

Thanks a lot.

Sincerely,
Shannon

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20121228/079be122/attachment-0001.html> -------------- next part --------------
A non-text attachment was scrubbed...
Name: luaintf.tar.gz
Type: application/x-compressed-tar
Size: 30720 bytes
Desc: not available
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20121228/079be122/attachment-0001.bin>

Pointer and dynamic array are not the same, even though dynamic array is built on top of pointer (to a structure). When you pass a dynamic array variable, you're passing pointer TO THE STRUCTURE, NOT THE DATA. To correctly do it, you have to pass pointer to the first data, which is @lib[0] in your case.

The Assembler shows that @lib and @lib[0] are exactly same thing. ? The problem is in lua52.pas, which I am still tracing. ?Now the most strange thing is why one of my two programs caused SIGSEGV while another one using same code block does not? ?Is it because one of them is gui program and the other is cli?

I will post a debug report of assembly call points for these 2 programs on both Linux-64 and Win-32 platforms.

Thanks,
Shannon

? ?, 12? 29, 2012 at 12:26 ???leledumbo ???
Pointer and dynamic array are not the same, even though dynamic array is built on top of pointer (to a structure). When you pass a dynamic array variable, you're passing pointer TO THE STRUCTURE, NOT THE DATA. To correctly do it, you have to pass pointer to the first data, which is @lib[0] in your case.

The Assembler shows that @lib and @lib[0] are exactly same thing.

The following program shows that they are NOT the same

#!/usr/bin/instantfpc
var
  a1: array of Integer;
begin
  setlength(a1,1);
  writeln(ptruint(@a1));
  writeln(ptruint(@a1[0]));
end.

output:
$ chmod +x test.pas
$ ./test.pas
134651632
3077795880

Please see attached log file I copied from the assembler window. ?I am yet to test this on windows.

Thanks,
Shannon

? ?, 12? 29, 2012 at 1:26 ???leledumbo ???

The Assembler shows that @lib and @lib[0] are exactly same thing.

The following program shows that they are NOT the same

#!/usr/bin/instantfpc
var
a1: array of Integer;
begin
setlength(a1,1);
writeln(ptruint(@a1));
writeln(ptruint(@a1[0]));
end.

output:
$ chmod +x test.pas
$ ./test.pas
134651632
3077795880

Hi leledumbo,

You are right that they are not same:

begin
? setlength(a1,1);
? a1[0] := 12345;
? getvalue(@a1);
? getvalue(@a1[0]);
end. ?

The assembly code for the 2 getvalue() are:

project1.lpr:14 getvalue(@a1);
000000000040022F 48bf40a2620000000000 movabs $0x62a240,%rdi 0000000000400239 e862ffffff ? ? ? ? ? callq 0x4001a0 <GETVALUE> project1.lpr:15 getvalue(@a1[0]);
000000000040023E 488b3c2540a26200 ? ? mov 0x62a240,%rdi 0000000000400246 e855ffffff ? ? ? ? ? callq 0x4001a0 <GETVALUE>

One used movabs, and the other use mov. ?But why the generated assembly in my code are same?

Shannon

? ?, 12? 29, 2012 at 1:26 ???leledumbo ???

The Assembler shows that @lib and @lib[0] are exactly same thing.

The following program shows that they are NOT the same

#!/usr/bin/instantfpc
var
a1: array of Integer;
begin
setlength(a1,1);
writeln(ptruint(@a1));
writeln(ptruint(@a1[0]));
end.

output:
$ chmod +x test.pas
$ ./test.pas
134651632
3077795880

> The Assembler shows that @lib and @lib[0] are exactly same thing.

That's true for static arrays.

The following program shows that they are NOT the same

That's true for dynamic (managed) arrays.

You're talking about different array types.

DoDi

Sorry, we are talking about same thing. In my program I am using dynamic array, NOT static, but the assembly is same. In leledumbo's example, the assembly is differnt.

? ?, 12? 29, 2012 at 2:55 ???Hans-Peter Diettrich ???

> The Assembler shows that @lib and @lib[0] are exactly same thing.

That's true for static arrays.

The following program shows that they are NOT the same

That's true for dynamic (managed) arrays.

You're talking about different array types.

DoDi

Your code sample doesn't have a declaration for PluaL_Reg (and you don't use the TluaLReg type). Was it a simple copy'n'paste error or is it declared by the Lua wrapper?

PluaL_Reg is defined by lua52.pas. ? In the email I omitted irrelevant code, but full source is attached in the first email.?

? ?, 12? 29, 2012 at 3:13 ???Fl?vio Etrusco ???
Your code sample doesn't have a declaration for PluaL_Reg (and you don't use the TluaLReg type). Was it a simple copy'n'paste error or is it declared by the Lua wrapper?

Reading luaL_newlib documentation and lua52.pas implementation of it (it's a macro in the original C code), I see that you need a sentinel value to mark the end of the array by putting nil in the name part. I was already skeptic since the API doesn't mention the number of elements in the array as commonly required by a C function.

The end of array mark is added by lua52.pas, if you read the source code... ?My problem is solved if I remove the overloaded luaL_newlib function and use only the original pointer version, i.e.

var lib: array of luaL_Reg;
luaL_newlib(L, PluaL_Reg(lib));

It seems that while converting from dynamic array to pointer, the compiler knows to point to the first element of the array, not the structure. ?Could anyone confirm this behavior?

The remaining problems are: 1) why the lupas program works well even with the problematic overloaded version? ?I tested on Windows 7 (32bit), both programs failed. ?2) I will email the author of lua52.pas to investigate the overloaded functions, and personally I will suggest to not use

? ?, 12? 29, 2012 at 9:39 ???leledumbo ???
Reading luaL_newlib documentation and lua52.pas implementation of it (it's a macro in the original C code), I see that you need a sentinel value to mark the end of the array by putting nil in the name part. I was already skeptic since the API doesn't mention the number of elements in the array as commonly required by a C function.

Sorry, I think I was too fast in reading the unit. After reading slowly, the APIs actually uses open array, not dynamic array. In this case @lib should be the same as @lib[0], but here's something interesting: I tried your code (converted as program):

uses
  lua52;
type
  TluaLReg = array of luaL_Reg;
var
  LUA: Plua_State;
  lib : array of luaL_Reg;
  //lib : PluaL_Reg;
  i: Integer;
function l_dat_upd(LUA: Plua_State): LongInt; cdecl;
begin
end;
function l_dat_clr(LUA: Plua_State): LongInt; cdecl;
begin
end;
begin
LUA := luaL_newstate();
luaL_openlibs(LUA);
SetLength(lib, 2);
//SetLength(TluaLReg(lib), 2);
lib[0].name := 'update';
lib[0].func := @l_dat_upd;
lib[1].name := 'clear';
lib[1].func := @l_dat_clr;
writeln('actual = $'+HexStr(ptruint(@lib),8));
luaL_newlib(LUA, lib);
lua_setglobal(LUA, 'data');
lua_close(LUA);
end.

inserting some writelns in lua52 unit near problematic functions and running in gdb results in:

luaL_newlib oa = $BFFFEE30
luaL_setfuncs oa = $BFFFECC0

Program received signal SIGSEGV, Segmentation fault. 0xb7e386d2 in ?? () from /lib/i386-linux-gnu/libc.so.6 (gdb) bt
#0 0xb7e386d2 in ?? () from /lib/i386-linux-gnu/libc.so.6 #1 0xb7fa2d12 in ?? () from /usr/lib/i386-linux-gnu/liblua5.2.so.0 #2 0xb7f95f09 in lua_setfield () from
/usr/lib/i386-linux-gnu/liblua5.2.so.0
#3 0xb7fa9a9a in luaL_setfuncs () from
/usr/lib/i386-linux-gnu/liblua5.2.so.0
#4 0x08066821 in LUAL_SETFUNCS (L=0x8071008, LR=..., highLR=1, NUP=0) at lua52.pas:835
#5 0x080669e4 in LUAL_NEWLIB (L=0x8071008, LR=..., highLR=1) at lua52.pas:859
#6 0x0804a3ea in main () at test.pas:31

As you can see, the pointer changes when you pass an open array into another function expecting open array, even when static array is used (I've tried that one too). So, somebody with enough compiler knowledge should analyze this. Maybe you should create a bug report.

Hi leledumbo,

I have a few questions:

1) what's the difference between open array and dynamic array? Is "array of Integer" open or dynamic?

2) How to report a bug, and I believe this maybe a bug in the fpc compiler?

Thanks

? ?, 12? 30, 2012 at 12:30 ???leledumbo ???
Sorry, I think I was too fast in reading the unit. After reading slowly, the APIs actually uses open array, not dynamic array. In this case @lib should be the same as @lib[0], but here's something interesting: I tried your code (converted as program):

uses
lua52;
type
TluaLReg = array of luaL_Reg;
var
LUA: Plua_State;
lib : array of luaL_Reg;
//lib : PluaL_Reg;
i: Integer;
function l_dat_upd(LUA: Plua_State): LongInt; cdecl;
begin
end;
function l_dat_clr(LUA: Plua_State): LongInt; cdecl;
begin
end;
begin
LUA := luaL_newstate();
luaL_openlibs(LUA);
SetLength(lib, 2);
//SetLength(TluaLReg(lib), 2);
lib[0].name := 'update';
lib[0].func := @l_dat_upd;
lib[1].name := 'clear';
lib[1].func := @l_dat_clr;
writeln('actual = $'+HexStr(ptruint(@lib),8));
luaL_newlib(LUA, lib);
lua_setglobal(LUA, 'data');
lua_close(LUA);
end.

inserting some writelns in lua52 unit near problematic functions and running in gdb results in:

luaL_newlib oa = $BFFFEE30
luaL_setfuncs oa = $BFFFECC0

Program received signal SIGSEGV, Segmentation fault. 0xb7e386d2 in ?? () from /lib/i386-linux-gnu/libc.so.6 (gdb) bt
#0 0xb7e386d2 in ?? () from /lib/i386-linux-gnu/libc.so.6 #1 0xb7fa2d12 in ?? () from /usr/lib/i386-linux-gnu/liblua5.2.so.0 #2 0xb7f95f09 in lua_setfield () from
/usr/lib/i386-linux-gnu/liblua5.2.so.0
#3 0xb7fa9a9a in luaL_setfuncs () from
/usr/lib/i386-linux-gnu/liblua5.2.so.0
#4 0x08066821 in LUAL_SETFUNCS (L=0x8071008, LR=..., highLR=1, NUP=0) at lua52.pas:835
#5 0x080669e4 in LUAL_NEWLIB (L=0x8071008, LR=..., highLR=1) at lua52.pas:859
#6 0x0804a3ea in main () at test.pas:31

As you can see, the pointer changes when you pass an open array into another function expecting open array, even when static array is used (I've tried that one too). So, somebody with enough compiler knowledge should analyze this. Maybe you should create a bug report.

1) what's the difference between open array and dynamic array? Is "array of Integer" open or dynamic?

Open arrays are parameters only, compatible with all kinds of arrays.

AFAIK the (Delphi) compiler adds hidden (implementation specific) information about the array dimension, to the accessible array base address. I'm not sure what's the base address of an array[-10..10] in this case. Try find out yourself...

DoDi

1) what's the difference between open array and dynamic array? Is "array of

Integer" open or dynamic?

Open array can only be declared as parameter type, it can't be declared as a variable type. When done in latter case, it will always be dynamic array. The following example will explain:

type
  TDynIntArray = array of Integer;
procedure p1(a: TDynIntArray); // here a is dynamic array
procedure p2(a: array of Integer); // here a is open array

a good article explaining open array:
http://rvelthuis.de/articles/articles-openarr.html so "array of Integer" can mean both, depending where and how it's declared.

2) How to report a bug, and I believe this maybe a bug in the fpc compiler?

http://wiki.lazarus.freepascal.org/How_do_I_create_a_bug_report

or just go to bugs.freepascal.org, login (create account if you don't have one) and press report issue (the fields are mostly intuitive).

Hi leledumbo,

Apparently, I've been wrong for all these years using Delphi :-S. ? I always think that open arrays are same as dynamic arrays. So,

1) from the source code of lua52.pas, how do you know the array parameter to luaL_newlib() accepts only static array but not dynamic array? ? If you don't read the source code or document of lua API?

2) If the api only accept static array but I passed dynamic array and hence it crashed, then it is my fault, not FPC compilers. ? In such case, shall I still file a bug report?

3) Finally, if I still file a bug, can I just ask the reviewer to read this mail thread as it is archived?

Thanks

? ?, 12? 30, 2012 at 7:41 ???leledumbo ???

1) what's the difference between open array and dynamic array? Is "array of

Integer" open or dynamic?

Open array can only be declared as parameter type, it can't be declared as a variable type. When done in latter case, it will always be dynamic array. The following example will explain:

type
TDynIntArray = array of Integer;
procedure p1(a: TDynIntArray); // here a is dynamic array
procedure p2(a: array of Integer); // here a is open array

a good article explaining open array:
http://rvelthuis.de/articles/articles-openarr.html so "array of Integer" can mean both, depending where and how it's declared.

2) How to report a bug, and I believe this maybe a bug in the fpc compiler?

http://wiki.lazarus.freepascal.org/How_do_I_create_a_bug_report

or just go to bugs.freepascal.org, login (create account if you don't have one) and press report issue (the fields are mostly intuitive).

1) from the source code of lua52.pas, how do you know the array parameter

to luaL_newlib() accepts only static array but not dynamic array?

No, I didn't say that. Only the pointer version is correct. The array version is wrong despite the array type passed, this is due to the pointer addresses being invalid (from one function call to another) maybe.

If you don't read the source code or document of lua API?

Lua API is consistent, it uses whatever C provides. Thus it can only accept pointer, since C can only pass that as well.

2) If the api only accept static array but I passed dynamic array and hence it crashed, then it is my fault, not FPC compilers. In such case, shall I still file a bug report?

We don't know yet, it's just my investigation. We need people with more knowledge.

3) Finally, if I still file a bug, can I just ask the reviewer to read this mail thread as it is archived?

Better copy and paste the relevant parts, so the report is concise.

If the code of that lua52.pas is the one available here: http://lua-users.org/files/wiki_insecure/lua52.pas the error is understandable if one looks at the luaL_newlib function (the array overload):

=== source begin ===

procedure luaL_newlib(L: Plua_State; lr: array of luaL_Reg);
begin
    luaL_newlibtable(L, lr);
    luaL_setfuncs(L, @lr, 0);
end;

=== source end ===

The address of the open array parameter is passed, not the array to the first element. If you change the luaL_setfuncs call to

=== source begin ===

luaL_setfuncs(L, @lr[0], 0);

=== source end ===

it should work...

Regards,
Sven

Hi Sven,

I think that's the library we are talking about... However, make that call to @lr[0] will NOT fix the problem, as I mentioned in previous emails. ? To help you see what happened, I attach the debug log (from lazarus assembly window) again.

Also, it is very strange that my programs act very differently on Linux64 and Win32. ?On windows, even the "correct" version (which runs well on command line) will cause access violation in the IDE!

Thanks!

? ?, 12? 30, 2012 at 9:34 ???Sven Barth ???

If the code of that lua52.pas is the one available here: http://lua-users.org/files/wiki_insecure/lua52.pas the error is understandable if one looks at the luaL_newlib function (the array overload):

=== source begin ===

procedure luaL_newlib(L: Plua_State; lr: array of luaL_Reg);
begin
luaL_newlibtable(L, lr);
luaL_setfuncs(L, @lr, 0);
end;

=== source end ===

The address of the open array parameter is passed, not the array to the first element. If you change the luaL_setfuncs call to

=== source begin ===

luaL_setfuncs(L, @lr[0], 0);

=== source end ===

it should work...

Regards,
Sven

Xiangrong Fang wrote

lua52.pas does not append a sentinel for you. You must do it yourself. Forgetting to append a sentinel will lead to unpredictable behavior of Lua internals.

Inside API functions in lua52.pas is always equal to because *lr* is an open array parameter.