Hello,

So I advanced my CGI using HTTP GET so far, but I'd like to migrate to HTTP POST. But I always get this error message in the apache logs:

Premature end of script headers: openpoker3.cgi, referer: http://localhost:8080/cgi-bin/openpoker3.cgi?module=MainPage

The request doesn't even get to my handler in the CGI =(

Here is my JavaScript which initiates the communication:

  SendSyncRequestViaGet = function(RequestURL, RequestContent)
  {
    var request = new XMLHttpRequest();
    request.open('GET', RequestURL+"&data="+RequestContent, false);
    request.setRequestHeader('Content-Type', 'text/html');
    request.send("");
    //debugDiv.innerHTML = debugDiv.innerHTML + "after send<br>";
    return request.responseText;
  }
  SendSyncRequestViaPost = function(RequestURL, RequestContent)
  {
    var request = new XMLHttpRequest();
    request.open('POST', RequestURL, false);
    request.setRequestHeader('Content-Type', 'text/html');
    request.setRequestHeader("Content-length", RequestContent.length);
    request.setRequestHeader("Connection", "close");
    debugDiv.innerHTML = debugDiv.innerHTML + "before send<br>";
    request.send(RequestContent);
    debugDiv.innerHTML = debugDiv.innerHTML + "after send<br>";
    return request.responseText;
  }
  var lobbyTimer;
  function lobbyTimerCallback()
  {
    // Load all Lobby table data
    var tableInnerHTML =
SendSyncRequestViaPost("openpoker3.cgi?module=PokerServer","00_FFFFFFFFFFFFFFFF_0000002E_00000000_");
 ...

If I just use SendSyncRequestViaGet then everything goes fine. If I use Post, I get apache errors and that previously mentioned message in the Apache log.

The JavaScript executes correctly as I already verified, and the error is not in my Handler code, because it doesn't even get there.

Any ideas? I am sure that the JavaScript code is correct, because if I substitute my program main with this:

var
  Strings: TStringList;
  i: Integer;
begin
  Strings:= TStringList.Create;
  Strings.LoadFromFile(ExtractFilePath(ParamStr(0)) + 'mainpage.html');
  WriteLn('Content-Type: text/html');
  WriteLn('');
  for i := 0 to Strings.Count-1 do
    WriteLn(Strings.Strings[i]);
  try
  OPDebugLn('[AAA]');
  Application.Initialize;
  OPDebugLn('[B]');
  Application.Run;
  OPDebugLn('[C]');
  except
    OPDebugLn('[D]');
  end;
end.

And then the POST works fine! But of course that I destroyed fcl-web then...

Updating to a recent 2.5 didn't help. I tryed to rebuild fcl-web with cgidebug but I couldn't find any trace of an output log.

There is no exception. It just runs fine and doesn't call my module and does not print any error messages, does not print anything after the headers actually =(

Any ideas are welcome...

thanks,

Where is the action ? If you specify a module, then you must specify an action as well.

What you can try to do is to create an OnShowException handler for the application, and see what that prints. (normally you should get a normal error page in case of an error, not the 'premature end of script headers')

Michael.

Michael Van Canneyt wrote

Where is the action ? If you specify a module, then you must specify an action as well.

actions are not necessary. In my module there is a property DefActionWhenUnknown=True (it is like this by default).

Also if this was the problem I would expect that it fails when using GET too. I tryed adding an action to see if it helps, but the only thing that changed is that now I get this in the response to the POST:

exception at 0043004B: Unsupported content type: "text/html; charset=UTF-8".

I am now using this call:

    var tableInnerHTML =
SendSyncRequestViaPost("openpoker3.cgi?module=PokerServer&action=TFPWebAction0","00_FFFFFFFFFFFFFFFF_0000002E_00000000_");

What specifies the name of the action? This seams to be missing from the wiki. I just supposed it is the "Name" property.

What you can try to do is to create an OnShowException handler for the application, and see what that prints. (normally you should get a normal error page in case of an error, not the 'premature end of script headers')

It seams that this didn't help either. The OnShowException was never called.

Ok, the exception was a great addition =)

It corners the error here:

procedure TRequest.InitPostVars;
Var
  M  : TCapacityStream;
  Cl : Integer;
  B  : Byte;
  CT : String;
begin
{$ifdef CGIDEBUG}
  SendMethodEnter('InitPostVars');
{$endif}
  CL:=ContentLength;
  M:=TCapacityStream.Create;
  Try
    if CL0 then
      begin
      M.Capacity:=Cl;
      M.WriteBuffer(Content[1], Cl);
      end;
    M.Position:=0;
    CT:=ContentType;
    if Pos('MULTIPART/FORM-DATA',Uppercase(CT))0 then
      ProcessMultiPart(M,CT, ContentFields)
    else if Pos('APPLICATION/X-WWW-FORM-URLENCODED',Uppercase(CT))0 then
      ProcessUrlEncoded(M, ContentFields)
    else if CL0 then
      begin
{$ifdef CGIDEBUG}
      SendDebug('InitPostVars: unsupported content type:'+CT);
{$endif}
      Raise Exception.CreateFmt(SErrUnsupportedContentType,[CT]);
<------- error here

If I remove the line from JavaScript which sets the content type I just get:

exception at 0043004B: Unsupported content type: "text/plain; charset=UTF-8".

=/

Ok, I figured part of it out. I can get past the exception if I use 2.5.1

And the action was necessary, which is really strange because it worked nicely without it when using GET

But now my problem is that I cannot get my POST data. I tryed this so far:

lData := ARequest.ContentFields.Text;

but it is empty.

Are you sure your form data is passed properly with POST? You should check the communication in your browser using FireBug or the debug tool in your browser if you use something other than FF (Opera, Chromium, Chrome has also built in debugger with pressing F12), and check if the query string parameters (module, action) as well as your POST fields (form data) are really passed the same way to the server with both GET and POST.
The only difference should be the Request Method . I have a feeling that your form data does not come at all from the javascript caller for some reason.

AB

There is no form and no form data here. I want to pass raw data via post. And yes, I checked that it is sent correctly.

Felipe Monteiro de Carvalho wrote

There is no form and no form data here. I want to pass raw data via post. And yes, I checked that it is sent correctly.

Then your Content-Type header is neither "MULTIPART/FORM-DATA" nor "APPLICATION/X-WWW-FORM-URLENCODED" when you use POST. It must be one of these two.

http://www.w3.org/TR/html4/interact/forms.html#h-17.13.4

AB

If you'd look at the current version of TRequest.InitPostVars, you'd know why it is empty ;)

=== source begin ===

procedure TRequest.InitPostVars;
Var
   M  : TCapacityStream;
   Cl : Integer;
   CT : String;
begin
{$ifdef CGIDEBUG}
   SendMethodEnter('InitPostVars');
{$endif}
   CL:=ContentLength;
   if CL0 then
     begin
     M:=TCapacityStream.Create;
     Try
       if CL0 then
         begin
         M.Capacity:=Cl;
         M.WriteBuffer(Content[1], Cl);
         end;
       M.Position:=0;
       CT:=ContentType;
       if Pos('MULTIPART/FORM-DATA',Uppercase(CT))0 then
         ProcessMultiPart(M,CT, ContentFields)
       else if Pos('APPLICATION/X-WWW-FORM-URLENCODED',Uppercase(CT))0
then
         ProcessUrlEncoded(M, ContentFields)
     finally
      M.Free;
     end;
     end;
{$ifdef CGIDEBUG}
   SendMethodExit('InitPostVars');
{$endif}
         end;

=== source end ===

So it seems that you MUST set the contenttype either to MULTIPART/FORM-DATA or APPLICATION/X-WWW-FORM-URLENCODED.

Maybe TRequest should be extended by something like an "unknown content type handler"...

Regards,
Sven

[snip]

Yes, this is as per the HTML specs.

Maybe TRequest should be extended by something like an "unknown content type handler"...

This can of course be done, the question is what this handler should do ?

Michael.

Michael Van Canneyt wrote

> Maybe TRequest should be extended by something like an "unknown > content type handler"... This can of course be done, the question is what this handler should do ?

Handler gets ContentType, transmitted data (the TMemoryStream) and the ContentFields field as arguments and processes them accordingly.

This is the theory. In practice this of course can be a problem if the transmitted data is e.g. binary data, a XML or something else that can't be stuffed in a TStrings... But on the other hand the handler could at least do something else with that data (e.g. put it into a field in the class the handler belongs to).

Regards,
Sven

Michael Van Canneyt wrote

> So it seems that you MUST set the contenttype either to > MULTIPART/FORM-DATA or APPLICATION/X-WWW-FORM-URLENCODED. Yes, this is as per the HTML specs.

They actually prohibit sending arbitrary data? oh, boy, these people like rules and inneficiency =D But there must be something missing here, because file upload is possible, and there you send arbitrary data.

And also it seams pretty safe to ignore such restriction, only fcl-web blocked me here. I doubt that the browser will try to syntax check my data to see if it is XML and block me.

This can of course be done, the question is what this handler should do ?

You mean the user application? Whatever it wants? Receiving a TStringList or a TStream I can just process it. It expects a particular format and reacts to it, or gives an error if the format is worng. I already have a data structure ready which I adapted, but using GET is rather inneficient, you have to encode it, so I thought that I should just pass to POST to be able to send more arbitrary data. HTTP per se should pose no restrictions to what I am sending via POST.

2011/11/1 Felipe Monteiro de Carvalho :

You can send arbitrary data using MULTIPART/FORM-DATA

Vincent

Hello,

Ok, it seams that I found a workaround. It works correctly if I use this:

  lData := ARequest.Content;

instead of

 lData := ARequest.ContentFields.Text;

It seams that the string is filled correctly, even if the TStringList isn't filled.

I think this could be improved by adding this code:

procedure TRequest.InitPostVars;
Var
...
  DataStr: string;
begin
...
    else if Pos('APPLICATION/X-WWW-FORM-URLENCODED',Uppercase(CT))0 then
      ProcessUrlEncoded(M, ContentFields)
    // For other data just load it to a TStringList and let the
    // user application process it
    else
    begin
      SetLength(DataStr, M.Size);
      M.ReadBuffer(DataStr[1], M.Size);
      ContentFields.Text := DataStr;
    end;

But I was not able to test it yet, because my build script (located in C:\Programas\lazarus31\fpc\2.5.1\source\packages):

set path=C:\Programas\lazarus31\fpc\2.5.1\bin\i386-win32 make fcl-web
pause

Fails with:

C:\Programas\lazarus31\fpc\2.5.1\source\packages>set path=C:\Programas\lazarus31 \fpc\2.5.1\bin\i386-win32

C:\Programas\lazarus31\fpc\2.5.1\source\packages>make fcl-web make -C fcl-web all
make[1]: Entering directory `C:/Programas/lazarus31/fpc/2.5.1/source/packages/fc l-web'
C:/Programas/lazarus31/fpc/2.5.1/bin/i386-win32/ppc386.exe fpmake.pp -n -Fu../.. /rtl -Fu../../packages/hash/units/i386-win32 -Fu../../packages/paszlib -Fu../../ packages/fcl-process -Fu../../packages/fpmkunit
Fatal: Can't find unit system used by fpmake
Fatal: Compilation aborted
make[1]: *** [fpmake] Error 1
make[1]: Leaving directory `C:/Programas/lazarus31/fpc/2.5.1/source/packages/fcl -web'
make: *** [fcl-web] Error 2

Even while a nearly identical script works for FPC 2.4.2 for building fcl-web.

Vincent Snijders wrote

You can send arbitrary data using MULTIPART/FORM-DATA

Indeed, but I'd rather specify my own mime-type and not use any pre-defined format for packaging my data. This kind of thing just adds overhead and slowness.

Note the "per the HTML specs". HTTP specs aren't that strict.

HTTP per se should pose no restrictions to what I am sending via POST.

It doesn't. Application/octet-stream is a common MIME used to send binary data.

Ludo

Ludo Brands wrote

Note the "per the HTML specs". HTTP specs aren't that strict.

But what exaxtly do these specs ban? Because I am sending via XMLHttpRequest in JavaScript, and there you can specify any mime-type that you want. You can send any data and I haven't seen any restriction related to the mime-type so far. Probably this restrictions don't apply to JavaScript but only to form data and things like that (purely in HTML)?

Not to mention that a CGI application can also communicate with native clients, not only HTML pages, and the native client is not subject to HTML rules.

Yes. The HTML spec defines the use of these 2 MIME types for a user agent uploading a form. XMLHttpRequest, as the name indicates, isn't limited to HTML.

Not to mention that a CGI application can also communicate with native clients, not only HTML pages, and the native client is not subject to HTML rules.

There are plenty of applications using http as a communication channel and that have nothing to do with html.

Ludo

Actually this code is wrong, for it to be proper it would need to first check if the mime-type is "text*", then it would need to read the encoding, convert between encodings ...

But I am fully satisfied with just Content, because my message is in UTF-8 and my application also uses UTF-8, so adding such conversions would only make things slower and probably wrong too (by making wrong conversions).

I will simply document all of this stuff in the wiki.

All this is correct, but the primary function of HTTP remains the transport of web-pages. fcl-web was/is geared towards that.

That said, it's of course not so that fcl-web should restrict itself to HTML. I will look into having an event or so to handle unknown encodings.

For the time being, you can do the following:

1. create a descendent of TRequest (actually TCGIRequest) and override InitPostVars.
2. Then create a TCGIHandler descendent to override TCgiHandler.CreateRequest to instantiate your TCGIRequest descendant
3. Make a descendent of TCustomCGIApplication that overrides InitializeWebHandler which instantiates your TCGIHandler descendent. Use that descendent as your application object.

A bit cumbersome, but it gets the job done. (I had to do it myself on one particular occasion).

I'll look into making it a bit easier to do. (including the event to handle unknown encodings)

Michael.