I need to create some kind of bot for automatic testing against our web app. The following app target real online web application:

{$mode objfpc}{$H+}
uses
  {$ifdef unix}
  cthreads,
  {$endif}
  Classes,SysUtils,fphttpclient;
procedure Log(http: TFPHTTPClient);
begin
  WriteLn('BEGIN
**********************************************************');
  WriteLn(http.RequestHeaders.Text);
  WriteLn(http.ResponseStatusCode);
  WriteLn(http.ResponseHeaders.Text);
  WriteLn('END **********************************************************');
end;
var
  http: TFPHTTPClient;
  formdata: TStrings;
begin
  http := TFPHTTPClient.Create(nil);
  try
    // initial GET request to initialize cookies
    http.Get('http://lelelagi.mysirclo.com');
    Log(http);
    // POST: add item to cart
    formdata := TStringList.Create;
    formdata.Values['cmd'] := 'add';
    formdata.Values['item_id'] := 'montauk-tote';
    formdata.Values['member_email'] := ' ';
    http.FormPost('http://lelelagi.mysirclo.com/cart',formdata); // #1
    Log(http);
    // POST: set country, city and shipping
    formdata.Clear;
    formdata.Values['cmd'] := 'shipping_country';
    formdata.Values['shipping_value'] := 'ID';
    http.FormPost('http://lelelagi.mysirclo.com/cart',formdata); // #2
    Log(http);
    formdata.Clear;
    formdata.Values['cmd'] := 'shipping_city';
    formdata.Values['shipping_value'] := 'Kota Depok - Depok';
    http.FormPost('http://lelelagi.mysirclo.com/cart',formdata);
    Log(http);
    formdata.Clear;
    formdata.Values['cmd'] := 'shipping';
    formdata.Values['shipping_value'] := 'JNE REG';
    http.FormPost('http://lelelagi.mysirclo.com/cart',formdata);
    Log(http);
    // POST: place order
    formdata.Clear;
    formdata.Values['first_name'] := 'Test Name';
    formdata.Values['phone'] := '088712342345';
    formdata.Values['address_line1'] := 'Certain Street';
    formdata.Values['country'] := 'ID';
    formdata.Values['state'] := 'DKI Jakarta';
    formdata.Values['city'] := 'DKI Jakarta - Jakarta';
    formdata.Values['postal_code'] := '12424';
    formdata.Values['email'] := '';
    formdata.Values['delivery_first_name'] := 'Test Name';
    formdata.Values['delivery_phone'] := '088712342345';
    formdata.Values['delivery_address_line1'] := 'Certain Street';
    formdata.Values['delivery_country'] := 'ID';
    formdata.Values['delivery_state'] := 'DKI Jakarta';
    formdata.Values['delivery_city'] := 'DKI Jakarta - Jakarta';
    formdata.Values['delivery_postal_code'] := '12424';
    formdata.Values['delivery_email'] := '';
    formdata.Values['shipping_value'] := 'JNE REG';
    formdata.Values['payment_method'] := 'bank-transfer';
    formdata.Values['message'] := '';
    formdata.Values['agreement'] := 'on';
http.FormPost('http://lelelagi.mysirclo.com/cart/place_order/guest',formdata);
    Log(http);
  finally
    http.Free;
  end;
end.

The app always dies with "EHTTPClient: Error reading data from socket" after the first FormPost, any of them (e.g. if you comment #1, it will die after #2). Did I miss something? Is this the correct way to make such a simulation?

Yes it is.
If the problem disappears after you do a Free and Create of the http class, then it means something is left hanging after the call. In that case I would need to check what is happening.

If the problem persists, then it means there is some mechanism on the server that is causing this.

Michael.

If the problem disappears after you do a Free and Create of the http class,

then it means something is left hanging after the call.

If I Free and re-Create the class, how can I ensure the request header of subsequent requests contain the same cookies? I need to maintain 2 cookie values (which is given from first GET request in the response header) throughout all requests.

This is a sample output of the program:
BEGIN ********************************************************** [Request Headers]

[Response Status Code]
200
[Response Headers]
Server: nginx
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Vary: Accept-Encoding
Set-Cookie: sirclo_session_id=71e58d13b4124c71a54c0eb6438b8007; path=/ Set-Cookie: cart_code=f654373c175ac6e86da22c7400a146f0; expires=Fri, 29-Apr-2016 16:40:05 GMT; path=/
Pragma: no-cache
Date: Wed, 29 Apr 2015 16:40:05 GMT
X-Page-Speed: ngx_pagespeed 1.8
Cache-Control: max-age=0, no-cache, no-store
Connection: Keep-alive
Via: 1.1 ID-0002262071501332 uproxy-3

END ********************************************************** BEGIN ********************************************************** [Request Headers]
Content-Type: application/x-www-form-urlencoded
Content-Length: 79

[Response Status Code]
302
[Response Headers]
Server: nginx
Date: Wed, 29 Apr 2015 16:40:07 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Set-Cookie: sirclo_session_id=0c9f312bc97c0332105c2accc0feede5; path=/ Location: http://lelelagi.mysirclo.com/cart
Cache-Control: no-cache, no-store, must-revalidate Pragma: no-cache
Expires: 0
Connection: Keep-alive
Via: 1.1 ID-0002262071501332 uproxy-6

END ********************************************************** An unhandled exception occurred at $000000000046DCFB: EHTTPClient: Error reading data from socket
$000000000046DCFB
$000000000046DA9B
$000000000046F344
$000000000046FAA8
$000000000047027E

Each request log is delimited by BEGIN-END line. As you can see, in the first request, the response indicates 2 Set-Cookie:, which must be preserved for subsequent requests. But in the next request, the request header contains no Cookie: line, which is why the server issues another Set-Cookie:, though this time it's only sirclo_session_id issued, the cart_code is not issued again. Still, this is wrong as both must appear instead of just one of them.

P.S.:
1. I guess RequestHeaders.Text doesn't contain Cookie: line, does it?
2. 302 response status code is expected, should not be a problem

Just extract the and save the value of the Set-Cookie header ? I always store the cookies explicitly.

The class doesn't automatically handle cookies.
It doesn't store and retrieve the cookies.
Specifically, if it gets a set-cookie, it will not apply it to the next request.

We can think of a "TCookieJar" class implementation as an option, but for now you must handle cookies yourself.

It must

2. 302 response status code is expected, should not be a problem

I have plans to handle this in TFPHTTPClient.

Michael.

Just extract the and save the value of the Set-Cookie header ? I always store the cookies explicitly.

If that's so, the current Cookies property implementation seems broken. I try to debug and indeed I found that each Set-Cookie line overrides the previous if there are more than one Set-Cookie (procedure DoCookies, FCookies gets cleared at the start).

I try to log Cookies.Text, and for instance:

Set-Cookie: cart_code=f654373c175ac6e86da22c7400a146f0; expires=Fri, 29-Apr-2016 16:40:05 GMT; path=/

results in:
cart_code=f654373c175ac6e86da22c7400a146f0
expires=Fri, 29-Apr-2016 16:40:05 GMT
path=/

which is not right since expires and path are cookie attributes instead of cookie value (I even wonder if TStrings is the right choice to store cookies). Guess I'll have to use Free + re-Create + search and parse Set-Cookie manually then.

We can think of a "TCookieJar" class implementation as an option

I guess so, but it's better integrated in fphttpclient so it can act as a true browser session instead of just a one time request caller.

It must

It actually does, but actually one can't prove that by inspecting RequestHeaders.Text BEFORE doing http method, because Cookies gets merged there DURING the http method.

I have plans to handle this in TFPHTTPClient

I didn't ask if 302 is a problem, it's a statement that 302 is expected and is OK. Does what you say indicates that probably the cause of the stuck error?

I will check this.

I will check this. Sounds like something in the cookie parsing is wrong.

That I will definitely not do. I prefer the component to be stateless as much as possible. Which is why I would separate out the hypothetical TCookieJar.

If you want that you'll need to make a separate class which combines the two into one class. It you can then also handle saving cookies for later retrieval, which will be pretty much application specific anyway.

I meant that I want to have automated handling of 302 (based on a property to enable/disable this, obviously);
i.e. resubmit the request at the 302 location.

Michael.