I was having some problems using vanilla sockets and OpenSSL to read a few specific web pages using HTTPS. Normally I don't have any problems using HTTPS, but with a few sites my code was not working. I ending up finding the RTL unit LibCurl in one of the FPC packages and got it working without too much trouble. LibCurl is a library allowing programmers to get the functionality of the curl program without launching an external process.

If anyone is interested, here is a small bit of code to GET a page over HTTP or HTTPS using LibCurl.

Interface:

function CurlGet(const Url: string; out Data: string; UserAgent: string =
''): Boolean;

Implementation:

uses
  LibCurl;
function WriteData(Ptr: PChar; MemberSize, MemberCount: UIntPtr; var Data:
string): UIntPtr; cdecl;
var
  S: string;
begin
  SetString(S, Ptr, MemberSize * MemberCount);
  Data := Data + S;
  Result := MemberSize * MemberCount;
end;
function CurlGet(const Url: string; out Data: string; UserAgent: string =
''): Boolean;
var
  Curl: PCURL;
begin
  Data := '';
  Result := False;
  if Url = '' then
    Exit;
  Curl := curl_easy_init();
  if Curl = nil then
    Exit;
  try
    curl_easy_setopt(curl, CURLOPT_URL, [PChar(Url)]);
    if UserAgent  '' then
      curl_easy_setopt(curl, CURLOPT_USERAGENT, [PChar(UserAgent)]);
    curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, );
    curl_easy_setopt(curl, CURLOPT_WRITEDATA, );
    curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, [0]);
    curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, [0]);
    Result := curl_easy_perform(curl) = CURLE_OK;
  finally
    curl_easy_cleanup(Curl);
  end;
end;

At first sight, this does not differ from the teststream example already in FPC ?

Michael.

Michael,

You are right, they are very similar. I didn't know of the existence of the example to which you referred. Thank you for pointing that file out. I searched for it just now and took a look. One difference is that my example ignores certificates problems, but this difference is important in my tests as without those lines the HTTPS sites I am retrieving will mysteriously fail. Another is that I am concatenating to a string passed by reference in the curl `write function`.

For the benefit of others, there are a few problems with the FPC example to note.

1) The callback CURLOPT_WRITEFUNCTION should return the number of bytes handled.
2) You should check the result of curl_easy_perform to determine if the curl perform operation (HTTP GET) was successful.

It does that ?

  Result:=TStream(Data).Write(Ptr^,Size*nmemb);

Write returns the number of bytes written.

2) You should check the result of curl_easy_perform to determine if the curl perform operation (HTTP GET) was successful.

I added a check.

Michael.

Does that mean curl is always waiting until the whole operation (download) is complete?

Is it possible to do it in a nonblocked way? For example with a continuous nonblocked loop call and a flag when the operation is finished?

Am Dienstag, 28. September 2021, 02:01:00 CET schrieb Anthony Walter via fpc- pascal:

Am Sonntag, 5. Dezember 2021, 20:38:30 CET schrieb Rainer Stratmann via fpc- pascal:

Found it already. It seems that a nonblocking operation mode is possible!

https://everything.curl.dev/libcurl/drive
https://everything.curl.dev/libcurl/drive/multi