Hello Lazarus Community,

I am implementing asynchronous file download using WinINet's STDCALL callback. However, when the callback function recursively calls itself, it generates a crash around fpc_popaddrstack. I hypothesize that Pascal does not understand how to handle callbacks that use the stdcall convention (which I must use since the callback interfaces with the Win32 API and is itself set via the Win32 WinINet procedure InternetSetStatusCallback).

I was wondering if anyone might graciously offer guidance on how to implement asynchronous download to file using the Win32 API or associated libraries only. Note that I have tried WinHTTP and a different implementation of WinINet using InternetConnect/InternetOpenRequest/InternetSendRequest/InternetReadFile). In the previous two implementations I have encountered the same segmentation fault around fpc_popaddrstack. Based on my experience with the below code and other STDCALL implementations, I do not think that Free Pascal can handle the case where a STDCALL function interrupts itself recursively. In the STDCALL convention, the callee is responsible for cleaning up the stack upon function exit. However, because the function in question (HttpCallback) has been interrupted by a second call to it from the Windows API, it has not had a chance to clean up the stack (because it has not had a chance to exit and thereby execute the epilogue code that cleans up the stack). The condition that causes the callback to interrupt itself is when InternetReadFile errors out with ERROR_IO_PENDING. In this circumstance, InternetReadFile also sends back an INTERNET_STATUS_REQUEST_COMPLETE message which interrupts the currently executing HttpCallback function.

To state the exact circumstance of the crash: when not stepping through, InternetReadFile below errors out frequently with ERROR_IO_PENDING. This is fine except that InternetReadFile ALSO recursively triggers the callback, which after the second recursive call causes a crash.

My current code is as follows. I've added inline comments around the parts that seem to crash. Thank you very much for any asssistance you can provide.

<PRE>

	PROCEDURE HttpCallback(hInternet: HINTERNET;
													dwContext: DWORD_PTR;
													dwInternetStatus: DWORD;
													lpvStatusInformation: LPVOID;
													dwStatusInformationLength: DWORD); STDCALL;
		TYPE
			PMAIN_CONTEXT = ^MAIN_CONTEXT;
			PAPP_CONTEXT = ^APP_CONTEXT;
		VAR
			dwError, dwBytes, dwBytesRead, dwStructType: DWORD ;
			pDwStructType: DWORD_PTR;
			pMainContext: PMAIN_CONTEXT;
			pAppContext: PTR;
			bQuit: BOOLEAN;
			mainContext: MAIN_CONTEXT;
			appContext: APP_CONTEXT;
			handlePtr: LongWord;
			internetAsyncResult1: INTERNET_ASYNC_RESULT;
			responseString: Str;
			statusCode, length, lpdwIndex: DWORD;
			success: BOOLEAN;
			breakpoint: INT;
			hasError: BOOLEAN;
	BEGIN
		dwBytes := 0;
		bQuit := FALSE;
		dwStructType := Ptr(dwContext)^;
		IF dwStructType = STRUCT_TYPE_MAIN_CONTEXT THEN
		BEGIN
			mainContext := PMAIN_CONTEXT(dwContext)^
		END
		ELSE
		BEGIN
			appContext := PAPP_CONTEXT(dwContext)^;
		END;
		CASE dwInternetStatus OF
			INTERNET_STATUS_HANDLE_CREATED:
			BEGIN
				{Verify we've a valid pointer}
				IF lpvStatusInformation  NIL THEN
				BEGIN
					internetAsyncResult1 := INTERNET_ASYNC_RESULT(Pointer(lpvStatusInformation)^);
					IF (dwStructType = STRUCT_TYPE_MAIN_CONTEXT) THEN
					BEGIN
						IF mainContext.hConnect = NIL THEN
						BEGIN
							mainContext.hConnect := Ptr(internetAsyncResult1.dwResult);
							{ This saves the hConnect handle to the global gNetworkStream.mainContext variable. }
							PMAIN_CONTEXT(dwContext)^.hConnect := Ptr(internetAsyncResult1.dwResult);
						END;
					END;
				END;
      END; { end CASE dwInternetStatus of INTERNET_STATUS_HANDLE_CREATED }
      INTERNET_STATUS_REQUEST_COMPLETE:
      BEGIN
      	internetAsyncResult1 := INTERNET_ASYNC_RESULT(Pointer(lpvStatusInformation)^);
				IF internetAsyncResult1.dwError = ERROR_SUCCESS THEN
				BEGIN
					ZeroMemory(@gABuffer, sizeof(gABuffer));
					dwError := ERROR_SUCCESS;
					dwBytes := sizeof(gABuffer);
					dwBytesRead := 0;
					success := InternetReadFile(gNetworkStream.mainContext.hConnect,
																			@gABuffer,
																			dwBytes,
																			dwBytesRead);
					IF success THEN
					BEGIN
						gNetworkStream.theOutputFile.PosWriteFile(@gABuffer,
								                                kTFileAtMark,
				                                        dwBytesRead, { nBytes: LONGINT; }
								                                TRUE, { showIOErrAlert: BOOLEAN; }
								                                hasError);
						WHILE success AND (dwBytesRead > 0) AND (NOT hasError) DO
						BEGIN
							{ When not stepping through, InternetReadFile below errors out frequently with
							  ERROR_IO_PENDING.  This is fine except that InternetReadFile ALSO recursively
							  triggers the callback, which after the second recursive call causes a crash }
							success := InternetReadFile(gNetworkStream.mainContext.hConnect,
																			@gABuffer,
																			dwBytes,
																			dwBytesRead);
							IF success THEN
							BEGIN
					  		gNetworkStream.theOutputFile.PosWriteFile(@gABuffer,
										                                kTFileAtMark,
						                                        dwBytesRead, { nBytes: LONGINT; }
										                                TRUE, { showIOErrAlert: BOOLEAN; }
										                                hasError);
							END
							ELSE
							BEGIN
								dwError := GetLastError;
								breakpoint := 2;
								IF dwError = ERROR_IO_PENDING THEN
								BEGIN
									{ breakpoint to make the code stop here so we can verify that dwError is ERROR_IO_PENDING }
									breakpoint := 3;
								END;
							END;
						END;
						breakpoint := 3;
						{ the below case (bytes read = 0 and no dwError) means the file has finished downloading }
						IF (dwBytesRead = 0) AND (dwError = ERROR_SUCCESS) THEN
						BEGIN
							breakpoint := 4;
						END;
					END; { end first IF success }
				END; { END IF internetAsyncResult1.dwError = ERROR_SUCCESS }
			END; { end CASE dwInternetStatus of INTERNET_STATUS_REQUEST_COMPLETE }
		END; { end CASE dwInternetStatus }
	END;
	//snippet of code from the program that sets up the download session and callback
	PROCEDURE DownloadURL (theURLString: Str; { Helper function, use DownloadURLTo___ instead. }
							outputStream: TMemoryStream;
							toFile: TFile;
							outputFile, name, pass: Str);
		CONST
			kPOSTTypeGET = 'GET';
			kPOSTTypePOST = 'POST';
			kPOSTTypeHEAD = 'HEAD';
		VAR
      thePOSTType: Str;
	  hasError: BOOLEAN;
	  dwError 				 : DWORD;
      dwFlags: DWORD;
      cbStatusCode: DWORD;
      lpdwIndex: DWORD;
      status: Integer;
      acceptTypesArray: ARRAY[0..1] OF Str;
	BEGIN (* DownloadURL *)
		gTheBytesRead := 0;
    hasError := FALSE;
	acceptTypesArray[0] := '*/*';
  	acceptTypesArray[1] := '';
    gNetworkStream.hRequest := Nil;
	dwError := ERROR_SUCCESS;
	gTheStream := outputStream;
    thePOSTType := kPOSTTypeGET;
    { Create Session handle and specify Async Mode }
    gNetworkStream.mainContext.hSession := InternetOpen('WinInet HTTP Async Session', { User Agent }
														dwOpenType, { Preconfig or Proxy }
														g_proxy,  { g_proxy name }
														NIL, { g_proxy bypass, do not bypass any address }
														INTERNET_FLAG_ASYNC); { 0 for Synchronous }
    IF gNetworkStream.mainContext.hSession = Nil THEN
	BEGIN
	    dwError := GetLastError();
	    GOTO quit;
	END;
    { Set the dwStatus callback for the handle to the Callback function }
    httpCallbackPtr := InternetSetStatusCallback(gNetworkStream.mainContext.hSession,
													INTERNET_STATUS_CALLBACK( HttpCallBack ));
    IF httpCallbackPtr = INTERNET_INVALID_STATUS_CALLBACK THEN
    BEGIN
      dwError := GetLastError();
      GOTO quit;
    END;
    IF CrackHostAndPath(theURLString, g_hostName, g_resource)  ERROR_SUCCESS THEN
    BEGIN
			dwError := GetLastError();
			GOTO quit;
    END;
    IF toFile  NIL THEN
	BEGIN
		OpenTFile(toFile, hasError);
		IF NOT hasError THEN
			SetFilePosition(toFile, 0, fsFromStart, hasError);
		gNetworkStream.theOutputFile := toFile;
	END;
    { Create Connection handle and provide context for async operations }
	InternetOpenUrl(gNetworkStream.mainContext.hSession,
						PCHAR(theURLString),
						NIL, { do not provide additional Headers }
						0, { dwHeadersLength }
						INTERNET_FLAG_RELOAD OR INTERNET_FLAG_PRAGMA_NOCACHE OR INTERNET_FLAG_NO_CACHE_WRITE,
						DWORD_PTR(@gNetworkStream.mainContext)); {Provide the context to be used during the callbacks}

</PRE>

Recursive asynchronous calls deserve special handling. They should be avoided in the first place, nowadays *threads* can be used instead.

This IMO is not a language problem, but misuse of asynchronous processing.

I was wondering if anyone might graciously offer guidance on how to implement asynchronous download to file using the Win32 API or associated libraries only.

I'd suggest to move the download into an thread, and use synchronous processing there.

DoDi

conditio
n that causes the callback to interrupt itself is when InternetReadFile errors out with ERROR_IO_PENDING. In this circumstance, InternetReadFile also sends back an INTERNET_STATUS_REQUEST_COMPLETE message which interrupts the currently executing HttpCallback function.

You are trying to read the complete HTTP reply inside the callback which will not work. ERROR_IO_PENDING just means that the data for the next buffer is not there yet. The callback function is called for every buffer you receive from the server and you should have only one InternetReadFile per INTERNET_STATUS_REQUEST_COMPLETE. So when the HTTP reply takes many buffers you will get many callbacks and in every callback you are supposed to read the buffer received with one single InternetReadFile,

Take a look at this example on how to set up async processing: http://msdn.microsoft.com/en-us/library/windows/desktop/cc185684(v=vs.85).aspx. You'll notice the AcquireRequestHandle function that uses a EnterCriticalSection to lock the context. In your code you are not locking anything and your "recursive" entry in the callback is just using the same context which is the probable cause of your crash.

STDCALL callbacks are used commonly in Freepascal and are not a problem at all.

If you just want to download an url to file the easy way using the windows framework take a look at this post
http://www.lazarus.freepascal.org/index.php?topic=18506.15#msg105226

Ludo

Ludo, Hans-Peter,

Many thanks for your insights.

Ludo, I am reviewing the msdn code example you posted; I have read quite a few but don't remember seeing that one.

Hans-Peter, our project has so far been able to avoid the additional complexity posed by threads. It would be with reluctance that we would turn to threading; however, if experimenting with Ludo's post content (see below) does not work, then would will use threads. Might you know of an example implementation or code for asynchronous HTTP and/or FTP downloads?

Danke,
Matthew

In principle you could do it as follows:

=== example begin ===

type
   TDownloader = class(TThread)
   public type
     TCallback = procedure(aSender: TDownloader; aData: TStream) of object;
   private
     fCallback: TCallback;
     fUrl: String;
   protected
     procedure Execute; override;
   public
     constructor Create(const aUrl: String; aCallback: TCallback);
   end;
constructor TDownloader.Create(const aUrl: String; aCallback: TCallback);
begin
   // check that aCallback and aUrl are given and set up local variables
   // ...
   FreeOnTerminate := True;
   inherited Create(False);
end;
procedure TDownloader.Execute;
var
   ms: TMemoryStream;
begin
   ms := TMemoryStream.Create;
   try
     // here you use what ever synchronous API you want and store it's
result in ms
     // when all data is received
     fCallback(Self, ms);
   finally
     ms.Free;
   end;
end;
// somewhere else
TDownloader.Create('http://example.org/whatever', @MyCallback);
// fire and forget

=== example end ===

Note: if you want to be able to cancel the download (which could be done by calling the thread's "Terminate" method which sets "Terminated" to "True") you should not set "FreeOnTerminate" to "True", check for "Terminated" in the thread's "Execute" method and capture the instance returned by "TDownloader.Create".

Regards,
Sven

In an thread you implement *synchronous* downloads, sample code should be available. The entire thread then works asynchronously to the main thread, until it has finished its task (terminates).

DoDi