Async Requests

[t***@gmail.com]

I'm trying to write an application that sends many (about 5000) http
GET requests without waiting for each call to finish.

I am using the following sequence:

1. InternetOpen (with INTERNET_FLAG_ASYNC flag)
2. InternetSetStatusCallback
3. InternetConnect
4. I loop 5000 times each time calling:
4.1. HttpOpenRequest
4.2. HttpSendRequest

I then wait to get INTERNET_STATUS_REQUEST_COMPLETE on my callback
function,
once i get it I call:

1. InternetQueryDataAvailable
2. InternetReadFile

in a loop, until there is no more data available.
I keep getting different errors, usually I fail on reading the internet
file, and then get stuck (wininet won't continue sending me
REQUEST_COMPLETE statuses.

Can someone please help me understand where my problem is? Maybe there
is a sample out there that does just this (send many GET requests)?

Thanks!

[Vladimir Scherbina]

Hi testgames,

You're coding a typical DOS attack - server might think that you're an
attacker and closes the connection. Getting 5k of documents is very
agressive policy!

Can you tell me what errors do you get?
--
Vladimir

Post by t***@gmail.com
I'm trying to write an application that sends many (about 5000) http
GET requests without waiting for each call to finish.
1. InternetOpen (with INTERNET_FLAG_ASYNC flag)
2. InternetSetStatusCallback
3. InternetConnect
4.1. HttpOpenRequest
4.2. HttpSendRequest
I then wait to get INTERNET_STATUS_REQUEST_COMPLETE on my callback
function,
1. InternetQueryDataAvailable
2. InternetReadFile
in a loop, until there is no more data available.
I keep getting different errors, usually I fail on reading the internet
file, and then get stuck (wininet won't continue sending me
REQUEST_COMPLETE statuses.
Can someone please help me understand where my problem is? Maybe there
is a sample out there that does just this (send many GET requests)?
Thanks!

[t***@gmail.com]

Hi Vladimir,
Thanks for your reply.


I am trying to get chunks of many files from my server (I use Range
header for my requests).

I don't think the server (IIS) is thinking this is an attack because I
created three tests:

1. multithreaded test - each thread works with WinInet in synchronous
mode - each time i create a new internet connection and connect it
2. single threaded test - I call Wininet synchronously - I create only
one connection an then many HTTP requests on this connection
3. single threaded test - I call WinInet asynchronous mode - I create
only one connection an then many HTTP requests on this connection


The asynchronous mode test acts unexpectedly:
First, if I call HttpOpenRequest/HttpSendRequest, once I receive the
INTERNET_STATUS_REQUEST_COMPLETE, the dwResult points to an invalid
handle - so I use the hInternet passed to the callback.
If I use InternetOpenUrl - the handle is a valid one.

But that is the least of my problems, other then that, whenever I get
an INTERNET_STATUS_REQUEST_COMPLETE callback, and run my data retreival
loop (InternetQueryDataAvailable, InternetReadFile), sometimes
InternetQueryDataAvailable returns a huge number (even though I query
the header and see the data is much smaller), sometimes
InternetReadFile fails with a different error code than
ERROR_IO_PENDING, sometiems it fails with ERROR_IO_PENDING, but even
though I wait for a second it keeps returning ERROR_IO_PENDING.

Other times, InternetQueryDataAvailable returns 0 bytes, even though
there is more bytes to read - in this case, if i stop looping for data,
I won't get anymore callbacks.


I'm certain that I'm doing something wrong (because MS doesn't document
wininet in async mode at all).

Could you help me understand the correct sequence I should use for my
application?

Maybe I should not call InternetQueryDataAvailable at all.
Maybe I should work with InternetReadFileEx.
Maybe I should open a new thread for retreiving data after gettting the
INTERNET_STATUS_REQUEST_COMPLETE?

So many possiblities :)

Thanks for your time

Post by Vladimir Scherbina
Hi testgames,
You're coding a typical DOS attack - server might think that you're an
attacker and closes the connection. Getting 5k of documents is very
agressive policy!
Can you tell me what errors do you get?
--
Vladimir

Post by t***@gmail.com
I'm trying to write an application that sends many (about 5000) http
GET requests without waiting for each call to finish.
1. InternetOpen (with INTERNET_FLAG_ASYNC flag)
2. InternetSetStatusCallback
3. InternetConnect
4.1. HttpOpenRequest
4.2. HttpSendRequest
I then wait to get INTERNET_STATUS_REQUEST_COMPLETE on my callback
function,
1. InternetQueryDataAvailable
2. InternetReadFile
in a loop, until there is no more data available.
I keep getting different errors, usually I fail on reading the internet
file, and then get stuck (wininet won't continue sending me
REQUEST_COMPLETE statuses.
Can someone please help me understand where my problem is? Maybe there
is a sample out there that does just this (send many GET requests)?
Thanks!

[Vladimir Scherbina]

Hi! Here is me.

When wininet works in async mode it uses statically allocated buffers (for
it's own purposes) in heap that should be properly synchronized. Leak of
documentation does not answer the question: how it is syncronized and should
one think about syncronization itself? When dealing with simular to yours
task I syncronized each call to wininet function to prevent race conditions.
Try to wrap each call to wininet to critical sections and make the tests one
more time.

[t***@gmail.com]

Hey Vladimir,

Thanks for the reply.

I use 1 Critical section to lock every call to wininet, and still, I
get access violations.
I keep getting violation in my StatusCallback...

Is there any examples out there that work with WinInet async such as I
want to?

Thanks!

Post by Vladimir Scherbina
Hi! Here is me.
When wininet works in async mode it uses statically allocated buffers (for
it's own purposes) in heap that should be properly synchronized. Leak of
documentation does not answer the question: how it is syncronized and should
one think about syncronization itself? When dealing with simular to yours
task I syncronized each call to wininet function to prevent race conditions.
Try to wrap each call to wininet to critical sections and make the tests one
more time.