InternetReadFile much slower than IE

[meanfox]

Hi,

When I use IE to download a test file it takes about 3 seconds (non-
cached).
My own WinInet code using InternetReadFile takes about 7 seconds (also
non-cached).

This is repeatable. It is a 644K file. This is on XP SP2 with IE7.

Can anyone explain this and tell me how to speed up my WinInet
download? I thought IE used WinInet?

I have experimented with a few things that made no difference:

* Running synchronously vs running in a thread
* Downloading in 1 call to InternetReadFile vs. downloading many 1K
chunks

Please advise!

Tom

[Scherbina Vladimir]

IE7 still uses wininet. Can you show us a snippet of your code?
--
--Vladimir, Windows SDK MVP

Post by meanfox
Hi,
When I use IE to download a test file it takes about 3 seconds (non-
cached).
My own WinInet code using InternetReadFile takes about 7 seconds (also
non-cached).
This is repeatable. It is a 644K file. This is on XP SP2 with IE7.
Can anyone explain this and tell me how to speed up my WinInet
download? I thought IE used WinInet?
* Running synchronously vs running in a thread
* Downloading in 1 call to InternetReadFile vs. downloading many 1K
chunks
Please advise!
Tom

[meanfox]

Here's a minimal test app that demonstrates the problem.
Right now I am getting around 15-25 seconds for the WinInet download!
IE7 is taking 3-4s for the same file.
I need the INTERNET_FLAG_PRAGMA_NOCACHE because without it I get
caching errors. Another can of worms.

What am I doing wrong?

Thanks -

Tom



#include <windows.h>
#include <wininet.h>
#include <assert.h>
#include <stdio.h>
#include <conio.h>

void main(void)
{
HINTERNET session = ::InternetOpen("wininet_test",
INTERNET_OPEN_TYPE_PRECONFIG, NULL, NULL, 0);
assert(session != NULL);

HINTERNET h_http_file = InternetOpenUrl(session, "http://
www.fundictive.com/gamepatch/client_r.exe", NULL, 0,
INTERNET_FLAG_PRAGMA_NOCACHE, 0);
assert(h_http_file != NULL);

char szSizeBuffer[32] = "";
DWORD dwLengthSizeBuffer = sizeof(szSizeBuffer);
BOOL bQuery = ::HttpQueryInfo(h_http_file,
HTTP_QUERY_CONTENT_LENGTH, szSizeBuffer, &dwLengthSizeBuffer, NULL);
assert(bQuery == TRUE);

DWORD dwFileSize = atol(szSizeBuffer);
char * pdata = new char[dwFileSize];
DWORD length = 0;
DWORD time_start_ms = timeGetTime();
BOOL bRead = ::InternetReadFile(h_http_file, pdata, dwFileSize,
&length);
printf("InternetReadFile took %d ms\n", timeGetTime() -
time_start_ms);
assert(length == dwFileSize);

BOOL rc_ich = ::InternetCloseHandle(h_http_file);
assert(rc_ich == TRUE);

rc_ich = ::InternetCloseHandle(session);
assert(rc_ich == TRUE);

FILE * file = ::fopen("client_r.exe", "wb");
::fwrite(pdata, 1, length, file);
::fclose(file);

delete [] pdata;

getch();
}

[Paul Baker [MVP, Windows - Networking]]

If you're using INTERNET_FLAG_PRAGMA_NOCACHE and there is a proxy, you're
getting it from the origin server.

Internet Explorer may be getting it from the proxy cache or from its own
cache, depending on settings. This would of course be faster.

Paul

Post by meanfox
Here's a minimal test app that demonstrates the problem.
Right now I am getting around 15-25 seconds for the WinInet download!
IE7 is taking 3-4s for the same file.
I need the INTERNET_FLAG_PRAGMA_NOCACHE because without it I get
caching errors. Another can of worms.
What am I doing wrong?
Thanks -
Tom
#include <windows.h>
#include <wininet.h>
#include <assert.h>
#include <stdio.h>
#include <conio.h>
void main(void)
{
HINTERNET session = ::InternetOpen("wininet_test",
INTERNET_OPEN_TYPE_PRECONFIG, NULL, NULL, 0);
assert(session != NULL);
HINTERNET h_http_file = InternetOpenUrl(session, "http://
www.fundictive.com/gamepatch/client_r.exe", NULL, 0,
INTERNET_FLAG_PRAGMA_NOCACHE, 0);
assert(h_http_file != NULL);
char szSizeBuffer[32] = "";
DWORD dwLengthSizeBuffer = sizeof(szSizeBuffer);
BOOL bQuery = ::HttpQueryInfo(h_http_file,
HTTP_QUERY_CONTENT_LENGTH, szSizeBuffer, &dwLengthSizeBuffer, NULL);
assert(bQuery == TRUE);
DWORD dwFileSize = atol(szSizeBuffer);
char * pdata = new char[dwFileSize];
DWORD length = 0;
DWORD time_start_ms = timeGetTime();
BOOL bRead = ::InternetReadFile(h_http_file, pdata, dwFileSize,
&length);
printf("InternetReadFile took %d ms\n", timeGetTime() -
time_start_ms);
assert(length == dwFileSize);
BOOL rc_ich = ::InternetCloseHandle(h_http_file);
assert(rc_ich == TRUE);
rc_ich = ::InternetCloseHandle(session);
assert(rc_ich == TRUE);
FILE * file = ::fopen("client_r.exe", "wb");
::fwrite(pdata, 1, length, file);
::fclose(file);
delete [] pdata;
getch();
}

[meanfox]

Oh I thought I had stopped IE7 from doing any caching at all. Is there
actually a way to do that so that I can compare IE7 properly with my
WinInet app?

I turned on INTERNET_FLAG_PRAGMA_NOCACHE because when I updated the
files by uploading them, neither IE nor my WinInet application
detected that the files had changed for some time period. I think some
small text files (my manifests) were detected correctly, but my exe's
were not. Is that expected? How should I resolve that? My manifest
implied that the files had been updated, but the fetch retrieved stale
files...

Tom

[Paul Baker [MVP, Windows - Networking]]

The easiest thing to do at this point would be to see if removing the
INTERNET_FLAG_PRAGMA_NOCACHE flag offers comparable performance. This should
give you the same caching behaviour as Internet Explorer.

The behaviour you describe may simply be the behaviour of the proxy. If
you're always forcing it to load the page from the origin server, it's could
be slower. If you're not, the proxy might not give you a fresh copy.

There are a lot of things going on here that I cannot claim to understand
fully, so let's take it a step at a time.

Paul

Post by meanfox
Oh I thought I had stopped IE7 from doing any caching at all. Is there
actually a way to do that so that I can compare IE7 properly with my
WinInet app?
I turned on INTERNET_FLAG_PRAGMA_NOCACHE because when I updated the
files by uploading them, neither IE nor my WinInet application
detected that the files had changed for some time period. I think some
small text files (my manifests) were detected correctly, but my exe's
were not. Is that expected? How should I resolve that? My manifest
implied that the files had been updated, but the fetch retrieved stale
files...
Tom

[meanfox]

I appreciate your help.

Omitting INTERNET_FLAG_PRAGMA_NOCACHE does indeed speed it up heaps
but timings are still poorer than IE:

In ms:

8375
3125
4282
6140
11859
5265

As opposed to IE which is 2-4 s every time.

Tom

[Paul Baker [MVP, Windows - Networking]]

Next, I would experiment with buffer size. I use a buffer size of 1 KB for
InternetReadFile, but I don't know if I have any real reason for that
number. I do know that 32 bytes just seems like it is too small.

How big is the file you are testing with?

If you have Internet Explorer 4 or higher, you could look at using
InternetReadFileEx. I have not used this function, but my impression is that
it would not help performance. In this thread, Microsoft Developer Support
suggests that "There is not really any advantages to using either one":

http://groups.google.com/group/microsoft.public.inetsdk.programming.wininet/browse_thread/thread/ddef133a21a2c4ec

Paul

Post by meanfox
I appreciate your help.
Omitting INTERNET_FLAG_PRAGMA_NOCACHE does indeed speed it up heaps
8375
3125
4282
6140
11859
5265
As opposed to IE which is 2-4 s every time.
Tom

[meanfox]

Changing the buffer size to 1K helped marginally if at all. The file
is 644K.

I tried the InternetReadFileEx code but saw no improvement. (BTW that
code needs a memset for InetBuffers).

I wonder if tcpip would be faster...

I guess we can just conclude that INTERNET_FLAG_PRAGMA_NOCACHE slows
things down either significantly or drastically, but solves cacheing
errors.

Thanks again Paul...

Tom

[Paul Baker [MVP, Windows - Networking]]

Yes, but you said that even when you didn't use INTERNET_FLAG_PRAGMA_NOCACHE
, it was slower than Internet Explorer.

This is unsolved mystery, because Internet Explorer uses WinInet just like
you.

Paul

Post by meanfox
Changing the buffer size to 1K helped marginally if at all. The file
is 644K.
I tried the InternetReadFileEx code but saw no improvement. (BTW that
code needs a memset for InetBuffers).
I wonder if tcpip would be faster...
I guess we can just conclude that INTERNET_FLAG_PRAGMA_NOCACHE slows
things down either significantly or drastically, but solves cacheing
errors.
Thanks again Paul...
Tom

[meanfox]

It is unsolved yes.

But now I am more worried about how slow it can be if I use
INTERNET_FLAG_PRAGMA_NOCACHE, which I seem to need to use.

Is WinInet the best option I have?

My game will check a manifest for updates every time it runs.

Tom

[Paul Baker [MVP, Windows - Networking]]

This post might be inappropriate. Click to display it.

[Scherbina Vladimir]

Just wondering, did you tried to use winhttp for your purposes?
--
--Vladimir, Windows SDK MVP

Post by meanfox
It is unsolved yes.
But now I am more worried about how slow it can be if I use
INTERNET_FLAG_PRAGMA_NOCACHE, which I seem to need to use.
Is WinInet the best option I have?
My game will check a manifest for updates every time it runs.
Tom

[meanfox]

Paul,

Yes I'm sure there is a proxy at my ISP; and
INTERNET_FLAG_PRAGMA_NOCACHE certainly seems to fix the stale files
problem. It appears not to interfere with local timestamp-based
cacheing because I can see the local cache also being used, and reset
when I clear the temporary internet files in IE. If I do see any stale-
file problems I will add INTERNET_FLAG_RELOAD and possibly other
flags.

Vladimir,

I elected WinInet over WinHTTP because from what I read WinInet seemed
the appropriate API. My code is doing autoupdate for a game client. Do
you think I should give WinHTTP a go?

Cheers,

Tom

[Paul Baker [MVP, Windows - Networking]]

Hi meanfox,

Yes, I should have mentioned INTERNET_FLAG_RELOAD. It will certainly prevent
it from looking in the local cache, and I would expect that a "Pragma:
no-cache" header is also added in this case, preventing it from looking in
the proxy cache. You can verify that using Fiddler.

Then you can just look at the modified time and decide whether the file
changed yourself. You might consider using a HEAD request. If you do that,
you will get only the headers and will have to submit another, GET, request
if you actually want the file. So it's more efficient on both client and
server when there is not a newer version but less efficient if there is.
Probably not a big deal either way ;)

If this works for you, I see no reason to rewrite everything WinHTTP. It
depends on how much you've invested in WinInet already. WinHTTP is more
service oriented and will not try to keep a cache, use Internet Options or
otherwise act like Internet Explorer. This might be beneficial.

Paul

Post by meanfox
Paul,
Yes I'm sure there is a proxy at my ISP; and
INTERNET_FLAG_PRAGMA_NOCACHE certainly seems to fix the stale files
problem. It appears not to interfere with local timestamp-based
cacheing because I can see the local cache also being used, and reset
when I clear the temporary internet files in IE. If I do see any stale-
file problems I will add INTERNET_FLAG_RELOAD and possibly other
flags.
Vladimir,
I elected WinInet over WinHTTP because from what I read WinInet seemed
the appropriate API. My code is doing autoupdate for a game client. Do
you think I should give WinHTTP a go?
Cheers,
Tom

[Paul Baker [MVP, Windows - Networking]]

Hi Tom,

You might think I'm helping *you*, but actually you helped *me* too :)

I have something similar to this and I am not using any flags. I happen not
to be going through a proxy and my caching behaviour seems to be okay, so
it's not a big deal. But I should be using INTERNET_FLAG_RELOAD. I verified
using Fiddler that this flag does cause there to be a "Pragma: no-cache"
header in the request, even in the absence of INTERNET_FLAG_PRAGMA_NOCACHE.

RFC 3143 Known HTTP Proxy/Caching Problems which may be of interest:
http://tools.ietf.org/html/rfc3143#section-2.2.1

It implies that if there is a proxy, you would expect a "Cache-control:
no-cache" or "Cache-control: must-revalidate" header in the response. So
this can presumably be used to determine whether or not there is proxy
involvement.

Paul

Post by Paul Baker [MVP, Windows - Networking]
Hi meanfox,
Yes, I should have mentioned INTERNET_FLAG_RELOAD. It will certainly
prevent it from looking in the local cache, and I would expect that a
"Pragma: no-cache" header is also added in this case, preventing it from
looking in the proxy cache. You can verify that using Fiddler.
Then you can just look at the modified time and decide whether the file
changed yourself. You might consider using a HEAD request. If you do that,
you will get only the headers and will have to submit another, GET,
request if you actually want the file. So it's more efficient on both
client and server when there is not a newer version but less efficient if
there is. Probably not a big deal either way ;)
If this works for you, I see no reason to rewrite everything WinHTTP. It
depends on how much you've invested in WinInet already. WinHTTP is more
service oriented and will not try to keep a cache, use Internet Options or
otherwise act like Internet Explorer. This might be beneficial.
Paul

Post by meanfox
Paul,
Yes I'm sure there is a proxy at my ISP; and
INTERNET_FLAG_PRAGMA_NOCACHE certainly seems to fix the stale files
problem. It appears not to interfere with local timestamp-based
cacheing because I can see the local cache also being used, and reset
when I clear the temporary internet files in IE. If I do see any stale-
file problems I will add INTERNET_FLAG_RELOAD and possibly other
flags.
Vladimir,
I elected WinInet over WinHTTP because from what I read WinInet seemed
the appropriate API. My code is doing autoupdate for a game client. Do
you think I should give WinHTTP a go?
Cheers,
Tom

[meanfox]

Thanks Paul,

I may tinker with it some more at a later date. At least now I have a
better understanding of what is going on. The rfc link you gave looks
useful.

Normally there will be no update for my game.

Tom

[Paul Baker [MVP, Windows - SDK]]

You're welcome. Just post here again if more WinInet questions should arise.

Paul

Post by meanfox
Thanks Paul,
I may tinker with it some more at a later date. At least now I have a
better understanding of what is going on. The rfc link you gave looks
useful.
Normally there will be no update for my game.
Tom