HELP! Bit by WinINet HTTPS... again

[EMonaco]

All,

Well, I'm confounded having just dealt with 3 serious issues concerning
WinINet and the 832894 security patch, (Took all of February!) and it looks
like I've run into another! I will try to simplify everything as much as
possible.

Have an app that goes to a servlet using HTTPS (wininet). This app first
does a GET request and then POST requests (sending data in 132 byte packets)
till done. We do a simple date check for server cert validity as well as the
server requiring a client cert.

Normal operation should be:

Open Request. Do GET, receive error that client cert is needed, find & set
client cert, do HttpSendRequest() again. Get Response. Check Response.
Validate server cert. Close Request handle.

Open Request. Set client cert obtained above. Send Request. Get Response.
Check Response. Close Request.
This continues until done or 5 minute timeout.
Each 132 byte request is an item lookup, items are entered by user in
realtime.

Scenerio 1: FAILS!

Open Request. Do GET, receive error client cert needed. find&set client
cert, do HttpSendRequest() again. Get Response. Check Response. Validate
server cert. Close Request handle.

Open Request. Set client cert obtained above. Send Request. Get Response.
Check Response. Close Request.
Wait over 30 seconds. (server closes socket)
Open Request. Set client cert obtained above. Send Request. Get Error
"HttpSendRequest() Error(0x2f7d): An error occurred in the secure channel
support".

Now if I do each lookup before the server closes the socket due to
inactivity, everything works as expected.

Observations: First socket is opened SSL negotiation starts- is closed,
SendRequest() returns with client cert needed.
When I set the client cert and submit request again, a new socket is opened,
and everything proceeds. Wininet now uses that same socket for subsequent
request (as long as its not closed) however if it is closed, wininet opens
and closes a socket, but the wininet log does not show it "Sending" any data
on the attempt. (mind you we are setting INTERNET_FLAG_KEEP_CONNECTION! So
on the face of it, it seems that WinInet can't deal with the socket being
closed. (that is can't open a new socket and resume or start a new SSL
session).

If your still with me, here is where things get REALLY weird. My app has the
ability to retrieve a dword from the registry that is OR'd with the
(INTERNET_FLAG_KEEP_CONNECTION|INTERNET_FLAG_SECURE) flags on the
HttpOpenRequest. This is because I use a local IIS to test and its
certificate CN is by DNS name, but I use the IP address... hence I'll get an
INVALID_CN_NAME error if I don't include 0x1000
(INTERNET_FLAG_IGNORE_CERT_CN_INVALID) along with the HttpOpenRequest. Which
leads us to:

Scenario 2: Successful!

Do the same as scenario 1, only set the registry value for extra flags to
0x1000 (INTERNET_FLAG_IGNORE_CERT_CN_INVALID).

After 30 seconds, the server still closes the socket- but I don't get an
error and everything works. Investigation Wininet log and TDIMON dump- I see
a big difference, something about this flag- or the combination, causes
wininet to open a socket, do all the ssl handshake, server/client cert
exhange for every batch of HttpOpen/HttpSend Request!!!! I would think this
would be a function of only the INTERNET_FLAG_KEEP_CONNECTION, however I
removed it and get the same results- without the
INTERNET_FLAG_IGNORE_CERT_CN_INVALID, it reuses the same socket unless the
socket is disconnected by the remote at which point it fails- with the flag-
it uses a new socket for every request and does the whole SSL handshake cert
swap (which works fine- but adds a lot of extra data to the network).


So- just what the heck is goin on!? I'm sure this isn't right and whats it
gonna take to get it fixed!? PS now would be a good time to discuss finder
fees :)

Machine Info: Dell Inspiron 8200, 1.6Ghz, 512Meg. XP Pro, ALL critical
patches applied. (but this doesn't seem to matter, at least not entirely)
I've tried 3 debug versions of WinINet 6.0.2800.1106, 6.0.2800.1400, and
6.0.2800.1405 and I get the exact same results!


Erin.

[Stephen Sulzer]

I think I can explain this :)

The first issue--the 0x2f7d (ERROR_INTERNET_SECURITY_CHANNEL_ERROR) error
from HttpSendRequest--is a long-standing problem in WinInet related to
handling persistent SSL connections. Whenever WinInet tries to reuse a
persistent connection, it uses a simple test to see if the connection is
still alive: it peeks at the socket for a byte of data. If the socket recv()
call to read from the socket data does not fail, WinInet assumes the socket
connection is still alive. In addition, WinInet assumes that connections
that have been idle less than a certain period (1 minute) since the last
request completed are also still valid.

Normally this heuristic of determining if the socket is still alive works
well enough, at least for regular HTTP. But it doesn't always work correctly
for SSL connections--it can provide false positives to WinInet.

When the server closes an SSL connection, it may send a packet of data to
notify the client that the server has closed its SSL connection. After
sending this "SSL close notify" command, the server can then close the
underlying TCP connection. The server does not have to send this SSL close
notification, however, but it is considered the "clean" way to shutdown SSL.
If the server does send the SSL close notification, it will appear to
WinInet as data in the socket when it tests to see if the socket is still
alive. So WinInet then reuses this socket for the new request. When WinInet
gives the socket to the SChannel.dll SSL subsystem to prepare (encrypt) the
request data, SChannel will process the close notification and inform
WinInet that the connection is gone.

If you Google the web & groups for "IE SSL close notify" you will find
references to this IE/WinInet problem.

IE/WinInet has long had this problem of not dealing properly with the SSL
close notification in some situations. If the server closes the SSL
connection (and sends the close notify) immediately following the response
data of the current request (meaning no keep-alive), then there is no
problem as WinInet will close the socket (in this case WinInet will process
the close notification). If the server sends the close notify after, say,
more than 60 seconds of idle time, then WinInet's other test--60 second
expiration on an idle socket--will cause WinInet to assume the socket is
dead and avoid the problem. But if the close notify is sent not as part of
the response to the previous HTTPS request and within WinInet's 60-second
socket idle expiration, this will trick WinInet into thinking that the
closed SSL connection is still open.

Some webservers workaround this by not using persistent SSL connections with
IE. I doubt this problem with WinInet will get fixed anytime soon. You can
work around it by disabling keep-alive (which I know is unsatisfactory if
your program is trying to send a lot of requests in a short time).

Then 2nd issue--setting the INTERNET_FLAG_IGNORE_CERT_CN_INVALID seems to
fix or avoid the SECURITY_CHANNEL_ERROR problem--is also related to
WinInet's management of persistent connections. Setting the IGNORE flags
will cause WinInet to not reuse the socket connection. WinInet does this to
avoid conflicts in which different HTTPS requests specify different sets of
IGNORE flags. For example, suppose a request ignores the various server
certificate validation problems and afterwards WinInet caches the socket
connection. A subsequent HTTPS request which does want the server
certificate to be fully validated (so no IGNORE flags are given) would not
want the socket from the previous request because it would not know whether
the server's cert was good. (Reusing a socket connection usually skips the
server certificate validation.) To simplify tracking what security flags
were specified on which sockets, WinInet simply disables reusing SSL
connections in which the server certificate was not fully validated.

Hope that helps.

Stephen

[EMonaco]

Stephen,

Yes it does help. I guess the thing I'm having a hard time with
understanding though is why when it does fail with the first scenario,
WinINet can't establish a new SSL session on the new socket it opens so we
can continue on. Every attempt after that will fail with the 0x2f7d error
until I completely close out the app. When its not reusing sockets it can
and does negotiate a new SSL session- or is this in fact the "long standing
problem" your actually talking about? If this is the case I can't imagine
why Microsoft wouldn't fix this. In our case the HTTP headers are already
bigger than the content we are passing back and forth, add to that the
overhead (in time and resources) of opening a new socket, renegotiating SSL
(exchanging server and root then client certs) and that means MOST of the 4K
or so per request/response is overhead and useless to the app, and also
because of the extra reads and writes, slows down the whole lookup process.

Reusing the connection cuts out a lot of this overhead and speeds things up
noticeably. The only problem is when the connection does time out, WinINet
needs to be able to deal with that by opening a new socket, establishing a
new SSL session and continuing on. Surely thats not too much to ask?

Are there alternatives that can handle this properly? (for example doing
it in .NET, using OPENSSL or SChannel directly, etc)


Erin.

[EMonaco]

Stephen,

Our production server is NetScape-Enterprise/4.1. Based on your post- if
possible and we can change the server socket timeout (or ssl connection
timeout or whatever) to say 70 seconds, you are saying that WinInet wont
have the issue we are seeing? In other words it will discard the current
socket, open a new one, renegotiate SSL and happily continue?

Erin

[EMonaco]

Stephen,

More bad news too. I do the first lookup and wait over a minute to do the
next and it fails too. So it appears that once the server disconnects the
socket we are toast, period.


Erin.

[Stephen Sulzer]

This sounds like pretty bad bug and worse than I thought. You should contact
Microsoft PSS; hopefully they can debug the problem and maybe get you a
hotfix.

The only workaround I can think of right now is that when you encounter the
0x2f7d error code, force WinInet to unload its socket connection pool. So
you can continue to use keep-alive, but when the connection problem occurs,
force WinInet to discard all its cached socket connections. To do this you
need to do the following:

- close all open WinInet handles, including the session handle returned by
InternetOpen.
- call InternetSetOption(NULL, INTERNET_OPTION_END_BROWSER_SESSION, NULL,
0);
- call InternetSetOption(NULL, INTERNET_SETTINGS_CHANGED, NULL, 0);

After doing that, your app can then start over, creating new WinInet handles
and send the initial GET request.

Stephen

Post by EMonaco
Stephen,
More bad news too. I do the first lookup and wait over a minute to do the
next and it fails too. So it appears that once the server disconnects the
socket we are toast, period.
Erin.

[Stephen Sulzer]

minor correction - that should be:

InternetSetOption(NULL, INTERNET_OPTION_SETTINGS_CHANGED, NULL, 0);

Post by Stephen Sulzer
This sounds like pretty bad bug and worse than I thought. You should contact
Microsoft PSS; hopefully they can debug the problem and maybe get you a
hotfix.
The only workaround I can think of right now is that when you encounter the
0x2f7d error code, force WinInet to unload its socket connection pool. So
you can continue to use keep-alive, but when the connection problem occurs,
force WinInet to discard all its cached socket connections. To do this you
- close all open WinInet handles, including the session handle returned by
InternetOpen.
- call InternetSetOption(NULL, INTERNET_OPTION_END_BROWSER_SESSION, NULL,
0);
- call InternetSetOption(NULL, INTERNET_SETTINGS_CHANGED, NULL, 0);
After doing that, your app can then start over, creating new WinInet handles
and send the initial GET request.
Stephen

Post by EMonaco
Stephen,
More bad news too. I do the first lookup and wait over a minute to do

the

Post by EMonaco
next and it fails too. So it appears that once the server disconnects the
socket we are toast, period.
Erin.

[EMonaco]

All, I have logged an incident with MS Developer support (WinINet/ISAPI
group) regarding this issue. When I have more information I will post an
update for the newgroup.


Regards,
Erin.
--
Visit us on the web at: http://www.databasementsoftware.com

Post by Stephen Sulzer
This sounds like pretty bad bug and worse than I thought. You should contact
Microsoft PSS; hopefully they can debug the problem and maybe get you a
hotfix.
The only workaround I can think of right now is that when you encounter the
0x2f7d error code, force WinInet to unload its socket connection pool. So
you can continue to use keep-alive, but when the connection problem occurs,
force WinInet to discard all its cached socket connections. To do this you
- close all open WinInet handles, including the session handle returned by
InternetOpen.
- call InternetSetOption(NULL, INTERNET_OPTION_END_BROWSER_SESSION, NULL,
0);
- call InternetSetOption(NULL, INTERNET_SETTINGS_CHANGED, NULL, 0);
After doing that, your app can then start over, creating new WinInet handles
and send the initial GET request.
Stephen

Post by EMonaco
Stephen,
More bad news too. I do the first lookup and wait over a minute to do

the

Post by EMonaco
next and it fails too. So it appears that once the server disconnects the
socket we are toast, period.
Erin.

[EMonaco]

All,

Microsoft is still working on this problem. Will let you know.
--
Visit us on the web at: http://www.databasementsoftware.com

Post by EMonaco
All, I have logged an incident with MS Developer support (WinINet/ISAPI
group) regarding this issue. When I have more information I will post an
update for the newgroup.
Regards,
Erin.
--
Visit us on the web at: http://www.databasementsoftware.com

Post by Stephen Sulzer
This sounds like pretty bad bug and worse than I thought. You should

contact

Post by Stephen Sulzer
Microsoft PSS; hopefully they can debug the problem and maybe get you a
hotfix.
The only workaround I can think of right now is that when you encounter

the

Post by Stephen Sulzer
0x2f7d error code, force WinInet to unload its socket connection pool. So
you can continue to use keep-alive, but when the connection problem

occurs,

Post by Stephen Sulzer
force WinInet to discard all its cached socket connections. To do this you
- close all open WinInet handles, including the session handle returned by
InternetOpen.
- call InternetSetOption(NULL, INTERNET_OPTION_END_BROWSER_SESSION, NULL,
0);
- call InternetSetOption(NULL, INTERNET_SETTINGS_CHANGED, NULL, 0);
After doing that, your app can then start over, creating new WinInet

handles

Post by Stephen Sulzer
and send the initial GET request.
Stephen

Post by EMonaco
Stephen,
More bad news too. I do the first lookup and wait over a minute to do

the

Post by EMonaco
next and it fails too. So it appears that once the server disconnects

the

Post by Stephen Sulzer

Post by EMonaco
socket we are toast, period.
Erin.