FtpFindFirstFile/NextFile returns stale directory listings

Discussion:

(too old to reply)

George V. Reilly

2006-03-25 00:06:06 UTC

I'm enumerating a directory with FtpFindFirstFile +
InternetFindNextFile. If I directly add a new file to the remote
directory, then re-enumerate it, I see the original files; I do not see
the new file. If I directly remove a file from the remote directory,
then re-enumerate it; again, I still see the original list of files.

I'm guessing that WinInet is maintaining a cache. Am I right? I'm
closing both the hConnect and the hInternet after each directory
enumeration.

hInternet = InternetOpen(.., OPEN_TYPE_DIRECT, ...)
hConnect = InternetConnect(hInternet, ..., FTP_SERVICE,
INTERNET_FLAG_PASSIVE...
hFind = FtpFindFirstFile();
loop using InternetFindNextFile and GetLastError
InternetCloseHandle(hConnect)
InternetCloseHandle(hInternet)

--
/George V. Reilly Seattle, WA

Paul Baker

2006-03-26 21:29:41 UTC

Permalink

Remember that Internet Explorere uses WinInet?

Can you reproduce the same behaviour in WinInet? If not, you're probably
doing something wrong.

Paul

Post by George V. Reilly
I'm enumerating a directory with FtpFindFirstFile +
InternetFindNextFile. If I directly add a new file to the remote
directory, then re-enumerate it, I see the original files; I do not see
the new file. If I directly remove a file from the remote directory,
then re-enumerate it; again, I still see the original list of files.
I'm guessing that WinInet is maintaining a cache. Am I right? I'm
closing both the hConnect and the hInternet after each directory
enumeration.
hInternet = InternetOpen(.., OPEN_TYPE_DIRECT, ...)
hConnect = InternetConnect(hInternet, ..., FTP_SERVICE,
INTERNET_FLAG_PASSIVE...
hFind = FtpFindFirstFile();
loop using InternetFindNextFile and GetLastError
InternetCloseHandle(hConnect)
InternetCloseHandle(hInternet)
--
/George V. Reilly Seattle, WA

Scherbina Vladimir

2006-03-27 15:35:08 UTC

Permalink

The cache might be enabled on remote server and | or on your proxy, keep
this in mind. Another issue is to force wininet not to use cache
(INTERNET_FLAG_NO_CACHE_WRITE for FtpFindFirstFile, etc)
--
Vladimir
http://spaces.msn.com/vladimir-scherbina/

Paul Baker

2006-03-27 15:40:14 UTC

Permalink

It is documented that the INTERNET_FLAG_NO_CACHE_WRITE can be used with
FtpFindFirstFile, however it is not clear from the documentation what its
behaviour is.

Paul

Post by Scherbina Vladimir
The cache might be enabled on remote server and | or on your proxy, keep
this in mind. Another issue is to force wininet not to use cache
(INTERNET_FLAG_NO_CACHE_WRITE for FtpFindFirstFile, etc)
--
Vladimir
http://spaces.msn.com/vladimir-scherbina/

George V. Reilly

2006-03-28 00:44:44 UTC

Permalink

Thanks, Paul & Vladimir.

I found that passing INTERNET_FLAG_NO_CACHE_WRITE |
INTERNET_FLAG_RELOAD to FtpFindFirstFile worked for me. It's documented
at
http://msdn.microsoft.com/library/en-us/wininet/wininet/ftp_sessions.asp#Enumerating_director

/George

Scherbina Vladimir

2006-03-28 07:56:05 UTC

Permalink

its documentation is clear - when enumerating items their pathes are not
added to cache
--
Vladimir
http://spaces.msn.com/vladimir-scherbina/

Post by Paul Baker
It is documented that the INTERNET_FLAG_NO_CACHE_WRITE can be used with
FtpFindFirstFile, however it is not clear from the documentation what its
behaviour is.
Paul

Paul Baker

2006-03-28 14:21:29 UTC

Permalink

I cannot agree. There is no mention of what the flag does when passed to
FtpFindFirstFile in the documentation.

The generic definition of the flag is just that. To generic to be clear.

What URL does it cache? The directory? Each file? What is the
CacheEntryType? How does it use that entry? Does it check the
LastModifiedTime? ExpureTime? Or, maybe it is an in-memory cache totally
unrelated to this cache.
http://msdn.microsoft.com/library/en-us/wininet/wininet/commiturlcacheentry.asp

These questions are legitimate, as the cache is not necessarily geared
towards FTP directory listings. It's clear only in as much as it is not
suprising that using the flag affects the behaviour.

Actually, I suspect it's the INTERNET_FLAG_RELOAD flag that resolved it by
forcing a proxy to refresh its cache.

Paul

Post by Scherbina Vladimir
its documentation is clear - when enumerating items their pathes are not
added to cache
--
Vladimir
http://spaces.msn.com/vladimir-scherbina/

Post by Paul Baker
It is documented that the INTERNET_FLAG_NO_CACHE_WRITE can be used with
FtpFindFirstFile, however it is not clear from the documentation what its
behaviour is.
Paul

Paul Baker

2006-03-28 14:28:27 UTC

Permalink

Actually, there is even an inconsistency in the documentation.

According to this page, INTERNET_FLAG_NO_CACHE_WRITE cannot be used with
FtpFindFirstFile
http://msdn.microsoft.com/library/en-us/wininet/wininet/api_flags.asp

According to FtpFindFirstFile documentation, it can:
http://msdn.microsoft.com/library/en-us/wininet/wininet/ftpfindfirstfile.asp

I have sent feedback to MSDN.

Paul

Post by Paul Baker
I cannot agree. There is no mention of what the flag does when passed to
FtpFindFirstFile in the documentation.
The generic definition of the flag is just that. To generic to be clear.
What URL does it cache? The directory? Each file? What is the
CacheEntryType? How does it use that entry? Does it check the
LastModifiedTime? ExpureTime? Or, maybe it is an in-memory cache totally
unrelated to this cache.
http://msdn.microsoft.com/library/en-us/wininet/wininet/commiturlcacheentry.asp
These questions are legitimate, as the cache is not necessarily geared
towards FTP directory listings. It's clear only in as much as it is not
suprising that using the flag affects the behaviour.
Actually, I suspect it's the INTERNET_FLAG_RELOAD flag that resolved it by
forcing a proxy to refresh its cache.
Paul

Post by Scherbina Vladimir
its documentation is clear - when enumerating items their pathes are not
added to cache
--
Vladimir
http://spaces.msn.com/vladimir-scherbina/

Post by Paul Baker
It is documented that the INTERNET_FLAG_NO_CACHE_WRITE can be used with
FtpFindFirstFile, however it is not clear from the documentation what
its behaviour is.
Paul

Post by Scherbina Vladimir
The cache might be enabled on remote server and | or on your proxy,
keep this in mind. Another issue is to force wininet not to use cache
(INTERNET_FLAG_NO_CACHE_WRITE for FtpFindFirstFile, etc)
--
Vladimir
http://spaces.msn.com/vladimir-scherbina/

Scherbina Vladimir

2006-03-28 15:05:01 UTC

Permalink

Sometimes I realize that wininet is "dying" technology, there are some
questions sent by me to MSFT and they're still hunging, it's clear that MS
decided to move to winhttp

--
Vladimir
http://spaces.msn.com/vladimir-scherbina/

Paul Baker

2006-03-28 15:06:01 UTC

Permalink

Yes, the comparisons between WinInet and WinHTTP I have read seem to suggest
that.

However, I think Internet Explorer will continue to use WinInet for a long
time yet as it is, in as sense *made* for Internet Explorer.

FTP seems to be much less a priority in WinInet development than HTTP, and
Gopher development is non-existant.

Paul

Post by Scherbina Vladimir
Sometimes I realize that wininet is "dying" technology, there are some
questions sent by me to MSFT and they're still hunging, it's clear that MS
decided to move to winhttp
--
Vladimir
http://spaces.msn.com/vladimir-scherbina/

Scherbina Vladimir

2006-03-28 15:15:45 UTC

Permalink

Yes, you're right, at least for IE7 - it uses wininet.dll (a newer
version) - http://blogs.msdn.com/ie/search.aspx?q=wininet&p=1
--
Vladimir
http://spaces.msn.com/vladimir-scherbina/

Post by Paul Baker
Yes, the comparisons between WinInet and WinHTTP I have read seem to
suggest that.
However, I think Internet Explorer will continue to use WinInet for a long
time yet as it is, in as sense *made* for Internet Explorer.
FTP seems to be much less a priority in WinInet development than HTTP, and
Gopher development is non-existant.
Paul

Post by Scherbina Vladimir
Sometimes I realize that wininet is "dying" technology, there are some
questions sent by me to MSFT and they're still hunging, it's clear that
MS decided to move to winhttp
--
Vladimir
http://spaces.msn.com/vladimir-scherbina/

Scherbina Vladimir

2006-03-28 14:34:16 UTC

Permalink

Paul,

I agree with you that there're a lot of missed issues there (black spots),
but general acception is if you want to get the latest information from FTP
directory then use INTERNET_FLAG_NO_CACHE_WRITE | INTERNET_FLAG_RELOAD
flags - this issue is exactly documented at
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/wininet/wininet/ftp_sessions.asp#Enumerating_director

--
Vladimir
http://spaces.msn.com/vladimir-scherbina/

Paul Baker

2006-03-28 15:03:08 UTC

Permalink

It is documented in the page you mention that "If the application makes
changes on the FTP server or if the FTP server changes frequently, the
INTERNET_FLAG_NO_CACHE_WRITE and INTERNET_FLAG_RELOAD flags should be set in
FtpFindFirstFile. These flags ensure that the directory information being
retrieved from the FTP server is current.".

However, all it really does is say that you should use them (as if they are
magic). It is by no means exact. I stand by my point that "it is not clear
from the documentation what its behaviour is". Are you able to clearly
explain what affect this has in terms of low-level protocols? Are you able
to guarantee what the behaviour is in any given situation, or are you only
able to say "you should try it, it should work, etc.".

Also, this kind of information should be in the FtpFindFirstFile
documentation or at least cross-referenced there.

Paul

Post by Scherbina Vladimir
Paul,
I agree with you that there're a lot of missed issues there (black spots),
but general acception is if you want to get the latest information from
FTP directory then use INTERNET_FLAG_NO_CACHE_WRITE | INTERNET_FLAG_RELOAD
flags - this issue is exactly documented at
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/wininet/wininet/ftp_sessions.asp#Enumerating_director
--
Vladimir
http://spaces.msn.com/vladimir-scherbina/

Scherbina Vladimir

2006-03-28 15:10:47 UTC

Permalink

yes, I agree with you about leak of such explanations
--
Vladimir
http://spaces.msn.com/vladimir-scherbina/

Post by Paul Baker
It is documented in the page you mention that "If the application makes
changes on the FTP server or if the FTP server changes frequently, the
INTERNET_FLAG_NO_CACHE_WRITE and INTERNET_FLAG_RELOAD flags should be set
in FtpFindFirstFile. These flags ensure that the directory information
being retrieved from the FTP server is current.".
However, all it really does is say that you should use them (as if they
are magic). It is by no means exact. I stand by my point that "it is not
clear from the documentation what its behaviour is". Are you able to
clearly explain what affect this has in terms of low-level protocols? Are
you able to guarantee what the behaviour is in any given situation, or are
you only able to say "you should try it, it should work, etc.".
Also, this kind of information should be in the FtpFindFirstFile
documentation or at least cross-referenced there.
Paul

Post by Scherbina Vladimir
Paul,
I agree with you that there're a lot of missed issues there (black
spots), but general acception is if you want to get the latest
information from FTP directory then use INTERNET_FLAG_NO_CACHE_WRITE |
INTERNET_FLAG_RELOAD flags - this issue is exactly documented at
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/wininet/wininet/ftp_sessions.asp#Enumerating_director
--
Vladimir
http://spaces.msn.com/vladimir-scherbina/

Paul Baker

2006-03-28 15:10:04 UTC

Permalink

I agree with you too that there is some documentation here - just not any
that is up to my standards.

Paul

Post by Scherbina Vladimir
yes, I agree with you about leak of such explanations
--
Vladimir
http://spaces.msn.com/vladimir-scherbina/

Post by Paul Baker
It is documented in the page you mention that "If the application makes
changes on the FTP server or if the FTP server changes frequently, the
INTERNET_FLAG_NO_CACHE_WRITE and INTERNET_FLAG_RELOAD flags should be set
in FtpFindFirstFile. These flags ensure that the directory information
being retrieved from the FTP server is current.".
However, all it really does is say that you should use them (as if they
are magic). It is by no means exact. I stand by my point that "it is not
clear from the documentation what its behaviour is". Are you able to
clearly explain what affect this has in terms of low-level protocols? Are
you able to guarantee what the behaviour is in any given situation, or
are you only able to say "you should try it, it should work, etc.".
Also, this kind of information should be in the FtpFindFirstFile
documentation or at least cross-referenced there.
Paul

Post by Scherbina Vladimir
Paul,
I agree with you that there're a lot of missed issues there (black
spots), but general acception is if you want to get the latest
information from FTP directory then use INTERNET_FLAG_NO_CACHE_WRITE |
INTERNET_FLAG_RELOAD flags - this issue is exactly documented at
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/wininet/wininet/ftp_sessions.asp#Enumerating_director
--
Vladimir
http://spaces.msn.com/vladimir-scherbina/