To know how many actual visits one gets to their website now seems impossible because of caches. It would be nice to know, if anyone does, roughly what percent of visits are directed to caches. Those are visits we never know about. Perhaps a study was made... anyone know? Or perhaps I can put it another way, since my website gets AOL robots about every three hours, what is this telling me? Are these page requests or just normal scans looking for updated pages?

I don't know if AOL bots are sent out as a result of requests or as a result of some sort of lunar schedule. AOL might be able to tell you if you check with their customer service, but I doubt anyone that knows for certain would be willing to tell you.

Unfortunately, the caching situation is even more grim. I don't believe it's possible to design a site around this issue, since it's essentially a web browser setting.

For those of you unfamilar with the topic, here's a short summary:

Caching happens when your web browser saves pages that you visit to your computer's hard drive. This becomes a problem when you visit a particular page repeatedly, and instead of taking a current copy from the web server, your browser uses the copy held 'in cache' on your hard drive.

Sometimes this is favorable, as it saves loading time if you are on a slower connection (or love that bandwidth-consuming streaming audio, as I do), or it can be a bad thing if the website in question has changed content since you last visited.

How does this relate to web traffic and statistics? Well, if your browser takes the page from cache, you've never contacted the remote webserver, and in turn you don't generate any statistics.

AOL complicates the issue by performing it's own caching - it's web bots cache pages that are frequently requested by AOL clients in order to speed up their connection times. Sure, the bots generate traffic, but it's unclear how much bot traffic equates to real traffic that your site missed.

Sorry if I ramble - just as there are no stupid questions, rarely is there a simple one.

Thank you Jamie. Anyway, for me, the effects of caching on visitor counts is a frustrating matter. Although my traffic continues to climb, that is, according to my log stats, some of that is due to an increase in bots from AOL, Inktomi and everyone else. Whatever this more frequent caching means I'll try to find out. Anyway, one source said that because of caching, one's actual visitors may be substantially more. In the end, the actual count doesn't matter so much I suppose, but the direction one's website is going does. As you know the problem is the inability to guage the effectiveness of changes.
About 6 months ago, I tried the Deep Metrix Intel service which claims the ability to count cache visits while eliminating bots. Because of Java Script, I can see eliminating the bot count, which they did, but I wasn't convinced they were effectively counting cache requests. Anyway, according to those results, my visitor count was averaging 70-80% less than my log stats!!! It seems accuracy is not for sale at any price.

If you discover anything definitive, please by all means post it here - I'd love to know the results of this inquiry as well!

AOL determines the proxy caching of their web pages through HTTP header information, this must be set either via the server or by using a scripting language such as ASP for your site.

Information on how to set the headers can be found at AOLs webmaster resource site.
http://webmaster.info.aol.com/

This article discusses caching and setting browser cache settings via meta tags.
http://www.netmechanic.com/news/vol4/design_no5.htm

While I didn't find any 'study' on the effect of caching, that is, no one that would hazzard a guess on the percentage effect on visitor counts, I did find an enlightening forum exchange at

http://www.webmasterworld.com/forum39/163.htm