Well,

We have a dedicated server and every week for the past three weeks one or two ip's just start to hit pages on our site every second all day long. So we started researching and seeing what we could do about it. But seeing as we dont know exactly what it is i wanted to ask what other people do about this problem and what is the problem. In one day one ip can drag our bandwidth down 9 gigs from accessing one page all day and put the resources into overtime. We contacted hms twice, but they said it was our problem. Our partner company says it is a robot so we put a simple robots.txt file our there, with little confidence it did not work.

So i guess we are looking for an answer even, what is the problem? a robot? spider?
How can you stop this from happening? software? code?

Thanks for the help.

Well, you can reverse lookup the IP to see who they are registered to:

http://www.whois.sc

If it turns out they are registered to google, yahoo or another search company, a robots.txt file would be a good idea. Also, if that's the case, a REVIST meta tag set to like 7 days would be helpful (sometimes they ignore the revist tag tough).

My Windows VPS gets a lot of SQL server hits (20,000 in 30 mins) So i just block the IP addresses via my firewall. That's another alternative for you.

Jason101,

We are currently just blocking the IPs in IIS. But there has to be a better way to stop this from happening. Right?
We look up all the ip's that do this and hms told us about dnsstuff so we use that.

What are these programs called that do this?
Is there any solution that can pick out ips that do this and block them automatically?

Blocking them after or while the damage is being done does not sound like a solution to me.

Thanks for replying Jason.

UN101,
I am sorry to hear you are having this problem. It is really sad that someone would do this to you, but that is the nature of the internet; everything is connected, so good and bad will come from it. I may have my jargon mixed up, but I believe this is a malicious robot, and not a spider as even if your pagerank (or equivalent) is ten on google and other spiders, you're IMPOSSIBLE to get it to crawl you that much. This (server) that is attempting to pull a PoD (Ping of Death attack using pings to overwhelm the resources of a server including but not limited to bandwith and CPU) or another connect and cut connection attack, DDoS (Distributed Denial of Service attack which basically aims to do the same thing as PoD.) Usually blocking at the firewall level is good, but if you can't block at that, an IIS level would work too... Unfortunately otherwise, your only way of stopping them would probably be to bring this to court. However, in my experience, you are not sacrificing speed that much for this as IP blockings are common, and IIS or the firewall by default checks if an IP is banned, so this will not add any more speed than there is already. I hope for the best outcome of this issue. Thanks.

comprug, thanks for the reply.

It may be the nature of the internet but there still has to be something a person can do or buy to stop or lessen the amount or hits of the attacks. One happens to us every week for the past month. To say you have to find out about it then do something about it seems wrong to us.

There has to be another way.

UN101,
I see what you are saying now; it would be great if we could stop Viruses, Spam, Phishing, Spyware, PoD, and DDoS, the whole lot rather than spend billions (cumilative) together to stop are prevent against them, not including sacrifice speed, but in this case, the other solution is far more costly than even 9gb of bandwith a day; the only way to stop the attacks would be in a court of law, becuase if you got their ISP to suspend them, they'd find another one, so basically you'd have to file a lawsuit, and if you win ask for legal fees, but that is pretty harsh on you and is a waste of time, and I wish there was a way to STOP attacks not prevent them when they happen, there are just some phenomenons that are impossible to fully defeat; you can only defend against them.
UPDATE: Now that I think about it, consider this: 1. Get their ISP to suspend them, erase their db of sites to hit; you may not get back on, or at least you will give yourself more time to evaluate the situation. And if the ISP has any questions, threaten Legal Action even if you don't mean it, it always works). I believe ISPs are required by law to do something about stuff like that.

If you have access to your raw logs you may be able to deny those people who have visited your site x number of times in the past y minutes; x and y being some variable you set in a script. Perhaps you could ask in the programming section of this forum for help on the issue. Some one there may be able to help you.