In this post I wanted to share the wonders of WWW:Mechanize, a gem for automating web browsing through Ruby. Instead of making direct GET and POST requests, you tell it to click on a node, fill in a form, or submit a form. In fact, this post was made using WWW:Mechanize! Here is the code:

Notice that when you submit the forms to post a new topic, you also post hidden fields, and other stuff. Mechanize takes care of all that for you, and the cookies and session ids, as well as redirects. All you have to tell it is which links and buttons to click, and which forms and fields to fill in. All I had to fill in was my username, password, topic title, and body.

W O W

That really tickles my gibblies. Being of the ASP.Net, ColdFusion, and ASP world, I sure wish there was some equivilent to that. That's ridiculously simple... and powerful.

Although, as I was sitting here pondering all of the wonderful possiblities of such a simplified mechanic, I began conjuring as many malicious uses for something like this. This is putting ALOT of power in almost everybody's hands no matter how versed they are at programming.

Interesting nonetheless. Thanks for sharing!

Mechanize is a great substitute for an API on sites that you need to automate. Although the site is subject to change, by searching for links via regexp's, you can be compatible with the latest version of a site without updating your code.

I have used this for several projects, two of which relate to HMS. I created an API for their CP that would automate various tasks, including the retrieval of database passwords (in the html source). This worked well until they (ahem) installed a captcha. Another project you have probably heard of. I used this to automate my moderation duties - I would crawl the forum with mechanize, classify the text with a homemade program, and use its form processing capabilities to delete spammy posts. If you are interested in Mechanize, I can send you the code to some sample projects.

Not necessary. I won't be cracking Ruby on Rails open for myself for quite some time. But it's interesting to see these little tidbits in action as when I start doing anything with RoR I'm sure to remember some of these items.

Josh,
by the way.. What malicious uses can you come up with?

Well there are ALOT of possibilities that come with scraping and/or brute force form submissions. I know that stuff goes on now, but this makes it hella easy for even a novice.

I know Perl has a Mechanize function that inspired this but I don't know how often Perl is the tool of choice for these types of things. That would probably be a good indicator of how many people would want to utilize this for similar reasons.