Hi, I've written huge spiders and web crawlers using python, perl, php etc. I've used Urllib2, LWP, CookieJar, Mechanize library for hit web url. And for scraping data used regular expressions which is so faster. After R&D on your site it'll be clear which tools/library should use. Please check your inbox for more.