Dear web programmers,
This should be an easy task for a programmer who know what s/he's doing:
I need a web crawler that will crawl Australian websites (all websites ending in .au) making a record of whether or not each website:
- has a custom favicon (first look for <link rel="shortcut icon"...> in the HTML and then look for the /[url removed, login to view] file)
- works as [url removed, login to view] but not [url removed, login to view] (a comparison of the HTML served will need to be done)
- has a <title> (verses no title or one that includes "Untitled")
- has a meta description and meta keywords
- uses Google Analytics
The results will be stored in a mysql database.
The script should be able to pick up where it left off if it is stopped/interrupted.
I would prefer it to be written in PHP
When you bid, please begin your comment with "I am an experienced programmer from [enter your country]" so that I know that you have read the above requirements.
14 freelancers are bidding on average $171 for this job
Hi! I am an experienced programmer from Ukraine. Ready to develop the solution you need, it will query the whois and parse the sites one by one storing the data grabbed to database. Regards, Igor.
I am an experienced programmer from the United States. I have most of this code already written from a crawler I recently finished. A few slight code modifications and I can have exactly what you want.