There is a website for which I need to have its search results scraped into a csv file.
There is quite a bit of data to be captured. Altogether, I need just one set of search results captured. However, there are 1.7 million rows of data with 10 columns. Each data point is small, containing maybe 10 characters of text. This is spread out at 250 rows/page for about 6800 total pages of search results.
*** This is not just another scrape.*** The form parameters and cookies are inconsistent and quite tricky to work with. I work with a skilled web scraper and even he was unable to come up with a solution for this website. I think the best method is if you have some sort of visual scraper that manually clicks through each page just like a human would. When you apply, tell me what technology you are planning to use to accomplish the task.
The site requires you to be logged into an account to view the search results, so I'll send you the login details after I've selected you for the project. It would be best if you also had some sort of way to regulate the rate at which the scraper moves and to possibly have a list of IPs to go through to avoid being blocked by the website.
I need to have the project completed quickly, within 5 days after awarding you the job.
Budget: $60 - $120