I need some to write a script that will crawl this website:
[login to view URL]~cx~7E77839DD37678338BD84FD5A6EEE3651895944D5E15C9600A0CFF4FCAE1ADB1FADC93DFDDEBEC53
We need to collect the following in an SQL database
- Product Description
- Product Name
- Product Code
- Mfr Code
- Image link
There are 25,000 products that need to be index into an SQL file so you will need to ensure that you don't blacklist us (I don't mind the crawler running over a period of time)
I would also like the crawler to download each image to a folder on our server
Once the database has been fully populated I would require the crawl to run once a we to collect new product details and download new images, I will provide the new product codes that need crawl by uploading a TXT file to the server.
Please give examples of other crawlers that you have done before and PMB if you have any questions