I have a group of sites that utilize the same software build and are thus all the same (just different URL. Without hacking the dB I want to be able to run scraping code from command line to periodically scrape high-level data points from each of them either manually or via a cron job using a set/list of search term(s). The data is three levels down and has various data points to be retrieved, put is appropriate category/table.
Currently there is a recapta required for each search
Current steps for retrieving data is as follows:
1 - Select advanced filtering option
2 - Set search criteria requirements
3 - complete recapta
4 - click search
5 - Search results (list) are displayed
6 - Walk through each search result Click on an item in list and view results
7 - Grab data in each of 4 sections
Items in results are in 4 specific categories
There are 18 total items that are to be retrieved from the results from step 6 above (each data set)
A Non-Disclosure Agreement (NDA) will need to be executed prior to full details such as URL and specific data to be provided.