Web scraper -- 2

Main goal The software should mimic a human who logs in to a secure web site, navigates to a download page after successfully answering any security questions there might be and downloads the desired information being a file or the web page itself. Once the data acquired it has to be saved either in a file on the local hard drive or in a database. Details Create a Windows or Linux based software (preferable to support both) that will be able to scrape any information from a given web location. The web source might be protected with a user name and a password as well as with security questions that could change every time the URL is accessed so the program should be intelligent enough to detect the questions and answer with the right response. Once the source page containing the desired information has been reached, the software should be able to extract this information and save it into a file on a local hard drive or insert into a database. Specifics 1) In general, the program should be able to handle successfully: logging in to web sites having the user name and the password choosing the right response to a random security question download files through popup dialogues. download files by following URLs download data off a page. 2) The program has to be able to run in a batch mode without any user intervention and without a graphical interface, scheduled as a task on the Windows task manager or in cron in case it is running on Linux 3) The program has to be able to manage with the huge variety of possible scenarios for downloading information from the web and It has to be easy to configure e.g. via a conf file. I will provide a demo of one of the possible cases in a separate file 4) After a successful download save the file on the local disk or / and insert the data into a database thus the program has to be able to handle data sources as mysql, mariadb, postgres, etc. Code 1) The code should be clearly and well documented 2) The code should be written following the object model standards allowing easy scalability and further extendability. 3) All exceptions must be handled and a detailed logs produced when required for allowing quick and straightforward troubleshooting