I am looking for a developer(s) who can develop a web crawler or spider in PHP or using PHP frameworks.
The end-product will only require root domain names.. (eg [url removed, login to view]); which is stored in a table... and it will go through through it one by one.
The purpose of the spider is to finds all working links within that domain.
Would be nice to have a functionality that allows it to start at [url removed, login to view]\sub folderThe discovered links will be stored in a table.
It should knows if it fails on a site and continue where it left off...
Error logs if it encounters issue.
Should work on any domains.
Needs to be efficient.
The database used is mySQL.
The end product should work with php [url removed, login to view]
This can be billed as hourly, but I would need a quote on how many hours it takes and at what rate...
This can be on a project based also...
This is one of part of many development, would like to establish a long term relationship.