A web crawler that can pick up JS onclick events

I need a web crawler that can find links on a page and list them. Even links that are hidden by javascript onclick events.

It must

1) log the status code of the url given and any urls redirected through - example if given a url that redirects to another url with a 301 status code I need the 301 code and the 200 that it redirects to.

2) List the urls in a redirect chain if there is a chain.

3) Get all the links on the page given even ones hidden in onclick divs or other methods.

4) list all the rel, anchor text and image url elements for each link if they exist

5) follow redirects if required by meta redirects or [url removed, login to view] and list the urls in the redirect

6) We need to be able to run this from command line on a linux machine. I don't care too much what language but we need to be able to use it with php. Previously we were running HTML unit through shell_exec in php and then capturing what was echoed to the command line. Continuing like this is fine.

We had some luck with HTML unit but we have not got enough experience to get all our requirements.

Kemahiran: Pengikisan Web

Lihat lagi: web-crawler, what is a web, what is a crawler, hidden web, scraping crawler, pick 3, web scraping image, web scraping linux, web redirect, redirect status code, command web, find hidden url, crawler javascript, scraping experience, javascript image onclick, window events php, web scraping code, html unit, example web scraping, crawler code, scraping web, image onclick javascript, methods events, web crawler html, javascript links given url page

Tentang Majikan:
( 0 ulasan ) Stockport, United Kingdom

ID Projek: #4058819

Dianugerahkan kepada:


I have lots of experience with writing web automation software, please see PMB for examples of my previous projects related to web automation. Available to start immediately and finish as soon as possible. Best Rega Lagi

£500 GBP dalam 10 hari
(20 Ulasan)

8 pekerja bebas membida secara purata £390 untuk pekerjaan ini


I can help in your project, please check PMB and our ratings/reviews to get idea of our experience. Please let me know if you have any queries.

£250 GBP dalam 3 hari
(32 Ulasan)

I look forward to discuss further and can deliver the project

£520 GBP dalam 12 hari
(29 Ulasan)

Hi, Ready to start your work. Eagerly awaiting for your positive reply. Please check your inbox for further details. Thanks, Shaik.

£250 GBP dalam 5 hari
(25 Ulasan)

Hello, I can do this work for you and I'm ready to start. Please see pmb for details. Regards Raul

£250 GBP dalam 7 hari
(11 Ulasan)

Hello,Understood your scraping [url removed, login to view] check pmb for [url removed, login to view]

£250 GBP dalam 7 hari
(2 Ulasan)

Scrapping/Parsing/Automated engine Experts here. Check the message with attached samples and contact us. SI Team.

£750 GBP dalam 10 hari
(2 Ulasan)

Hi, Please check your PMB regards, Arun

£700 GBP dalam 5 hari
(0 Ulasan)

I'm an expert Webbot, Netbot creator and a Professional webscraper. .NET/C# My webscraping skills can be found at [url removed, login to view] I'll scrape any data from any website.

£400 GBP dalam 5 hari
(0 Ulasan)