Expired Domain Web Crawler Needed

$30-250 USD

Ditutup

Disiarkan

hampir 5 tahun yang lalu

$30-250 USD

Dibayar semasa penghantaran

I am looking for a software application that can crawl a list of websites for expired domains (any domains with a status of 404). The crawler should crawl a list of inputted names via a source text file. The source text file would look like this, 1 domain name per line: [login to view URL] [login to view URL] [login to view URL] [login to view URL] Once software grabs the source domains, it should fully crawl each site and report to a csv file any expired domains it finds. The format for the csv should be: domain where found, URL where found, expired domain An example line would be: [login to view URL],[login to view URL],[login to view URL] The crawler should use just one thread, and for the user agent, should just be chrome if that is possible. I plan to do a phase 2 later where the crawler will check an API to see if the expired domains are available for purchase, and also to add threads, proxies, and other features. This will all be phase 2 and 3. Right now, I just want phase 1 completed where it takes an input of "[login to view URL]" and then for domains it finds in 404 status, it should output "[login to view URL]" This needs to be desktop based running on Windows 10 Pro. It can run in Powershell or Python, and other languages are fine as well. For continued development, it would be beneficial to have it in Python, but any language you prefer is fine. Please include these with your bid: 1. Project Price in Full to do phase 1 2. Time to complete, since I am really needing this soon 3. What language you will be using 4. Any relevant experience you have with crawlers

Web Scraping

ID Projek: 20031872

Tentang projek

20 cadangan

Projek jarak jauh

Aktif 5 tahun yang lalu

Ingin menjana wang?

Alamat e-mel

Faedah membida di Freelancer

Tetapkan bajet dan garis masa anda

Dapatkan bayaran untuk kerja anda

Tuliskan cadangan anda

Ianya percuma untuk mendaftar dan membida pekerjaan

20 pekerja bebas membida secara purata $166 USD untuk pekerjaan ini

@e3d

Hi, I can make this crawler for you, but I actually already providing my clients with expired domains with backlinks found on sites of their choosing. also for the architecture that you've outlined you'll need lots of mem or hard limit on number of urls per site.

$250 USD dalam 3 hari

5.0

(259 ulasan)

8.7

@zekovicm

Hi there,I am Web Scraping expert from Bosnia & Herzegovina,Europe. I have created similar crawler in the past for checking expired domains . I can do it again for you ! Check out my profile and former clients feedback - that'll let you know everything about me. Please feel free to contact me so that we can discuss further details. Thank you for taking the time to read my proposal.I am looking forward to hearing from you. Best regards, Miljan

$220 USD dalam 5 hari

4.9

(137 ulasan)

7.6

@mtanveer2012

Hi, 1. Project Price in Full to do phase 1 Ans: 150 USD. 2. Time to complete, since I am really needing this soon Ans: 1-2 Days maximum. 3. What language you will be using Ans: Python. 4. Any relevant experience you have with crawlers Ans: My profile is full of crawling jobs, I have more than 5 years of experience.

$150 USD dalam 1 hari

4.9

(79 ulasan)

6.2

@schoudhary1553

Hello, I have gone through your job posting and become very much interested to work with you. I am an expert in this field. I have already completed several projects like this. For evidence you can see my profile. Please visit : https://www.freelancer.com/u/schoudhary1553 I have excellent command over English. I am a hard worker, productive and worthy of your attention I hope, I would be the right candidate for this post. Awaiting an affirmative response from you. Kinds Regards, Sandeep

$250 USD dalam 4 hari

5.0

(25 ulasan)

6.1

@brianconey

Hello How are you I have read your job description in detail . I am Xu It is a piece of cake i have done it before Please contact me and Let's discuss more thanks for your posting . hope to hear from you .

$140 USD dalam 7 hari

5.0

(17 ulasan)

5.6

@smsaurabhv

‌Hi, I have gone through your requirement to scrape lots of websites. I am EXPERT in building scraping tools /scripts. Hence, I can SURELY work on your project. I am having 4 YEARS of EXPERIENCE in developing PHP-PYTHON (Scrapy, Selenium) based web scraper as well as WINDOWS BASED web scraping software through which I have crawled many sites such as Craigslist, Amazon, Yelp and many others. I have also worked on complex site to bypass CAPTCHA with the use of PROXY IP bouncing techniques.. Let's work together :) Have a great day! I am glad to see your WORK HISTORY and positive reviews of other freelancers. I am really excited to work with you and would love to have a long-term business association for any of your data related needs less ,,,,,,,,,,,,. , ,,,,,,

$111 USD dalam 3 hari

4.9

(100 ulasan)

5.9

@writingapp

Hi. Great app writer for your projects. I have writen scraping app for many years. I am ready to write your project. Thank you for visiting my profile

$140 USD dalam 5 hari

4.9

(36 ulasan)

5.2

@shiyusun180409

I checked your description. As I am a software and scrap expert , I have a wealth of experience and skills in these projects. I prefer python language and i will complete it within 1~2 days. You will get good results. I hope to work with you. I will always wait for you. Thank you.

$150 USD dalam 2 hari

5.0

(18 ulasan)

4.7

@alexwmsoft

Dear Client Thank you for taking for your precious time to looking for my bid. I have fully read the job description and it is very clear to me. I am a senior web scraper who is good at python. I am using python to scrape data from the web site. Python selenium is very safe and lxml is faster than selenium. and I can also scrape by using PHP(cURL). I am free now. so I can start your project immediately. Please give a chance to work for you. Then, I will try my best to finish your project. I hope to discuss more details via chat. Looking forward to hear from you.

$155 USD dalam 3 hari

5.0

(5 ulasan)

4.5

@sky930320

Dear Sir. Glad to meet you. I'm Web developer specializing in web scraping crawling and indexing web pages. Skills: python, scrapy, selenium, requests, beautifulsoup, mechanize, lxml, urllib2, automation, bots, spiders I worked with 50+ differenct individuals/startups/companies and helped them archive their goals. Here's what you will get by working with me: * Working only on projects that I can deliver 100% * Ridiculous quality of work * Smart suggestions on increasing results/revenue and decreasing costs * Responsibility for tasks/project I work on * Reliability * Short response time * Good sense of humor ;) Hope I could awake you interest in me. Thank you for your attention. Shuaixing.(call me Shuaixing L.)

$250 USD dalam 7 hari

5.0

(9 ulasan)

4.2

@dhirajbaheti

Hi, My name is Dhiraj and I am a professional web scrapper and data mining expert in python with more than 6 years of experience. I have expertise in scraping complex/dynamic websites using requests, BeautifulSoup, Scrapy, proxy rotation, user agent rotation. I will be a able to build a windows desktop application using python which can crawl all your domains to look for expired domains. Also, yes all your requirements of single thread and chrome user agent can be easily implemented. It should not take me more than 1 day to give you the application as I had previously worked on similar project so I need to customise it as per your requirement. Please contact me for further discussion. Major highlights of my work are developing scrappers for Amazon, flipkart, wikepedia, google and bing search engines, linkedin, Airbnb, Yellopages, Justdial Airport timetable scraping etc.

$150 USD dalam 1 hari

5.0

(7 ulasan)

3.7

@drohanyadav

Hi, I will be using python language to do this project. I work fast, and I will probably finish it in few hours, but there are always some small complications, so it will be take me maximum of 2 days. I have been web scraping since 4 months now, and I have scraped lots of different kind of websites. My approach for this task is to first check the website response. If it's 404 then we can stop there. If it doesn't say that then the program will get all the text that is presented on the webpage and then look for any terms which can tell us that it's an error. I think this should work general. Please contact me if you are interested

$60 USD dalam 2 hari

4.9

(11 ulasan)

3.7

@BestService222

Hi I've carefully read the description of your project. I am very interested in your project As I am a professional Crawler Developer, I am sure I can help you I'm ready to start to work on your project right now. Please message me and let's discuss details. Thank you for your kindness.

$140 USD dalam 3 hari

5.0

(1 ulasan)

0.8

@alif121

Hello, I have 3 years of experience with Python and web scraping. I am very interested in your job and I can do this.

$140 USD dalam 7 hari

0.0

(0 ulasan)

0.0

@sevenstarservice

Hello, I'm a full time freelancer and am free now. I have read your project description I am very interested in this project. I have a lot of experience in this type of work. I'm sure I can complete it on time with high quality according to your request please get in touch so that we can discuss more and I can start to work immediately. I'm always ready.

$250 USD dalam 3 hari

0.0

(0 ulasan)

0.0

@sanidhyarajawat

1. $100(Negotiable after discussion). 2. 3-4 days(Even early if you pay for extra hours). 3. Python. 4. I have written scrapers and crawlers to get the data from various sites. I am a professional Python developer and I can complete this task within timeline and in less budget. I have more than 5 years of experience in this industry and I am very good with Linux systems. I also have experience in working with JavaScript along with Python. Please feel free to text me and we can discuss on this further.

$100 USD dalam 3 hari