
Cancelled
Posted
Paid on delivery
The goal of this project is to create a script, tool or program that is able to check if over 1 Million URLs are working or not. The URLs are stored in a MySQL table with the following fields: url_id, URL, status, massage, and last_check. The field status stores the HTTP Status Code. A message can be an optional field to store an error message if the URLs is invalid or not exist. The last_check contains the timestamp of the last URL check. The script/tool should use multi-threading to check URLs. The most important thing is to have a very fast tool to be able to check millions of URLs in a reasonable time. For example linkcheck 2.0.6 checks over 10,000 links within a few seconds. That should be our reference. The script/tool should be able to run on a web server automatically (cron) and by a single trigger. In that case, url_ids which should be checked will be passed to the script/tool. You are free to create this script/tool with python or any other programming language that will meet the requirements. Here are some reference implementations or similar tools: [login to view URL] [login to view URL] [login to view URL] Attached you'll find a CSV file with 1 Million URLs as an example for testing.
Project ID: 19467646
22 proposals
Remote project
Active 7 yrs ago
Set your budget and timeframe
Get paid for your work
Outline your proposal
It's free to sign up and bid on jobs
22 freelancers are bidding on average $159 USD for this job

Hi, How are you i have done scrapping almost on Half of Worldwide web including eCommerce giants (Amazon, eBay, craigslist) News Feed, Social media websites, API's. I develop my own tools based on client requirements with Multi-threading, a Bot with human behavior and Scrapping Applications with documents parsing. I Can do PDF Parsing and Capctha ByPass code as well. Contact me for further details. I have developed over 100+ Bots and Tools for my clients and made sure they got their data. I normally work with Python or C# Not convinced yet let me have your questions. Thanks Why choose me Experience: Have been very diverse in technology, learning new things and always ready for challenges. Attitude: Always looking at bright side, focus on getting things done and target towards goal. Passion: Never afraid to take risks and do everything with my 100% Ownership. Communication: I possess good English speaking as well as writing skills and honest in work.
$125 USD in 7 days
9.4
9.4

Hi I can implement a multi-threading desktop tool that can run in "Windows". The software can host on a windows VPS. Thanks
$300 USD in 2 days
7.9
7.9

Hi there,I am Automation expert from Bosnia & Herzegovina,Europe. I have carefully gone through with your requirements and I would like to help you with this project ! I can start immediately and finish it within the agreed deadline. Check out my profile, portfolio and former clients feedback - that'll let you know everything about me. Please feel free to contact me so that we can discuss further details. Thank you for taking the time to read my proposal.I am looking forward to hearing from you. Best regards, Miljan
$350 USD in 7 days
8.1
8.1

Hello, I have gone through your job posting and become very much interested to work with you. I am an expert in this field. I have already completed several projects like this. For evidence you can see my profile. Please visit : https://www.freelancer.com/u/schoudhary1553 I have excellent command over English. I am a hard worker, productive and worthy of your attention I hope, I would be the right candidate for this post. Awaiting an affirmative response from you. Kinds Regards, Sandeep
$250 USD in 5 days
7.5
7.5

Hi, I am an experienced python developer and I can write a python script to check 1 million urls to check for availability. It will use multi threading so the result will be very fast. I can complete this task in 4 days.
$150 USD in 4 days
6.2
6.2

can build distributed horizontal scaleable high multithread system with python. will need high cpu and ram server configuration. can check the validation if URL can explain more about architecture have done scraping of millions of products from Amazon Walmart eBay These are my skills related to web scraping and crawling Have done scraping in CasperJS Phantomjs, python. Have done testing and automation with selenium also. Know to deal with database like mongo, mysql, Elasticsearch. Also know to handle proxy and captcha while scrapping.
$250 USD in 3 days
6.2
6.2

Hi, I have gone through your requirement to scrape lots of websites. I am EXPERT in building scraping tools /scripts. Hence, I can SURELY work on your project. I am having 4 YEARS of EXPERIENCE in developing PHP-PYTHON (Scrapy, Selenium) based web scraper as well as WINDOWS BASED web scraping software through which I have crawled many sites such as Craigslist, Amazon, Yelp and many others. I have also worked on complex site to bypass CAPTCHA with the use of PROXY IP bouncing techniques.. Let's work together :) Have a great day! I am glad to see your WORK HISTORY and positive reviews of other freelancers. I am really excited to work with you and would love to have a long-term business association for any of your data related needs less ,,,,,,,,,,,,
$100 USD in 3 days
6.1
6.1

Hi, sir. am Gavrichkov. I have read carefully your requirements. I have the best skills and much experiences in building Python automation tools for Scraping. I have already completed so many projects and get good reviews from employers. Now I have some powerful tools which I have done recently. I am sure I can complete this project within 24 hours. I have always good communication with you. If you award this project to me, then you will get a great result! Looking to be your Good Venture along with your project! Regards.
$98 USD in 7 days
4.9
4.9

Hello, Hope you will be doing well!! I have read your description and I understand your requirement and I can handle your project. Please discuss with me for more clarification. I will do my best for you. Thanks
$155 USD in 3 days
4.2
4.2

Hi, I'm java developer with 3 years experience. I worked a lot with different projects and have a lot experience with such things as multithreading and databases (H2/SQLite/MySQL primary). Basically Java is crossplatform language so you can run it on unix systems without problems. I can also include own cron tasks in the app configured via json (so you dont need to create yours) if you want of course. I'm confident to solve this task in 1 (at maximum 2) days and produce good results. Good code quality, performance and support guaranteed. Feel free to contact me for further discussion. I'm interested. Thanks.
$60 USD in 1 day
3.2
3.2

I am a professional python developer and I can complete this task within timeline and in less budget. I have more than 5 years of experience in this industry and I am very good with Linux systems. I also have experience in working with JavaScript along with Python. Please feel free to text me and we can discuss on this further. If the job’s no longer available, I’d appreciate you throwing my hat into the ring for any similar opportunities in the future. Here is my url: https://www.freelancer.com/u/sanidhyarajawat Thank you for your time, I look forward to hearing from you soon
$80 USD in 3 days
3.0
3.0

Hello, I have 3 years of experience with Python and web scraping. I am very interested in your job and I can do this
$140 USD in 7 days
2.2
2.2

Hi, i've been using python for 6 years now .. my main field of expertise is in web scraping/task automation /db manipulations. this project looks like a nice challenge , as in have alot of experience in web interactions using python, i mainly use requests,urrlib,bs4... i can do all of the project descriptions in 12-36h . If quick,cheap and effective is what you're looking for , then i'm your guy ! P.S: message me anytime i'm not a bot lol
$80 USD in 0 day
2.6
2.6

I am an experienced developer/designer working in the industry from last 10 years, Now I am a full-time freelancer. I have also an expert team of Developers, Designer and SEO . I believe in transparency in costing and customer satisfaction on project delivery. Please initiate a chat for further discussion on the requirement, quote, and timeline. We look forward to your response. We will be happy to assist you! Best Regards, Mike
$200 USD in 4 days
0.0
0.0

Hi I am here new. I have 5 year experience about python web and desktop bot making and data analysis . I have done most complex soft Craigslist lead generator software
$140 USD in 7 days
0.0
0.0

Hi have experience doing a lot of scrapings but am really interested in your project sir/ma. And i assure you that you wont regret working with me.
$70 USD in 2 days
0.0
0.0

Esslingen am Neckar, Germany
Payment method verified
Member since Mar 25, 2019
€250-750 EUR
$250-750 USD
€30-250 EUR
$10-30 USD
$30-250 USD
$2-8 USD / hour
$1500-3000 USD
£20-250 GBP
$750-1500 USD
$500-1000 USD / hour
$250-750 USD
$2-8 USD / hour
₹1500-12500 INR
₹37500-75000 INR
$15-25 USD / hour
£10-15 GBP / hour
$30-250 USD
$30-250 USD
$30-250 USD
$30-250 USD
₹1250-2500 INR / hour
₹12500-37500 INR
€12-18 EUR / hour