Urgent: Scrapy sitemap parsing gig

I have a huge list of domains that we need to parse to get all of the sitemap data out of.

I’ll provide csv of all the domains. You might need to normalize them (checking http/https protocol) and check www or not.

We need two outputs:

Summary csv with the following

Proper url to the sitemap | total pages in sitemap | list of dates for the last year and count of pages updated on those dates.

So the csv will have 367 columns

Next output I need

You can hit the sitemap for each site and dump to csv a file per domain. The csv should have the sitemap data in it.

Url / modified

I have about 160k domains that we need to process for this.

I’ll provide you a Ubuntu Aws machine to run your solution on. Thinking scrapy or similar running for a few days.

To apply for this job your proposal must include the following

1- questions

2- what framework will your solution use?

3- ballpark how much time to get the solution running?

4- how many domains per 10 sec do you think we can process?

Kemahiran: Python, Pengikisan Web, Kejuruteraan Perisian, Linux, Scrapy

Lihat lagi: mysql modify sql dump file, dump file, php script update database dump file, examine dump file, examining dump file, plesk error dump file retrieving info filefinder, mssql dump file bak, windows mini dump file, cut mysql dump file, dump file location examine, split mysql dump file, sql dump file splitter, rebooting dump file, convert oracle dump file csv, spliting mysql dump file, split database dump file, windows dump file dmp, urgent software engineer web developer 5 days east 2.3 k 2.8 k ref ky jobs in singapore

Tentang Majikan:
( 460 ulasan ) Austin, United States

ID Projek: #22451100

31 pekerja bebas membida secara purata $36/jam untuk pekerjaan ini


Hello there We are top quality full-stack developers and we are ready to work on this project, we use Version Control Systems, Staging Servers, Team Slack Channel and Task Management Tool Can you send me a message? T Lagi

$40 USD / jam
(95 Ulasan)

Hi there, I am interested in your project. 1- Plz send me a sample data 2- what framework will your solution use? ==> Python3.7/Scrapy 3- ballpark how much time to get the solution running? ==> about 48 hrs 4- how ma Lagi

$38 USD / jam
(94 Ulasan)

Hello, I have gone through your job posting and become very much interested to work with you. I am an expert in this field. I have already completed several projects like this. For evidence you can see my profile. Pl Lagi

$25 USD / jam
(66 Ulasan)

I can start work right now and I can show you perfect result in a short time. Please contact me freely. Waiting for you with your great news.

$38 USD / jam
(72 Ulasan)

Hi, I have gone through your requirement to scrape lots of websites. I am EXPERT in building scraping tools /scripts. Hence, I can SURELY work on your project. I am having 4 YEARS of EXPERIENCE in developing PHP-PYTHON Lagi

$33 USD / jam
(81 Ulasan)

Hi, I am good in your required project; I also have a great working experience of more than 10 years. To ensure please visit my profile and check customers satisfaction level. I will complete your project within your Lagi

$25 USD / jam
(44 Ulasan)

Hi there! I am interested to do this project for you.'' 1- questions Ans: Please send me atleast 5 different url so i can check 2- what framework will your solution use? Ans: Scrapy will be best for this 3- ballpark h Lagi

$25 USD / jam
(22 Ulasan)

Hello I am working in Scripting technologies and administration for very long of 20+ years. I worked in various modules in Python includes numpy, scipy and pandas for Bigdata. Worked in both Django and Flask framework. Lagi

$40 USD / jam
(26 Ulasan)

Hi, I would love the opportunity to work on this project with you. I have vast programming experience, recently specialising in MetaQutoesLanguage, but i have previously deployed several Python programs commercially. Lagi

$45 USD / jam
(13 Ulasan)

Hi. I have writen a similar app but for windows. I am ready to write your project 1- questions? Can you run it in windows? 2- what framework will your solution use? .NET 3- ballpark how much time to get the solution r Lagi

$50 USD / jam
(24 Ulasan)

hi i saw you your post regarding web [login to view URL] scrap site map link we can fetch from [login to view URL],and use that url response and parse it in sitemap parser i can make a scraper for you maybe i can add a feature like scra Lagi

$27 USD / jam
(10 Ulasan)

Hi I have carefully read your job description with great interests. I have experience with python, django,selenium(scrappy framework) and bs4(beautifulSoup) for about 5 years. Please visit my past references here. Lagi

$38 USD / jam
(4 Ulasan)

Hi. I've checked your project description and I'm interested in your job. I fully understand your requirement. I'm very skilled with: JS frameworks & libraries like Angular, React, Vue; PHP frameworks such as Laravel Lagi

$30 USD / jam
(2 Ulasan)

Hello! I have worked on a few web scraping projects in the past, some with VB and some with scrapy. To answer your questions in order: - I would use Python 3 with scrapy, and possibly modules like urllib to sanitise t Lagi

$25 USD / jam
(11 Ulasan)

Software Engineering Guru awarded Bachelor's Degree in Computer Science and Technology, I am. Having checked the requirement of this project, I can notice that these types of projects are very familiar to me. I am read Lagi

$38 USD / jam
(4 Ulasan)

Hello. I have just reviewed your job description carefully. ALL SKILLS you need have never been problem for me. Anyhow, I can solve any problem there as I have long years experience in web development. I'll be great Lagi

$38 USD / jam
(7 Ulasan)

Hi. Dear I read your job description in detail and feel I can help your project. I have full experience and skills for the python. I have done the many projects as same as your project with Flask, Django project and M Lagi

$25 USD / jam
(4 Ulasan)

Dear Client! I'm a senior web developer with over 5 years of experience and very strong in this field. I can complete your project as you want. Please check my portifolio: scrapy:[login to view URL] Lagi

$25 USD / jam
(2 Ulasan)

Hello 1- questions : please give me at least one site link and csv file for all domains. 2- what framework will your solution use? : Core PHP / DOMDocument Parser, Python scrapy framework 3- ballpark how much time Lagi

$40 USD / jam
(1 Ulasan)

Greetings. I am an expert in software architecture. I have rich experiences in machine learning, AI, image processing ,openCV and google apis and extensions. I have many experiences in programming languages such as c Lagi

$38 USD / jam
(2 Ulasan)