Scrapy, find and fix a bottleneck: low throughput


The scraper is based on Scrapy and it's almost complete, we are scraping products from AMZ (Italy at the moment) but we are struggling with overall requests throughout.

We are using backconnect rotating proxies from 2 vendors and we have about 150 threads but we are able to scrape only 1/2 URLs per second, in addition we can use more threads/proxies with TOR but the speed is the same.

Kemahiran: Pengikisan Web, Python, Kejuruteraan Perisian, Perlombongan Data, Scrapy

Lihat lagi: php read html fix mistakes, read display shapefile use java, clipshare description fix, description fix joomla, php bug fix job description, read description, read description match, javascript read description, read description match photos, fix wordpress google site description, python read description file, read description browsergame, product read description write objective, hotel description english use, read description bidding, read description twitter, tbulit 3 features in core php website budget is 60 fixed read description, tbulit 3 features in core php website budget is 60$ fixed read description

Tentang Majikan:
( 12 ulasan ) Cagli, Italy

ID Projek: #21115494

9 pekerja bebas membida secara purata €11/jam untuk pekerjaan ini


Hi there, I may give you some suggestion about proxy. I have 4+ years’ experience at Web scraping. If you'd like to view my previous work, take a look at my profile https://www.freelancer.com/u/etuannv or check some de Lagi

€12 EUR / jam
(86 Ulasan)

Hi, Client!! @$@$@$PLEASE CONTACT ME. I CAN DO IT WONDERFULLY.$@$@$@ ##% Your idea is attracted me to your world.%## I am Python expert. i have various skills. @@PLEASE GIVE ME CHANCE TO PROVE MY SKILLS@@ :-):-):-) Lagi

€15 EUR / jam
(45 Ulasan)

Hello, I am a professional web scraper with more than 6 years of experience and have completed many scraping tasks successfully with client satisfaction I can scrape any kind of website for you and can output data i Lagi

€12 EUR / jam
(5 Ulasan)

Dear Sir. Glad to meet you. I'm Web developer specializing in web scraping crawling and indexing web pages, data entry. Skills: python, scrapy, selenium, requests, beautifulsoup, mechanize, lxml, urllib2, automation, Lagi

€9 EUR / jam
(10 Ulasan)

Dear Sir Nice to meet you. I am interesting in your project. I can build your project perfectly with my experience. I am full time developer and my goal was always delivery perfectly results on time I will do my best Lagi

€10 EUR / jam
(3 Ulasan)

Hi. Please survey my reviews. I saw your description carefully and I am confident I can full fill your requirement. So I provide my skills in hopes of working with you. I am full-stack developer and I have 2+ years Lagi

€10 EUR / jam
(4 Ulasan)

Hi there Hope this finds you Hale and Hearty I am a python developer, having more than 5 years of experience in Machine Learning, Deep Learning, and Data Science engineering. With my good experience in core python, I c Lagi

€10 EUR / jam
(2 Ulasan)

Hello There Good Day! I've carefully read your project description. I am very interested in doing this project. I am a professional Scraping expert. I will give 100% guaranteed satisfaction from my past experience Lagi

€9 EUR / jam
(2 Ulasan)

⭐LOW RATE! HIGH QUALITY&SPEED!⭐ I read your project . Your project is very interesting. I have ever been similar project with your project. I have already finished many project like yours and I have enough experience i Lagi

€9 EUR / jam
(0 Ulasan)