Ditutup

An expert developer needed to develop firmware files and metadata scrapers for various vendors websites

This project will develop set of crawlers based on Scrapy framework that can download and synchronize all of products' firmware (including all versions) from web pages of a given list of predefined vendors and store the firmware information (meta data) in PostgreSQL DB. Final number of crawlers would be ~100 and project milestones are defined per vendor and each milestone is max 65€ which is paid after we verify the completeness of each crawler and see no errors.

The mandatory metadata fields include (Manufacturer, Model, Version, Type, Name, Release Date (if available), Download link, (calculated Sha2 hash of the file)i.e. ( Cisco, Video Surveillance 6030 IP Camera, 2.7.0, IP Camera, [login to view URL], 21/08/2015, "link", “Sha2” ). There is a boolean field which indicates if the device is discontinued or not depending on the availability of such information on the website of the vendor. The firmware files itself will be stored in the file system and will be referenced in PostgreSQL.

The developer is required to extend an existing scraping framework that was partially developed based on Scrapy framework and follow DB schema and code templates provided by us. It's also the responsibility of the developer to test crawler and ensure completeness of the solution in terms of full coverage of the firmware files and product pages. There are no GUI components on the server that runs crawlers. Therefore, headless browsing mode should be used.

Project Scope

1. Crawlers will be written per vendor. This is required because each vendor website will have its own implementation of the firmware download page.

2. The user should be able to pause and resume crawling jobs.

3. Crawlers should detect previously downloaded files and only download updated and new content and firmware files. At first execution of each crawler, it will download all the available firmware files but the subsequent crawler runs will only download new firmware files which are added since the last crawling.

4. The developer is required to manually analyze each provided vendor site before writing a crawler to identify the following required information:

a. URLs for the firmware download page including all of the firmware versions for each product

b. URLs/files for each product that include the following information, required to be scraped: "Manufacturer", "Model", "Version", "Type", "Release Date", "if the product is discontinued"

c. Credential Requirements (Simple Signups, Specific Signups, No Signups)

d. Any Captcha on the page

e. Any honeypot traps

5. If a vendor site requires credential for firmware download, the developer is required to sign up an account using an email address dedicated for this project

6. Script will try to imitate human like behaviour (to a limit) while scraping the web page as well as using Tor if required

Important Notes

The developer MUST test the completeness of each crawler before delivering to us and present test completion evidence in the form of a populated PostgreSQL database of that vendor.

*An NDA and a contract must be signed before the beginning of the project. A copy of the developer's identification document is required to verify the identity.

*Please apply just when you fully read and understand the project and agree with the conditions.

Kemahiran: Pengikisan Web, Scrapy, Python, Linux, Docker

Lihat lagi: qbfc freelance developer needed, iphone developer needed, softwares needed develop net application, expert artist needed, outsource developer needed, cakephp developer needed, php oscommerce developer needed, ajax developer needed, flash games developer needed, expert developer needed, expert php developer needed, expert android developer needed immediately, I need an android developer to develop a learning app for school children, expert magento developer needed, Need an expert JS developer to develop webviews and modifying a web bot, how to become an expert software developer, how to be an expert web developer, how to be an expert developer, how to be an expert android developer

Tentang Majikan:
( 4 ulasan ) Brussels, Belgium

ID Projek: #27844582

17 pekerja bebas membida secara purata €6912 untuk pekerjaan ini

IDS008

Hi! I am a Python expert with over 5 years experience dealing with Software, I can help working with your projects. I've previously worked on the exact same project for another employer. Web Automation with Selenium, B Lagi

€6000 EUR dalam 70 hari
(12 Ulasan)
4.9
yanakhokhlova199

Hello I agree your proposal with 65€ per one, I have rich experience in python and django for webscraping platform developent. Please discuss in details and share target urls. Thanks

€6500 EUR dalam 30 hari
(3 Ulasan)
4.3
drishinfotech

Hello I have read the job description carefully. I am ready to sign NDA and will fulfill other formalities. I can create the crawler for each vendor but before all these , I need to check the feasibility to trespass t Lagi

€9000 EUR dalam 7 hari
(12 Ulasan)
4.8
Alexod

Hi. I can build all crawlers you need. I'm doing such job since 2010. I have done many crawlers. I have many skills and I can provide it for sure !

€7500 EUR dalam 7 hari
(5 Ulasan)
3.8
tartas2

Hi,I can create any kind of scrapper that meets your needs.I read your description and I think it is very easy to be done ASAP.I will provide you a sample [login to view URL] contact me on chat so we can discuss more about th Lagi

€5000 EUR dalam sehari
(4 Ulasan)
2.8
marinaparshina05

Hi I have just read your description in detail so I understood what you want fully. I have 5+ years of experience in this field and I feel very much suitable for your project. It's possible to your all proposals to me. Lagi

€7000 EUR dalam 30 hari
(1 Ulasan)
2.4
Puneeths1996

Hi, My portfolio site, [login to view URL] My Github site: [login to view URL] Please leave a message, would be interested in hearing about the requirement and the objective th Lagi

€7500 EUR dalam 7 hari
(2 Ulasan)
2.1
harlamovi

Dear Sir. I understood all the scope of project and agreed the conditions. I am a senior software developer/System Developer/Security Expert with 10 years experiences. Also I have enough knowledge about SHA, PostgreSQL Lagi

€5000 EUR dalam 15 hari
(2 Ulasan)
1.6
jonnathanceballo

Hello! I am happy to put my bid on your project. I have read your requirement and I noticed that I am appropriate to this project. As a skillful software developer, I have rich experience with C# .net, python web scrap Lagi

€7500 EUR dalam 7 hari
(1 Ulasan)
1.2
Darwish001

hi, i have rich experienced of C/C++, Python so that i can give you the perfect result. C/C++, Python is my top skill and i am majoring it for over 10 years. i have completed the several projects already so i have con Lagi

€7500 EUR dalam 7 hari
(0 Ulasan)
0.0
nalliancetech

Narinder Alliance Technologies LLC An IT Consulting and Software Development company. We have a team expert in Web Designing, Application Development and Databases. Major offerings include Digital Transformation, Appl Lagi

€10000 EUR dalam 120 hari
(0 Ulasan)
0.0
sstarcorp

Hi, Its our routine job to do web-scrapping to collect data to build AI based use-cases. We readily have multiple web-scrapper which we can give demo to you. We are a team ofFull Stack, data science and ML/AI exper Lagi

€7500 EUR dalam 45 hari
(0 Ulasan)
0.0
mobiusteknologi

I read your project detail carefully, and fully understood what you want. i am an expert of web scraping, especially Scrapy. You can test me, else i can make the demo task. I want to work with you. Let's discuss . Lagi

€5000 EUR dalam 30 hari
(1 Ulasan)
1.5
arslanimt

Hello! As a passionate full stack developer, I will make your needs to be done in a no time. We may discuss details via chat time by time to make the solution fit your needs well. I will try to deliver good result only Lagi

€9000 EUR dalam 60 hari
(0 Ulasan)
0.0
shreeramy2020

Greetings!!! I've come across the project requirement that matches my skills and experience and I'm ready to work part-time or full-time with your company. I have a strong 5 years of experience with back-end technologi Lagi

€7000 EUR dalam 7 hari
(0 Ulasan)
0.0
Shadab262

hello sir, am cheeked plz come inbox for more discussion jawed,[login to view URL] ------------------------------------------ MY HANDWORK BACKEND & FRONTEND BOTH ARE (Design & development ) [login to view URL] Lagi

€5000 EUR dalam 7 hari
(0 Ulasan)
0.0
dinoknowsbetter

Greetings, I specialize in building professional bots and websites with reliable and turnaround times. I run a team of experienced and professional developers striving to take on new projects and deliver ready to roll Lagi

€5500 EUR dalam 10 hari
(0 Ulasan)
0.0