Lengkap

Developing a file scraper to scrape firmware files from various vendors websites

A python based CLI script that can download all product’s firmware (including all versions) from web pages for a given list of predefined vendors and store the information (meta data) in SQLite [login to view URL] mandatory metadata fields include ( Manufacturer, Model, Version, Type, Name, Release Date(if available), Download link ) i.e. ( Cisco, Video Surveillance 6030 IP Camera, 2.7.0, IP Camera, [login to view URL], 21/08/2015, "link" ) There is a non-mandatory binary field which indicates if the device is discontinued or not depending on the fact that vendor mention that on the website or not. The firmware files itself will be stored in the file system and will be referenced by index ID in SQLite.

The arguments to the script should be a list of comma separated vendor names or the location of a text file containing the vendor name.

There are no GUI components in the server where the script will run hence headless mode for browser should be used by the script

Solution Scope

1. Script will be written per vendor. This is required because each vendor website will have its own implementation of the firmware download page. However, efforts will be put to identify and implement reusable components, if any.

2. The script will only download new firmware that have been added by the vendor. Hence first execution of script will download all the firmware available but the subsequent runs will only download new ones which will get added. This will be achieved by analysing data available in SQLite and skipping the files that are already been downloaded and processed.

3. Each vendor, that will be provided, will be analysed manually to identify the following, which will be required to develop the script:

a. URL for the firmware download page

b. Credential Requirements (Simple Signups, Specific Signups, No Signups)

c. Any Captcha on the page

d. Any honeypot traps

4. If there are credential required to download the firmware and the credentials are simple ones where a simple sign up is required, the signup will be done manually as part of the manual analysis using a gmail account dedicated for this work.

5. Script will try to imitate human like behaviour (to a limit) while scraping the web page as well as uses Tor, so that if the vendor site has scraper/crawler detection logic implemented, it can be skipped. This will be achieved by adding random delays, random view time, avoiding honeypot traps through manual analysis

Solution Brief

A Python Selenium and SQLite based solution will be developed which will have the following features/components:

1. File Management Module: Responsible for storing and managing the downloaded files and meta data. Firmware and installer files will be stored on the filesystem which will have a structured folder hierarchy. Meta data of the files will be stored in SQLite. Meta Data will refer to the stored files through paths on the file system and file index/name.

2. Vendor Scrappers: Python Selenium based scrapper will be written for each of the vendor, responsible for downloading the files and grabbing the meta data from the vendor’s site. This will make use of the file management module to store the file and meta data to SQLite.

3. Configuration File: All the configurations for the framework (including vendor specific like credentials, url etc) will be stored in a json file which can be easily modified manually.

4. Execution Script: The configuration file can be setup to represent the polling interval for each of the vendor scraper and when the execution script is run it will go and schedule each of the vendor scripts individually according the polling interval defined in the config.

Deliverable:

1) Python Source Code including the comments in the code explaining each function & its details. We should be able to give any required input as an argument and execute it as one line command in the Linux terminal.

2) Dependencies

3) Manual to install, configure and use the scraper

Kemahiran: Python, Linux, Pengikisan Web, SQLite, Kejuruteraan Perisian

Lihat lagi: aspnet word file scraper, java file parser find text files, scrape mp3 files web page, file upload php large files, scrape pdf files, scrape php files site, vbs file copying large amount files, website file scraper, file upload perl big files, java scrape directory files, file upload handling separates files, scrape excel files, vbs file read values xml files, uplodad file various files various folders aspnet, php upload file using ajax large files, wget scrape website files, scrape media files mp3 mp4 from file directory

Tentang Majikan:
( 0 ulasan ) Schaarbeek, Belgium

ID Projek: #22076792

Dianugerahkan kepada:

Prutheus

Hello, I am Mr. Martin F, an experienced german developer. I do web scraper developing for years, this is why I can say that I would be the perfect guy for this project. I want to build several scraper for you, whic Lagi

€18 EUR / jam
(27 Ulasan)
5.5

21 pekerja bebas membida secara purata €21/jam untuk pekerjaan ini

helmot

I have +12 years of experience in Python programming. I reviewed your detailed project description. I'm expert in webscraping and have completed tens of webscraping tasks here on freelancer. How many vendors do you hav Lagi

€15 EUR / jam
(220 Ulasan)
8.2
soner

I can offer my high quality web scraping services to you. I have huge experience in web scraping area. Please contact if interested

€17 EUR / jam
(23 Ulasan)
7.9
etuannv

Hi there, I am interested in your project. I would approach your project by making a program to collect the data automatically. The program will be written in Python/Scrapy. It could be run on any operating system. I h Lagi

€15 EUR / jam
(92 Ulasan)
6.7
pixelonline

Hi there, We are a leading Mobile App development company from INDIA, Please check some of our projects here under, [login to view URL] <social media App> [login to view URL] Lagi

€44 EUR / jam
(7 Ulasan)
5.8
emizentech

Hello, We have 6+ years of experienced full-stack (Python, Django, Machine learning, Anaconda) developers team for your existing website projects. Also We have 8+ years of experienced JavaScript MEAN/MERN stack deve Lagi

€18 EUR / jam
(60 Ulasan)
5.8
smsaurabhv

Hi, I have gone through your requirement to scrape lots of websites. I am EXPERT in building scraping tools /scripts. Hence, I can SURELY work on your project. I am having 4 YEARS of EXPERIENCE in developing PHP-PYTHON Lagi

€16 EUR / jam
(76 Ulasan)
5.6
SecureForYou

Dear client. Thanks for taking your valuable time for reviewing my proposal.I have read your project description and I think I can be a candidate of your project. I have been confident and sure that I can complete your Lagi

€16 EUR / jam
(6 Ulasan)
5.2
mrlinh2710

Hello sir! As a Python expert, I am glad to place the bid on your project. As you can see in my profile, I am fully experienced and lots of skills in automation scripts. I want to discuss more via chat. Regards.

€12 EUR / jam
(13 Ulasan)
5.5
pandread1

Hi there, this is Pandelis, Im a Python developer and Web-Scraping specialist. Just finished a similar project that required scraping metadata in SQL DB, downloading documents for each record, storing them in an organ Lagi

€17 EUR / jam
(13 Ulasan)
5.0
Jng93Slv3

Hi! I'm interesting your project very well. I am a full time developer and i can work more than 50 hours in a week. And also i can work any timezone you want. I have more than 8 years experience about web development. Lagi

€15 EUR / jam
(7 Ulasan)
4.5
oswaldoodavidal

Hello, I have experience in web scraping with Python. I have read your project's requirements and I'm capable of delivering the Python scripts with the config files, instructions to install and dependencies. I can u Lagi

€15 EUR / jam
(4 Ulasan)
4.1
wangxiangyu1231

Hi... I read your brief carefully. I'm python expert and my last project on freelancer.com is also python scraping task. My past works: Youtube comment scrapping Real estate property list to csv Job-site content to csv Lagi

€15 EUR / jam
(9 Ulasan)
3.6
MuhammadRohailP

Hi I am an expert and have scrapped 100+ sites till date. I have worked with C#, Selenium, Curl and scrappy with Python to scrap sites and get perfect results. Best scrapped sites: [login to view URL], Microsoft Partners, and Lagi

€15 EUR / jam
(5 Ulasan)
3.6
dev7fares

Hello my name is Fares, I can get it done perfectly as you want let's chat when you are online, Thanks

€15 EUR / jam
(16 Ulasan)
3.6
KDC999

Hello. How are you? I am Meiling Cai, developer with long time experiences. I am skilled in web scraping with python, xpath, selenium, and BS4. I am confident in doing in time. I will use python scrapy to get data f Lagi

€15 EUR / jam
(9 Ulasan)
2.9
jjsanchezb

Hello. I am a software developer with strong habilities in scraper to web applications using automated browser solutions like selenium in python. Also, I have a strong knowledge of the Linux ecosystem in any distributi Lagi

€12 EUR / jam
(2 Ulasan)
2.5
NaeemRajde

Hello! I am very interested in your post project. While I read your description carefully, I was excited with feeling that I would be able to satisfy for your requirements in this job. We can negotiate on price/Budget Lagi

€15 EUR / jam
(2 Ulasan)
2.5
Aperseus

Hi! I have read your description and feel confident to fulfill your requirement. I have 9 years of C, C++, C#,QT , ASP.NET, Python, Arduino ,VB,Meta Trader,Raspberry Pi,Meta Trader, Block Chain..... Programming Experie Lagi

€100 EUR / jam
(0 Ulasan)
0.0
viniciuscunha3

Nice to meet you Hi I am automation expert If there is any website you need to scrape data from (no matter the format, what blocking tools there are). I can deliver the data. If you do the same thing on the internet Lagi

€17 EUR / jam
(0 Ulasan)
0.0
liamstevens

Hi there, I have worked with creating custom web scraping solutions in the past to suit varying webpage requirements, specifically with Python (multithreaded). Can provide link to Github repository showing an example Lagi

€17 EUR / jam
(0 Ulasan)
0.0