Python windows app - hit a list of domains and check [login to view URL]

We have a big list of domains we need to check and get an activity score for. To do this our thought was to attempt to get the [login to view URL] file and then parse it to get a last modified date. then we would do some basic math on this to come up with a score.

Assuming that the pages are edited in groups of dates i think we can dump in to excel a month and count so we have something like

1/2019 - 10

2/2019 - 37

3/2019 - 0

so in the excel output we have

domain | http response | found sitemap | page count from sitemap | months for the last 24.... (make a column for each month for the last 24 months)

one thing to make sure you got in the sitemap is that sometimes sitemaps are nested. you will need to follow the link if it leads to another sitemap for that section. for example using google [login to view URL] this main map links to sub maps.

if the domain redirects or fails we want to log that I think we can log the http response codes for this so something like

2xx - fine

4xx - failed

5xx - failed

3xx- redirect, then log the redirect name it sends back

app should run on windows. and would be cool if we can have config file to put the path to the csv of domains, and maybe some thread count so we can set the performance characteristic of this. We will be running this on some big lists of domains so it might be good to make sure its able to handle running a big list like 10K in some controlled threaded fashion. also we should think about a timeout after a minute or something reasonable so the app doesnt freeze for a dead or bad url?

I found this lib which might help you if you needed it - [login to view URL]

Kemahiran: Python, Pengikisan Web

Lihat lagi: windows app inventory, windows messenger contact list, convert vbnet windows app web app, timer windows app, stock windows app, windows file open windows app, iphone notes sync windows app, postgresql con python windows, windows hide service list, server 2005 report windows app, post asp windows app, iphone app scan upc price check, convert windows app web app, windows mobile todo list synchronisation, windows application play list, developing windows app sits system tray, python tray app windows, python systray app windows, windows phone grocery list app

Tentang Majikan:
( 460 ulasan ) Austin, United States

ID Projek: #22349323

28 pekerja bebas membida secara purata $298 untuk pekerjaan ini


First of all thank you for an excellent description! I can provide you a Scrapy (Python) based web scraping tool that will read a list of domain from a CSV file and allow you to change number of threads to use. It wil Lagi

$350 USD dalam 3 hari
(380 Ulasan)

Hello Sir, I am expert who understands the value of time. I pride myself in my attention to detail. I am very hard working and aim to deliver in less time than quoted. I want to make you, my employer happy without cha Lagi

$350 USD dalam 8 hari
(36 Ulasan)

Hello, Hi, Nice to meet you! I have read your requirements carefully and I am very interesting for your project. I am confident of this project as I'm a professional Python expert with over 5 years of experience. gith Lagi

$300 USD dalam 7 hari
(42 Ulasan)

Hello. As a web scraping and Python and Django expert, I am glad to place the bid on your project. As you can see in my profile, I am fully experienced and lots of skills in web development. I want to discuss more via Lagi

$300 USD dalam 7 hari
(15 Ulasan)

I can write python app for Windows with GUI using the mentioned python lib and multi threading . It will save results in csv as per description . I have 6+ experience in Python .

$250 USD dalam 3 hari
(65 Ulasan)

Hi, I have gone through your requirement to scrape lots of websites. I am EXPERT in building scraping tools /scripts. Hence, I can SURELY work on your project. I am having 4 YEARS of EXPERIENCE in developing PHP-PYTHON Lagi

$250 USD dalam 3 hari
(78 Ulasan)

Hi there, I like that project. Im a Python developer with lots of knowledge in Web Scraping and Python Libraries. I do not exactly understand how you are going to calculate the 'activity score' from sitemaps but Im aw Lagi

$280 USD dalam 7 hari
(14 Ulasan)

Hi. I am ready to write your project Write apps on your demand in many languages (Visual Basic, VBA, VBS, .NET, C#, JS, Python, Java, PowerShell) Write database apps including many db formats: MS Access, MS SQL, SQL Lagi

$345 USD dalam 7 hari
(19 Ulasan)

Hi. 10 k urls or 10 M ?

$277 USD dalam 3 hari
(9 Ulasan)

Hi I offer a wide range of services, including , Python , Web Scraping I can create and deliver the project as per the information.I have skilled, expert programmers I'm very excited to assist you in making your Lagi

$400 USD dalam 7 hari
(7 Ulasan)

Hello, dear Customer. I am interested in your project and feel confident after reading your project description. Please contact me so that we can discuss it in more detail. Looking forward to hearing from you soon. Tha Lagi

$300 USD dalam 7 hari
(10 Ulasan)

hi i saw you your post regarding web scraping.i have done many similar tasks like [login to view URL] config .i will give you excel file and you can set parameters on that file(like file name,site name some filters maybe) i can m Lagi

$300 USD dalam 7 hari
(9 Ulasan)

Hi. I am a freelancer working in 24/7 service. I have much experience for 6+ years in Scraping Field, I will produce the output in any format you desire. I think this project is for me and I can finish it for a short t Lagi

$250 USD dalam 2 hari
(10 Ulasan)

It's a piece of cake for me. Hi, sir I am a Web scraping expert. I have many experiences like your project. I am very confident to complete this task in a few days. I am ready for your project and I will provide the b Lagi

$300 USD dalam 7 hari
(9 Ulasan)

Hello. I am an expert of python scrape.I am very interested in your job and I can do it quickly. Please contact me. Thank you.

$300 USD dalam 7 hari
(9 Ulasan)

Hi, I'm Chunzuo. I read your description and understand what u need. I have already completed python task. Please contact with me to discuss more.

$300 USD dalam 7 hari
(5 Ulasan)

Greetings, This project is very similar to one that I recently completed, using Python to test a list of domain names and recording the results of their DNS lookups and HTTP server response status including redirects f Lagi

$300 USD dalam 3 hari
(5 Ulasan)

Hi! there. ⭐Thanks⭐ good post. I am web scrapper. I am ready in any language such as PHP, Python. I have rich experience in scraping many sites. I want to meet you on the chatting. Regards.

$300 USD dalam 7 hari
(3 Ulasan)

I can start your project immediately. I can provide full-time communication and work your time-zone. If you give me a chance to serve you, I will provide a high quality product within the deadline. Best Regards

$300 USD dalam 7 hari
(4 Ulasan)

Hi,Thank you for your job posting. I have read your job post just before. I have a strong knowledge of Python, Django VPython, Jupyter, Keras, Flask, Python-OpenCV, Tkinter GUI... I am very interested in our work and Lagi

$250 USD dalam 7 hari
(4 Ulasan)