Sedang Disiapkan

Scraping web content with Selenium in Java

I want to download the content of a website but it's is a single page site (like instagram) and I cant just download an html file. We have to programmatically access the website through a browser (preferably in a headless environment like a Linux machine running in aws) and then use xpath to find the information we need and save this information somewhere (text files would be ok as long as the data is consistent).

Assuming we continue with our Instagram example, the objective is to take a snapshot of all the information in a public account. This can be split into several steps:

* Start by going to the main page of an Instagram user and get the number of posts, followers, following and bio/description

* Get the full list of followers (instagram ids). Note this requires a sub-window and scrolling down.

* Get the full list of following (instagram ids). Note this requires a sub-window and scrolling down.

* Then for each post get a link to the post, a link and maybe a thumbnail of the image, the number of likes, the number of comments, the list of hastags, the list of people who commented, the list of people mentioned and possibly the text in the post plus all the comments. Note that since this has to be done for all posts we will need to scroll down the page all the way to the end.

This is a side project for me and I will have to maintain the code so very simple and easy to read source code is preferred. With plenty of comments explaining why things are done. Please take that into account when making an offer. Unit tests would be a very nice addition, it would be great to include them (starting from the very obvious to more complicated cases).

A payment schedule will be established for each one of the steps, starting from the most basic of just getting the number of posts, number of followers, number of following and bio/description.

Please only apply if you are very familiar with this kind of work. I am software engineer myself and although I am not familiar with the details of web automation I do a very decent job reading java code.

Kemahiran: Java, JUnit, Selenium Webdriver, Pengikisan Web, xpath

Lihat lagi: scraping web content, i want template website for my scientific company, i want order website, i want a website developed how much cost, i want a website account, i want a website, i want personal website, i want make website, i want create website, i want similar website, i want a website mockup for solar powered products, i want a website landing page that has the following posted creatively enough that my target market will want to submit their na, i want a website developer, i want a website designer, i want a website designed for selling watches free, java retrieve web content, liferay add web content java customization, web content management java spring, screen scraping web app java, web content extractor java application

Tentang Majikan:
( 7 ulasan ) Zurich, Switzerland

ID Projek: #16630199

Dianugerahkan kepada:


Hello, I have a lot of experience with selenium and java . Take a look at my profile to see previous projects I have done successfully . Message me to discuss any details and start working on your project immediately. Lagi

€211 EUR dalam 5 hari
(16 Ulasan)

19 pekerja bebas membida secara purata €190 untuk pekerjaan ini


Hi, my name is Sevinc. My bid covers all the phases you explained above... I read your "Scraping web content with Selenium in Java" project descriptions carefully before bidding. I checked your requirements as well... Lagi

€500 EUR dalam 7 hari
(249 Ulasan)

Dear Sir, I'm very much delighted to let you know that i did data scraping with PHP-cURL, PhantomJS, Node.js, Selenium from many sites. I just scraped the data from web site and then wrote the data in mysql database Lagi

€100 EUR dalam 3 hari
(65 Ulasan)

Hi there,I am Miljan,Web Scraping expert from Bosnia & Herzegovina,Europe. I have carefully gone through with your requirements and I would like to help you with this job ! I can start immediately and finish it within Lagi

€222 EUR dalam 3 hari
(69 Ulasan)

hi , I have scrapped 100+ websites . Amazon , adidias and many more r among them . knk me so that we can discuss . I can do this using python not java , if language is not a problem for me then knk me up .

€166 EUR dalam 3 hari
(115 Ulasan)
€155 EUR dalam 3 hari
(47 Ulasan)

Hi there..... Warm Greetings We came along with your request for Scraping web content with Selenium in Java and we reviewed your project description. We'd like to help you with confidence and satisfying results.. Lagi

€300 EUR dalam 5 hari
(26 Ulasan)

Hello. Hope you are doing well. I am a professional bot developer and have 10+ years experience. [login to view URL] [login to view URL] [login to view URL] Waiting for your positive response to make a grea Lagi

€155 EUR dalam 3 hari
(6 Ulasan)

Hi. My name is Oleg and I am senior C#/.NET software developer from Ukraine. Please contact to discuss details regarding the project.

€250 EUR dalam 3 hari
(3 Ulasan)

I am a python developer. I have great experience in web scraping and I am an expert in it. I have all necessary skills to scrape almost any website. I have even scraped sites like google, whatsapp web, instagram which Lagi

€125 EUR dalam 2 hari
(14 Ulasan)

Hi, I'm interested in this job also as a side job. In my current job wehave selenium with chrome web driver in a headless environment running in ec2 already setup. We use this environment to run our end to end tests. Lagi

€166 EUR dalam 3 hari
(15 Ulasan)
€155 EUR dalam 3 hari
(6 Ulasan)

A proposal has not yet been provided

€155 EUR dalam 3 hari
(2 Ulasan)

I am experienced SELENIUM Expert. Hi, I’ve carefully gone through your job posting. I am Web Scraping Expert.... I will get all data you want. I can scrapy even Ebay site. I can work with you anytime you want. Lagi

€222 EUR dalam 3 hari
(2 Ulasan)

Hello, I have briefly read the description on Scraping web content with Selenium in Java development, and I can deliver as per the requirements however I need us to discuss for more clarity on the details, deadline Lagi

€155 EUR dalam 3 hari
(2 Ulasan)

Hello. I spend a lot of time with python+selenium and I can do it for you. I need more details about your task. Best regards, Serge

€166 EUR dalam 5 hari
(0 Ulasan)
€155 EUR dalam 3 hari
(1 Ulasan)
€88 EUR dalam 2 hari
(0 Ulasan)

I have more than 10 years of experience in automation testing using selenium/webdriver, Java, Python, Appium etc. My specialities include building automation frameworks from scratch, writing automated test scripts and Lagi

€155 EUR dalam 3 hari
(0 Ulasan)