Expert to build a web using Scrapy + djangoCMS + AngularJS4


1) Please read the project details and attachment before you bid.

2) Full requirements as in attachment, which is fairly clear

3) Budget currency is NZD

4) Total price of this job is as high as you bid, no-negotiable.


1)[login to view URL]

2)[login to view URL]


A1: 1) scrap by using regex to complete the ID for the whole URL

A2: 2) can only be scrap based on looping each category/subcategory to get all listings in each page

If you have a better way for A2, that is great.


Functional requirements:


R1. scrap all categories (see Appendix 2: product categories in attachment)

R2. scrap all listings of a website

R3. scrap production information and delta information of an existing listing: price, stock^, sold qty^, see Appendix 1:schema tables below.

R4. Admin backend UI to:

4.1 list all existing sites, last ran schedule and add new site(name,site url)

4.1.1 a button for pop up to set up schedule to start scrapping a site and a button to start scrap the site

4.1.2 a button to drill down the siate and redirect a new pate for displaying categories of a site

4.2 add new category in 4.1.2 and a button to scrap new category

4.3 display all listing under the category.

4.5 a page to display fields and setting for a scrapy item. Modify existing css rule for existing field of a listing

4.6 add a new css rule to scrap a new field(document/write HOWTO steps for changing the code if needed)

^: if field/data exists in page

R5. webAPI to return result of listing query

R6. front end pages to:

6.1 display all categories of a site on left and display all listings a site - paged

6.2 click on one listing in 6.1 and open a new page to display detail of a listing. product name, brand, image, latest price/price history(scrapped by #R3)

R7. user login and add listing to watch list


Technical requirements:


Backbones: Scrapy + djangoCMS + MySQL + SQLAlchemy + AngularJS4

A. add scrapy module as an app for djangoCMS

1. support multi-threading

2. anti-scraping: [login to view URL]

tool:RandomProxy + RotateUserAgentMiddleware

3. each listing has an Unique identifier(foreign key for other tables)

4. data access layer: SQLAlchemy

B. admin backend

1. support login( djangoCMS support by default)

2. four *admin* pages integrated to *djangoCMS admin backend*, refer to in attachments(site page / rule page/ product page/ users)

C. WebAPI as an app for django (for R5)

1. input: text(product name) or unique identifier

2. return: array of listing in JSON format [{Id, productName, brand, last price, last date, image url, [category 1, category 2, ...]}]

3. authentication token(64bit)

4. security: SSL/https

D. front end to display (for R6)

1. UI(Angular4 + Bootstrap)

2. security: SSL/https

3. call webapi to retrieve data

4. support CORS: [login to view URL]

5. listing page to display products, display top 20 products by default order by last scrapped time.

6. detail page to display product details

E. naming convention: CamelCase or underscore

F(optional). it is great if WebAPI result can be used for both admin backend and frontend, if you want, you can use WebAPI for admin backend


Appendix 1: schema tables:

1. site (id, name, baseUrl, enable, schedule, lastScrapDateTime, lastStatus)

store site as source, enable flag indicate the schedule is active for scrapping, update lastScrapDateTime each time the spider ran)

2. siteRule(id, siteId, ruleName, enable, type^, url, minId**, maxId**, linkRegex^^)

^type: depend on scrap by Id or category links (see [sources] at the beginning above)

.... please find more in attachment

Please propose your milestones if you do not agree as below:
1. build & deliver scrapy as an app in djangoCMS
2. build & deliver admin backend pages
3. build & deliver front end pages

Kemahiran: Angular.js, Django, Python, Scrapy

Lihat lagi: making bot can retrieve data websites, insert data xml using vbnet, developing websites using wordpress, set product review websites, data entry using spss, parse data websites, amazon product data, data mining using aspnet, free product data feed website, manufacturer web product data extracting, product selling websites, can excel data websites, build web site badoo, click button web using 2008, build web video chat php, someone build web properties, build distro using busybox, scrap data website using aspnet, build web site selling furniture, build web base app data entry form

Tentang Majikan:
( 0 ulasan ) Wellington, New Zealand

ID Projek: #17441593

13 pekerja bebas membida secara purata $501 untuk pekerjaan ini


Hello sir Nice to meet you. As a senior web developer, i can help you. I have rich experiences in web development. Thanks Regard.

$619 NZD dalam 10 hari
(75 Ulasan)

Hello, sir Your job is exciting me. Let me do this job for you now just to prove my expertise in this field. Hope I could awake you interest in me. I am always online, work hard and very sensitive to details. Thank Lagi

$555 NZD dalam 10 hari
(39 Ulasan)

Dear,sir! nice to meet you. I’m very interested in your job post involving these skills I need to know the details of your code through a conversation with you thank you for you attention Relevant Skills and Exp Lagi

$175 NZD dalam 10 hari
(10 Ulasan)

Dear Sir Thanks for your inviting me to your job. If you want to work with me, I will do my best. Sincerely

$1111 NZD dalam 10 hari
(28 Ulasan)

Hi there, I have 4+ years experience at Python and Web scraping. I think I can do your project. If you'd like to view my previous work, take a look at my profile https://www.freelancer.com/u/etuannv or check some de Lagi

$1666 NZD dalam 10 hari
(21 Ulasan)

Hello , How are you? I just saw your project description carefully. I am very interested in your project. I have rich experience in Scrapy + djangoCMS + AngularJS4. I am ready to start your project immediately. Lagi

$175 NZD dalam 10 hari
(11 Ulasan)
$175 NZD dalam 10 hari
(35 Ulasan)

Hello, Martin. I am Xiao Bob. I hope to discuss more details over chat. Please contact me. I am free now. Thanks.

$280 NZD dalam 5 hari
(12 Ulasan)

Hi, Sir. I am a senior web developer with 5 years of experience in building website. I am very experienced in web development with PHP(CodeIgniter, Laravel, WordPress, ...), Java(JSP, Spring, Hibernate, Struts, JSF.. Lagi

$500 NZD dalam 10 hari
(1 Ulasan)

We have experienced python, django developers. we have experience with BeautifulSoup, scrapy. You can see our work [login to view URL], [login to view URL]

$500 NZD dalam 10 hari
(12 Ulasan)

Hello, Hope you are doing well. The last 8 year's I have focused on high-quality Web, IOS, Android and Computer Vision development. Please see my range of skill sets below. After coming on chat discussions I wi Lagi

$144 NZD dalam 10 hari
(4 Ulasan)

Hi, Thank you for giving me a chance to bid on your project. i am a serious bidder here and i have already worked on a similar project before and can deliver as u have mentioned I have got Rich experience in Jooml Lagi

$175 NZD dalam 10 hari
(0 Ulasan)
$444 NZD dalam 5 hari
(0 Ulasan)