Need Node.JS / Database / Amazon AWS expert to help with Data Collection project - 05/12/2018 02:34 EST


I need an expert in Node.JS, Amazon AWS, and Database management to help with a data collection project.

Project Overview:

We have created a script which queries a third-party API, using a specific keyword, and returns JSON data related to that keyword.

Task #1:

We need that script to run once every 60 seconds.

Task #2:

We need a Database set up to store the JSON data. From speaking with experts, it sounds like this should be a relational database such as MySQL.

Task #3 (hard part):

We need to duplicate this automated script for approximately 5,000 different keywords, all running queries every 60 seconds and storing their data in the main Database.


This presents a few issues, on which we need advice.

Problem #1: The third-party API has a rate limit of 60 seconds per request. Presumably, this means we need to create 5,000 different scripts, all running from different IP addresses (need advice). From our research, some people have had success doing this on a smaller scale using Amazon’s Lambdas service. We need advice on how to run all of these API queries concurrently, efficiently, and at the lowest cost. Recommendations on services and solutions are welcome. Note: This is not an attempt to avoid paying for API access - it is a public API that has a rate limit.

Problem #2: These queries are very quickly going to create a very large database. We need advice on the best Database service / solution to use, for maximum performance at the lowest cost.

Problem #3: The goal for this data is to send it to a Front-End which will be able to present the Data in real time. We are still building that Front-End, but have a general idea of what it will look like and how the data will be presented. We need advice on the best way to Collect, Process, Analyze, Store, and Present this data in a real-time environment. From our research, there are various services and solutions for this type of problem. For example, Amazon Kinesis seems like it could be a solution? Need advice.

If you are interested, please get in touch and we can chat further. Thank you for your time.

Kemahiran: Amazon Web Services, Azure, JavaScript, node.js, RESTful

Lihat lagi: deploy angular 4 app to aws ec2, elastic beanstalk, deploy express app to aws, express ec2, aws console, aws node js example, node.js aws load balancer, deploy node app to ec2, grade `12 java project, project data collection, data collection process method petrol bulk required project, need collection script, project data collection sheet, project data collection dubai, project data collection entry, bid project data need list top universities, project data collection tool, need lots content writers bulk content writing project, automobile project data collection, need sap abap developer can support finishing project

Tentang Majikan:
( 35 ulasan ) faridabad, India

ID Projek: #18298402

9 pekerja bebas membida secara purata $199 untuk pekerjaan ini


Hello, I am a Technical Developer working on following Technologies :- #- Basic Skills(HTML5,CSS3,Bootstrap4,AJAX, Javascript, jQuery / Prototype) #- CMS(WordPress, Prestashop, Shopify) #- MEAN S Lagi

$370 NZD dalam 3 hari
(14 Ulasan)

Respected Sir , Warm Greetings , As i have gone through your initial requirement and have understood that you need an excellent developer for your JS work and i am confident to deliver you the best desired result. Lagi

$172 NZD dalam 3 hari
(4 Ulasan)

Hi, I have gone thru description....We will discuss on chat window. I am Full stack developer having 6 years experience in technologies like Html 5, Css3, Bootstrap, JavasScript, jQuery, Angular.js, React.js, Node.js, Lagi

$250 NZD dalam 3 hari
(5 Ulasan)

Hey, I have 5+ years experience in web development and two years experience in blockchain development. I am able to make your project world class. Check my URL: [login to view URL] (website) tripagot Lagi

$155 NZD dalam 3 hari
(0 Ulasan)

Hello Project Manager, **Immediate Start** No Upfront Payment Payment After Delivery I am expert with 7+ years of experience in creating & designing online store, shopify/eCommerce websites,Node JS, PHP , Mage Lagi

$250 NZD dalam 6 hari
(0 Ulasan)

Hi dear, I have 6+years of rich experience with several complex and easy projects alone or with a team, I liked your project details and already done and completed successfully with UK clients offsite. Please share m Lagi

$155 NZD dalam 3 hari
(0 Ulasan)

Dear Sir/Madam. My name is George Mburu. I am a full stack web developer. I am proficient in nodejs and amazon AWS. I am available full time and I believe I can do the job. Give me more details. Awaiting to here fro Lagi

$177 NZD dalam 3 hari
(0 Ulasan)
$222 NZD dalam 3 hari
(0 Ulasan)

Hello Recruiter I am interested in the Job and I have 5 years of experience in Web Development and have completed 30+ projects till now in Web Development. I also have experience with front end designing. You Lagi

$36 NZD dalam 3 hari
(0 Ulasan)