Data Processing Design a pipeline to ingest data into an operational data store.

Design a pipeline to ingest data into an operational data store which accounts for monitoring and logging auditing for completeness (source records should not be dropped) ability to configure and replicate the pipeline for different sources with minimal changes Implement specific part of the pipeline using the tool "Apache AirFlow/Spark"

Document the critical choices and decisions, preferably using Git Data for this procedure:

TMDB - The Movie Database (TMDb) is a community built movie and TV database. This can be used to demonstrate the design and implementation.

APIs Introduction:

[login to view URL] File dumps -

[login to view URL]

Design and standards:

1. I am looking for a pipeline implementation that works on either or high availability. With the requirement, I want if you can design a solution which ingest data with integrity when run on tuned production setup.

2. You must choose Python v3 (latest) programming language or framework or libraries.

3. You can choose Docker (dockerfile, docker-compose) to setup the environment and add the same in the repository, if chosen.

4. You are free to choose any flavor of Git workflow, ideally something that can be extended by a team as well.

Submitting solution:

Please email me your solution which contains: summary for the design and implementation code (ie: DDLs, dockerfiles, pipeline implementation) Ideal case submission to share a link to public git repository with all docs and code described by a README.

Kemahiran: Python, MySQL, Pengkomputeran Awan, Docker, Apache

Lihat lagi: data processing skills, data processing spreadsheet, data processing forum, data processing health care, excel data processing home, yahoo store data, contract bidding canada data processing, market reaserch simple system simple data processing, bookkeeping website design maintenance online research data processing, data processing companies german doutch post type data, data processing design payroll system, data processing job german deutsche post type data, prposal writing skills questionare design data processing, data processing logo design, data processing bpo process non voice process data, toptuned net london uk computer programming data processing and inaugurated systems design services, tunedspace net london uk computer programming data processing and inaugurated systems design services, python data processing pipeline, data processing pipeline

Tentang Majikan:
( 0 ulasan ) Dubai, United Arab Emirates

ID Projek: #22790009

Dianugerahkan kepada:


Hey, I'm Arnav I'm an experienced Python Fullstack Developer with a skillset comprising of web development (frontend & backend), automation, machine learning, deep learning, data mining, data analysis, API development Lagi

$50 USD dalam 5 hari
(3 Ulasan)

4 pekerja bebas membida secara purata $65 untuk pekerjaan ini


Hi, Nice to meet you! I have read your requirements carefully and I am very interesting for your project. I am confident of this project as I'm a professional Python expert with over 5 years of experience. [login to view URL] Lagi

$100 USD dalam 7 hari
(44 Ulasan)

Hi. Dear. Your job posting has caught my attention. I have an expertise in Python and Python Frameworks such as Flask,Django. I have many experiences in python, tkinter, django. I can complete your project successfully Lagi

$100 USD dalam 2 hari
(1 Ulasan)

Today I have read your job, I am very interested in your job. I have rich experience for over 6 years in webscraping projects I believe my skills would be ideal for your project. I can complete this job within the req Lagi

$10 USD dalam sehari
(3 Ulasan)