develop end to end data pipeline in hadoop

1. Develop/Propose various ETL pipelines for the combination of hadoop eco-system with/NOT spark using AWS cluster;

2. Implement the multi-purpose ELT pipeline (like kappa or lambda architecture) that ingress data from various sources (NOT limited to mySQL, Oracle, NoSQL, Flat files like PDF, Excel, CSV, .docx, Live/streaming) to hadoop/hdfs by using wide range of bigdata tools;

3. Ensure the data quality like format, fields, precision, No. of rows etc.., during data migration from data sources to HDFS;

4. Pipeline must able to handle huge data in GB's;

5. Troubleshoot by creating breakpoint in the ETL pipeline at various levels (at each hadoop eco-system tool) like resource level & code level towards memory management, performace tuning, optimization;

Kemahiran: Perkhidmatan Web Amazon, Pemprosesan Data, Penggudangan Data, Hadoop, Spark

Lihat lagi: data entry front end excel, develop linux multimedia front end, develop data extract ssis, data pipeline course, data engineering pipeline, what is data pipeline in hadoop, data analysis pipeline, data pipeline tutorial, data pipeline definition, data pipeline examples, how to create data pipeline in hadoop, use vba access develop data centre, develop data access layer vbnet, collect user data joomla front end, excel data input front end, data entry front end, develop data access layer asp net , develop data entry screen mobile phone data entry sending sms, spreadsheet data entry front end, collect data joomla front end

Tentang Majikan:
( 0 ulasan ) Atlanta, United States

ID Projek: #19016439

8 pekerja bebas membida secara purata $176 untuk pekerjaan ini


Hello, I have gone through your job posting and become very much interested to work with you. I am an expert in this field. I have already completed several projects like this. For evidence you can see my profile. Lagi

$250 USD dalam 4 hari
(179 Ulasan)

Have 10 years of IT experience with more than 4.5 years of experience in hadoop technologies like hive,pig,spark,sqoop,map reduce and [login to view URL] have very good experiemcence in Java,scala,Python and shell scripting. Lagi

$222 USD dalam sehari
(11 Ulasan)

Hello there, We are a team of expert Big Data developers with more than 10 years of rich inductry experience & have succesfully delivered multiple projects in the past like a)recipe recommendation b)movie recom Lagi

$333 USD dalam 3 hari
(2 Ulasan)

Hi, I have 8 years of experience and working on hadoop, spark, nosql, java, BI tools(tableau, powerbi), cloud(Amazon, Google, Microsoft Azure)... Done end to end data warehouse management projects on aws cloud with ha Lagi

$55 USD dalam sehari
(3 Ulasan)

We have great young and dedicated team, trustworthy and proper completion of project is our moto. Please let me know if you are intesred

$30 USD dalam sehari
(0 Ulasan)

Hello there, Gone through your details and after reading over your application this looks like a perfect fit for my skill sets!! // Please ping me here so we can discuss details and after analysis, I will provid Lagi

$200 USD dalam 3 hari
(0 Ulasan)

I have built several data pipelines using AWS and can definitely design and develop the solution. I will need more details however on this.

$166 USD dalam 2 hari
(0 Ulasan)

I work with similar use cases on day to day basis. I have around 3 years of hands on experience with big data technologies. Along with work I can help you with understanding. Reach out to me if you are interested.

$155 USD dalam 3 hari
(0 Ulasan)