
Closed
Posted
Paid on delivery
Responsible for designing and implementing large-scale data migration and ingestion pipelines to move high-volume data from diverse sources into cloud platforms. Sources include HDFS, relational databases such as MySQL and PostgreSQL, and real-time streaming systems like Kafka. Develop and maintain robust data pipelines using PySpark, ensuring efficient processing of batch and streaming data. Implement automated scheduling mechanisms to orchestrate data workflows on daily and monthly intervals, ensuring reliability and timely data availability. Optimize data ingestion and storage through advanced performance tuning, partitioning, and compaction strategies to handle large-scale datasets efficiently. Ensure data quality, consistency, and fault tolerance across all pipelines. Deploy and manage data processing workloads in Kubernetes, leveraging containerization for scalability, resilience, and resource optimization. Monitor job performance and implement improvements based on system metrics and evolving requirements.
Project ID: 40367480
1 proposal
Remote project
Active 9 days ago
Set your budget and timeframe
Get paid for your work
Outline your proposal
It's free to sign up and bid on jobs
1 freelancer is bidding on average ₹1,050 INR for this job

I am Neelam, a highly experienced and reputable freelance developer. Although my background primarily involves Python, which is a perfect match for your project, I have also extensively worked in web services, web development and other relevant areas. My skills include not only the technical aspects of the job but also a comprehensive understanding of their complexes and nuances. For instance, my expertise in using Linux, Unix, MS Windows Server and different cloud solutions (AWS, MS Azure, Google cloud) would help me leverage containerization using Kubernetes skillfully – guaranteeing scalability, resilience and resource optimization when working with large-scale datasets. In summary, I offer you mastery over Python as well as other relevant technologies/products that makes me uniquely capable of meeting all aspects of this project with top-notch quality within optimal time frames. Additionally, my commitment to reputation over mere task completion ensures an end product that is scalable, reliable for even future evolving requirements made by you or signifiers made by evolving environment conditions that we will diligently monitor based on system metrics model leveraging our experience for your benefit. Now let's get together and build something amazing!
₹1,050 INR in 1 day
0.0
0.0

Visakhapatnam, India
Payment method verified
Member since Apr 13, 2026
£2-5 GBP / hour
$30-250 USD
₹1500-12500 INR
$30-250 USD
$250-750 USD
$15-25 USD / hour
€12-18 EUR / hour
€12-18 EUR / hour
₹12500-37500 INR
₹600-1500 INR
₹1500-12500 INR
₹400-750 INR / hour
$250-750 USD
$30-250 USD
€5000-10000 EUR
₹750-1250 INR / hour
$250-750 USD
$30-250 USD
₹12500-37500 INR
$30-250 USD