PySpark Jobs

Hiring: Sr. Data Engineer Job ID: GKS1805_3 --- Role: Senior Data Engineer Mode: Work From Home Budget: ₹50,000 – ₹65,000/month Availability: Immediate Joiners Only Open Positions: 1 Interview Process: AI Interview → L1 → 2 Client Rounds --- Role Overview We are looking for a high-impact Senior Data Engineer to design and build scalable, multi-tenant data systems on AWS. You will work across data engineering, backend systems, APIs, and ML workflows, contributing to end-to-end product delivery. --- Key Responsibilities - Design & build scalable multi-tenant systems on AWS (Glue, Lambda, Step Functions, ECS, S3, DynamoDB, SageMaker) - Develop robust data pipelines (ETL/ELT) for high-volume datasets - Build Lambda-based microservi...

I need a cloud-native ETL pipeline built end-to-end on AWS, coded in PySpark and designed for production reliability. The pipeline will ingest data from three sources—databases, APIs, and file systems—then standardise and load it into an analytics-ready destination. Source files arrive in a mix of CSV, JSON, and Parquet, so the job must include automatic format detection, schema inference, and efficient column-wise writes. Beyond raw transformation, I want solid engineering practices: parameter-driven jobs, modular Spark code, unit tests, logging, alerting, and retry logic. Leveraging AWS native services such as Glue, EMR, Lambda, and S3 is expected, but I’m open to other AWS components if they shorten development time or lower cost. Candidates must have expertise in da...

Hiring: ML Engineer – Test & Learn Platform Experience: 3+ Years Location: Remote (1–2 visits to Bangalore required) Salary: ₹40,000 – ₹50,000/month --- Role Overview: We are looking for an ML Engineer to build and scale experimentation and causal inference systems. You will work on statistical engines, APIs, and cloud-based pipelines to enable data-driven decision-making. --- Key Responsibilities: - Develop ML/statistical models (DID, Synthetic Control, A/B Testing) in Python - Build and integrate FastAPI-based services - Design large-scale data pipelines using PySpark, Delta Lake, and Azure Data Lake - Optimize Spark jobs (memory, partitioning, performance tuning) - Work with Databricks for job orchestration and data workflows - Containerize and deploy applica...