Spark Jobs

We are looking for a PySpark Expert to help optimise our PySpark pipeline. We are using Kafka for ingestion, PySpark for Stream processing and Elasticsearch for storage. Our pipeline has a couple of bottlenecks, which we want to unclog. We want someone who understands how to use the various optimisation techniques in Spark, knows how to set the configurations for hardware resources and understan...

Hi there, We need a team of Big data specialist with Python(ML) Spark(mandatory) skills. Preferably looking for Bangalore team. Complete requirement will be shared after initial discussion.

using apriori implement frequent items with the specifications given

I need you to develop some software for me. I would like this software to be developed for Windows.

Hello Hadoop Expert: I have a project that involves the following technologies. See below. Hadoop, Pig, Hive, Sqoop, Oozie, Spark. I need someone to help me with my project and through that, I would like to get a better understanding of the above technologies. You do not have to be an expert in ALL of them. If you know some of them and can help me that should be fine. But when you bid for this pr...

Ingest data from Azure SQL Database. Do the transformation in Azure Databricks using SQL. Schedule the Notebooks in Azure Data Factory. Load the transformed data into Azure SQL Database.

[log masuk untuk melihat URL] for a data engineer resource with minimum 2+ Yrs of experience in Hadoop. [log masuk untuk melihat URL] 4 Years of experience. [log masuk untuk melihat URL] is an immediate requirement [log masuk untuk melihat URL] required for 1 month [log masuk untuk melihat URL] can work on this remotely

Need Help in fixing the issue with the HiveQL script.

Hello, I will need big data engineers in order to work remotely. Please send me your CVs in your responses, copy paste messages will be ignored. Thanks

Need a utility which can fetch live twitter data using pyspark and create a dashboard which has top trending hashtags dynamically on AWS EMR .