Lengkap

[PYTHON & GCP EXPERT] Data ingestion and processing from Pub/sub to Bigquery -- 4

I'm trying to automate the ingestion of data from Pub/sub to Bigquery. I need help writing a cloud function to subscribe to a pub/sub, do some processing on that data to apply a certain logic, and then insert the data into Bigquery. I intend to use cloud scheduler, cloud pub/sub, cloud functions, and big query. Do not intend to use an apache beam as the volume of data is very low. (<100 records per day)

Deliverables

Complete writing cloud function which does the following:

1. Picks up data from pub/sub.

2. Processes the data and compares it to data in another table in BigQuery.

3. Based on certain conditions and criteria, the cloud function will insert that record into a separate table with an additional column that describes the status as 'Met', 'Not Met', 'Partially Met'

Please find project details in the attached excel file.

[login to view URL]

1. Data flow -> Pub/sub - Cloud function - Big Query

2. Flow logic - Every time data is pulled, the cloud function must use logic to map the ID, category, and status from the source table(highlighted in green) to Vul-CCF mapping, Vul-status to Control Health status, vul codename to app name with the environment and insert/update record in the destination table.

Kemahiran: Python, Kejuruteraan Perisian, MySQL, Google Cloud Platform, Pemprosesan Data

Lihat lagi: data entry processing officer, excel data entry processing controls, data entrydata processing, data entry processing skills, python script extract data web page, python script crape data, simply data email processing, python read input data file, scarborough data entry processing, python script extract data, python script read data text file, python script extract data website, expert data processing, python data stream processing, gcp pub sub ack, data being published to pub/sub must be sent as a bytestring., data being published to pub sub must be sent as a bytestring, google functions pub sub python

Tentang Majikan:
( 48 ulasan ) Siem Reap, Cambodia

ID Projek: #30987653

Dianugerahkan kepada:

abhishekkarna007

Hi. I am an experienced Full stack developer with expertise in Python, GCP, Bigquery and many more. I have done a similar project in my past and am sure to complete this within reasonable time. In previous project , a Lagi

$250 USD dalam 2 hari
(4 Ulasan)
3.0

4 pekerja bebas membida secara purata $135 untuk pekerjaan ini

SunshineConsult

Hello,I work on GCP,AWS and Python having 14 years of professional experience,can create a cloud function using python as required,Thanks.

$156 USD dalam 2 hari
(5 Ulasan)
3.3
agupta2812

Hello there! Good day! I am interested in this project. I have experience working on similar projects for more than 8 years. I am an expert in Python/GCP. Please reach out to me to discuss further. Thank you!

$125 USD dalam 7 hari
(3 Ulasan)
2.4
Shone15Paul

AN EXPERT IN ABOVE NAMED SKILLS, DEAR CLIENT, Thank you for sharing your project requirements. After KEENLY and PROFESSIONALLY reading your description and being able to comprehend CLEARLY with Mentioned REQUIRED SKIL Lagi

$10 USD dalam sehari
(1 Ulasan)
0.5