SQL View to PySpark Notebook

@koolvishruth

Hi , I am from Bangalore. I am certified in databricks. I will get this done.I am data engineer with 7 years of experience. Let’s connect

@bizmanas

Hi, Your Azure Databricks/PySpark migration project is a great match for my experience. I have 18+ years of experience in SQL Server, Azure, Python, and production ETL, with a 4.9 rating and long-term repeat clients. Approach: Convert the SQL Server CTE logic into optimized PySpark DataFrame transformations. Preserve the existing notebook framework, logging, audits, validation, and exception handling. Write the output to Delta Lake and create the SQL View. Validate results against the existing SQL view for functional parity. One question: approximately how many CTEs and source tables are involved in the SQL view? Looking forward to discussing the project!

@hoangvanp

Hello, Refreshing the SQL Server view with a PySpark transformation in Azure Databricks is about seamlessly transitioning the existing business logic to a new PySpark implementation while maintaining the current notebook structure. I would start by reviewing the SQL view's CTEs and window selection logic to replicate them accurately in PySpark. The main focus will be on ensuring data integrity, performance, and compatibility with Databricks Runtime. My experience includes similar projects where I modernized SQL transformations into PySpark workflows, emphasizing clean, optimized code and logic alignment. A few questions: - What specific aggregations and calculations should be performed? - Are there any additional requirements for the output Delta table? - Do you have any preferences for testing frameworks? I look forward to discussing the implementation details further. Best regards,

@meddhif202

As a data automation developer, I can convert your SQL view into a PySpark transformation while preserving Databricks audits, Delta output, and SQL view creation. My Python and Spark data work fits this, and I can start now with one CTE converted free for you to judge. Please get in touch.

@sanddep5

Converting SQL to PySpark isn't just a syntax change—it's about preserving business logic while optimizing for Spark performance. Your existing Databricks framework should remain untouched, with only the transformation layer replaced. That's exactly the type of migration our team has delivered for enterprise data platforms. Hi, We've reviewed your requirements and can seamlessly convert your SQL Server view into an optimized PySpark transformation while preserving your existing Databricks Asset Bundle framework, including logging, metadata, schema validation, audit columns, exception handling, and Delta write logic. Our team has over 10 years of experience working with multiple clients on ETL pipelines, SQL optimization, PySpark, Delta Lake, Databricks, API integrations, and enterprise data solutions. References can be shared if you'd like. We'll replicate the CTEs, window functions, CASE logic, aggregations, and calculated measures using efficient PySpark DataFrame APIs, ensuring the output matches the existing SQL view. We'll also create the Delta table, SQL View, and assist with validation to confirm parity between both implementations. Let's have a short discussion to review the SQL view and notebook so we can deliver a clean, production-ready migration.

@jblignau42

The risk in a SQL to PySpark conversion is not writing the code, it is that the converted version looks correct and quietly returns different rows. Null ordering in window functions, the default frame when you leave out ROWS BETWEEN, and how Spark treats ties all differ from SQL Server, and those gaps only show up on real data. So I would build it, then run both side by side and reconcile row counts and aggregate totals per grouping key before calling it done. If the numbers do not tie out exactly, it is not finished. I would keep your existing logging, schema validation and audit column patterns exactly as they are and slot the transformation into them rather than reworking your framework. Can you share the view definition and roughly how many rows it returns? That tells me whether this is a straight translation or whether the CTEs need restructuring for Spark.