Data Engineer+ MS Fabric

Posted on August 5, 2026

Apply Now Share Job

Job Description

Role: Data Engineer+ MS Fabric

Overview

• Years of Experience: 8+ Years
• Start date: 13th July
• Duration: 3 Months
• Work Timings: 1:30pm to 10:30 pm
• Remote Opportunity and Contract Role

Key Responsibilities

• Perform data analysis to understand source systems and business requirements
• Translate requirements into scalable data models and pipeline designs
• Build and maintain data ingestion pipelines from multiple sources (batch and incremental/delta loads) into Microsoft Fabric
• Experience in writing scripts from multiple sources including database tables, flat files and APIs
• Ingest and land raw data into bronze layer, ensuring reliability, scalability, and traceability
• Transform raw data into clean, standardised silver datasets (deduplication, conformance, validation)
• Design and develop gold layer models (fact/dimension tables, aggregates) for analytics and reporting
• Design end-to-end ETL/ELT pipelines across bronze, silver, and gold layers
• Support creation of semantic models and reporting-ready datasets
• Implement data quality rules and validation checks (completeness, accuracy, freshness, reconciliation)
• Implement robust error handling mechanisms including retry logic, logging, exception handling, and alerting
• Design self-healing pipelines (auto-retry, checkpointing, idempotent processing, and failure recovery)
• Optimise pipeline performance and ensure reliable, production-grade data processing
• Monitor pipelines and proactively identify and resolve failures or performance bottlenecks
• Work with CI/CD and DevOps practices for deployment across environments (Dev/Test/Prod)
• Able to write, amend and debug scripts using both PySpark and SQL
• Have a good working knowledge of software design standards
• Collaborate with business, BI, and architecture teams to deliver end-to-end data solutions

Required Skills

• Requires extensive hands-on experience with Microsoft Fabric, including designing and developing reusable boilerplate ingestion frameworks for Bronze, Silver, and Gold layer data pipelines using PySpark notebooks.
• Strong expertise in implementing Medallion Architecture (Bronze, Silver, Gold) following lakehouse design principles.
• Experience building metadata-driven and configurable ingestion frameworks to support scalable and reusable data pipelines.
• Understanding of performance tuning, cost optimization, and best practices for scalable enterprise-grade Fabric lakehouse solutions.

Other Details

• NB: The candidates interviewed so far either lack strong Spark fundamentals or are unfamiliar with Fabric best practices and deployment using Azure DevOps.
• While they demonstrate strengths in data modelling, SQL, and ADF concepts, Fabric extends beyond ADF and SQL.
• It requires substantial PySpark coding, effective workspace strategies, adherence to lakehouse best practices, lakehouse maintenance, retrieving secrets from Key Vault, and broader platform expertise.

Required Skills

No specific skills listed.