Data Engineer

Description

Design, model, and implement data warehousing activities to deliver the data foundation that drives impact through informed decision making. • Work with engineers, product managers, and data analysts to understand data needs, propose or enhance existing framework and platform.
Lead planning and designing solutions for multiple project initiatives, provide technical management to other Data Engineers • Handle Client communications and settle expectations.
Solve challenging data integration problems utilizing optimal ETL, ELT patterns, frameworks, query techniques, and sourcing from structured and unstructured data sources.
Leverage MongoDB as a NoSQL database solution, managing data modelling, indexing, and query optimization.
Develop Azure Logic Apps to automate business workflows, integrating with various external systems through connectors and custom APIs. • Design and implement Azure Service Bus solutions for enterprise-scale applications, enabling seamless communication between microservices and distributed components.
Optimize pipelines, dashboards, frameworks, and systems to facilitate easier development of data artifacts. • Build data expertise and own data quality for allocated areas of ownership including managing SLAs and supporting processes running in production.

Responsibilities

• A degree (preferably an advanced degree) in Computer Science, Engineering or a related field.
• Capable in planning, designing and leading small team of data Engineers including mentoring and guiding development.
• 7+ years of industry experience in architecting and developing data warehousing solutions, preferably in large production environments.
• Azure Data Service - ADF or Synapse Analytics.
• Extensive experience in designing, developing, and maintaining data solutions/pipelines in the Azure ecosystem, including Azure Service Bus, Logic Apps, SQL, Mongo DB & ADF. • Expertise in data ETL/ELT, schema design, dimensional modeling, slowly changing dimensions.
• Capability in coding and optimizing PySpark programs, partitioning and scaling parquet, and delta formatted data structures.
• Creating and optimizing large scale datasets, and familiarity with modern data warehousing techniques Mandatory Skills Data Engineering, Logic Apps, Azure Service Bus, SQL, Azure data factory, Mongo db