Data Engineer

Description

Job Description The Data Engineer for the Technology + Innovation Lab a new group within Enbridge tasked with positively disrupting its core businesses with innovative and impactful products using advanced technologies will help ingest, transform and store clean and enriched data in ready for business intelligence consumption. Fully capable specialist able to solve complex, datacentric problems and drive solutions to completion independently. An ideal candidate should be able to design, model and maintain data warehouses and data marts, as well as maintain and configure data engineering and ETL tools (e.g., ODI, DataServices, SSIS).
Responsibilities Works independently on complex data engineering problems to support data science strategy of products Use broad and deep technical knowledge in the data engineering space to solve complex data problems for product teams, with a core focus on leveraging technical expertise Improves the data availability by acting as a liaison between Lab teams and source systems Collects, blends, and transforms data using ETL tools, database management system tools, and code development Implements data models and structures data in readyfor business consumption formats Performs aggregations on data across various warehousing models (e.g. OLAP cubes, star schemas) for BI purposes Interacts with business teams and capable of understanding how data needs to be structured for consumption Required Skills We are looking for a candidate with 5+ years of experience in a Data Engineer role, who has attained a Graduate degree in Computer Science, Statistics, Informatics, Information Systems or another quantitative field. Create and maintain optimal data pipeline architecture. Assemble large, complex data sets that meet functional / nonfunctional business requirements. Identify, design, and implement internal process improvements automating manual processes, optimizing data delivery, redesigning infrastructure for greater scalability, data quality checks, minimize Cloud cost, etc. Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL, Data Bricks, NoSQL Build analytics tools that utilize the data pipeline to provide actionable insights into customer acquisition, operational efficiency and other key business performance metrics. Keep our data separated and secure across national boundaries through multiple data centers and Azure regions. Document and communicate standard methods, best practices and tools used. Work with other data engineers, data ingestion specialist, & subject matter experts across the company to consolidate methods and tool standards where practical. Experience using the following software/tools Experience with big data tools Hadoop, HDI, & Spark Experience with relational SQL and NoSQL databases, including COSMOS Experience with data pipeline and workflow management tools Data Bricks (Spark), ADF, Dataflow Experience with Microsoft Azure Experience with streamprocessing systems Storm, StreamingAnalytics, IoT Hub, Event Hub Experience with objectoriented/object function scripting languages Python, Scala, SQL Ability to produce formal written deliverables including data models, technical standards and guidelines, investigations and recommendations, presentations and briefings, future state roadmaps, and metadata catalogs for business and technical use.