Data engineer

Skills

Ensure adherence to architecture patterns and be a key contributor in the delivery of full data lifecycle frameworks and patterns including data acquisition storage transformation prep and consumption to support Enterprise Data Lake build out for use in analytics solutions at different maturity levels between data science and operational teams Senior hands-on technical resource in data engineering design and implementations including data ingestion data models data structures data storage high-throughput data processing data pipelines and data monitoring at scale Aid in the development of and follow data engineering best practices with considerations for high data availability computational efficiency cost and quality Key contributor on a team in the build and maintenance of environments processes functionalities and tools to improve all stages of analytics solution development e.g. proof of concepts prototypes and production Design and build capability for repeatable configuration driven automation throughout the data lifecycle Mentor associate resources across data engineering teams

Description

Client is seeking a Senior Data Engineer for the data engineering team driving a ground up data lake build. This role will be responsible for providing senior level engineering work in a team of data engineers in the design and implementation of patterns supporting a ground up data lake build. The position is a hands-on role requiring proven expertise in data engineering best practices including modern cloud data lake ingestion, migration, management, and consumption patterns to enable analytics solutions development and delivery at scale.

Responsibilities

• 3-5+ years of professional experience in data engineering related roles - full software development lifecycle, DBA, data architect, data engineer
• 3+ years specific experience in modern data engineering and cloud data practices
• Working experience in data analytics (data wrangling, mining, integration, analysis, visualization, data modeling, analysis/analytics, and reporting) using BI (Business Intelligence) tools
• Extensive knowledge of general IT concepts, strategies, methodologies, modern application and data engineering architectures and approaches including cloud, streaming, event based, IoT data, and edge server capability
• Experience with AI/ML concepts, tooling, and data prep for data discovery tools, Jupyter Notebooks, and AWS SageMaker
• Solid communication skills - verbal and written. Ability to discuss and explain data concepts, practices, architectures, and tooling to data engineering teams and ensure adherence to architectural patterns and standards
• Experience working closely with and maintaining alignment to data engineering and enterprise architecture teams
• Expertise in AWS services including S3, EC2, SQS, RDS, Neptune, EMR, Kinesis, Lambda, Step Functions, Terraform, EventBridge, Glue, Redshift, Athena, DynamoDB, API Gateway, ElasticSearch, Service Catalog, Cloudwatch, EKS, IAM, and API Gateway
• Experience with security, encryption, and data access best practices in AWS
• Experience writing complex programs, implementing architectures, and enabling automation in cloud environments
• Experience implementing configuration driven, reusable, automation frameworks and patterns
• Programming background and ability to utilize a variety of software/languages/tools, e.g., Python, Scala, Java, Spark, C++, Hive, SQL, etc.
• Linux skills including Korn Shell, scripting, regex,
• Experience with DevOps best practices
• Demonstrated pursuit of optimizing CI/CD process and tools, testing frameworks and practices
• Experience working with leading cloud vendors such as Amazon, Google, or Microsoft
• Experience with microservice development, Docker, Kubernetes, EKS
• Experience with engineering and programming best practices and enforcing at team level
• Experience optimizing data solutions for multi-petabyte data systems in batch, streaming, and event approaches
• Experience working with agile methodologies, cross-functional teams (Product Owners, Scrum Masters, Developers, Test Engineers), backlog grooming, and tooling
• Must be proactive, demonstrate initiative, and be a logical thinker
• Must be team oriented with strong collaboration, prioritization, and adaptability skills required
• Degree in Computer Science or equivalent work experience, degree preferred