Python Pyspark Developer

Description

● Collaborate with the team to build out features for the data platform and consolidate data

assets

● Build, maintain and optimize data pipelines built using Spark

● Advise, consult, and coach other data professionals on standards and practices

● Work with the team to define company data assets

● Migrate CMS’ data platform into Chase’s environment

● Partner with business analysts and solutions architects to develop technical

architectures for strategic enterprise projects and initiatives

● Build libraries to standardize how we process data

● Loves to teach and learn, and knows that continuous learning is the cornerstone of every

successful engineer

● Has a solid understanding of AWS tools such as EMR or Glue, their pros and cons and

is able to intelligently convey such knowledge

● Implement automation on applicable processes

Responsibilities

• ● 5+ years of experience in a data engineering position
• ● Proficiency is Python (or similar) and SQL
• ● Strong experience building data pipelines with Spark
• ● Strong verbal & written communication
• ● Strong analytical and problem solving skills
• ● Experience with relational datastores, NoSQL datastores and cloud object stores
• ● Experience building data processing infrastructure in AWS
• ● Bonus: Experience with infrastructure as code solutions, preferably Terraform
• ● Bonus: Cloud certification
• ● Bonus: Production experience with ACID compliant formats such as Hudi, Iceberg or
• Delta Lake
• ● Bonus: Familiar with data observability solutions, data governance frameworks