BigData Engineer

Description

• Build a highly functional and efficient Big Data platform that brings together data from disparate sources and allows FinThrive to design and run complex algorithms providing insights to Healthcare business operations.
• Build ETL Data Pipelines in Aws Cloud using Aws ADF and Databricks using PySpark and Scala.
• Migrate ETL Data pipelines from On Prem Hadoop Cluster to Aws Cloud.
• Build Data Ingestion Pipelines in Aws to pull data from SQL Server.
• Perform Automated and Regression Testing.
• Partner with internal business, product and technical teams to analyze complex requirements and deliver solutions.
• Participate in development, automation and maintenance of application code to ensure consistency, quality, reliability, scalability and system performance.
• Deliver data and software solutions working on Agile delivery teams

Responsibilities

• • Bachelor's degree in Computer science or a related discipline
• • 6+ years of data engineering in an enterprise environment
• • 6+ years of experience writing production code in Python, PySpark or Scala
• • Strong knowledge of Aws platform. Should have worked in Aws ADF, Deployed ADF and Databricks code to production and be able to troubleshoot production issues.
• • Experience with SQL.
• • Experience with Big Data technologies in Aws such as Spark, Hive, Sqoop, Databricks or any other equivalent components.
• • Experience working with git and CI/CD tools
• • Proven background in Distributed Computing, ETL development, and large-scale data processing