Spark Admin

Description

We are seeking an Engineer specializing in Platform Support to ensure the seamless operation and optimization of our infrastructure. The ideal candidate will possess expertise managing Hadoop/Spark Clusters, Spark Application with an understanding of Helm and Kubernetes would be good. This role requires adept problem-solving skills, the ability to collaborate across teams, and a passion for maintaining high-performance systems.

Responsibilities

• • Provide advanced support for the company's platform infrastructure, focusing on Spark technologies.
• • Monitor system performance, diagnose, and troubleshoot issues, and implement solutions to maintain high availability and reliability.
• • Manage Hadoop clusters, including HDFS, YARN, and related ecosystem components, to support big data processing and analytics workloads.
• •Configure and fine-tune Spark clusters for optimal performance and resource utilization, including memory management, scheduling, and data locality optimizations.
• •Monitor cluster health and performance metrics using tools like Cloudera Manager, Ambari, or custom monitoring solutions, and proactively address any issues that may arise.
• • Conduct regular performance assessments, identify areas for improvement, and implement optimizations to enhance platform scalability and stability.
• • Exposure to HELM, Kubernetes is good to have.
• • Participate in on-call rotation to provide timely resolution of critical incidents and ensure 24/7 system availability.
• Document procedures, configurations, and best practices to facilitate knowledge sharing and ensure operational continuity.
• • Stay updated on emerging technologies and industry trends related to platform support and distributed computing.