Hadoop Developer

Description

10+ years of hands-on experience in the Hadoop ecosystem with expertise in PySpark for data processing and transformation.

Proficient in writing and optimizing complex Hive and Spark queries for efficient data retrieval and manipulation.

Strong experience handling large volumes of data, particularly in the banking domain, with a focus on data governance, security, and compliance.

Solid working knowledge of Linux commands for file handling, process management, and system troubleshooting.

Experience working with Cloudera distribution, including cluster management, troubleshooting, and performance tuning, along with expertise in YARN for resource management and Zookeeper for cluster coordination and synchronization.

Hands-on experience with Oozie for scheduling and managing complex ETL workflows and job orchestration.

Proven ability to manage real-time data processing pipelines and large-scale financial data for critical business applications.

In-depth experience with Apache Spark distribution (e.g., Spark on YARN, Spark SQL) for high-performance distributed data processing, tuning, and optimization.

Basic understanding of Hadoop administration tasks, including cluster monitoring, tuning, and troubleshooting, with familiarity in Ambari for cluster management and Ganglia for system performance monitoring.

Familiar with integrating data streaming platforms like Kafka for real-time data ingestion and processing, ensuring scalability and fault tolerance.

Collaborate with cross-functional teams to align data solutions with business requirements.

Create and maintain comprehensive documentation for processes, workflows and configurations.

Participate in code reviews, quality assurance and release planning processes.

Experience in the banking domain is good to have. Strong analytical and problem-solving skills.

Excellent communication and collaboration abilities. Attention to detail and focus on delivering high-quality solutions.