Big Data Engineer
American Express Company
Contract Phoenix, Arizona, United States Posted 3 months ago
About Position
Big Data Engineer (Contract)
$75.00 / Hourly
Phoenix, Arizona, United States
Big Data Engineer
Contract Phoenix, Arizona, United States Posted 3 months ago
Skills
Big Data; ETL - Big Data / Data Warehousing; PySpark; Spark; HiveDescription
6+ years of software development experience and leading teams of engineers and scrum teams
3+ years of hands-on experience of working with Map-Reduce, Hive, Spark (core, SQL and PySpark)
Solid Datawarehousing concepts
Knowledge of Financial reporting ecosystem will be a plus
Expert on Distributed ecosystem
Hands-on experience with programming using Core Java or Python/Scala
Expert on Hadoop and Spark Architecture and its working principle
Hands-on experience on writing and understanding complex SQL(Hive/PySpark-dataframes), optimizing joins while processing huge amount of data
Experience in UNIX shell scripting
Ability to design and develop optimized Data pipelines for batch and real time data processing
Should have experience in analysis, design, development, testing, and implementation of system applications
Demonstrated ability to develop and document technical and functional specifications and analyze software and system processing flows.
Responsibilities
- Knowledge of cloud platforms like GCP/AWS, building Microservices and scalable solutions, will be an advantage
- 1 + years of experience in designing and building solutions using Kafka streams or queues
- Experience with GitHub/Bitbucket and leveraging CI/CD pipelines
- Experience with NoSQL i.e., HBase, Couchbase, MongoDB is good to have
- Excellent technical and analytical aptitude
- Good communication skills
- Excellent Project management skills.
- Results driven
By applying to a job using PingJob you are agreeing to comply with and be subject to the PingJob Terms and Conditions for use of our website. To use our website, you must agree with the Terms and Conditions and both meet and comply with their provisions.