Big Data Engineer

Description

Fundamental Components/Job Description:

Develops large scale data structures and pipelines to organize, collect and standardize data that helps generate insights and addresses reporting needs.
Writes ETL (Extract / Transform / Load) processes, designs database systems and develops tools for real-time and offline analytic processing.
Collaborates with data science team to transform data and integrate algorithms and models into automated processes.
Uses knowledge in Hadoop architecture, HDFS commands and experience designing & optimizing queries to build data pipelines.
Uses strong programming skills in Python, Java or any of the major languages to build robust data pipelines and dynamic systems.
Builds data marts and data models to support Data Science and other internal customers.
Integrates data from a variety of sources, assuring that they adhere to data quality and accessibility standards.
Analyzes current information technology environments to identify and assess critical capabilities and recommend solutions.
Experiments with available tools and advises on new tools in order to determine optimal solution given the requirements dictated by the model/use case.
Work with Data Scientists and Engineers to drive improvements in code quality via manual and automated testing.
Identify and analyze suitable opportunities for test automation
Utilize efficient and effective processes and standards in order to continuously improve quality
Support DevOps implementations. Implement end-to-end automation
Support tool research, analysis and selection. Support automation assessments
Implement automation frameworks and approaches
Support development of automation at all layers of the application stack
Support Testing and quality activities within Agile delivery
Design, develop and execute test automation frameworks and test automation suites
Evaluate and maintain existing manual and automated test cases & procedures
Perform load and stress tests using enterprise level test automation tools
Interpret test results and prepare comprehensive test reports
Act as key point of contact for other test engineers with automation needs
Manage and maintain physical, virtualized and simulated test environments
Identify and resolve issues that arise when integrating the application software with other software components
Assist in reviewing and determining amount of code coverage of unit tests
Analyzes current information technology environments to identify and assess critical capabilities and recommend QA solutions.
Representing QA lead at technical meetings as needed