Data Engineer with Gen AI/RAG

Skills

As a Data Engineer you will bring to life implement architecture blueprints and help our team by mapping out solutions to some of their complex technical challenges. You'll provide technical expertise mitigate risk and offer solutions tailored for their needs. From migrations of existing workloads to building advanced cloud solutions you'll help shape and build to increase agility improve security reduce costs and meet utilization targets. You will work closely with the Data Engineering Data Science Infrastructure Architecture & Platform teams and will focus on creating blueprints to nourish a culture of engineering excellence. You will report into the Sr. Managing Director Big Data Governance. Responsible for developing testing and generating reports or dashboards from a host of systems and delivering results to clients. Plan and conduct comprehensive analytics services including reporting and predictive analytics. Provide expert level technical and project leadership. May manage and be a key contributor on multiple projects simultaneously

Description

Be a pioneer in business, education, and global impact by joining the Harvard Business School Digital Transformation team - a “startup with assets,” where you will have the chance to deploy cutting-edge digital- and emerging technology education solutions. Where else can you make a difference at the intersection of cutting-edge technology, world-class education, noble purpose, and timeless legacy?

Responsibilities

• Support data engineering needs across HBS, including technical support and training in Big Data frameworks and ways of working, revision and integration of source code, to release and source code quality control
• Designing and producing high performing stable end-to-end applications to perform complex processing of batch and streaming massive volumes of data in a multi-tenancy big data platform in the cloud, and output insights back to business systems according to their requirements.
• Design and implement core platform capabilities, tools, processes, ways of working and conventions under agile development to support the integration of data sourcing and use cases implementation, towards reusability, to ease up delivery and ensure standardization across data deliverables in the platform.
• Work with the architecture team to define the strategy for evolving the Big Data
• Evaluate and help determine the technologies to be used on the Big Data Platform. Investigate new technologies for improving and future-proofing the platform. Collaborate on implementation efforts with various technical teams within HBS, Harvard, and vendors as needed
• Responsible for other duties as assigned
• Interview business owners to understand business problem, questions, and pain points; Collect, analyze, and present findings to identify trends, causes, risks and opportunities
• Conversant with qualitative and quantitative techniques, including predictive analysis and visualization
• Develop business applications to provide a comprehensive suite of self-service solutions; may test prototype software and participate in approval and release process for software
• Implement solutions to standardize and systemize routine reports, dashboards, and metrics
• Advocate for and engage in the implementation of data civics, including building data catalogs, documentation efforts, educating stakeholders, and standardizing access to data. Abide by and follow the Harvard University IT technical standards, policies and Code of Conduct

Educational Requirements

• Bachelor’s degree in Mathematics, Physics, Computer Science, Engineering, Statistics, or an equivalent technical discipline or 5 years of experience is required
• 3-5 years of experience in a hands-on technical role as an engineer and/or data engineering working at scale in collaborative coding and cloud environments is required
• Public projects or demos a plus
• Experience with common build tools, unit, integration, functional and performance testing from automation perspective, and continuous delivery, under agile practices is necessary
• Experience with AWS strongly preferred, including data stores, compute engines, container services, and build tools. For example, S3, RDS, Redshift, Glue, EMR (Elastic MapReduce), Kinesis, Lambda, Step Functions, DynamoDB, Data Pipeline, CloudFormation.
• Must be fluent in Python, SQL, and coding best practices. Must be comfortable working programmatically with complex data structures (e.g., highly nested JSON or XML, nested Python lists and dictionaries).
• Must be familiar with Object Oriented design patterns
• Familiarity with scientific computing packages and platforms such as pandas and Spark preferred.
• Experience with Gen AI prompt engineering and RAG architecture preferred
• Expert-level experience in designing, building, and managing applications to process large amounts of data in a cloud ecosystem or other big data frameworks is a must