Data/Scala Engineer

Skills

Description

We’re looking for an engineer who will lead the effort to register all the datasets generated within Twitter with the Metadata system called DAL. You can find more information about this system via this blog post (link below) The registration of the datasets is critical for legal (GDPR laws) and security purposes. Registration will also enable engineers and Product Managers to discover all datasets offered by various different teams that can be used for experimentation purposes.

This role will be critical as it doesn’t only entail writing code to help integrate the datasets, but will also involve collaborating with customers from various teams from Revenue, Consumer, and Machine Learning orgs. You will benefit from learning and gaining experience with Twitter’s data pipelines and technology behind them.

The primary responsibilities will include:
● Understand DAL
● Understand Twitter’s Data format
● Understand internal data discovery tool
● Write code to integrate batch applications (Scalding jobs) with DAL
● Collaborate with various teams to understand what their datasets do and plan next steps
● Help teams modernize their existing scripts using newer Scalding sources

Responsibilities

• This initial work is projected to take eight months; there are about ten other pipeline projects. This is an extendable contract, and there is a ton of work to be done, so if someone can come in and make an impact, they have a good shot at sticking around. This is the first of 10 projects, all of which could follow; this is an extendable position.
• Twitter's Data environment is unique. It takes time to ramp up; these people will have exposure, so some of the largest Hadoop clusters in the world but will become vital to Twitter's data infrastructure.