Data Engineer

Description

We have a flexible model for team organization at Crunchbase, centered around the principles of agile software development, allowing us to fluidly adapt to new requirements and situations as they arise. We tend towards two main types of teams Featureoriented teams operate independently and own their projects from cradle to grave, building everything they need to execute. While they don't necessarily have to do all the work themselves, it's their responsibility to make sure the business needs are met all the way into production. Functionoriented teams provide support to feature teams and have deeper expertise on particular parts of the stack. It's their responsibility to build platforms that are so good the other teams want to use them, rather than being forced into it.We also ship code as early as we possibly can; getting betas into customer hands is a top priority. Our goal is to tune our process to customer needs as efficiently and quickly as possible. Crunchbase is built with a variety of tools a core part of our engineering philosophy is to use the right tool for the job Ruby, Scala, Java Node.js, Angular.js, Typescript Redis, PostgreSQL, ElasticSearch Git, Nginx, Docker, NewRelic, Jenkins AWS EC2, ELB, VPC, S3, R53, RDS, SQS, etc.Data Engineering at CrunchbaseOur data team is responsible for building and maintaining the infrastructure for our data needs. As Crunchbase grows, so too does the amount of data we process, the number of sources it comes from, and the number of ways that people want to slice and dice it. We are currently building out a robust pipeline for our core data and were continually expanding use cases for it, so it must be both scalable and flexible.At its core, Crunchbase is a data company, and data engineering is at the heart of our platform and will propel us into the future. The responsibilities of data engineers at Crunchbase include Architect and build new dimensional data models and schema designs to improve accessibility, efficiency, consistency and quality of both internal and production data. Build, monitor, and maintain analytics and production data ETL pipelines. Provide the foundation for a datadriven culture by empowering other engineers and the Product team to ask questions of the dataset in an easy, reliable way. Enable data scientists to implement NLP and ML algorithms at scale, in faulttolerant, highly available systems.Qualifications Solid understanding of computer science and software engineering fundamentals. Motivated to participate in ongoing learning and growth through pair programming, code reviews, application of new technologies and best practices. Excellent verbal and written communication skills. Familiarity with tools we presently use is a plus, but not required if you know something better, we may use that instead!