Data Engineer III

As a Data Engineer at ACV Auctions you HAVE FUN You will design, develop, write, and modify code.  You will be responsible for development of ETLs, application architecture, optimizing databases & SQL queries.  You will work alongside other data engineers and data scientists in the design and development of solutions to ACV's most complex software problems.  It is expected that you will be able to operate in a high performing team, that you can balance high quality delivery with customer focus, and that you will have a record of delivering and guiding team members in a fast-paced environment.

• Design, develop, and maintain scalable ETL pipelines using Python and SQL to ingest, process, and transform data from diverse sources.
• Write clean, efficient, and well-documented code in Python and SQL.
• Utilize Git for version control and collaborate effectively with other engineers.
• Implement and manage data orchestration workflows using industry-standard orchestration tools (e.g., Apache Airflow, Prefect)..
• Apply a strong understanding of major data structures (arrays, dictionaries, strings, trees, nodes, graphs, linked lists) to optimize data processing and storage.
• Support multi-cloud application development.
• Contribute, influence, and set standards for all technical aspects of a product or service including but not limited to, testing, debugging, performance, and languages.
• Support development stages for application development and data science teams, emphasizing in MySQL and Postgres database development.
• Influence company wide engineering standards for tooling, languages, and build systems.
• Leverage monitoring tools to ensure high performance and availability; work with operations and engineering to improve as required.
• Ensure that data development meets company standards for readability, reliability, and performance.
• Collaborate with internal teams on transactional and analytical schema design.
• Conduct code reviews, develop high-quality documentation, and build robust test suites
• Respond-to and troubleshoot highly complex problems quickly, efficiently, and effectively.
• Participate in engineering innovations including discovery of new technologies, implementation strategies, and architectural improvements.
• Participate in on-call rotation

What you will need:

• Bachelor's degree in computer science, Information Technology, or a related field (or equivalent work experience)
• Ability to read, write, speak, and understand English.
• 3+ years of experience programming in Python
• 3+ years of experience with ETL workflow implementation (Airflow, Python)
• 3+ years work with continuous integration and build tools.
• 2+ year of experience with Cloud platforms preferably in AWS or GCP
• Knowledge of database architecture, infrastructure, performance tuning, and optimization techniques.
• Knowledge in day-day tools and how they work including deployments, k8s, monitoring systems, and testing tools.
• Proficient in version control systems including trunk-based development, multiple release planning, cherry picking, and rebase.
• Proficient in databases (RDB), SQL, and can contribute to table definitions.
• Self-sufficient debugger who can identify and solve complex problems in code.
• Deep understanding of major data structures (arrays, dictionaries, strings).
• Experience with Domain Driven Design.
• Experience with containers and Kubernetes.
• Experience with database monitoring and diagnostic tools, preferably Data Dog.
• Hands-on skills and the ability to drill deep into the complex system design and implementation.
• Proficiency in SQL query writing and optimization.
• Familiarity with database security principles and best practices.
• Familiarity with in-memory data processing
• Knowledge of data warehousing concepts and technologies, including dimensional modeling and ETL frameworks
• Strong communication and collaboration skills, with the ability to work effectively in a fast paced global team environment.

• Experience working with:

• SQL data-layer development experience; OLTP schema design

• Using and integrating with cloud services, specifically: AWS RDS, Aurora, S3, GCP
• Github, Jenkins, Python

Nice to Have Qualifications:

• Experience with Airflow, Docker, Visual Studio, Pycharm, Redis, Kubernetes, Fivetran, Spark, Dataflow, Dataproc, EMR
• Experience with database monitoring and diagnostic tools, preferably DataDog.
• Hands-on experience with Kafka or other event streaming technologies.
• Hands-on experience with micro-service architecture