Sr Software Engineer II- Data Engineering and Platform

LG Ad Solutions

LG Ad Solutions is a global leader in connected TV (CTV) and cross-screen advertising. We pride ourselves on delivering state-of-the-art advertising solutions that integrate seamlessly with today's ever-evolving digital media landscape.

The Opportunity:

We are seeking a highly skilled and experienced developer to join our team in the role of Senior Software Engineer II – Data Engineering & Platform. This pivotal role focuses on providing technical vision and leadership for a major business component, managing end-to-end processes that operate at a high scale, processing several petabytes of data daily. The ideal candidate will have a robust background in data engineering, coupled with a proven track record in technical leadership. You will collaborate with cross-functional teams to drive technical innovation and deliver robust, scalable solutions that meet our business needs.

Responsibilities:

• Architect and Implement Scalable Data Solutions:
  Design, develop, and deploy comprehensive data solutions encompassing ETL pipelines, data warehousing, data architecture, data cataloguing, and data delivery mechanisms. Ensure these solutions are scalable, reliable, and maintainable to support evolving business needs.
• Lead Data Platform Strategy:
  Develop and execute the strategic vision for the data platform, ensuring alignment with organizational objectives. Oversee the architecture and management of data systems to support efficient data storage, processing, and utilization.
• Collaborate Cross-Functionally:
  Work closely with product managers, engineering teams, data scientists, and other stakeholders to design and implement effective data integration and hand-off mechanisms. Facilitate seamless data flow and accessibility across various systems and teams.
• Optimize Performance and Scalability:
  Identify and address performance bottlenecks within the data platform. Implement strategies to enhance system performance, scalability, and reliability, ensuring the infrastructure can handle large-scale data processing efficiently.
• Oversee Data Governance and Quality:
  Implement and maintain data governance frameworks to ensure data quality, accuracy, consistency, and security. Establish policies and procedures for data management, ensuring compliance with regulatory requirements and industry best practices.
• Mentor and Lead Engineering Teams:
  Provide technical leadership and mentorship to data engineers and other team members. Foster a culture of innovation, ownership, and continuous learning within the team, guiding them through complex technical challenges and project deliveries.
• Conduct Code and Architecture Reviews:
  Lead code and architecture reviews to maintain high standards of code quality and system design. Provide constructive feedback and ensure adherence to best practices and organizational standards.
• Stay Abreast of Industry Trends:
  Keep up-to-date with emerging technologies, tools, and methodologies in data engineering and platform development. Evaluate and recommend new solutions to enhance the data platform's capabilities and efficiency.

Requirements:

• Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
• Over 5 years of software development experience, with at least 2 years in a technical leadership role overseeing data engineering or data platform teams.
• Strong proficiency in big data technologies, including Apache Spark/PySpark, Apache Airflow and Apache Kafka.
• Experience with programming languages such as Scala and Python for developing robust data pipelines.
• In-depth knowledge of both relational and non-relational databases, with the ability to design and optimize complex queries.
• Hands-on experience with data visualization tools and techniques to present data insights effectively.
• Proficiency in developing and maintaining CI/CD pipelines using tools like Jenkins and GitHub Actions.
• Strong experience in writing and maintaining unit, integration, and end-to-end tests to ensure data pipeline reliability and accuracy.
• Excellent problem-solving abilities with a keen attention to detail, capable of diagnosing and resolving complex technical issues.
• Strong communication skills, with the ability to convey complex technical concepts to diverse audiences.
• Proven ability to collaborate cross-functionally with product managers, data scientists, and other engineering teams to deliver cohesive data solutions.
• Familiarity with data governance and security practices to ensure compliance and data integrity.

Preferred:

• Experience with Infrastructure as Code (IaC) tools such as Pulumi and Terraform for managing cloud resources.
• Prior experience working with Databricks and AWS cloud services.
• Experience with Agile development methodologies and leading Agile teams.
• Experience in working with Distributed Graph Databases.