Senior Data Engineer

Role Overview:The Senior Data Platform Engineer is an experienced professional who designs and optimizes complex data processing solutions to address sophisticated business needs. In this role, you will tackle large -scale batch and streaming data projects, ensuring that data pipelines and platforms are scalable, high -performing, and secure. A Senior Data Platform Engineer acts as a technical expert in the team, often mentoring junior engineers and leading critical implementation efforts. This role requires deep expertise in data engineering tools and frameworks – including proficiency in Apache Spark, Trino, or Apache Iceberg for big data processing and analytics – and the ability to leverage these technologies to build robust data products. This position is also client -focused, demanding excellent communication to collaborate with client stakeholders on a near -daily basis. Key Responsibilities: Architect and Optimize Data Pipelines: Lead the design, development, and optimization of large -scale data pipelines, handling both batch data movement and real -time streaming data. Ensure pipelines are built for scalability (able to handle growing data volumes) and low -latency data delivery where needed.Technical Leadership: Serve as a subject matter expert in the data platform team. Review code, enforce engineering best practices, and mentor junior data engineers in improving pipeline reliability, data modeling, and coding standards. Proactively introduce improvements or new technologies to enhance the data platform.Advanced Data Processing & Analytics: Utilize advanced big data frameworks and query engines (e.g., Spark, Trino, etc.) to implement complex data transformations and aggregations. For example, use Apache Spark (a unified analytics engine for large -scale data processing) or Trino (a distributed SQL query engine for big data analytics) to process and query massive datasets efficiently. Leverage Apache Iceberg (a high -performance table format for huge analytic tables) for managing large -scale data lake storage and enabling fast analytical queries.Data Flow Management: Oversee the scheduling and orchestration of data workflows to ensure timely availability of data for analytics. Implement monitoring and observability on data pipelines to detect issues, and ensure data reliability and quality for downstream use.Collaborate with Stakeholders: Work closely with data analysts, data scientists, and other engineering team members to understand data needs and ensure the platform meets business requirements. Liaise with client teams regularly to gather requirements and present data solutions that address their problems. As a senior engineer, you may lead technical discussions with clients, offering guidance on data best practices and platform capabilities.Data Governance and Security: Champion data governance standards within projects – implement data validation, access controls, and compliance measures. Ensure that the data platform adheres to security best practices and that sensitive data is handled in accordance with corporate and regulatory guidelines.Data Infrastructure Management: Support the underlying data infrastructure (e.g., databases, cloud storage, processing frameworks) to ensure high availability and optimal performance of data systems. Uphold best practices in data security and governance to protect sensitive information and comply with relevant policies. RequirementsExtensive Data Engineering Experience: Several years (around 5 years) of hands -on experience in data engineering or related fields. Demonstrated ability to design and manage complex data ingestion frameworks for both structured and unstructured data.Expertise in Big Data Technologies: Proven expertise in one or more modern big data processing technologies. Experience with Apache Spark, Presto/Trino, or Apache Iceberg is required for this role, as you will be leveraging these systems to handle large -scale data processing and querying. Familiarity with distributed data storage and indexing techniques is expected.Proficient in SQL and Python: Mastery of SQL for complex querying (including performance tuning on large datasets) and Python for building data pipeline scripts or tooling. Ability to work with other programming languages (Scala, Java, etc.) for Spark or similar frameworks is a plus.Pipeline and API Integration Skills: Strong experience in developing ETL/ELT pipelines and using workflow orchestration tools. Comfortable working with streaming data platforms (e.g., Kafka or Flink) and event -driven architectures. Also, proficient in integrating external data sources via APIs and handling real -time data feeds.Data Architecture and Modeling: Solid understanding of data modeling principles and the design of data warehouses/data lakes. Ability to design efficient schema and storage solutions for analytics. Experience with optimizing query performance and resource utilization in big data environments.Problem Solving and Leadership: Excellent problem -solving skills applied to debugging data issues and optimizing pipeline performance. Demonstrated leadership in taking ownership of projects, and ability to communicate technical concepts to both technical teams and non -technical stakeholders clearly. Experience in a client -facing environment or consulting projects is highly valued, as the role involves guiding clients through technical decisions.Education: Bachelor's or Master's degree in Computer Science, Data Engineering, or a related discipline. Continuous learning mindset to stay updated with the latest data engineering technologies and best practices.