Lead Data Engineer

At INDI, we're passionate about empowering individuals and businesses worldwide. Our cutting-edge recruiters connect leading companies with top talent, fostering a dynamic environment where innovation thrives. Join us in shaping the future of work.

Overview of the role:

The Lead Data Engineer position focuses on serving as core contributors to internal Data Platforms, responsible for building scalable, resilient pipelines that support Business Operations, Analytics, and cutting-edge AI/ML initiatives. This role involves designing, implementing, and optimizing data pipelines using a wide range of Big Data and cloud-native technologies while providing technical leadership, guiding personnel, and ensuring architectural consistency across projects.

Key responsibilities:

• Providing technical leadership and mentorship to other data engineers, helping them develop and succeed in their roles.

• Leading design reviews, enforcing coding standards, and guiding architectural decisions to ensure platform scalability and maintainability.

• Partnering with senior stakeholders and engineering leadership to prioritize and plan technical work that aligns with strategic business goals.

• Acting as technical points of contact for cross-team collaborations, participating in project planning, scoping, and retrospectives.

• Demonstrating curiosity, initiative, and capability to integrate AI-powered tools into day-to-day engineering workflows.

• Supporting AI/GenAI projects by building pipelines that feed LLM apps and recommendation systems.

• Driving adoption of data engineering best practices across organizations, including testing, CI/CD, and observability.

• Designing, building, and maintaining batch and streaming data pipelines using native data platforms.

• Ingesting data from various sources, including REST APIs, cloud services, and enterprise SaaS platforms.

• Implementing robust transformation logic using Python, PySpark, SQL, and Java across structured and semi-structured datasets.

• Ensuring data integrity, lineage, and performance across ingestion, transformation, and delivery layers.

Requirements:

• Data Engineering Experience: Substantial background (6+ years) as Data Engineer in enterprise-scale data infrastructures.

• Leadership Skills: Proven leadership in technical project management and personnel mentorship.

• Architectural Guidance: Track record of guiding architectural conversations in high-throughput data ecosystems.

• Programming Mastery: Advanced proficiency in SQL and Python programming, with expertise in PySpark or Java.

• Data Architecture: Comprehensive knowledge of data modeling, transformation methodologies, and data warehouse architectures.

• Hadoop Ecosystem: Proven expertise with Hadoop ecosystem technologies.

• Pipeline Development: Successful implementation of resilient, high-performance data pipelines meeting strict service level agreements.

• Integration Expertise: Deep understanding of API integration, performance optimization, and data schema adaptation.

• Language Proficiency: Advanced level of English.

Additional skills preferred:

• Workflow Tools: Practical implementation experience with Apache NiFi and Apache Airflow.

• Development Environment: Expertise in containerized environments, developer tooling, and API frameworks.

• AI/ML Platforms: Demonstrated capabilities in AI/ML platform support and generative AI pipeline development.

• Innovation Contributions: Proven contributions to internal technological innovations within data engineering workflows.

What to expect from us:

• Competitive Compensation: Excellent payment in USD or your preferred local currency.

• Paid Leave: Parental leave, vacation, and national holidays.

• Dynamic Work Culture: Innovative and multicultural environment.

• Elite Collaboration: Work with the global top 1% of talent.

• Growth and Support: Mentorship and career development opportunities.

If you are interested in being part of a team composed of the best professionals and working 100% goal-oriented in an innovative environment, do not hesitate to apply