Data Engineering Director

Job Title: Senior Data Engineering Lead (Databricks)

Company Overview:

At eClerx, we are a leading IT firm specializing in innovative technologies and solutions that drive business transformation. Leveraging expertise in business process management, advanced analytics, and smart automation, we empower our clients to achieve operational excellence and competitive advantage in fast-evolving markets.

Role Overview:

We are seeking a highly experienced Senior Data Engineering Lead with a strong focus on Databricks and cloud-based data engineering to lead our data engineering team. This leadership role requires a visionary who can design, develop, and manage scalable data infrastructure and pipelines, while mentoring and inspiring a team of data engineers. You will work closely with cross-functional teams including data scientists, analysts, and software engineers to enable robust data-driven decision-making and support business goals.

Key Responsibilities:

- Lead and manage a team of data engineers, providing mentorship, technical guidance, and fostering a culture of collaboration and innovation.
- Architect, design, and oversee implementation of large-scale data pipelines, data lakes, and cloud-based data warehouses using Databricks, Apache Spark, and Snowflake.
- Develop and optimize ETL/ELT workflows ensuring performance, reliability, and scalability of data infrastructure.
- Collaborate with business stakeholders, data scientists, and software teams to understand requirements and translate them into scalable, efficient data solutions.
- Implement best practices for data quality, governance, security, and compliance.
- Drive continuous improvement of data engineering processes, standards, and tools across the organization.
- Support presales activities by contributing to RFPs, technical proposals, and client engagements.
- Stay abreast of emerging data technologies and trends, recommending innovative solutions to enhance analytics capabilities.
- Manage resource planning, project prioritization, and delivery timelines ensuring alignment with business objectives.
- Lead performance reviews, identify skill gaps, and champion professional development within the data engineering team.
- Facilitate cross-team communication to streamline data workflows and improve overall delivery.

Qualifications & Skills:

- Bachelor's or Master's degree in Computer Science, Data Science, Engineering, or a related discipline.
- Minimum 15 years of professional experience in data engineering with at least 9 years in leadership or senior technical roles.
- Deep hands-on expertise with Databricks and Apache Spark for large-scale data processing.
- Strong programming skills in Python, Scala, or Java.
- Extensive experience with cloud data platforms such as AWS, Azure, or GCP, including services like S3, Redshift, BigQuery, Snowflake.
- Solid understanding of data modeling, data warehousing, ETL/ELT design, and data lakes.
- Experience with big data technologies like Hadoop, Kafka, and Databricks ecosystem.
- Knowledge of CI/CD pipelines, data orchestration tools (e.g., Apache Airflow), and data governance best practices.
- Proven experience managing high-performing teams and delivering complex data engineering projects on time.
- Familiarity with analytics solutions and the ability to translate business needs into technical requirements.
- Strong communication skills, capable of engaging with both technical teams and senior leadership.
- Experience supporting presales efforts and client technical discussions is a plus.
- Bonus: Exposure to machine learning lifecycle and model deployment on Databricks.

---