Lead Data Engineer

Technical Lead (Client-Side Role)

Experience : 8+ years overall in Data Engineering, Analytics, and Python development with at least 3+ years in a Technical Lead / Project Management role

About the Role :

We are seeking an experienced Technical Lead with strong client-facing skills to drive data engineering and analytics projects for our customers. This role requires hands-on expertise in cloud data architectures, Python-based pipelines, real-time streaming integrations, security best practices, and effective project delivery. The ideal candidate will be an excellent communicator who can design robust solutions, guide teams, and ensure smooth collaboration between technical and business stakeholders.

Key Responsibilities :

Technical Leadership & Solution Architecture :

- Design and implement secure, scalable data architectures on AWS, Azure, or GCP.

- Architect and deliver cloud-based data engineering solutions covering data ingestion, transformation, and storage.

- Define best practices for integrating diverse data sources and databases securely, considering SSL, firewalls, IAM policies, VPNs, and encryption.

- Oversee security aspects of integrations, ensuring compliance with organizational and regulatory requirements.

- Optimize and maintain data warehouse solutions (BigQuery, Snowflake, Redshift, etc.) with a focus on performance, governance, and cost efficiency.

- Lead design reviews and architecture discussions with both internal and client teams.

Data Engineering & Real-Time Streaming :

- Develop and manage robust ETL/ELT pipelines using Python, SQL, and modern orchestration tools.

- Integrate real-time streaming data using Apache Kafka, Spark Structured Streaming, or cloud-native services.

- Work on data ingestion strategies from APIs, flat files, databases, and streaming sources with focus on reliability and scalability.

- Design delta and CDC (Change Data Capture) strategies for near real-time updates.

- Ensure efficient data processing for large-scale, distributed systems.

Data Analysis & Business Intelligence :

- Perform advanced data analysis to generate actionable insights for business use cases.

- Design and deliver intuitive Tableau dashboards and data visualizations with effective storytelling.

- Define and implement data quality checks, anomaly detection, and validation frameworks to ensure high integrity data pipelines.

- Collaborate with analysts and business users to translate requirements into technical designs.

Software Development & API Engineering :

- Develop and deploy REST APIs and backend services for data products and integrations.

- Contribute to full-stack integration as needed, including basic front-end implementation.

- Ensure secure and efficient API development with proper authentication, authorization, and monitoring.

- Conduct code reviews, enforce best practices, and maintain high-quality standards across the development team.

AI, Machine Learning & Advanced Analytics :

- Support ML/AI project implementation, including predictive modeling, deep learning, and Generative AI/LLM use cases.

- Collaborate with data scientists to integrate AI models into production pipelines and cloud infrastructure.

- Enable secure and scalable deployment of AI models using cloud services.

Required Skills & Qualifications :

- Cloud Platforms : Deep hands-on experience with AWS, Azure, or GCP for designing secure, scalable data solutions.

- Programming & Data Engineering : Expert in Python, SQL, Spark, Kafka, and streaming integration.

- Data Warehousing : Proven ability with BigQuery, Snowflake, Redshift, including ETL/ELT pipeline design and optimization.

- Integration Security : Strong understanding of network configurations, VPNs, firewalls, SSL/TLS, IAM policies, and secure service account management.

- Real-Time Streaming : Experience designing and operating streaming data pipelines with minimal latency.

- Data Analytics & BI : Skilled in data visualization with Tableau, capable of delivering clear, actionable insights.

- Software Engineering : Proficiency in REST API development, backend services, and integrating APIs securely.

- AI/ML Integration : Exposure to ML, deep learning, LLM, and Generative AI with cloud-based deployment.

- Quality Assurance : Strong focus on automated testing, data validation, and performance monitoring.

Preferred Qualifications :

- Prior experience managing enterprise-scale data engineering projects and leading technical teams.

- Experience with DevOps practices including CI/CD pipelines, Infrastructure as Code (Terraform, CloudFormation).

- Familiarity with modern front-end frameworks (React, Angular) for UI integration.

- Understanding of regulatory compliance requirements for data handling (e.g., GDPR, HIPAA).