Data Engineer

Role : Staff Data Engineer

Location : Hyderabad, India

Experience : 9-14 Years

Employment Type : Full-time

Job Overview :

We are seeking an exceptionally skilled and experienced Staff Data Engineer with 9-14 years of hands-on experience in data engineering and/or software development. The ideal candidate will be a technical leader, responsible for designing, building, and optimizing our large-scale data infrastructure and pipelines. This role demands deep expertise in programming languages, big data technologies, cloud services, and real-time data systems, with a strong emphasis on driving data-driven insights for product and AI/ML Data Architecture & Strategy : Lead the design and implementation of highly scalable, robust, and efficient data architectures, pipelines, and data models to support analytical, reporting, and AI/ML needs.

- ETL/ELT Development : Develop, optimize, and maintain complex ETL/ELT processes for large-scale data ingestion and transformation using Python, Spark, and SQL.

- Big Data Ecosystem Management : Drive the utilization and optimization of compute and orchestration tools such as Databricks, Airflow, and Talend across the entire data lifecycle.

- Real-time Data Systems : Design and build real-time data systems, including leveraging streaming OLAP engines like Druid and ClickHouse, to support immediate insights and operational analytics.

- Cloud Data Solutions : Architect and implement solutions leveraging a wide range of AWS services, including EMR Spark, Redshift, Kinesis, Lambda, Glue, S3, and Athena.

- Data Quality & Governance : Establish and enforce best practices for data quality, governance, and metadata management, ensuring accuracy and reliability of data.

- Performance Optimization : Proactively identify and resolve performance bottlenecks in data pipelines and analytical queries, managing Terabytes of data and billions of records effectively.

- BI & Analytics Enablement : Collaborate with BI analysts and product teams to enable effective reporting, dashboarding, and analytics using tools like Tableau, Looker, and Preset.

- AI/ML Personalization Support : Contribute to the infrastructure and data pipelines required for building and deploying AI/ML personalization products.

- Event Data Management : Work proficiently with event data collection tools such as Snowplow, Segment, Google Tag Manager, Tealium, and mParticle.

- Security & Compliance : Implement and ensure adherence to stringent data privacy and security standards, particularly in high-security environments like HIPAA, PCI, or similar contexts.

- Mentorship & Leadership : Provide technical leadership and mentorship to junior data engineers, fostering a culture of excellence and continuous learning.

Requirements :

- Experience : 9-14 years of hands-on experience in data engineering and/or software development roles.

- Programming Expertise : Highly skilled in programming languages such as Python, Spark, and SQL.

- BI Tool Familiarity : Comfortable utilizing Business Intelligence (BI) tools like Tableau, Looker, Preset, and others for data validation and analysis.

- Event Data Collection : Proficient in utilizing event data collection tools such as Snowplow, Segment, Google Tag Manager, Tealium, mParticle, and more.

- Compute & Orchestration : Comprehensive expertise across the entire lifecycle of implementing and managing compute and orchestration tools like Databricks, Airflow, Talend, and other similar platforms.

- Streaming OLAP Engines : Skilled in working with streaming OLAP engines such as Druid, ClickHouse, and similar technologies.

- AWS Services : Extensive experience leveraging AWS services, including but not limited to EMR Spark, Redshift, Kinesis, Lambda, Glue, S3, and Athena.

- Real-time Data Systems : Good understanding and practical experience in building real-time data systems.

- AI/ML Data Products : Experience with data pipelines supporting AI/ML personalization products.

- Customer Data Platforms : Experience with Customer Data Platforms (CDPs) and Data Management Platforms (DMPs), contributing to holistic data strategies.

- Data Security : Familiarity with high-security environments like HIPAA, PCI, or similar contexts, highlighting a commitment to data privacy and security.

- Large-Scale Data Management : Accomplished in managing large-scale data sets, effectively handling Terabytes of data and billions of records.

Nice to Have :

- Exposure to GCP services like BigQuery, Google Storage, Looker, Google Analytics, and similar.