Senior Data Engineer

Years of Experience: 6

Job Description: We are seeking an experienced and detail-oriented Senior Data Engineer with strong expertise in QA process automation for data validation and testing. The ideal candidate will bring hands-on experience with QuerySurge and a solid foundation in ETL/ELT processes, data integration, and data quality assurance. In this role, you will be responsible for designing, developing, and automating data validation and testing workflows across our data pipelines. You will ensure data integrity, accuracy, and reliability by leveraging automated testing frameworks and working closely with cross-functional teams.A strong background in data engineering, SQL, and QA automation practices is required.

Primary Skills: AWS, ETL Concepts,Data Integration Tool (Any),QA

Secondary Skills: Python, DataStage, Informatica IICS, Informatica , SSIS

Role Description:

The Lead Data Engineer will be responsible for overseeing the design, development, and maintenance of our data infrastructure and pipelines. This role requires strong technical expertise, leadership skills,and the ability to work collaboratively with cross-functional teams to ensure efficient and reliable data processes.

Role Responsibility:

Lead the design and architecture of scalable and robust data integration solutions on-premise and in the cloud.

Ensure solutions meet business requirements and industry standards.

Engage with executive leadership and key stakeholders to understand business needs and translate them into technical solutions.

Define data models, schemas, and structures to optimize data storage, retrieval, and processing for analytical workloads.

Work with data architects and solution architects to ensure alignment with overall data strategy and architecture principles.

Lead a team of data engineers by providing technical guidance, mentorship, and support.

Plan, prioritize, and manage data engineering projects, tasks, and timelines.

Design, develop, and maintain data integration pipelines and ETL/ELT workflows.

Lead more than one project or manage a larger team that might have sub-tracks.

Role Requirement:

Proficient in basic and advanced SQL.

Proficient in using Python for data integration and data engineering tasks.

Proficiency in ETL/ELT tools such as Informatica, Talend, DataStage, SSIS, DBT, Databricks, or equivalent.

Experience with relational databases (like SQL Server, Oracle, MySQL, PostgreSQL) and NoSQL databases (like MongoDB, Cassandra), cloud databases (RedShift, Snowflake, Azure SQL).

Familiarity with big data technologies like Hadoop, Spark, Kafka, and cloud platforms such as AWS, Azure, or Google Cloud.

Solid understanding of data modeling, data warehousing concepts, and practices.

Good Knowledge and Understanding of Data warehouse concepts (Dimensional Modeling, change data capture, slowly changing dimensions etc.).

Knowledgeable in performance tuning and optimization

Experience in Data Profiling and Data validation

Experience in requirements gathering and documentation processes, and performing unit testing.

Understanding and implementing QA and various testing processes in the project.

Knowledge of any BI tools will be an added advantage.

Sound aptitude, outstanding logical reasoning, and analytical skills.

Willingness to learn and take initiative.

Ability to adapt to a fast-paced Agile environment.

Relevant certifications in data engineering or cloud platforms are a plus.

Additional Requirement:

6+ years of experience in Data Engineering with exposure to data quality and QA processes.

2+ years of hands-on experience in QuerySurge for test automation and data validation.

Strong proficiency in SQL for validation queries and test case design.

Experience with ETL/ELT tools (Informatica, Talend, DataStage, SSIS, DBT, or equivalent) and Redshift.

Proficiency in Python for scripting, automation, and test integration.

Familiarity with CI/CD pipelines, version control (Git), and Agile environments.

Strong understanding of data warehousing concepts, dimensional modeling, and data governance.

Experience in requirements gathering, test case creation, and defect lifecycle management.

Excellent analytical skills, logical reasoning, and problem-solving capabilities.

Relevant certifications in QA automation, data engineering, or cloud platforms are preferred.