Data Engineer(Azure)

Position:  Data Engineer

Location:  Chennai/Banglore/Hyderabad/Trivandrum, PST Timing

Job Summary 

We are seeking a Data Engineer to work on a Generative AI initiative to join our team. The ideal candidate will have a deep understanding of data modeling, data schemas and has developed ETLs from various sources eensuring high data availability, fault tolerance and security and governance.

Responsibilities: 
- Collaborate with stakeholders to understand data requirements and design scalable and efficient data models, schemas, and architectures on the Azure platform.
- Develop and implement data integration and ETL (Extract, Transform, Load) processes to ingest, transform, and load data from various sources into Azure data storage systems.
- Build and maintain scalable and reliable data pipelines using Azure Data Factory, Azure Databricks, or similar tools to automate the movement and transformation of data.
- Optimize data storage and retrieval mechanisms, ensuring high data availability, fault tolerance, and efficient query performance.
- Hands-on experience with cloud platforms (MS Azure, AWS or Google Cloud Services)
- Hands-on experience with distributed computing (preferably using Apache Spark or Databricks)
- Monitor and troubleshoot data pipelines, identifying and resolving performance issues, data quality problems, and data integration challenges.
- Collaborate with data scientists and analysts to provide them with access to reliable and well-structured data sets for analysis and reporting purposes.
- Implement data governance and security measures to protect sensitive data, ensure compliance with regulatory standards, and adhere to data privacy policies.
- Work closely with the DevOps team to automate deployment processes, ensure continuous integration and delivery of data solutions, and maintain infrastructure as code.
- Stay updated with the latest advancements in Azure data services, evaluate and implement new tools and features to enhance data engineering processes.

Requirements 
- Bachelor's or masters degree in computer science, Engineering, or a related field.
- Proven experience as a Data Engineer, with a strong focus on Azure cloud platform.
- In-depth knowledge of Azure data services such as Azure Data Factory, Azure Databricks, Azure Synapse Analytics, Azure SQL Database, Azure Data Lake Storage, and Azure Cosmos DB.
- Proficiency in designing and implementing scalable data models, data warehouses, and data lakes.
- Strong understanding of ETL processes, data integration patterns, and data transformation techniques.
- Experience with programming languages such as Python, SQL, and PowerShell for data manipulation, scripting, and automation.
- Familiarity with big data technologies and frameworks like Hadoop, Spark, and NoSQL databases is a plus.
- Strong analytical and problem-solving skills, with the ability to troubleshoot and optimize data pipelines.
- Excellent communication and collaboration skills to work effectively in a team environment and interact with stakeholders from various disciplines.

Preferred Qualifications :
- Bachelor's or master's degree in computer science, Data Science, Statistics, Math, Physics, or other Science related discipline with course work in AI/ML.
- Cloud certifications preferred.
- Contribute to the design and implementation of new data-driven solutions, using cutting-edge machine learning and artificial intelligence techniques.
- Self-motivated with a strong desire to learn and stay up to date with new technologies in the field.
- Ability to work collaboratively in a team environment and communicate complex technical concepts to non-technical stakeholders.