Opportunity For Data Engineer

Here is the job description for Data Engineer resources we are looking for:

Key Responsibilities:

• Lead Data Engineering Projects: Oversee the development and deployment of end-to-end data ingestion pipelines using Azure Databricks, Apache Spark, and related technologies, ensuring scalability, performance, and efficiency.
• Design & Architecture: Design high-performance, resilient, and scalable data architectures for data ingestion and processing using best practices for Azure Databricks and Spark.
• Team Leadership: Provide technical guidance and mentorship to a team of data engineers, fostering a culture of collaboration, continuous learning, and innovation.
• Collaboration: Work closely with data scientists, business analysts, and other stakeholders to understand data requirements and ensure smooth integration of various data sources into the data lake/warehouse.
• Optimization & Performance Tuning: Ensure data pipelines are optimized for speed, reliability, and cost efficiency in an Azure environment. Conduct performance tuning, troubleshooting, and debugging of Spark jobs and Databricks clusters.
• Code Quality & Best Practices: Enforce and advocate for best practices in coding standards, version control, testing, and documentation.
• Integration with Azure Services: Work with other Azure services such as Azure Data Lake Storage, Azure SQL Data Warehouse, Azure Synapse Analytics, and Azure Blob Storage to integrate data seamlessly.
• Continuous Improvement: Stay current with industry trends and emerging technologies in the field of data engineering and make recommendations for improvements to the teams tools and processes.
• Ensure Data Quality: Implement data validation and data quality checks as part of the data ingestion process to ensure consistency, accuracy, and integrity of ingested data.
• Risk & Issue Management: Proactively identify risks and blockers and resolve complex technical issues in a timely and effective manner.

Qualifications:

• Education: Bachelor's or master's degree in computer science, Information Technology, Engineering, or a related field.

• Experience:

• 8+ years of experience in data engineering or a related field.

• Strong hands-on experience with Azure Databricks, Spark, Python/Scala, CICD, Scripting for data processing.
• Experience working in multiple file formats like Parquet, Delta, and Iceberg.
• Knowledge of Kafka or similar streaming technologies for real-time data ingestion.
• Experience with data governance and data security in Azure.
• Proven track record of building large-scale data ingestion and in cloud environments, specifically Azure.
• Deep understanding of Azure Data Services (e.g., Azure Blob Storage, Azure Data Lake, Azure SQL Data Warehouse, Event Hubs, Functions etc.).
• Familiarity with data lakes, data warehouses, and modern data architectures.
• Experience with CI/CD pipelines, version control (Git), Jenkins and agile methodologies.

• Understanding of cloud infrastructure and architecture principles (especially within Azure).

• Technical Skills:

• Expert-level proficiency in Spark, SPARK Streaming, including optimization, debugging, and troubleshooting Spark jobs.

• Solid knowledge of Azure Databricks for scalable, distributed data processing.
• Strong coding skills in Python and Scala for data processing.
• Experience working with SQL, especially for large datasets.

• Knowledge of data formats like Iceberg, Parquet, ORC, and Delta Lake.

• Leadership Skills:

• Proven ability to lead and mentor a team of data engineers, ensuring adherence to best practices.

• Excellent communication skills, capable of interacting with both technical and non-technical stakeholders.
• Strong problem-solving, analytical, and troubleshooting abilities.

Soft Skills:

• Excellent communication and interpersonal skills.
• Strong organizational skills with the ability to manage multiple tasks and priorities.
• Ability to work in a fast-paced, constantly evolving environment.
• Strong collaborative skills and ability to work effectively across teams.