Azure & Big Data - Lead

Job Description

Why Join Us
Are you inspired to grow your career at one ofIndia's Top 25 Best Workplaces in IT industryDo you want to do the best work of your life at one of thefastest growing IT services companies Do you aspire to thrive in an award-winning work culture thatvalues your talent and career aspirations It'shappening right hereat Iris Software.

About Iris Software
At Iris Software, our vision is to be our client's most trusted technology partner, and the first choice for the industry's top professionals to realize their full potential. With over 4,300 associates across India, U.S.A, and Canada, we help our enterprise clients thrive with technology-enabled transformation across financial services, healthcare, transportation & logistics, and professional services.

Our work covers complex, mission-critical applications with the latest technologies, such as high-value complex Application & Product Engineering, Data & Analytics, Cloud, DevOps, Data & MLOps, Quality Engineering, and Business Automation.

Working at Iris
Be valued, be inspired, be your best.
At Iris Software, we invest in and create a culture where colleagues feel valued, can explore their potential, and have opportunities to grow.
Our employee value proposition (EVP) is about Being Your Best - as a professional and person. It is about being challenged by work that inspires us, being empowered to excel and grow in your career, and being part of a culture where talent is valued. We're a place where everyone can discover and be their best version.

Job Description

Job Summary:
We are seeking a highly skilled and experienced Lead Data Engineer to join our data engineering team. The ideal candidate will have a strong background in designing and deploying scalable data pipelines using Azure technologies, Spark, Flink, and modern data lakehouse architectures. This role demands hands-on technical expertise, leadership in managing offshore teams, and a strategic mindset to drive data-driven decision-making across financial and regulatory domains

Key Responsibilities:
. Design, develop, and deploy scalable batch and streaming data pipelines using PySpark, Flink, Scala, SQL, and Redis.
. Lead migration of complex on-premise workflows to Azure cloud ecosystem (Databricks, ADLS, Azure Data Factory), optimizing infrastructure and deployment processes.
. Implement performance tuning strategies to reduce job runtimes and enhance data reliability, including optimization of Unity Catalog tables.
. Collaborate with product stakeholders to deliver high-priority data features and ensure alignment with business goals.
. Manage and mentor an 8-member offshore team, fostering best practices in data engineering and agile development.
. Conduct internal training sessions on modern data architecture, cloud-native deployments, and data engineering best practices.

Required Skills & Technologies:
. Big Data Tools: PySpark, Spark, Flink, Hive, Hadoop, Delta Lake, Streaming, ETL
. Cloud Platforms: Azure (ADF, Databricks, ADLS, Event Hub), AWS (S3)
. Orchestration & DevOps: Airflow, Docker, Kubernetes, GitHub Actions, Jenkins
. Programming Languages: Python, Scala, SQL, Shell
. Other Tools: Redis, Solace, MQ, Kafka, Grafana, Postman
. Soft Skills: Team Leadership, Agile Methodologies, Stakeholder Management, Technical Training

Certifications (Good to have):
. Databricks Certified: Data Engineer Associate, Lakehouse Fundamentals
. Microsoft Certified: Azure Fundamentals (AZ-900), Azure Data Fundamentals (DP-900)

Preferred Qualifications:
. Bachelor's degree in Engineering (E.C.E.) with strong academic performance.
Proven experience in financial data pipelines, regulatory reporting, and risk analytics

Mandatory Competencies

Big Data - Big Data - Pyspark
Big Data - Big Data - SPARK
Beh - Communication
Big Data - Big Data - HIVE
Programming Language - Scala - Scala
Big Data - Big Data - Hadoop
DevOps/Configuration Mgmt - DevOps/Configuration Mgmt - Containerization (Docker, Kubernetes)
DevOps/Configuration Mgmt - DevOps/Configuration Mgmt - Jenkins
Cloud - AWS - AWS S3, S3 glacier, AWS EBS
DevOps/Configuration Mgmt - DevOps/Configuration Mgmt - GitLab,Github, Bitbucket

Perks and Benefits for Irisians

At Iris Software, we offer world-class benefits designed to support the financial, health and well-being needs of our associates to help achieve harmony between their professional and personal growth. From comprehensive health insurance and competitive salaries to flexible work arrangements and ongoing learning opportunities, we're committed to providing a supportive and rewarding work environment.
Join us and experience the difference of working at a company that values its employees success and happiness.

---