Sr. Data Engineer

Exigo Tech is a Sydney-based Technology Solutions Provider that is focused on providing solutions on three major verticals; Infrastructure, Cloud, and Application to businesses across Australia. We help companies reach operational efficiencies by empowering them with technology solutions that drive their business processes.

Exigo is looking for Full-time Sr. Data Engineer

We are ISO 27001:2022 certified organization

Visit our website: for more details….

LinkedIn:

Click Here to know more : LIFE AT EXIGO TECH

Roles and Responsibilities

• Install, configure, and manage Apache Spark (open-source) clusters on Ubuntu, including Spark master/worker nodes and Spark environment files.
• Configure and manage Spark UI and Spark History Server for monitoring jobs, analyzing DAGs, stages, tasks, and troubleshooting performance.
• Develop, optimize, and deploy PySpark ETL/ELT pipelines using DataFrame API, UDFs, window functions, caching, partitioning, and broadcasting.
• Deploy PySpark jobs using spark-submit in client/cluster mode with proper logging and error handling.
• Install, configure, and manage Apache Airflow including UI, scheduler, webserver, connections, and variables.
• Create, schedule, and monitor Airflow DAGs for PySpark jobs using SparkSubmitOperator, BashOperator, or PythonOperator.
• Configure and manage cron jobs for scheduling data processing tasks where needed.
• Install, configure, and optimize Trino (PrestoSQL) coordinator and worker nodes; configure catalogs such as S3, MySQL, or PostgreSQL.
• Maintain Linux/Ubuntu servers including services, logs, environment variables, memory usage, and port conflict resolution.
• Design and implement scalable data architectures using Azure Data Services including ADF, Synapse, ADLS, Azure SQL, and Databricks.
• Develop, manage, and automate ETL/ELT pipelines using Azure Data Factory (Pipelines, Mapping Dataflows, Dataflows).
• Monitor, troubleshoot, and optimize data pipelines across Spark, Airflow, Trino, and Azure platforms.
• Work with structured, semi-structured, and unstructured data across multiple data sources and formats.
• Implement data analytics, transformation, backup, and recovery solutions.
• Perform data migration, upgrade, and modernization using Azure and database tools.
• Implement CI/CD pipelines for data solutions using Azure DevOps and Git.
• Ensure data quality, governance, lineage, metadata management, and security compliance across cloud and big data environments.
• Design and optimize data models using star and snowflake schemas; build data warehouses, Delta Lake, and Lakehouse systems.
• Develop and rebuild reports/dashboards using Power BI, Tableau, or similar tools.
• Collaborate with internal teams, clients, and business users to gather requirements and deliver high-quality data solutions.
• Provide documentation, runbooks, and operational guidance.

Technical Skills:

1. Apache Spark (Open Source) & PySpark - Must

2. Apache Spark installation & cluster configuration (Ubuntu/Linux)

3. Spark master/worker setup (standalone & cluster mode)
4. Spark UI & History Server configuration and debugging
5. PySpark development (ETL pipelines, UDFs, window functions, DataFrame API)
6. Performance tuning (partitioning, caching, shuffles)

7. spark-submit deployment with monitoring and logging

8. Apache Airflow & Job Orchestration - Must

9. Airflow installation & configuration (UI, scheduler, webserver)

10. Creating and scheduling DAGs (SparkSubmitOperator, BashOperator, PythonOperator)
11. Retry logic, triggers, alerting, and log management

12. Cron job scheduling & process automation

13. Trino (PrestoSQL) - Must

14. Trino coordinator & worker node setup

15. Catalog configuration (S3, RDBMS sources)

16. Distributed SQL troubleshooting & performance optimization

17. Azure Data Services (nice to have)

18. Azure Data Factory

19. Azure Synapse Analytics
20. Azure SQL / Cosmos DB
21. Azure Data Lake Storage (Gen2)
22. Azure Databricks (Delta, Notebooks, Jobs)

23. Azure Event Hubs / Stream Analytics

24. Microsoft Fabric ( nice to have)

25. Lakehouse

26. Warehouse
27. Dataflows
28. Notebooks

29. Pipelines

30. Programming & Querying

31. Python

32. PySpark
33. SQL

34. Scala

35. Data Modeling & Warehousing

36. Star schema modeling

37. Snowflake schema modeling
38. Fact/dimension modeling
39. Data warehouse & Lakehouse design

40. Delta Lake / Lakehouse architectures

41. DevOps & CI/CD

42. Git / GitHub / Azure Repos

43. Azure DevOps pipelines (CI/CD)

44. Automated deployment for Spark, Airflow, ADF, Databricks, Fabric

45. BI Tools (Nice to have)

46. Power BI

47. Tableau

48. Report building, datasets, DAX

49. Linux/Ubuntu Server Knowledge

50. Shell scripting

51. Service management
52. Logs & environment variables

Soft Skills:

• Excellent problem solving and communication skills
• Able to work well in a team setting
• Excellent organizational and time management skills
• Taking end-to-end ownership
• Production support & timely delivery
• Self-driven, flexible and innovative
• Microsoft Certified: Azure Data Engineer Associate (DP-203 / DP -300)
• Knowledge of DevOps and CI/CD pipelines in Azure

Education:

• BSc/BA in Computer Science, Engineering or a related field

Work Location:
Vadodara, Gujarat, India