Senior Data Engineer

Job Title : Senior Data Engineer (Azure Databricks)Role Type : Fixed Term Direct Contract with TalproContract Duration : 6 months (Fixed Term Contract)Years of Experience : 6+ Yrs.Notice Period : Immediate Joiners OnlyWork Mode : Hybrid (3 Days from Office Weekly) Location : Manyata Tech Park, Bengaluru - Looking for Bengaluru Based Candidates onlyDomain : Any IT Domain About the Role : Talpro is seeking a Senior Data Engineer (Azure Databricks) for an internal position to be deployed at a client location. This is a 6-month contract-to-hire role if you excel during the contract, there is potential to transition into a permanent position. You will be on Talpros payroll and work with the clients team in Bangalore. The work model is hybrid (mix of on-site and remote as needed) and aligned to UK business hours (approximately 2 PM to 7 PM IST).Key Responsibilities : - Data Pipeline Development : Design and implement highly performant, metadata-driven data ingestion and transformation pipelines from multiple sources using Azure Databricks and Apache Spark. This includes building both streaming and batch data processes to handle large-scale data in real time and in bulk.- Framework & Optimization : Develop scalable and reusable frameworks for data ingestion and transformation of large datasets. Perform Spark performance tuning and optimization to ensure data pipelines are efficient and cost-effective. Provide technical guidance on complex data problems (including handling geospatial data) using Spark DataFrames and other big data tools.- Data Quality & Integration : Design and implement data quality checks and processes to ensure data integrity is maintained through the pipeline. Integrate end-to-end data pipelines from source systems to target data repositories (data lakes/data warehouses), ensuring consistency, quality, and reliability of data at all times.- Collaboration & Tooling : Work closely with cross-functional teams (data analysts, data scientists, application developers) to support additional project components such as reporting tools, APIs, and search applications. Evaluate various tools and technologies (e.g., Azure Data Factory, Azure Databricks features, etc.) against project requirements to recommend the best fit.- Agile Delivery : Operate within an Agile/DevOps delivery environment. Participate in iterative sprint planning, deliver proof-of-concepts (POCs), and guide the project from concept to production. Implement continuous integration/continuous deployment (CI/CD) practices for data pipelines and ensure solutions are delivered in a timely, iterative manner.Requirements : Required Qualifications : - Experience : 68 years of overall Data Engineering experience, preferably in building data solutions on Microsoft Azure. Within this, at least 34 years of hands-on Databricks experience, including delivering 2+ end-to-end Databricks projects from concept to production.- Azure Databricks Expertise : Proven experience in designing Databricks-based data platforms, with a strong understanding of Spark internals, performance tuning, and best practices for building data lakes/warehouses on Databricks. Familiarity with addressing non-functional requirements (scalability, reliability, security) in a data platform design.- Data Pipeline Skills : Hands-on experience building data pipelines using Azure Data Factory and Azure Databricks. Proficient in developing data integration workflows using Python (including PySpark) for ETL/ELT processes.Certifications : Databricks Certified Data Engineer Associate (at least one Databricks certification is required). Microsoft Azure Data Engineer Associate certification is highly desired.Data Warehousing & ETL : Experience designing and delivering solutions on the Azure Data Analytics stack (e.g., Azure Data Lake, Azure Synapse Analytics). Background in building data warehouse solutions using ETL/ELT tools such as Informatica or Talend.Data Management : Strong understanding of data management best practices data profiling, data cleansing, data transformation, and data quality assurance. Experience implementing routines for data standardization, deduplication, and matching to ensure high data quality.Preferred Qualifications (Nice to Have) : - DevOps & CI/CD : Experience working in a DevOps-oriented environment using tools like Azure DevOps (formerly VSTS), Chef, Puppet, or Terraform for infrastructure automation and deployment. Knowledge of setting up CI/CD pipelines for data projects is a plus.- Diverse Data Experience : Exposure to working with structured and unstructured data, including experience handling geospatial data or large imaging datasets. Ability to incorporate such data types into pipelines and ensure performance.- NoSQL & Big Data Tech : Familiarity with open-source NoSQL databases and graph databases (e.g., MongoDB, Cassandra, Neo4j) and understanding of how to integrate or migrate data to/from these systems.- Streaming & IoT : Experience with streaming data platforms and event ingestion services such as Azure Event Hub, Azure IoT Hub, Apache Kafka, or Apache NiFi. Ability to design pipelines for real-time data processing and event-driven architectures.Why Join Talpro?Innovative Projects : Work on cutting-edge data engineering projects using Azure Databricks and the modern Azure data stack, tackling challenges in big data and analytics.Contract-to-Hire Opportunity : The 6-month C2H model allows you to prove your skills with the possibility of a long-term role. Youll get a foot in the door with a leading client and the stability of Talpros backing.Flexible Work Environment : Enjoy a hybrid work setup with the flexibility of remote work combined with collaborative on-site days. Align with UK timing for better work-life balance if you prefer later start times.Professional Growth : Leverage your expertise and certifications while also having opportunities to learn new technologies (cloud services, DevOps tools, streaming platforms). Talpro supports continuous learning and certification for our employees.Note : This position is for Bangalore-based candidates (or those willing to relocate) as on-site presence is required on a case-to-case basis. (ref:hirist.tech)