Lead Data Pipeline Engineer

2 weeks ago

Ahmedabad, India MSBC Group Full time

Join us as a Senior DataOps Engineer (AWS)—Drive High-Performance Data Systems for Financial Services Lead the E-Comms data pipeline within Compass’s Application Simplification workstream: design, build, and optimize AWS-native ingestion and enrichment of high-volume communications (Bloomberg, ICE, LSEG/Refinitiv, Symphony, etc.). Ensure robust retention, security, and data integrity, and deliver reliable, analytics- and AI-ready datasets for downstream platforms. Role Responsibilities: Build AWS infrastructure using Terraform or CDK Develop, maintain, test and optimize data pipelines and workflows between the source data points and target destinations. Implement data management and governance frameworks to ensure compliance with regulatory and legal requirements, and security protocols. Collaborate with other team members such as the Project Manager and Business Analysts to identify and address risks and challenges in real time. Write unit, integration, and infrastructure tests Monitor, troubleshoot and resolve issues. Automate the data pipeline where possible to reduce manual risk and improve efficiency. Work closely with stakeholders and subject matter experts to ensure success criterion and definition of done is met. Stay updated on industry trends and emerging tech to ensure continuous improvement. Experience / Competences Essential: 10+ years of hands-on AWS development and infrastructure experience with a financial services background. Proficiency in programming Python and/or Java. Proficient with Terraform or CDK (infrastructure-as-code). Experience with EKS, Lambda, EventBridge, Step Functions, S3, DynamoDB, AWS Glue, and Transfer Family. Solid understanding of CI/CD pipelines, ideally with a background in software engineering, product management or data analytics. Experience with market data ingestion, metadata extraction, and event-driven architectures. Experience in Business Communications Technology e.G. Bloomberg, ICE, Symphony, Teams Chat, etc. Familiarity with security best practices, IAM, and VPN configuration. Excellent problem-solving skills and ability to work in a fast-paced environment. Bachelor’s degree in Computer Science, Data Science, Engineering, or related field. Strong communication and collaboration skills to engage with technical and non-technical stakeholders. Strong experience with Agile software delivery and proven ability to take initiative. Passionate about data and being the best in class. Desired: Experience with regulatory compliance and data security in financial services. Familiarity with Alation, Snowflake, or other analytics platforms Knowledge of financial markets and trading platforms. Certifications in DataOps, cloud platforms, or related areas. MSBC Group has been a trusted technology partner for over 20 years, delivering the latest systems and software solutions for financial services, manufacturing, logistics, construction, and startup ecosystems. Our expertise includes Accessible AI, Custom Software Solutions, Staff Augmentation, Managed Services, and Business Process Outsourcing. We are at the forefront of developing advanced AI-enabled services and supporting transformative projects, such as state-of-the-art trading platforms, seamless application migrations, and integrating real-time data analytics. With offices in London, California, and Ahmedabad, and operating in every time-zone, MSBC Group is your AI and automation partner.

  • Data Pipeline Engineer

    5 days ago

    Ahmedabad, Gujarat, India TIGI HR Full time

    Data EngineerExperience: 4–6 YearsLocation: AhmedabadWe're looking for a skilledData Engineerto design, build, and maintain scalable data pipelines and systems. The ideal candidate will have hands-on experience withSnowflake,DBT, andAirflow, with a passion for building efficient, high-performance data workflows.Key Responsibilities:Develop, optimize, and...

  • Lead data engineer

    4 weeks ago

    Ahmedabad, India Advantmed Full time

    Lead Data Engineer – Azure CloudWe are looking for an experienced Lead Data Engineer to build and manage a robust data platform on Azure Cloud. The ideal candidate will design, develop, and maintain scalable data pipelines, ensure data quality, and leverage cloud technologies for efficient data processing.Key ResponsibilitiesDevelop and maintain data...

  • Lead Engineer/Data Engineer

    2 weeks ago

    Ahmedabad, India Advantmed Full time

    Lead Data Engineer – Azure Cloud We are looking for an experienced Lead Data Engineer to build and manage a robust data platform on Azure Cloud . The ideal candidate will design, develop, and maintain scalable data pipelines, ensure data quality, and leverage cloud technologies for efficient data processing. Develop and maintain data ingestion pipelines...

  • Lead Data Engineer

    3 days ago

    Ahmedabad, Gujarat, India Tecblic Full time

    Job Description:Lead Data Engineer with Azure CloudLocation:AhmedabadExperience:6 to 7 yearsEmployment Type:Full-TimeWe are looking for a highly motivated and experienced Data Engineer to join our  team. As a Data Engineer, you will play a critical role in designing, building, and optimizing data pipelines that ensure the availability, reliability, and...

  • Lead Data Engineer

    2 weeks ago

    Ahmedabad, India Advantmed Full time

    Lead Data Engineer – Azure Cloud We are looking for an experienced Lead Data Engineer to build and manage a robust data platform on Azure Cloud . The ideal candidate will design, develop, and maintain scalable data pipelines, ensure data quality, and leverage cloud technologies for efficient data processing. Key Responsibilities Develop and maintain data...

  • Chief Data Pipeline Architect

    2 weeks ago

    ahmedabad, India beBeeDataEngineer Full time

    Job OverviewWe are seeking a highly skilled Senior Data Engineer to design, develop, and maintain high-performance data pipelines and data systems. The ideal candidate will have strong experience with cloud platforms, modern ETL/ELT tools, and deep technical skills in Python, SQL, and distributed data frameworks.Key ResponsibilitiesDesign and implement...

  • Senior Data Pipeline Specialist

    2 weeks ago

    ahmedabad, India beBeeDataEngineer Full time

    Data Engineer PositionAs a Data Engineer, you will be working on a global engineering team for an international enterprise organisation. This is a greenfield multi-year project initiative requiring experienced Data Engineers with strong skills in Palantir Foundry, PySpark, and TypeScript.Key Responsibilities:Developing data pipelines using Palantir Foundry...

  • Data Engineering Lead

    2 weeks ago

    ahmedabad, India Kanerika Inc Full time

    Job Location: AhmedabadWhat You Will Do:Develop and maintain data pipelines using Azure Data Factory (ADF) /Databricks for data integration and ETL processes.Design, implement, and optimize Power BI /Fabric reports and dashboards to deliver actionable business insights.Collaborate with data analysts, business users, and other teams to understand data...

  • Data Engineering Lead

    2 weeks ago

    Ahmedabad, India Kanerika Inc Full time

    Job Location: Ahmedabad What You Will Do: Develop and maintain data pipelines using Azure Data Factory (ADF) /Databricks for data integration and ETL processes. Design, implement, and optimize Power BI /Fabric reports and dashboards to deliver actionable business insights. Collaborate with data analysts, business users, and other teams to understand data...

  • Data Engineering Lead

    2 weeks ago

    Ahmedabad, India Kanerika Inc Full time

    Job Location: Ahmedabad What You Will Do: Develop and maintain data pipelines using Azure Data Factory (ADF) /Databricks for data integration and ETL processes. Design, implement, and optimize Power BI /Fabric reports and dashboards to deliver actionable business insights. Collaborate with data analysts, business users, and other teams to understand data...