Lead Data Pipeline Engineer

2 weeks ago


Odhav, India MSBC Group Full time

Join us as a Senior DataOps Engineer (AWS)—Drive High-Performance Data Systems for Financial Services Lead the E-Comms data pipeline within Compass’s Application Simplification workstream: design, build, and optimize AWS-native ingestion and enrichment of high-volume communications (Bloomberg, ICE, LSEG/Refinitiv, Symphony, etc.). Ensure robust retention, security, and data integrity, and deliver reliable, analytics- and AI-ready datasets for downstream platforms.Role Responsibilities:Build AWS infrastructure using Terraform or CDKDevelop, maintain, test and optimize data pipelines and workflows between the source data points and target destinations.Implement data management and governance frameworks to ensure compliance with regulatory and legal requirements, and security protocols.Collaborate with other team members such as the Project Manager and Business Analysts to identify and address risks and challenges in real time.Write unit, integration, and infrastructure testsMonitor, troubleshoot and resolve issues.Automate the data pipeline where possible to reduce manual risk and improve efficiency.Work closely with stakeholders and subject matter experts to ensure success criterion and definition of done is met.Stay updated on industry trends and emerging tech to ensure continuous improvement.Experience / CompetencesEssential:10+ years of hands-on AWS development and infrastructure experience with a financial services background.Proficiency in programming Python and/or Java.Proficient with Terraform or CDK (infrastructure-as-code).Experience with EKS, Lambda, EventBridge, Step Functions, S3, DynamoDB, AWS Glue, and Transfer Family.Solid understanding of CI/CD pipelines, ideally with a background in software engineering, product management or data analytics.Experience with market data ingestion, metadata extraction, and event-driven architectures.Experience in Business Communications Technology e.G. Bloomberg, ICE, Symphony, Teams Chat, etc.Familiarity with security best practices, IAM, and VPN configuration.Excellent problem-solving skills and ability to work in a fast-paced environment.Bachelor’s degree in Computer Science, Data Science, Engineering, or related field.Strong communication and collaboration skills to engage with technical and non-technical stakeholders.Strong experience with Agile software delivery and proven ability to take initiative.Passionate about data and being the best in class.Desired:Experience with regulatory compliance and data security in financial services.Familiarity with Alation, Snowflake, or other analytics platformsKnowledge of financial markets and trading platforms.Certifications in DataOps, cloud platforms, or related areas.MSBC Group has been a trusted technology partner for over 20 years, delivering the latest systems and software solutions for financial services, manufacturing, logistics, construction, and startup ecosystems. Our expertise includes Accessible AI, Custom Software Solutions, Staff Augmentation, Managed Services, and Business Process Outsourcing. We are at the forefront of developing advanced AI-enabled services and supporting transformative projects, such as state-of-the-art trading platforms, seamless application migrations, and integrating real-time data analytics. With offices in London, California, and Ahmedabad, and operating in every time-zone, MSBC Group is your AI and automation partner.


  • Lead Data Engineer

    1 week ago


    Odhav, India Advantmed Full time

    Lead Data Engineer – Azure CloudWe are looking for an experienced Lead Data Engineer to build and manage a robust data platform on Azure Cloud. The ideal candidate will design, develop, and maintain scalable data pipelines, ensure data quality, and leverage cloud technologies for efficient data processing.Key ResponsibilitiesDevelop and maintain data...

  • Lead Data Architect

    2 weeks ago


    Odhav, India ConsultMeIn Full time

    Role: Group Lead – Data EngineeringExperience: 15 – 18 YearsCTC: Up to 90 LPA (including variable)Location: Ahmedabad (Work from office, Hybrid)Primary ResponsibilitiesManage strategy and allocation of data engineering resources for cross-functional delivery teams within the regionHiring governance for external and internal data engineer resources...

  • Data Architect

    1 week ago


    Odhav, India ProductSquads Full time

    About the CompanyCompany Profile: ProductSquads was founded with a bold mission: to engineer capital efficiency through autonomous AI agents, exceptional engineering, and real-time decision intelligence. We’re building an AI-native platform that redefines how software teams deliver value—whether through code written by humans, agents, or both. Our stack...


  • Odhav, India Kanerika Inc Full time

    Job Location: AhmedabadWhat You Will Do:Develop and maintain data pipelines using Azure Data Factory (ADF) /Databricks for data integration and ETL processes.Design, implement, and optimize Power BI /Fabric reports and dashboards to deliver actionable business insights.Collaborate with data analysts, business users, and other teams to understand data...


  • Odhav, India Armakuni Full time

    Armakuni (AWS Premier Partner), a trusted partner in helping organizations leverage cloud-native technologies to achieve agility, scalability, and resilience.Cloud-Native Transformation : Guiding businesses through adopting cloud-native practices to optimize their operationsDevOps and Platform Engineering : Building robust, secure, and scalable platforms...


  • Odhav, India Software Co Full time

    Lead the World’s Most Advanced AI-Powered Full-Stack Application Designer & Developer Platform.Software Co is on a mission is to redefine the way software is built. Over the past decade, we’ve delivered hundreds of enterprise-grade projects, gaining unparalleled experience, insights, and technical depth. Now, we’re channelling that expertise into...


  • Odhav, India Tech Holding Full time

    The Role:We are looking for a highly skilled Senior Frontend Engineer to join our growing team at Tech Holding. You will be responsible for designing, developing, and maintaining high-quality, responsive, and scalable user interfaces across multiple products. The ideal candidate is passionate about modern front-end technologies, performance optimization, and...

  • AI Innovation

    1 week ago


    Odhav, India ACL Digital Full time

    Job Function, Roles & Responsibilities: - Lead strategic initiatives and own the practice for Edge AI/ML, data pipelines, and intelligent embedded systems - Define and build the competency roadmap for machine learning, deep learning, model deployment, and real-time inferencing on edge platforms - Oversee data creation — including data collection, dataset...


  • Odhav, India Indemn Full time

    Company DescriptionIndemn.Ai is building the future of insurance conversations. We develop an AI-powered conversational platform that simplifies how insurers, agencies, brokers, and MGAs interact with customers. Our platform enables real-time policy queries, quote generation, and service requests through intelligent AI agents—backed by human-in-the-loop...


  • Odhav, India Adani Energy Solutions Ltd. Full time

    Objective:The Head Engineering - responsible for leading all substation-related engineering functions, ensuring operational efficiency, safety, and cost-effectiveness. This role is pivotal in driving the design, execution, and optimization of substation projects, aligning with BU's strategic focus on expanding its transmission network and enhancing...