Data Engineer for Scalable Distributed Systems

2 weeks ago


Bengaluru, Karnataka, India Zeta Global Full time

Lead Software Engineer – Big Data

We are seeking a skilled Data Engineer to join our Data Cloud team at Zeta Global. Our Data Cloud team is responsible for the Zeta Identity Graph platform, which captures billions of behavioral, demographic, environmental, and transactional signals, enabling people-based marketing. As part of this team, you will design and grow our existing data infrastructure to democratize data access, enable complex data analyses, and automate optimization workflows for business and marketing operations.

Job Description:

Key Responsibilities:

  • Design, refine, tune, and maintain our real-time and batch data infrastructure
  • Utilize technologies such as HDFS, Spark, Snowflake, Hive, HBase, Scylla, Django, FastAPI, etc., on a daily basis
  • Maintain data quality and accuracy across production data systems
  • Collaborate with Data Engineers to optimize data models and workflows
  • Work with Data Analysts to develop ETL processes for analysis and reporting
  • Partner with Product Managers to design and build data products
  • Work with our DevOps team to scale and optimize our data infrastructure
  • Participate in architecture discussions, influence the roadmap, take ownership, and responsibility over new projects
  • Take part in 24/7 on-call rotation (be available by phone or email in case something goes wrong)

Requirements:

  • A minimum of 7 years of software engineering experience
  • Proven long-term experience and enthusiasm for distributed data processing at scale, eagerness to learn new things
  • Expertise in designing and architecting distributed low-latency and scalable solutions in cloud and on-premises environments
  • Exposure to the entire software development lifecycle from inception to production and monitoring
  • Fluency in Python or solid experience in Scala, Java
  • Proficiency in relational databases and Advanced SQL
  • Expertise in using services like Spark, HDFS, Hive, HBase
  • Experience in adequate usage of any scheduler such as Apache Airflow, Apache Luigi, Chronos, etc.
  • Experience in adequate usage of cloud services (AWS) at scale
  • Experience in agile software development processes
  • Excellent interpersonal and communication skills

Nice to Have:

  • Experience with large-scale/multi-tenant distributed systems
  • Experience with columnar/NoSQL databases – Vertica, Snowflake, HBase, Scylla, Couchbase
  • Experience in real-time streaming frameworks – Flink, Storm
  • Experience with web frameworks such as Flask, Django

Salary Estimate: $150,000 - $200,000 per year, depending on location and experience

Benefits: Comprehensive health insurance, 401(k) plan, paid time off, and opportunities for professional growth and development



  • Bengaluru, Karnataka, India Rippling Full time

    About RipplingRippling is a unified workforce platform that enables businesses to manage their HR and IT needs in one place. By connecting various business systems to a single source of truth, Rippling streamlines employee data management and automates manual tasks.With Rippling, employees can be onboarded quickly and efficiently. A new hire's payroll,...


  • Bengaluru, Karnataka, India MoEngage Inc. Full time

    MoEngage Inc. is a leading Marketing Technology Stack provider that assists brands in redefining their customer engagement in the mobile era.We are looking for an experienced Engineering Director to lead our efforts in building scalable data systems that can handle billions of data points generated by customers and their devices.The ideal candidate will have...


  • Bengaluru, Karnataka, India Nutanix Full time

    About the OpportunityWe are seeking highly skilled developers to join our team and contribute to the development of scalable distributed file systems. If you are passionate about solving complex technical problems and building innovative solutions in a fast-paced environment, this role is ideal for you.About the TeamAs a member of our team at Nutanix working...


  • Bengaluru, Karnataka, India Squareroot Consulting Pvt Ltd Full time

    We are Squareroot Consulting Pvt Ltd, a well-funded startup disrupting the P2P lending domain in India. Our mobile-only lending platform helps underserved banking consumers. We're looking to build a team in Engineering.**Position:** Senior Distributed Systems Engineer**Location:** Bangalore**Experience:** 10+ years**Compensation:** ₹50-70 Lacs + ESOP per...


  • Bengaluru, Karnataka, India LinkedIn Full time

    Lead a team of engineers in designing and developing scalable distributed systemsLeverage your technical expertise to create high-quality software, improve engineering productivity, and foster a culture of collaboration and results.About the RoleWe are seeking an experienced Software Engineer to lead our Ads Trust Engineering team in creating a safe and...


  • Bengaluru, Karnataka, India Zeta Global Full time

    Big Data and Distributed Systems EngineerWe are seeking a highly skilled Distributed Systems Engineer with expertise in big data to join our team at Zeta Global. This role will involve designing, developing, and owning the distributed systems CRM platform for Zeta's Zeta Marketing Platform.Your Impact:Implement the next version of our big data CRM platform,...


  • Bengaluru, Karnataka, India SuprSend Full time

    Company Overview:SuprSend is a cutting-edge communication platform based in Bengaluru, providing a centralized notification infrastructure that enables seamless creation, management, and delivery of notifications across multiple channels. The platform ensures reliable and scalable notification delivery, eliminating the need for in-house development of...


  • Bengaluru, Karnataka, India LinkedIn Full time

    Are you passionate about building high-performance infrastructure platforms? Do you have expertise in distributed systems and Rust programming?About the RoleWe're seeking a talented Data Engineer to join our team at LinkedIn. As a key member of our software engineering group, you'll be responsible for designing and building next-generation infrastructure...


  • Bengaluru, Karnataka, India Databricks Full time

    At Databricks, we empower data teams to tackle the world's most complex challenges. Our cutting-edge data and AI infrastructure platform enables customers to drive business growth through deep data insights.We're a customer-obsessed company founded by engineers, and we're passionate about solving technical challenges. From designing next-gen UI/UX to scaling...


  • Bengaluru, Karnataka, India Talentnauts Full time

    About UsTalentnauts is a cutting-edge technology firm seeking an experienced Backend Engineer to join our team.Estimated Salary:$120,000 - $180,000 per yearJob DescriptionWe are looking for a skilled and innovative Backend Engineer to design, build, and maintain high-performance backend services. Your focus will be on developing scalable systems using Golang...


  • Bengaluru, Karnataka, India Nutanix Full time

    The OpportunityAs a Distributed Systems Engineer at Nutanix, you will be part of the Acropolis Management team, which owns the control plane for the Nutanix operating system.Engineering / R&D at Nutanix:Nutanix is a global leader in cloud software and a pioneer in hyper-converged infrastructure solutions, making computing invisible anywhere.Your Role:Design...


  • Bengaluru, Karnataka, India Nutanix Full time

    The OpportunityWe are seeking passionate engineers to work on scalable distributed systems at Nutanix, a global leader in cloud software and hyper-converged infrastructure solutions.Engineering / R&D at Nutanix:Nutanix makes computing invisible anywhere, with AHV Hypervisor playing a crucial role in the Nutanix compute story at any scale for private, hybrid,...


  • Bengaluru, Karnataka, India NTT DATA Full time

    About NTT DATANTT DATA is a $30 billion trusted global innovator of business and technology services.We serve 75% of the Fortune Global 100 and are committed to helping clients innovate, optimize and transform for long term success.As a Global Top Employer, we have diverse experts in more than 50 countries and a robust partner ecosystem of established and...


  • Bengaluru, Karnataka, India LinkedIn Full time

    **Transform the Way the World Works with LinkedInWe are building a next-generation tracking platform to efficiently collect interaction data for our billion-plus users cutting across multiple technology stacks (mobile, applications, infrastructure) and generate insights in near real-time.You will be part of our world-class software engineering team,...


  • Bengaluru, Karnataka, India InMobi Full time

    At InMobi, we're pushing the boundaries of what's possible in mobile-first experiences. As a Senior Backend Engineer - Distributed Systems, you'll play a pivotal role in designing and building large-scale distributed systems and backend services that drive our products and services.About the RoleWe're seeking an experienced Senior Software Development...


  • Bengaluru, Karnataka, India LinkedIn Full time

    Company OverviewLinkedIn is a professional networking platform that connects people and helps them achieve their career goals. Our mission is to create economic opportunities for every member of the global workforce.Job DescriptionWe are seeking an experienced Principal Staff Software Engineer, Systems Infrastructure to join our world-class software...


  • Bengaluru, Karnataka, India LinkedIn Full time

    At LinkedIn, we are building the next-generation infrastructure and platforms to support our global workforce. As a Senior Distributed Systems Architect, you will be responsible for designing and implementing scalable, high-performance systems that operate 24/7.Company OverviewWe are a world-class software engineering team dedicated to creating economic...


  • Bengaluru, Karnataka, India Wayfair Full time

    We are looking for an exceptional Lead Software Architect for Distributed Systems to join our Supply Chain Retail tech team at Wayfair.Salary: $180,000 - $200,000 per yearAbout the RoleThis is a leadership position that requires a seasoned architect with expertise in designing and developing scalable distributed systems. As a Lead Software Architect, you...


  • Bengaluru, Karnataka, India LinkedIn Full time

    About the RoleAs a Chief Distributed Systems Architect at LinkedIn, you will be responsible for designing and building next-generation infrastructure and platforms. You will work with our world-class software engineering team to build massively scalable data storage and replication systems, cutting-edge search platforms, best-in-class AI platforms, and...


  • Bengaluru, Karnataka, India Squareroot Consulting Pvt Ltd. Full time

    Job Title: Senior Technical Staff MemberJob SummaryWe are seeking an experienced Distributed Systems Architect to join our team at Squareroot Consulting Pvt Ltd. As a key member of our Distributed Systems team, you will design and implement cutting-edge distributed, scale-out data infrastructure software systems.Responsibilities- Design and implement...