Highly Experienced Data Engineer

10 hours ago


Bengaluru, Karnataka, India Talent21 Management and Shared Service Pvt Ltd Full time

About Talent21 Management and Shared Service Pvt Ltd

We are a leading organization in the industry, seeking an experienced Principal Software Engineer to join our Epsilon Attribution/Forecasting Product Development team.

Job Overview:

The successful candidate will be responsible for designing, implementing, and optimizing data processing solutions using Scala, Spark, and Hadoop.

Main Responsibilities:

  • Design, implement, and optimize data processing solutions using Scala, Spark, and Hadoop
  • Collaborate with cross-functional teams to deploy big data solutions on our on-premises and cloud infrastructure
  • Build, schedule, and maintain workflows
  • Perform data integration and transformation, troubleshoot issues, document processes, and communicate technical concepts clearly
  • Continuously enhance our attribution engine/forecasting engine

Requirements:

To succeed in this role, you must have:

  • Strong experience (12+ years) in Scala programming language
  • Extensive experience with Apache Spark for Big Data processing
  • Proficiency in performance tuning of Spark jobs, optimizing resource usage, shuffling, partitioning, and caching for maximum efficiency in Big Data environments
  • In-depth understanding of the Hadoop ecosystem, including HDFS, YARN, and MapReduce
  • Expertise in designing and implementing scalable, fault-tolerant data pipelines with end-to-end monitoring and alerting
  • Using Python to develop infrastructure modules
  • Solid grasp of database systems and SQLs for writing efficient SQLs (RDBMS/Warehouse) to handle TBS of data
  • Familiarity with design patterns and best practices for efficient data modelling, partitioning strategies, and sharding for distributed systems
  • End-to-end ownership with definition, development, and documentation of software objectives, business requirements, deliverables, and specifications in collaboration with stakeholders
  • Experience in working on GIT (or equivalent source control) and solid understanding of Unit and integration test frameworks
  • Able to collaborate with stakeholders/teams to understand requirements and develop a working solution
  • Able to work within tight deadlines and effectively prioritize and execute tasks in a high-pressure environment
  • Able to mentor junior staff

Preferred Qualifications:

Candidates with the following skills and qualifications will be preferred:

  • Hands-on with Databricks for unified data analytics, including Databricks Notebooks, Delta Lake, and Catalogues
  • Proficiency in using the ELK (Elasticsearch, Logstash, Kibana) stack for real-time search, log analysis, and visualization
  • Strong background in analytics, including the ability to derive actionable insights from large datasets and support data-driven decision-making
  • Experience with data visualization tools like Tableau, Power BI, or Grafana
  • Familiarity with Docker for containerization and Kubernetes for orchestration

Estimated Salary: $180,000 - $250,000 per year, depending on location and experience



  • Bengaluru, Karnataka, India Mouri Tech (P) Ltd Full time

    Company Overview:Mouri Tech (P) Ltd is a renowned technology firm seeking an accomplished Senior MSBI Developer to spearhead our data engineering endeavors.Estimated Salary: $160,000 - $200,000 per year, based on industry standards and location.Job Description:We require a highly experienced professional with 9+ years of expertise in SSIS, SAAS, SSRS, and...


  • Bengaluru, Karnataka, India Groww Full time

    About GrowwWe are a passionate group of people driven by ownership, customer-centricity, integrity and the passion to constantly challenge the status quo. Our long-term vision is to become the trusted financial partner for millions of Indians.Our culture enables us to be what we are — India's fastest-growing financial services company. It fosters an...


  • Bengaluru, Karnataka, India Capgemini Full time

    About the RoleWe are seeking a highly skilled Highly Experienced Network Engineer to join our team at Capgemini. In this role, you will be responsible for designing and implementing robust network infrastructure solutions that meet the needs of our clients.Key ResponsibilitiesYou will have strong knowledge of VLAN, RIP, BGP, OSPF protocols and be able to...


  • Bengaluru, Karnataka, India Juniper Networks Full time

    Juniper Networks is committed to delivering a new network for the next decade. To achieve this goal, we are seeking a Highly Experienced Routing Engineer to join our team in Bangalore.The ideal candidate will have 8+ years of experience in the Networking industry and a thorough understanding of IP Packet flow, OSI layers, and related technologies. They will...


  • Bengaluru, Karnataka, India Kredivo Group Full time

    About the RoleWe are seeking a highly skilled and experienced software engineering lead to join our team at Kredivo Group. As a key member of our organization, you will be responsible for leading large-scale features and projects, collaborating with cross-functional teams, and driving innovation in software development.


  • Bengaluru, Karnataka, India Foundit Full time

    About the Role:We are seeking a highly skilled and experienced AWS Cloud Data Architect to join our dynamic team at Foundit. As a key member of our data engineering team, you will be responsible for designing, implementing, and maintaining large-scale data processing pipelines on AWS.Responsibilities: Develop and implement efficient data ingestion,...


  • Bengaluru, Karnataka, India NTT DATA Services Full time

    NTT DATA Services is a leading provider of digital and AI infrastructure, committed to helping clients innovate, optimize, and transform for long-term success. As a trusted global innovator of business and technology services, we serve 75% of the Fortune Global 100.We are seeking a highly experienced Pega BPM Developer to join our team in Bangalore, India....


  • Bengaluru, Karnataka, India Covenant Consultants Full time

    We are a leading consulting firm Covenant Consultants, and we are seeking an exceptionally skilled Senior Software Engineer to join our team.Salary: $150,000 - $200,000 per annum.Job Description:About the RoleThis is a rare opportunity for a highly experienced Senior Software Engineer to take on a challenging role that involves building and maintaining...


  • Bengaluru, Karnataka, India Squareroot Consulting Pvt Ltd Full time

    Job Title: Highly Experienced Software ArchitectAbout Squareroot Consulting Pvt LtdSquareroot Consulting Pvt Ltd is a reputable consulting firm that offers cutting-edge technology solutions to its clients. We are looking for highly skilled and experienced software architects to join our team.Salary Range: ₹1800000 - ₹2500000 per annumJob...


  • Bengaluru, Karnataka, India Ascent Force Full time

    About the RoleWe are seeking an experienced and highly skilled Azure Data Engineer with 5-10 years of expertise in Azure Data Factory (ADF), Databricks, and PySpark to join Ascent Force.


  • Bengaluru, Karnataka, India Oracle Full time

    At Oracle, we are building a future-ready Gen2 cloud Data Science service platform, and we need your expertise to help us succeed. As a highly skilled Artificial Intelligence (AI) engineer, you will play a crucial role in designing, implementing, and deploying machine learning models in production environments.The ideal candidate will have a strong...


  • Bengaluru, Karnataka, India AECOM Full time

    Role OverviewAECOM is seeking a highly skilled and experienced Water Resources Engineer to join our team. As a key member of our engineering team, you will be responsible for contributing to the design and development of large-scale water resource projects.


  • Bengaluru, Karnataka, India Synopsys Inc Full time

    OverviewSynopsys Inc. is a world-leading electronic design automation (EDA) software company that empowers the global electronics industry's innovation and productivity.Job Title: Highly Experienced Software Development LeadEstimated Salary: ₹45 Lakhs - ₹60 Lakhs per annum,Location: Bangalore, IndiaAbout the Role:We are looking for an exceptional Senior...


  • Bengaluru, Karnataka, India LinkedIn Full time

    Job OverviewWe are seeking a highly experienced Senior Software Engineering Manager, Systems Infrastructure to join our team at LinkedIn in Bangalore, India.


  • Bengaluru, Karnataka, India Egis Group Full time

    **Job Summary**We are seeking a highly experienced structural engineer to join our team at Egis Group. As a key member of our engineering team, you will be responsible for designing and developing structural solutions for complex projects.**Responsibilities:**Design and develop structural solutions for complex projects, ensuring accuracy and adherence to...


  • Bengaluru, Karnataka, India NTT DATA Full time

    Job OverviewWe are seeking a highly skilled software engineer to join our team at NTT DATA, specializing in Teamcenter.


  • Bengaluru, Karnataka, India Thomson Reuters Full time

    Role OverviewWe are seeking a highly skilled Data Engineer to join our team at Thomson Reuters.About the RoleThis is an exciting opportunity to work with a talented team of professionals who are passionate about data engineering and analytics.Main ResponsibilitiesDesign, build, and maintain large-scale data pipelines and architectures.Develop and implement...


  • Bengaluru, Karnataka, India SixCardinals Full time

    About SixCardinalsWe are a fast-paced and entrepreneurial startup that values innovation and creativity. Our team is passionate about building cutting-edge technology solutions that transform industries.As a Highly Experienced Full Stack Engineer at SixCardinals, you will lead a team of 4-5 engineers in delivering high-quality solutions end-to-end. This...


  • Bengaluru, Karnataka, India arm limited Full time

    About ArmArm is a world leader in semiconductor intellectual property (IP) and has more than 1,000 employees worldwide.We have an exciting opportunity for a Highly Experienced Silicon Validation Engineer to join our team!Job DescriptionAs a key member of our team, you will be responsible for driving validation specifications and methodology and delivering on...


  • Bengaluru, Karnataka, India HEVO DATA Full time

    About Hevo Data:We are a cutting-edge technology company that helps businesses grow by providing fast and intuitive data pipelines. Our mission is to simplify data access and make it easy for everyone to unlock its potential.At Hevo, we deal with massive amounts of data generated by over 2000 customers in 40+ countries. Our platform is used by brands like...