Technical Lead- Data Engineer

6 hours ago

Bengaluru, Karnataka, India Astar Data Full time ₹ 12,00,000 - ₹ 36,00,000 per year

Kindly find the Job Description Below.

Job Title:Technical Lead- Data Engineer

Location: Bangalore

Years of Experience: 8+ years of experience

Sigmoidworks with a variety of clients from start-ups to fortune 500 companies. We are looking for a detailed oriented self-starter to assist our engineering and analytics teams in various roles as a Software Development Engineer.

As an Technical Lead- Data Engineer, you will be responsible for building a highly-scalable and extensible big data platform that provides the foundation for collecting, storing, modeling, and analyzing massive data sets from multiple channels.

We are looking for a skilled Data Engineer with 6+ years of experience in big data technologies, particularly Python, PYSpark, SQL and data lakehouse architectures. The ideal candidate will have a strong background in building scalable data pipelines and experience with modern data storage formats, including Apache Iceberg. You will work closely with cross-functional teams to design and implement efficient data solutions in a cloud-based environment.Key Responsibilities:

  • Data Pipeline Development:
  • Design, build, and optimize scalable data pipelines using Apache Spark.
  • Implement and manage large-scale data processing solutions across data lakehouses.
  • Data Lakehouse Management:
  • Work with modern data lakehouse platforms (e.g.Apache Iceberg) to handle large datasets.
  • Optimize data storage, partitioning, and versioning to ensure efficient access and querying.
  • SQL & Data Management:
  • Write complex SQL queries to extract, manipulate, and transform data.
  • Develop performance-optimized queries for analytical and reporting purposes.
  • Data Integration:
  • Integrate various structured and unstructured data sources into the lakehouse environment.
  • Work with stakeholders to define data needs and ensure data is available for downstream consumption.
  • Data Governance and Quality:
  • Implement data quality checks and ensure the reliability and accuracy of data.
  • Contribute to metadata management and data cataloging efforts.
  • Performance Tuning:
  • Monitor and optimize the performance of Spark jobs, SQL queries, and overall data infrastructure.
  • Work with cloud infrastructure teams to optimize costs and scale as needed.

Qualifications:

  • Bachelors or Masters degree in Computer Science, Information Technology, or a related field.
  • 8+ years of experience in data engineering, with a focus on Java/python, Spark and SQL Programming languages.
  • Hands-on experience with Apache Iceberg, Snowflake, or similar technologies.
  • Strong understanding of data lakehouse architectures and data warehousing principles.
  • Proficiency in AWS data services.
  • Experience with version control systems like Git and CI/CD pipelines.
  • Excellent problem-solving and analytical skills.
  • Strong communication and collaboration skills.

Nice to Have:

  • Experience with containerization (Docker, Kubernetes) and orchestration tools like Airflow.
  • Certifications in AWS cloud technologies

  • Technical Lead-Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India Best Infosystems Ltd. Full time

    Technical Lead-Data Engineer_Full-Time_Bangalore/Pune/Navi Mumbai/Noida/Hyderabad/ChennaiJob Title: Technical Lead-Data EngineerLocation: Bangalore/Pune/Navi Mumbai/Noida/Hyderabad/ChennaiJob Type: Full-TimeExperience: 8+ yearsQualifications: Bachelor's/relevant exp - with Snowpro CertificationJob Description:Must Have Skills:•Overall – 8+...

  • Technical Lead-Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India Best Infosystems Ltd. Full time

    Technical Lead-Data Engineer_Full-Time_Bangalore/Pune/Navi Mumbai/Noida/Hyderabad/ChennaiJob Title: Technical Lead-Data EngineerLocation: Bangalore/Pune/Navi Mumbai/Noida/Hyderabad/ChennaiJob Type: Full-TimeExperience: 8+ yearsQualifications: Bachelor's/relevant exp - with Snowpro CertificationJob Description:Must Have Skills:•Overall – 8+...

  • Technical Lead-Data Engineer

    3 weeks ago

    Bengaluru, Karnataka, India Best Infosystems Ltd. Full time

    Technical Lead-Data Engineer_Full-Time_Bangalore/Pune/Navi Mumbai/Noida/Hyderabad/Chennai Job Title: Technical Lead-Data Engineer Location: Bangalore/Pune/Navi Mumbai/Noida/Hyderabad/Chennai Job Type: Full-Time Experience: 8+ years Qualifications: Bachelor's/relevant exp - with Snowpro Certification Job Description: Must Have Skills: •Overall –...

  • Data Engineer

    1 week ago

    Bengaluru, Karnataka, India NTT DATA, Inc. Full time ₹ 1,04,000 - ₹ 1,30,878 per year

    Req ID:321800We are currently seeking a Data Engineer (Talend &Pyspark) to join our team in Bangalore, Karntaka (IN-KA), India (IN)."Job Duties: Key Responsibilities: Design and implement tailored data solutions to meet customer needs and use cases, spanning from streaming to data lakes, analytics, and beyond within a dynamically evolving technical...

  • Technical Lead in Data Engineering

    2 weeks ago

    Bengaluru, Karnataka, India beBeeData Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Technical Lead in Data EngineeringAbout UsWe're a US-focused startup building innovative AI solutions for financial companies. Our mission is to transform complex data into actionable insights.We're at the forefront of developing cutting-edge technologies, and we're assembling a talented team to help us achieve our vision.The RoleYou'll be responsible for...

  • Digital Engineering Lead Engineer

    1 week ago

    Bengaluru, Karnataka, India NTT DATA Full time

    Req ID 299336NTT DATA strives to hire exceptional innovative and passionate individuals who want to grow with us If you want to be part of an inclusive adaptable and forward-thinking organization apply now We are currently seeking a Digital Engineering Lead Engineer to join our team in Bangalore Karn xc4 x81taka IN-KA India IN How We Will Help...

  • Lead Data Scientist

    2 weeks ago

    Bengaluru, Karnataka, India Data Science Full time ₹ 15,00,000 - ₹ 20,00,000 per year

    Get to know the team GrabFin is an aggregate of FinTech businesses spread across 6 countries in S.E. Asia, in the Lending, Payments and Insurance domains. We are excited to provide innovative financial services to all participants of the Grab Ecosystem be it our Drivers, Consumers or Merchants. Our products are built on fundamental market insights combined...

  • Data Entry Operator

    6 hours ago

    Bengaluru, Karnataka, India Design Engineer Full time ₹ 2,64,000 per year

    We are seeking a dedicated and detail-oriented Data Entry Operator (DEO) to support R&D projects. The role involves accurate data entry, documentation, and maintenance of research records in secure systems. The DEO will assist scientists, engineers, and administrative staff by ensuring timely and error-free handling of project information.Key...

  • Technical Project Lead – Data Engineering

    2 weeks ago

    Bengaluru, Karnataka, India beBeeProjectManager Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Job Opportunity:">We are seeking a skilled technical project manager to lead and deliver projects in data engineering and analytics.">About the Role:">The successful candidate will manage cross-functional teams to execute data platform, pipeline, and analytics initiatives, ensuring alignment with business goals and timely delivery.">Responsibilities:">Plan,...

  • Technical Data Lead

    1 week ago

    Bengaluru, Karnataka, India Zeero Full time ₹ 5,00,000 - ₹ 8,00,000 per year

    Job SummaryWe are seeking a highly skilled and experienced Technical Data Lead to join our innovative and dynamic team.This role is a unique blend of a Data Engineer, Data Analyst, and Data Scientist, requiring a deep understanding of the entire data lifecycle.The ideal candidate will be a hands-on leader, responsible for guiding a team of data professionals...