Data Engineer-Technical Lead

4 weeks ago

Bengaluru Karnataka India, Karnataka Sigmoid Full time

Kindly find the Job Description Below.

Job Title:Technical Lead-Azure Data Engineer

Location: Bangalore

Years of Experience: 10+ years of experience

Sigmoid works with a variety of clients from start-ups to fortune 500 companies. We are looking for a detailed oriented self-starter to assist our engineering and analytics teams in various roles as a Software Development Engineer.

As an Technical Lead- Data Engineer, you will be responsible for building a highly-scalable and extensible big data platform that provides the foundation for collecting, storing, modeling, and analyzing massive data sets from multiple channels.


We are looking for a skilled Data Engineer with 10+ years of experience in big data technologies, particularly Azure, Python, PYSpark, SQL and data lakehouse architectures. The ideal candidate will have a strong background in building scalable data pipelines and experience with modern data storage formats, including Apache Iceberg. You will work closely with cross-functional teams to design and implement efficient data solutions in a cloud-based environment.

Key Responsibilities:

  • Data Pipeline Development:
  • Design, build, and optimize scalable data pipelines using Apache Spark.
  • Implement and manage large-scale data processing solutions across data lakehouses.
  • Data Lakehouse Management:
  • Work with modern data lakehouse platforms (e.g.Apache Iceberg) to handle large datasets.
  • Optimize data storage, partitioning, and versioning to ensure efficient access and querying.
  • SQL & Data Management:
  • Write complex SQL queries to extract, manipulate, and transform data.
  • Develop performance-optimized queries for analytical and reporting purposes.
  • Data Integration:
  • Integrate various structured and unstructured data sources into the lakehouse environment.
  • Work with stakeholders to define data needs and ensure data is available for downstream consumption.
  • Data Governance and Quality:
  • Implement data quality checks and ensure the reliability and accuracy of data.
  • Contribute to metadata management and data cataloging efforts.
  • Performance Tuning:
  • Monitor and optimize the performance of Spark jobs, SQL queries, and overall data infrastructure.
  • Work with cloud infrastructure teams to optimize costs and scale as needed.

Qualifications:

  • Bachelor’s or Master’s degree in Computer Science, Information Technology, or a related field.
  • 9+ years of experience in data engineering, with a focus on Java/python, Spark and SQL Programming languages.
  • Hands-on experience with Apache Iceberg, Snowflake, or similar technologies.
  • Strong understanding of data lakehouse architectures and data warehousing principles.
  • Proficiency in AWS data services.
  • Experience with version control systems like Git and CI/CD pipelines.
  • Excellent problem-solving and analytical skills.
  • Strong communication and collaboration skills.

Nice to Have:

  • Experience with containerization (Docker, Kubernetes) and orchestration tools like Airflow.
  • Certifications in Azure cloud technologies

  • Technical Lead- Data Engineer

    3 days ago

    Bengaluru, Karnataka, India Astar Data Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Kindly find the Job Description Below.Job Title:Technical Lead- Data EngineerLocation: BangaloreYears of Experience: 8+ years of experienceSigmoidworks with a variety of clients from start-ups to fortune 500 companies. We are looking for a detailed oriented self-starter to assist our engineering and analytics teams in various roles as a Software...

  • Technical Lead

    3 days ago

    Bengaluru, Karnataka, India Astar Data Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Technical Lead Data EngineeringAs a Technical Lead, you will be responsible in building a highly-scalable and extensible big data platform that provides the foundation for collecting, storing, modeling, and analyzing massive data sets from multiple channels.This position reports to Engineering Manager.Responsibilities:Align Sigmoid with key Client...

  • Data & AI Engineer Lead

    3 days ago

    Bengaluru, Karnataka, India NTT DATA Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Framework Design & Architecture Architect a metadata-driven, Python/Spark-based framework for automated data validation across high-volume production datasets. Define DQ rule templates for completeness, integrity, conformity, accuracy, and timeliness. Establish data quality thresholds, escalation protocols, and exception workflows. Automation & Integration...

  • Data Engineer

    1 week ago

    Bengaluru, Karnataka, India NTT DATA Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Design and implement tailored data solutions to meet customer needs and use cases, spanning from streaming to data lakes, analytics, and beyond within a dynamically evolving technical stack. Provide thought leadership by recommending the most appropriate technologies and solutions for a given use case, covering the entire spectrum from the application layer...

  • Hiring Technical Lead

    3 days ago

    Bengaluru, Karnataka, India Astar Data Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Responsibilities:Align Sigmoid with key Client initiativesInterface daily with customers across leading Fortune 500 companies to understand strategic requirementsConnect with VP and Director level clients on a regular basis.Travel to client locationsAbility to understand business requirements and tie them to technology solutionsFacilitate in Technical...

  • Data Entry Operator

    3 days ago

    Bengaluru, Karnataka, India Design Engineer Full time ₹ 2,64,000 per year

    We are seeking a dedicated and detail-oriented Data Entry Operator (DEO) to support R&D projects. The role involves accurate data entry, documentation, and maintenance of research records in secure systems. The DEO will assist scientists, engineers, and administrative staff by ensuring timely and error-free handling of project information.Key...

  • Data Engineer

    1 week ago

    Bengaluru, Karnataka, India NTT DATA, Inc. Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Key Responsibilities:Design and implement tailored data solutions to meet customer needs and use cases, spanning from streaming to data lakes, analytics, and beyond within a dynamically evolving technical stack.Provide thought leadership by recommending the most appropriate technologies and solutions for a given use case, covering the entire spectrum from...

  • Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India NTT DATA, Inc. Full time ₹ 1,04,000 - ₹ 1,30,878 per year

    Req ID:321800We are currently seeking a Data Engineer (Talend &Pyspark) to join our team in Bangalore, Karntaka (IN-KA), India (IN)."Job Duties: Key Responsibilities: Design and implement tailored data solutions to meet customer needs and use cases, spanning from streaming to data lakes, analytics, and beyond within a dynamically evolving technical...

  • Technical Lead

    2 weeks ago

    Bengaluru, Karnataka, India, Karnataka Deluxe Full time

    Job Title: Technical LeadExperience: 9+ yearsLocation: OnsiteIndustry: Media & EntertainmentAbout the RoleWe are looking for a Tech Lead with deep expertise in modern application development and technologies to provide technical leadership and guidance to our engineering team. This role is primarily focused on technical direction, mentorship, and hands-on...

  • Associate Lead Data Scientist

    2 weeks ago

    Bengaluru, Karnataka, India Astar Data Full time ₹ 8,00,000 - ₹ 24,00,000 per year

    Sigmoid Analytics is a leading Data solutions company backed by Sequoia Capital. We offer best in- end-to-end data value chain spanning across Data Science, Data Engineering and Data Ops. With data and technology at the core of our solutions, we are solving some of the toughest problems out there. Our culture is modelled around expertise and mutual respect...