Data Engineer

5 hours ago

Chennai Tamil Nadu India, Tamil Nadu NielsenIQ Full time

Role Summary:


Design, build, and optimize large-scale ETL and data-processing pipelines handling GB–TB volumes. Operate within the Databricks ecosystem and drive migration of selected workloads to high-performance engines such as Polars and DuckDB. Maintain strong engineering rigor across CI/CD, testing, and code-quality enforcement. Apply analytical thinking to solve data reliability, performance, and scalability problems. AI familiarity is advantageous.


Core Responsibilities:

  • Develop and maintain distributed data pipelines using Scala, Spark, Delta, and Databricks.
  • Engineer robust ETL workflows tuned for high-volume ingestion, transformation, and publishing.
  • Profile pipelines, remove bottlenecks, and optimize compute, storage, and job orchestration.
  • Lead migration of suitable workloads to Polars, DuckDB, or equivalent high-performance engines.
  • Implement CI/CD workflows with automated builds, tests, deployments, and environment gating.
  • Enforce coding standards through code coverage targets, unit/integration tests, and SonarQube rules.
  • Ensure pipeline observability: logging, data quality checks, lineage, and failure diagnostics.
  • Apply analytical reasoning to triage complex data issues and deliver root-cause clarity.
  • Contribute to AI-aligned initiatives when required: RAG design, fine-tuning workflows, agentic patterns.
  • Collaborate with product, analytics, and platform teams to operationalize data solutions


Required Skills and Experience:

  • 3+ years in data engineering with strong command of Scala and Spark.
  • Proven background in ETL design, distributed processing, and high-volume data systems.
  • Hands-on experience with Databricks (jobs, clusters, notebooks, Delta Lake).
  • Proficiency in workflow optimization, performance tuning, and memory management.
  • Experience with Polars, DuckDB, or similar columnar/accelerated engines.
  • CI/CD discipline using Git-based pipelines; strong testing and code-quality practices.
  • Familiarity with SonarQube, coverage metrics, and static analysis.
  • Strong analytical and debugging capability across data, pipelines, and infra.
  • Exposure to AI concepts: embeddings, vector stores, retrieval-augmented generation, fine-tuning, agentic architectures.

Preferred:

  • Experience with Azure cloud environments .
  • Experience in metadata-driven or config-driven pipeline frameworks.

  • Data Engineer

    6 days ago

    Chennai, Tamil Nadu, India CX Data Labs Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    HiRole - Azure Data EngineerLocation - Chennai, Gurugram (5days Office)Full timeShift timing 2pm to 11pmKey Responsibilities:· Develop, maintain, and optimize scalable ETL/ELT pipelines using PySpark and Databricks.· Collaborate with cross-functional teams to design and implement data models and data integration solutions.· Create and maintain robust SQL...

  • Data Engineer

    6 days ago

    Chennai, Tamil Nadu, India CX Data Labs Full time ₹ 9,00,000 - ₹ 12,00,000 per year

    HiRole - Azure Data engineerLocation - Chennai, GurugramWork from officeFull timeImmediate joiner Only (15days Notice period only)Candidates with Certification in Databricks is good to haveTechnical Skills:o Proficiency in SQL for complex queries, database design, and optimization.o Strong experience with PySpark for data transformation and processing.o...

  • Data Engineer

    5 hours ago

    Chennai, Tamil Nadu, India, Tamil Nadu LTIMindtree Full time

    Data Engineer:Experience required: 8 to 12 yearsJob location: Chennai, Hyderabad, Pune (5 days work from office from client location)Seeking a highly skilled and motivated Engineering Lead to join our team as a Cloud and Data Engineer. The ideal candidate will possess extensive expertise in cloud technologies and data engineering, with a proven ability to...

  • Data Engineer

    5 hours ago

    Chennai, Tamil Nadu, India, Tamil Nadu Tata Consultancy Services Full time

    Big Data/Hadoop Experience particularly in ingesting data and implementing Data ingestion pipelines, SQOOP, HADOOP, HDFS, HIVE, IMPALA, Java, Scala, Spark Scala is Mandatory Data Engineer with below responsibilities:Lead Data Engineer to build data pipelines to support implementation of data science and analytics use cases. Candidate needs to be able to...

  • Data Engineer

    5 hours ago

    Chennai, Tamil Nadu, India, Tamil Nadu Tata Consultancy Services Full time

    TCS present an excellent opportunity for Data EngineerJob Location: Delhi, Hyderabad, Bangalore, Kochi, Chennai, Kolkata, PuneExperience required :4- 12 yrsSkills: Data Engineer - GCPSQL + PySparkWalk in Interview Date: 22-Nov-25 (Saturday)Address: TCS Chennai, MagnumMust-HaveStrong and working experience in data migration using GCPPython, Spark, Kafka and...

  • Data Engineer

    5 hours ago

    Chennai, Tamil Nadu, India, Tamil Nadu IntraEdge Full time

    Job Title: Data EngineerExperience: 3 to 7 YearsLocation: ChennaiWork Mode: Onsite / Hybrid (as applicable)Job Description:We are looking for a highly skilled and motivated Data Engineer with 3 to 7 years of experience to join our growing team in Chennai. The ideal candidate will have strong expertise in DBT, Snowflake, CI/CD pipelines, and SQL, with a good...

  • Data Engineer

    5 hours ago

    Chennai, Tamil Nadu, India, Tamil Nadu MRF Full time

    Azure/SQL/Application - Data EngineerJob description:Responsible to maintain the data required for managed application like Advanced Planning System, Dealer Management System, etc.,Responsible for user access management on data upload and downloadWorking with functional team to manage the required data in a structured mannerMake use of Azure services like...

  • Lead Data Engineer

    6 days ago

    Chennai, Tamil Nadu, India Eucloid Data Solutions Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Eucloid is looking for a Lead Data Engineer to join our Data Platform team supporting various business applications. The ideal candidate will support the development of data infrastructure for our clients by participating in activities that may include starting from upstream and downstream technology selection to designing and building different components....

  • Data Engineer

    5 days ago

    Chennai, Tamil Nadu, India AABM Cloud Data Solution Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    We are seeking a Senior Data Engineer with10+ yearsof experience to join our team.Required Qualifications10+ years of experience in data engineering or similar role.Hands on experience in snowflake, SSIS, Python ( including NumPy and Pandas) is mandatory.Expertise in SQL and any cloud environment (AWS & Azure, etc.) is mandatory.Proficiency in mordern BI...

  • Data Engineer

    5 hours ago

    Chennai, Tamil Nadu, India, Tamil Nadu Raretech Full time

    Company DescriptionRareTech is a leading IT Products and Services company catering to the needs of clients in Bahrain and the GCC region. The company focuses on providing tailored solutions, leveraging its expertise in mobile application development, networking, IT consulting, and outsourcing to meet evolving business requirements. RareTech has a...