Big Data Engineer

1 week ago

Chennai, Tamil Nadu, India Citi Full time US$ 90,000 - US$ 1,20,000 per year

Responsible for designing, developing, and optimizing data processing solutions using a combination of Big Data technologies. Focus on building scalable and efficient data pipelines for handling large datasets and enabling batch & real-time data streaming and processing.
 
Responsibilities:
> Develop Spark applications using Scala or Python (Pyspark) for data transformation, aggregation, and analysis.
> Develop and maintain Kafka-based data pipelines: This includes designing Kafka Streams, setting up Kafka Clusters, and ensuring efficient data flow.
> Create and optimize Spark applications using Scala and PySpark: They leverage these languages to process large datasets and implement data transformations and aggregations.
> Integrate Kafka with Spark for real-time processing: They build systems that ingest real-time data from Kafka and process it using Spark Streaming or Structured Streaming.
> Collaborate with data teams: This includes data engineers, data scientists, and DevOps, to design and implement data solutions.
> Tune and optimize Spark and Kafka clusters: Ensuring high performance, scalability, and efficiency of data processing workflows.
> Write clean, functional, and optimized code: Adhering to coding standards and best practices.
> Troubleshoot and resolve issues: Identifying and addressing any problems related to Kafka and Spark applications.
> Maintain documentation: Creating and maintaining documentation for Kafka configurations, Spark jobs, and other processes.
> Stay updated on technology trends: Continuously learning and applying new advancements in functional programming, big data, and related technologies.

Proficiency in:
Hadoop ecosystem big data tech stack(HDFS, YARN, MapReduce, Hive, Impala).
Spark (Scala, Python) for data processing and analysis.
Kafka for real-time data ingestion and processing.
ETL processes and data ingestion tools 
Deep hands-on expertise in Pyspark, Scala, Kafka
 
Programming Languages:
Scala, Python, or Java for developing Spark applications.
SQL for data querying and analysis.
 
Other Skills:
Data warehousing concepts.
Linux/Unix operating systems.
Problem-solving and analytical skills.
Version control systems

Job Family Group:

Technology

Job Family:

Applications Development

Time Type:

Full time

Most Relevant Skills

Please see the requirements listed above.

Other Relevant Skills

For complementary skills, please see above and/or contact the recruiter.

Citi is an equal opportunity employer, and qualified candidates will receive consideration without regard to their race, color, religion, sex, sexual orientation, gender identity, national origin, disability, status as a protected veteran, or any other characteristic protected by law.

If you are a person with a disability and need a reasonable accommodation to use our search tools and/or apply for a career opportunity review Accessibility at Citi.

View Citi's EEO Policy Statement and the Know Your Rights poster.

  • Big Data Engineer

    2 weeks ago

    Chennai, Tamil Nadu, India Insight Global Full time

    Position: Big Data EngineerDuration: 12+ monthsLocation: India - Chennai or GurgaonRate: 4-17 LPARequired Skills & ExperienceAt least 6+ Years of experience with PySpark and MapRProgramming background with PythonExperience with SQL (Relational Databases) doing data modeling, Oracle experience is a plus.Experience with PowerBI and/or TableauJob...

  • Big Data Engineer

    3 days ago

    Chennai, Tamil Nadu, India Impacteers Full time

    Position Overview :We are looking for a skilled and detail-oriented Big Data Engineer to design, develop, and maintain scalable data pipelines and architectures. The role involves working with large datasets, integrating diverse data sources, and ensuring data availability for analytics, machine learning, and business intelligence. The ideal candidate will...

  • Big Data Engineering Leader

    2 weeks ago

    Chennai, Tamil Nadu, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 25,00,000

    We are seeking a highly skilled professional to lead our data engineering initiatives. As a Big Data Engineer, you will be responsible for designing and developing large-scale data processing systems.Key Responsibilities:Design and implement scalable data pipelines using PySpark and HiveDevelop and maintain big data applications using Spark, HBase, and DQ...

  • Big Data Engineering

    5 days ago

    Chennai, Tamil Nadu, India DataHonDo Pte. Ltd. Full time

    Location: Chennai Walk-in Date: 20th September 2025 Joining: Immediate / Within 30 days Employment Type: Full-time / Contract (Open for both)Job Summary:We are hiring skilled and motivated Big Data Engineers with hands-on experience in SQL, Hadoop ecosystem, Spark, Python, and Kubernetes. The ideal candidates will demonstrate a strong learning attitude,...

  • Data Engineer

    1 week ago

    Chennai, Tamil Nadu, India CX Data Labs Full time ₹ 1,04,000 - ₹ 1,30,878 per year

    HiRole - Azure Data engineerLocation - Chennai, Gurugram (5days Onsite)Full timeImmediate joiner Only (15days Notice period only)Technical Skills:o Proficiency in SQL for complex queries, database design, and optimization.o Strong experience with PySpark for data transformation and processing.o Hands-on experience with Databricks for building and managing...

  • Big Data Engineering Lead

    1 week ago

    Chennai, Tamil Nadu, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Senior Big Data EngineerWe are seeking a highly skilled Senior Big Data Engineer to lead our data engineering team and drive the development of scalable, secure, and high-performance big data solutions.Experience in leading distributed teams across multiple geographies and time zones is essential.Strong technical expertise in Scala, Python, Apache Spark...

  • Big Data Engineering

    3 days ago

    Chennai, Tamil Nadu, India DataHonDo Pte. Ltd. Full time

     Location: Chennai Walk-in Date: 20th September 2025 Joining: Immediate / Within 30 days Employment Type: Full-time / Contract (Open for both)Job Summary:We are hiring skilled and motivated Big Data Engineers with hands-on experience in SQL, Hadoop ecosystem, Spark, Python, and Kubernetes. The ideal candidates will demonstrate a...

  • Big Data Engineering

    5 days ago

    Chennai, Tamil Nadu, India DataHonDo Pte. Ltd. Full time ₹ 15,00,000 - ₹ 28,00,000 per year

     Location:Chennai Walk-in Date:20th September 2025 Joining:Immediate / Within 30 days Employment Type:Full-time / Contract (Open for both)Job Summary:We are hiring skilled and motivated Big Data Engineers with hands-on experience in SQL,Hadoop ecosystem, Spark, Python, and Kubernetes. The ideal candidates will demonstrate a strong learning...

  • Senior Big Data Engineer

    2 weeks ago

    Chennai, Tamil Nadu, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Big Data Platform EngineerAt our organization, we are building a world-class big data platform to bring value out of data for us and for our customers.We aim to democratize data, support exploding business, provide reporting and analytics self-service tools, and fuel existing and new business critical initiatives.Our mission is to build scalable, available,...

  • Data Engineer

    2 weeks ago

    Chennai, Tamil Nadu, India Quantrail Data Full time US$ 60,000 - US$ 1,00,000 per year

    Location: Chennai, India (On-site)Duration: 3 monthsStipend: UnpaidEligibility:2025 graduates onlyConversion: Opportunity for a full-time role based on performance Note: Please apply only if you are serious about joining us. This is a learning-focused internship, and we expect interns to be committed to upskilling themselves and contributing to real-world...