Spark Developer

3 days ago

Chennai, Tamil Nadu, India Citi Full time ₹ 9,00,000 - ₹ 12,00,000 per year

ETL Developer will be responsible for designing, implementing, and optimizing distributed data processing jobs to handle large-scale data in Hadoop Distributed File System(HDFS) using Apache Spark and Python. This role required deep understanding of data engineering principles, proficiency in Python and hands-on experience with Spark and Hadoop ecosystems. Developer will collaborate with data engineers, analysts, and business stakeholders to process, transform and drive insights and data driven decisions.

Responsibilities:

  • Data Processing and Transformation:

    Design and Implement of Spark applications to process and transform large datasets in HDFS.

    Develop ETL Pipelines in Spark using Python for data Ingestion, cleaning, aggregation, and transformations.

    Performance Optimization:

    Optimize Spark jobs for efficiency, reducing run time and resource usage.

    Finetune memory management, caching, and partitioning strategies for Optimal performance

    Data Engineering with Hadoop and Spark:

    Load data from different sources into HDFS, ensuring data accuracy and integrity.

    Integrate Spark Applications with Hadoop frameworks like Hive, Sqoop etc.

    Testing and debugging:

    Troubleshoot and debug Spark Job failures, monitor job logs, and Spark UI to Identify Issues.

Qualifications:

  • 2-5 years of relevant experience
  • Experience in programming/debugging used in business applications
  • Working knowledge of industry practice and standards
  • Comprehensive knowledge of specific business area for application development
  • Working knowledge of program languages
  • Consistently demonstrates clear and concise written and verbal communication

  • Expertise in handling complex large-scale Warehouse environments

    •    Hands-on experience writing complex SQL queries, exporting and importing large amounts of data using utilities

Education:

  • Bachelor's degree in a quantitative field (such as Engineering, Computer Science) or equivalent experience

This job description provides a high-level review of the types of work performed. Other job-related duties may be assigned as required.

Job Family Group:

Technology

Job Family:

Applications Development

Time Type:

Full time

Most Relevant Skills

Please see the requirements listed above.

Other Relevant Skills

For complementary skills, please see above and/or contact the recruiter.

Citi is an equal opportunity employer, and qualified candidates will receive consideration without regard to their race, color, religion, sex, sexual orientation, gender identity, national origin, disability, status as a protected veteran, or any other characteristic protected by law.

If you are a person with a disability and need a reasonable accommodation to use our search tools and/or apply for a career opportunity review Accessibility at Citi.

View Citi's EEO Policy Statement and the Know Your Rights poster.

  • Spark Scala Developer

    4 days ago

    Chennai, Tamil Nadu, India Infotel UK Full time ₹ 9,00,000 - ₹ 12,00,000 per year

    Position: Spark Scala DeveloperCompany: Infotel IndiaAt Infotel UK, a premier technology consulting firm, we are seeking a skilled Spark Scala Developer to join our dynamic team. In this role, you will leverage your expertise in Spark and Scala to design and implement advanced data processing solutions that drive insights and provide value to our clients....

  • Hiring Bigdata Java Spark Professionals

    4 weeks ago

    Chennai, Tamil Nadu, India LTIMindtree Full time

    Greetings from LTIMindtree About the job Are you looking for a new career challenge? With LTIMindtree, are you ready to embark on a data-driven career? Working for global leading manufacturing client for providing an engaging product experience through best-in-class PIM implementation and building rich, relevant, and trusted product information across...

  • Apache Spark Senior Technical Lead

    2 days ago

    Chennai, Tamil Nadu, India Sopra Steria Full time US$ 1,50,000 - US$ 2,00,000 per year

    Company Description About Sopra SteriaSopra Steria, a major Tech player in Europe with 50,000 employees in nearly 30 countries, is recognised for its consulting, digital services and solutions. It helps its clients drive their digital transformation and obtain tangible and sustainable benefits. The Group provides end-to-end solutions to make large companies...

  • Scala Developer

    3 weeks ago

    Chennai, Tamil Nadu, India Techmynd Consulting Full time

    Pyspark, Scala DeveloperLocation : Bangalore, ChennaiExperience : 6- 8 YearsMust-Have :- Experience with Scala and Spark & its architecture.- Good SQL Knowledge.Job Description :We are seeking a highly skilled Scala & Spark Developer with strong expertise in distributed data processing and big data technologies. The ideal candidate will have hands-on...

  • Java Developer + Spark

    3 days ago

    Chennai, Tamil Nadu, India Citi Full time US$ 1,00,000 - US$ 1,50,000 per year

    The Applications Development Senior Supervisor is an intermediate management level position responsible for providing full leadership and direction to a team of employees in an effort to establish and implement new or revised application systems and programs in coordination with the Technology team. The overall objective of this role is to lead applications...

  • Experienced Data System Developer Wanted

    12 hours ago

    Chennai, Tamil Nadu, India beBeeDataSystemDeveloper Full time ₹ 20,00,000 - ₹ 25,00,000

    Job Title:Seeking an Experienced Data System Developer to spearhead the design and implementation of large-scale data processing systems utilizing Spark/Cloudera technologies.Key Responsibilities:Develop, implement and maintain end-to-end data pipelines using Databricks, Python, Delta Lake, and Apache SparkMigrate large datasets from legacy systems like...

  • Data Solutions Developer

    10 hours ago

    Chennai, Tamil Nadu, India beBeeDataDeveloper Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Job Title: Data Solutions Developer We are seeking a skilled Data Solutions Developer to join our team. In this role, you will be responsible for designing and implementing data pipelines using Big Data technologies. Requirements:1. Experience in Big Data technologies and working exposure in at least one cloud platform on related data services (AWS / Azure...

  • Software Development Engineer

    4 days ago

    Chennai, Tamil Nadu, India Workday Full time

    Job DescriptionYour work days are brighter here.At Workday, it all began with a conversation over breakfast. When our founders met at a sunny California diner, they came up with an idea to revolutionize the enterprise software market. And when we began to rise, one thing that really set us apart was our culture. A culture which was driven by our value of...

  • Informatica Developer

    4 days ago

    Chennai, Tamil Nadu, India Acabes International Full time

    We are looking for talented and passionate Informatica Developers to join our growing team.Years of experience required - 3 to 6 yrsLocation - ChennaiKey Skills & Technical Experience Required:• Proven experience as an Informatica developer with an IT company• Proficient in Informatica BDM development• Expert in SQL and Shell scripting• Informatica...

  • Sr Developer, Software Development

    3 weeks ago

    Chennai, Tamil Nadu, India transunion Full time

    Job DescriptionWhat You'll Bring:Key Responsibilities -- Develop and maintain front-end & back-end components of our fraud detection platform.- Implement real time data processing and streaming functionalities- Design and develop APIs for integrating various microservices- Collaborate with cross-functional teams to deliver high quality software solutions-...