Big Data Developer

3 weeks ago

Chennai Tamil Nadu India, Tamil Nadu IntraEdge Full time

Job Title: Senior Big Data Engineer (Python)

Experience: 5+ Years

Locations: Chennai, Bangalore, Gurugram

Employment Type: Full-Time

Notice Period: Immediate to 30 Days Preferred

Job Summary

We are seeking an experienced Big Data Engineer with strong expertise in Python to design, build, and manage large-scale data pipelines and analytics solutions. The ideal candidate will have hands-on experience working with Big Data technologies and cloud platforms, and a passion for writing efficient, scalable, and maintainable code.

Key Responsibilities

  • Design, develop, and maintain scalable big data pipelines using Python and other data processing tools.
  • Work with distributed data processing frameworks like Spark, Hadoop, Hive, or similar.
  • Implement ETL processes for structured and unstructured data from various sources.
  • Collaborate with data scientists, analysts, and other engineering teams to understand data needs.
  • Optimize data workflows and ensure data quality, integrity, and security.
  • Deploy solutions on cloud platforms (AWS/GCP/Azure) and automate data workflows.
  • Monitor data pipelines, troubleshoot issues, and ensure high availability and performance.

Required Skills & Qualifications

  • Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
  • Minimum 5 years of hands-on experience in Big Data engineering roles.
  • Strong proficiency in Python for data processing and scripting.
  • Experience with Apache Spark, Hadoop ecosystem (Hive, HDFS, HBase), Kafka.
  • Solid understanding of data warehousing concepts and data modeling.
  • Hands-on experience with SQL and NoSQL databases.
  • Familiarity with cloud data platforms like AWS (EMR, S3, Glue), Azure, or Google Cloud.
  • Good understanding of CI/CD, containerization (Docker, Kubernetes), and version control (Git).

Preferred Skills (Nice to Have)

  • Experience with Airflow or other workflow orchestration tools.
  • Knowledge of real-time data processing (e.g., Spark Streaming, Flink).
  • Exposure to data governance, data lineage, and catalog tools.

Why Join Us?

  • Opportunity to work on cutting-edge Big Data solutions with global clients.
  • Collaborative and innovation-driven work culture.
  • Competitive compensation and career growth opportunities.
  • Flexible work environment with hybrid (based on role).

  • Big Data Developer

    3 weeks ago

    Chennai, Tamil Nadu, India, Tamil Nadu Tata Consultancy Services Full time

    4+ years of hands on development experience in programming languages such as JAVA,SCALA using Maven, Apache Spark Frameworks and Unix Shell scriptingShould be comfortable with Unix File system as well as HDFS commandsShould have worked on query languages such as Oracle SQL, Hive SQL, Spark SQL, Impala, HBase DB Should be flexibleShould have good...

  • Big Data Developer

    1 day ago

    Chennai, Tamil Nadu, India Net2Source Full time ₹ 2,50,000 - ₹ 7,50,000 per year

    Big Data Developer Skills Required: Big Data & Hadoop Ecosystem, Python, PySpark Experience: 5-10 Years Share your updated resume now.

  • Big Data Developer

    2 weeks ago

    Chennai, Tamil Nadu, India, Tamil Nadu Tata Consultancy Services Full time

    Role: Big Data DeveloperExperience: 6Location: Chennai/GurgaonRequired Technical Skill Set:GCP , HDFS, Unix Script, Autosys ,Hive.Must Have:Hands on experience on Spark framework, Unix scripting, Hive queries, writing UDF in Hive. Theortical knowledge and POC alone will not suffice. Good knowledge in Software Development Life Cycle and Project Development...

  • Senior Big Data Developer

    2 weeks ago

    Chennai, Tamil Nadu, India MOURI Tech Foundation Full time ₹ 8,00,000 - ₹ 24,00,000 per year

    Looking for a Senior Big Data Developer to design, build, and optimize data pipelines and analytics solutions. Candidates with prior exposure to the Ericsson Wallet Platform will be given preference.Key ResponsibilitiesDevelop, test, and deploy high-performance data pipelines using Hadoop, Spark, Kudu, and HBase.Implement ETL/ELT workflows and ensure data...

  • Big Data Developer

    2 weeks ago

    Chennai, Tamil Nadu, India Tata Consultancy Services Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Greetings from TCSTCS has been a great pioneer in feeding the fire of young Techies like you. We are a global leader in the technology arena and there's nothing that can stop us from growing together.Your role is of key importance, as it lays down the foundation for the entire project.Position:Bigdata DeveloperJob Location:Bangalore, ChennaiExperience:5+Must...

  • Big Data Developer(Chennai)

    1 week ago

    Chennai, Tamil Nadu, India Tata Consultancy Services Full time ₹ 9,00,000 - ₹ 12,00,000 per year

    Dear Candidates,Greetings from Tata Consultancy ServicesTCS has been a great pioneer in feeding the fire of young techies like you. We are a global leader in the technology arena and there's nothing that can stop us from growing together.Kindly attach your updated resumesWhat we are looking for?Role: Big Data DeveloperExperience: 4 to 10 years...

  • Big Data Developer( Chennai)

    1 week ago

    Chennai, Tamil Nadu, India Tata Consultancy Services Full time ₹ 40,00,000 - ₹ 1,00,00,000 per year

    Dear Candidates,Greetings from Tata Consultancy ServicesTCS has been a great pioneer in feeding the fire of young techies like you. We are a global leader in the technology arena and there's nothing that can stop us from growing together.Kindly attach your updated resumesWhat we are looking for?Role: Big Data DeveloperExperience: 4 to 10 years...

  • Big Data Engineer

    1 week ago

    Chennai, Tamil Nadu, India Citi Full time ₹ 8,00,000 - ₹ 24,00,000 per year

    Responsible for designing, developing, and optimizing data processing solutions using a combination of Big Data technologies. Focus on building scalable and efficient data pipelines for handling large datasets and enabling batch & real-time data streaming and processing. Responsibilities:> Develop Spark applications using Scala or Python (Pyspark) for data...

  • Big Data Developer

    2 weeks ago

    Chennai, Tamil Nadu, India Grid Dynamics Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    Big Data DeveLoperResponsibilitiesDesign and implement data pipelines for migration from HDFS/Hive to cloud object storage (e.g., S3, Ceph).Optimize Spark (and optionally Flink) jobs for performance and scalability in a Kubernetes environment.Ensure data consistency, schema evolution, and governance with Apache Iceberg or equivalent table formats.Support...

  • Big Data Lead Engineer

    3 days ago

    Chennai, Tamil Nadu, India Synechron Full time ₹ 1,20,000 - ₹ 3,00,000 per year

    Big Data Lead Engineer - Hadoop - Spark & GCP (BigQuery/DataProc) with Java & Python ExpertiseJob SummarySynechron is seeking an experienced Big Data Lead Engineer to drive the design, development, and management of large-scale data solutionsThis full-time leadership role involves guiding a team of data engineers and architects to deliver scalable,...