Big Data Developer

1 day ago

tamil nadu, India IntraEdge Full time

Job Title: Senior Big Data Engineer (Python)

Experience: 5+ Years

Locations: Chennai, Bangalore, Gurugram

Employment Type: Full-Time

Notice Period: Immediate to 30 Days Preferred

Job Summary

We are seeking an experienced Big Data Engineer with strong expertise in Python to design, build, and manage large-scale data pipelines and analytics solutions. The ideal candidate will have hands-on experience working with Big Data technologies and cloud platforms, and a passion for writing efficient, scalable, and maintainable code.

Key Responsibilities

  • Design, develop, and maintain scalable big data pipelines using Python and other data processing tools.
  • Work with distributed data processing frameworks like Spark, Hadoop, Hive, or similar.
  • Implement ETL processes for structured and unstructured data from various sources.
  • Collaborate with data scientists, analysts, and other engineering teams to understand data needs.
  • Optimize data workflows and ensure data quality, integrity, and security.
  • Deploy solutions on cloud platforms (AWS/GCP/Azure) and automate data workflows.
  • Monitor data pipelines, troubleshoot issues, and ensure high availability and performance.

Required Skills & Qualifications

  • Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
  • Minimum 5 years of hands-on experience in Big Data engineering roles.
  • Strong proficiency in Python for data processing and scripting.
  • Experience with Apache Spark, Hadoop ecosystem (Hive, HDFS, HBase), Kafka.
  • Solid understanding of data warehousing concepts and data modeling.
  • Hands-on experience with SQL and NoSQL databases.
  • Familiarity with cloud data platforms like AWS (EMR, S3, Glue), Azure, or Google Cloud.
  • Good understanding of CI/CD, containerization (Docker, Kubernetes), and version control (Git).

Preferred Skills (Nice to Have)

  • Experience with Airflow or other workflow orchestration tools.
  • Knowledge of real-time data processing (e.g., Spark Streaming, Flink).
  • Exposure to data governance, data lineage, and catalog tools.

Why Join Us?

  • Opportunity to work on cutting-edge Big Data solutions with global clients.
  • Collaborative and innovation-driven work culture.
  • Competitive compensation and career growth opportunities.
  • Flexible work environment with hybrid (based on role).

  • Big Data Architect

    3 days ago

    Chennai, Tamil Nadu, India RiskInsight Consulting Pvt Ltd Full time

    **Responsibilities**: - Design and architect Big Data solutions to meet business requirements. - Collaborate with stakeholders to understand their data and analytics needs. - Create and maintain data models, data flow diagrams, and data integration strategies. - Develop and implement data ingestion, processing, and storage solutions using Big Data...

  • Big Data Engineer

    2 weeks ago

    Chennai, Tamil Nadu, India CIEL HR Full time

    Job Responsibilities Responsible for administering big data systems like EMR HDFS YARN Spark Hive Oozie and others Work closely with development teams at the design phase of big data jobs and pipelines to improve operational efficiency Troubleshoot big data systems for capacity bottlenecks of memory CPU OS storage and network Performance tuning...

  • Big Data Intern

    1 week ago

    Anna Nagar, Chennai, Tamil Nadu, India Vestel Telecom Services Full time

    **Job title**: Big Data Intern **Job type**: Full time, at Office **Schedule**: Day shift, Mon-Fri **Duration**: 4- 6 months **Stipend**: Nil **Work Location**: Chennai, Anna Nagar. 1. Working with Hadoop, Python, SQL and Spark. 2. Work on various ETL processing methods and tools used. 3. Will work on various cloud platforms for big data infra setup...

  • Big Data Engineer

    2 weeks ago

    Chennai, Tamil Nadu, India CIEL HR Full time

    Big Data Engineer Location Chennai Experience 5 - 10 Years Work Mode Work from office Interview 3 Rounds in total 2 Technical 1 Manager - Final round will be an F2F in Chennai office Primary Skills Hadoop Hive Spark Scala Sql

  • Urgent Search Big Data Developer

    3 weeks ago

    Hyderabad, Telangana - Chennai, Tamil Nadu, India Diverse Lynx Full time

    BIG DATA DEVELOPER EXPERIENCE - 4 LOCATION - CHENNAI HYDERABAD Experience in Python Spark and Hive Understanding of data warehousing and data modeling techniques Knowledge of industry-wide analytical and visualization tools Tableau and R Strong data engineering skills on the Azure Cloud Platform is essential Streaming frameworks like Kafka Knowledge of...

  • Big Data Developer

    1 week ago

    Chennai, Tamil Nadu, India FIS Full time

    Position Type Full timeType Of Hire Experienced relevant combo of work and education Education Desired Bachelor of Computer EngineeringTravel Percentage 0 Big Data Developer Spark HIVE Python and Impala The resource needs to work from Kolkata client location 7-10 yearsAre you curious motivated and forward-thinking At FIS youll have the...

  • Data Engineering

    7 days ago

    tamil nadu, India EXL Full time

    Responsibilities:Work with stakeholders to understand the data requirements to design, develop, and maintain complex ETL processes.Create the data integration and data diagram documentation. Lead the data validation, UAT and regression test for new data asset creation.Create and maintain data models, including schema design and optimization.Create and manage...

  • Data Analytics

    21 hours ago

    Chennai, Tamil Nadu, India NTT DATA Full time

    **Must Have**: - Scripting :Python, Pyspark, Unix scripting (Basic) - Cloud : AWS -Lambda, Glue, Athena,RDS,S3, DynamoDB,SNS, SQS, IAM, Step function, Pipeline monitoring, Security, Access and Role permissions, Logging and monitoring Services. - Database : SQL/Oracle - Tools : Pycham,VishualStudio,AWS CLI - Devops Tools : Terraform ,GIT,GIT actions, CICD...

  • Staff Software Engineer

    1 week ago

    Bangalore, Karnataka Chennai, Tamil Nadu, India Extreme Networks Full time

    Over 50 000 customers globally trust our end-to-end cloud-driven networking solutions They rely on our top-rated services and support to accelerate their digital transformation efforts and deliver unprecedented progress With double-digit growth year over year no provider is better positioned to deliver scalable outcomes than Extreme Inclusion is one of...

  • Senior Data Engineer

    2 weeks ago

    tamil nadu, India Eucloid Data Solutions Full time

    Job DescriptionEucloid is looking for a skilled Data Engineer to join our team and contribute to the design, development, and optimization of data frameworks pipelines using Python, Elasticsearch, SQL DB and Queue System. The ideal candidate will have a strong technical background, a passion for solving complex problems, and experience in building scalable...