Big Data with Airflow Python

4 days ago

Bengaluru Karnataka, India Achutha Associates Full time

Big Data Engineer - Airflow & Python

Location: Bangalore, India

Experience: 5+ years

Job Summary:
Key Responsibilities:
Design, develop, and maintain ETL/ELT pipelines using Apache Airflow.

Write efficient, scalable, and reusable Python scripts for data processing.

Work with big data technologies (Spark, Hadoop, Hive, etc.) to manage and analyze large datasets.

Optimize and automate workflows, ensuring reliability and performance.

Collaborate with cross-functional teams to design and implement data architecture solutions.

Ensure data quality, integrity, and security across pipelines.

Monitor, troubleshoot, and resolve data pipeline failures and bottlenecks.

Work with cloud platforms (AWS/GCP/Azure) to manage data storage and processing.

**Required Skills**:
5+ years of experience in Big Data Engineering.

Strong expertise in Apache Airflow for workflow automation.

Proficiency in Python for data processing and scripting.

Experience with SQL and database management (PostgreSQL, MySQL, etc.).

Hands-on experience with Hadoop, Spark, Hive, or other big data technologies.

Experience working with cloud-based data solutions (AWS S3, GCP BigQuery, Azure Data Lake).

Knowledge of CI/CD pipelines, Docker, and Kubernetes is a plus.

Strong problem-solving skills and ability to work in an agile environment.

Pay: ₹50,000.00 - ₹100,000.00 per month

Schedule:

- Day shift

**Experience**:

- Big data with airflow python: 6 years (required)

Work Location: Remote

  • Airflow Data Engineer

    3 weeks ago

    Bengaluru, India Virtusa Full time

    Description : We are looking for an experienced Senior Data Engineer to design, develop, and optimize scalable data pipelines and data processing frameworks that support rapidly growing data volume and complexity. The ideal candidate will have deep expertise in Big Data technologies, Azure cloud services, and data pipeline orchestration tools such as Airflow...

  • Big Data Developer

    3 weeks ago

    Bengaluru, India Sourcebae Full time

    Job Alert: Senior Big Data Engineer (5-8 Yrs) | Spark, Python, ScalaAre you a seasoned Big Data Engineer ready to build and optimize large-scale, distributed data processing applications? We are seeking a senior engineer with 5-8 years of experience in the core Big Data ecosystem to deliver high-performance solutions.This is a great opportunity to join a...

  • Big Data Developer

    3 weeks ago

    Bengaluru, India Sourcebae Full time

    Job Alert: Senior Big Data Engineer (5-8 Yrs) | Spark, Python, ScalaAre you a seasoned Big Data Engineer ready to build and optimize large-scale, distributed data processing applications? We are seeking a senior engineer with 5-8 years of experience in the core Big Data ecosystem to deliver high-performance solutions.This is a great opportunity to join a...

  • Big Data Developer

    3 weeks ago

    Bengaluru, India Sourcebae Full time

    Job Alert: Senior Big Data Engineer (5-8 Yrs) | Spark, Python, ScalaAre you a seasoned Big Data Engineer ready to build and optimize large-scale, distributed data processing applications? We are seeking a senior engineer with 5-8 years of experience in the core Big Data ecosystem to deliver high-performance solutions.This is a great opportunity to join a...

  • Big data developer

    3 weeks ago

    Bengaluru, India Sourcebae Full time

    ???? Job Alert: Senior Big Data Engineer (5-8 Yrs) | Spark, Python, Scala ???? Are you a seasoned Big Data Engineer ready to build and optimize large-scale, distributed data processing applications? We are seeking a senior engineer with 5-8 years of experience in the core Big Data ecosystem to deliver high-performance solutions. This is a great opportunity...

  • Big Data Developer

    3 weeks ago

    Bengaluru, India Sourcebae Full time

    🚀 Job Alert: Senior Big Data Engineer (5-8 Yrs) | Spark, Python, Scala 🚀Are you a seasoned Big Data Engineer ready to build and optimize large-scale, distributed data processing applications? We are seeking a senior engineer with 5-8 years of experience in the core Big Data ecosystem to deliver high-performance solutions.This is a great opportunity to...

  • Big Data Developer

    3 weeks ago

    Bengaluru, India Sourcebae Full time

    🚀 Job Alert: Senior Big Data Engineer (5-8 Yrs) | Spark, Python, Scala 🚀Are you a seasoned Big Data Engineer ready to build and optimize large-scale, distributed data processing applications? We are seeking a senior engineer with 5-8 years of experience in the core Big Data ecosystem to deliver high-performance solutions.This is a great opportunity to...

  • Big Data Developer

    3 weeks ago

    Bengaluru, India Sourcebae Full time

    🚀 Job Alert: Senior Big Data Engineer (5-8 Yrs) | Spark, Python, Scala 🚀Are you a seasoned Big Data Engineer ready to build and optimize large-scale, distributed data processing applications? We are seeking a senior engineer with 5-8 years of experience in the core Big Data ecosystem to deliver high-performance solutions.This is a great opportunity to...

  • Big Data Developer

    3 weeks ago

    Bengaluru, India Sourcebae Full time

    🚀 Job Alert: Senior Big Data Engineer (5-8 Yrs) | Spark, Python, Scala 🚀Are you a seasoned Big Data Engineer ready to build and optimize large-scale, distributed data processing applications? We are seeking a senior engineer with 5-8 years of experience in the core Big Data ecosystem to deliver high-performance solutions.This is a great opportunity to...

  • Big Data Developer

    3 weeks ago

    Bengaluru, India Sourcebae Full time

    🚀 Job Alert: Senior Big Data Engineer (5-8 Yrs) | Spark, Python, Scala 🚀Are you a seasoned Big Data Engineer ready to build and optimize large-scale, distributed data processing applications? We are seeking a senior engineer with 5-8 years of experience in the core Big Data ecosystem to deliver high-performance solutions.This is a great opportunity to...