Data Platform Engineer

9 hours ago

Pune, India Epergne Solutions Full time


Epergne Solutions is looking for Data Platform Engineer (Python + Airflow Data Engineer)


Total Yrs. of Experience* 6+ yrs

Relevant Yrs. of experience* 5+ yrs

Mandatory skills* Python / Airflow Data Engineer

Budget: As per market Standards

Work Location* India

Mode- Work from Office

Positions: Contract

Payroll: Epergne solutions


Detailed JD *(Roles and Responsibilities )


We are seeking a highly skilled with 3 to 6 years of experience, specifically with a strong background in AWS technologies. The ideal

candidate will have a deep understanding of Apache Airflow and its integration within AWS ecosystem, enabling efficient data pipeline orchestration and management.


Responsibilities:


  • Design, develop, and maintain complex data pipelines using Python for efficient data processing and orchestration.
  • Collaborate with cross-functional teams to understand data requirements and architect robust solutions within the AWS environment.
  • Implement data integration and transformation processes to ensure optimal performance and reliability of data pipelines.
  • Optimize and fine-tune existing data pipelines / Airflow to improve efficiency, scalability, and maintainability.
  • Troubleshoot and resolve issues related to data pipelines, ensuring smooth operation and minimal downtime.
  • Work closely with AWS services like S3, Glue, EMR, Redshift, and other related technologies to design and optimize data infrastructure.
  • Develop and maintain documentation for data pipelines, processes, and system architecture.
  • Stay updated with the latest industry trends and best practices related to data engineering and AWS services.


Requirements:

  • Bachelor s degree in Computer Science, Engineering, or a related field.
  • Proficiency in Python and SQL for data processing and manipulation.
  • Min 5 years of experience in data engineering, specifically working with Apache Airflow and AWS technologies.
  • Strong knowledge of AWS services, particularly S3, Glue, EMR, Redshift, and AWS Lambda.
  • Understanding of Snowflake is preferred.


  • Experience with optimizing and scaling data pipelines for performance and efficiency.
  • Good understanding of data modeling, ETL processes, and data warehousing concepts.
  • Excellent problem-solving skills and ability to work in a fast-paced, collaborative environment.
  • Effective communication skills and the ability to articulate technical concepts to non-technical stakeholders.


Preferred Qualifications:

  • AWS certification(s) related to data engineering o r big data.
  • Experience working with big data technologies like Snowflake, Spark, Hadoop, or related frameworks.
  • Familiarity with other data orchestration tools in addition to Apache Airflow.
  • Knowledge of version control systems like Bitbucket, Git


Preferred candidates who can join in 30 days/lesser Notice Period

  • Software platform engineer

    2 weeks ago

    Pune, India Data Dynamics Full time

    Overview We are seeking a skilled Platform Engineer to join our team and drive the development, deployment, and supportability of our Kubernetes-based microservices platform, deployed on-premises by customers. You will build comprehensive observability, enable log and report extraction for service cases without real-time access, and optimize our overuse of...

  • Software Platform Engineer

    2 weeks ago

    Pune, India Data Dynamics Full time

    Overview We are seeking a skilled Platform Engineer to join our team and drive the development, deployment, and supportability of our Kubernetes-based microservices platform, deployed on-premises by customers. You will build comprehensive observability, enable log and report extraction for service cases without real-time access, and optimize our overuse of...

  • Software Platform Engineer

    1 week ago

    Pune, India Data Dynamics Full time

    OverviewWe are seeking a skilled Platform Engineer to join our team and drive the development, deployment, and supportability of our Kubernetes-based microservices platform, deployed on-premises by customers. You will build comprehensive observability, enable log and report extraction for service cases without real-time access, and optimize our overuse of...

  • Software Platform Engineer

    1 week ago

    Pune, India Data Dynamics Full time

    OverviewWe are seeking a skilled Platform Engineer to join our team and drive the development, deployment, and supportability of our Kubernetes-based microservices platform, deployed on-premises by customers. You will build comprehensive observability, enable log and report extraction for service cases without real-time access, and optimize our overuse of...

  • Software Platform Engineer

    6 days ago

    Pune, India Data Dynamics Full time

    Overview We are seeking a skilled Platform Engineer to join our team and drive the development, deployment, and supportability of our Kubernetes-based microservices platform, deployed on-premises by customers. You will build comprehensive observability, enable log and report extraction for service cases without real-time access, and optimize our overuse of...

  • Software Platform Engineer

    4 weeks ago

    Pune, Maharashtra, India Data Dynamics Full time

    OverviewWe are seeking a skilled Platform Engineer to join our team and drive the development, deployment, and supportability of our Kubernetes-based microservices platform, deployed on-premises by customers. You will build comprehensive observability, enable log and report extraction for service cases without real-time access, and optimize our overuse of...

  • Software Platform Engineer

    1 week ago

    Pune, India Data Dynamics Full time

    OverviewWe are seeking a skilled Platform Engineer to join our team and drive the development, deployment, and supportability of our Kubernetes-based microservices platform, deployed on-premises by customers. You will build comprehensive observability, enable log and report extraction for service cases without real-time access, and optimize our overuse of...

  • Software Platform Engineer

    2 weeks ago

    Pune, India Data Dynamics Full time

    OverviewWe are seeking a skilled Platform Engineer to join our team and drive the development, deployment, and supportability of our Kubernetes-based microservices platform, deployed on-premises by customers. You will build comprehensive observability, enable log and report extraction for service cases without real-time access, and optimize our overuse of...

  • Software Platform Engineer

    2 weeks ago

    Pune, India Data Dynamics Full time

    OverviewWe are seeking a skilled Platform Engineer to join our team and drive the development, deployment, and supportability of our Kubernetes-based microservices platform, deployed on-premises by customers. You will build comprehensive observability, enable log and report extraction for service cases without real-time access, and optimize our overuse of...

  • Data Engineer

    2 weeks ago

    Pune, India Data Axle Full time

    About Data Axle: Data Axle Inc. has been an industry leader in data, marketing solutions, sales and research for 50 years in the US. Data Axle has set a strategic global centre of excellence in Pune. This centre delivers mission critical data services to its global customers powered by its proprietary cloud-based technology platform and leveraging...