Python Data Engineering Specialist

6 days ago

Nagpur, Maharashtra, India beBeeData Full time ₹ 1,00,00,000 - ₹ 2,00,00,000
Job Description

We are seeking a highly skilled Python Data Engineer to design and develop scalable data pipelines, real-time data streaming solutions, and optimize data processing tasks using Python.

This is an excellent opportunity to work on exciting projects that require developing data engineering environments, distributed computing architectures, and efficient data storage solutions.

Key Responsibilities:
  • Design, develop, and optimize ETL pipelines using open-source or cloud-based tools such as Apache NiFi, Talend, Pentaho, Airflow, and AWS Glue.
  • Write Python scripts to automate data extraction, transformation, and loading processes. Ensure code optimization for performance and scalability.
  • Work with Apache Spark and PySpark to process large datasets in a distributed computing environment, optimizing jobs for performance and resource efficiency.
  • Use orchestration tools like Apache Airflow to schedule, monitor, and automate data pipeline workflows.
  • Design and implement real-time data streaming solutions using technologies like Apache Kafka or AWS Kinesis for high-throughput, low-latency data processing.
  • Collaborate with database administrators to design and implement efficient data storage solutions using relational databases such as PostgreSQL, MySQL, and SQL Server.
Requirements:
  • Experience working with open-source ETL tools such as Apache NiFi, Talend, or Pentaho. Cloud-based tools like AWS Glue or Azure Data Factory are beneficial.
  • Proficiency in Python for automating data processing tasks, writing data pipelines, and working with libraries such as Pandas, Dask, PySpark, etc.
  • Strong knowledge of Apache Spark and PySpark for distributed data processing, along with optimization techniques.
  • Experience using Apache Airflow or similar tools for scheduling and automating data pipelines.
  • Experience with Apache Kafka or AWS Kinesis for building and managing real-time data pipelines.
  • Knowledge of data storage formats such as Parquet, Avro, Delta Lake, or similar open-source table formats.
  • Strong experience with relational databases (e.g., PostgreSQL, MySQL, SQL Server) and a solid understanding of database concepts like indexing, normalization, and query optimization.

  • Senior Data Engineering Specialist

    7 days ago

    Nagpur, Maharashtra, India beBeeDataEngineering Full time ₹ 15,00,000 - ₹ 19,00,000

    Job Title: Senior Data Engineering SpecialistWe are seeking a highly skilled professional to lead the design, implementation, and maintenance of large-scale data processing systems using Airflow and Python.Key Responsibilities:Design and develop complex data pipelines using Airflow and PythonCollaborate with cross-functional teams to integrate data...

  • Data Engineering Specialist

    7 days ago

    Nagpur, Maharashtra, India beBeeDataEngineering Full time ₹ 15,00,000 - ₹ 20,00,000

    Data Engineering Specialist\Job Summary:\We seek a skilled Data Engineer to contribute to our organization's data-driven decision-making process.\About the Role:\\Design and develop scalable data pipelines using distributed systems such as Hadoop, Spark, or NoSQL databases.\Maintain large-scale data processing applications using languages such as Java,...

  • Data Engineering Specialist

    5 days ago

    Nagpur, Maharashtra, India beBeeDataEngineering Full time ₹ 10,94,720 - ₹ 21,49,400

    Job Title: Data Engineering SpecialistKey Responsibilities:Develop scalable data solutions that meet customer needs while maintaining high standards of reliability, security, and performance.Contribute to the definition of product technical architecture.Maintain and document deliverables in a reliable system as needed.Participate in peer reviews for designs,...

  • Data Engineering Specialist

    5 days ago

    Nagpur, Maharashtra, India beBeeDataEngineering Full time ₹ 15,00,000 - ₹ 25,00,000

    Job OverviewWe are seeking a seasoned Data Engineering Specialist to spearhead the design, development, and implementation of large-scale data systems.

  • Senior IoT Data Specialist

    5 days ago

    Nagpur, Maharashtra, India beBeeDataEngineering Full time ₹ 15,00,000 - ₹ 25,00,000

    Job Opportunity:We are seeking a specialist with extensive experience in data engineering and strong Python skills to design and implement scalable IoT solutions.The ideal candidate should have hands-on experience in data transformation and ingestion, as well as proficiency in AWS IoT services and Python programming.Responsibilities include developing and...

  • Senior Data Engineer

    2 weeks ago

    Nagpur, Maharashtra, India beBeeData Full time US$ 8,00,000 - US$ 12,00,000

    Our team is seeking a skilled Data Specialist to contribute to the development of a FinTech application.The ideal candidate will have expertise in data engineering, with strong knowledge of Azure Databricks, SQL, Python, and Py-spark. Automation skills are also essential for this role.The selected individual will work closely with senior engineers, product...

  • Senior Data Engineering Professional

    2 weeks ago

    Nagpur, Maharashtra, India beBeeDataEngineering Full time ₹ 20,00,000 - ₹ 30,00,000

    Job Overview:We are seeking an experienced Data Engineering Specialist to work on a short-term contract. The ideal candidate will have 7+ years of experience in designing and implementing data pipelines using Azure Data Factory, developing optimized Databricks notebooks with SQL & Python, and working with various file formats such as Parquet, Delta, and...

  • Data Engineering Specialist

    3 days ago

    Nagpur, Maharashtra, India beBeeDataEngineering Full time ₹ 15,00,000 - ₹ 28,00,000

    Job Title: Data Engineering SpecialistAs a Data Engineering Specialist, you will be responsible for designing and implementing scalable data storage solutions using Azure SQL Database, Azure Data Lake Storage, Azure Cosmos DB, and Azure Blob Storage. Your expertise in ETL processes and data integration tools will enable us to efficiently move data from...

  • Data Science Specialist

    7 days ago

    Nagpur, Maharashtra, India beBeeData Full time ₹ 15,000 - ₹ 25,000

    Data Science SpecialistWe are seeking a highly motivated and detail-oriented Data Science Specialist to join our Research team. In this role, you will be responsible for researching, developing, and coding Agents using state-of-the-art LLMs with automated pipelines.You will write code for ML engines and micro-services pipelines.You will use, optimize, train,...

  • Senior Data Migration Engineer

    1 week ago

    Nagpur, Maharashtra, India beBeeData Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Job Title: Data Engineering SpecialistWe are seeking a skilled data engineer to join our team and lead the migration of large-scale SQL workloads from on-premise environments to Azure, ensuring high data integrity, minimal downtime, and performance optimization.The ideal candidate will have a strong foundation in data integration, transformation, and...