Senior Data Engineer Position

2 days ago

Tiruchi, Tamil Nadu, India beBeeDataArchitect Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

Job Summary:

We are seeking a skilled Data Architect to design and develop scalable data pipelines, real-time data streaming solutions, and optimize data processing tasks using Python.

This is an excellent opportunity to work on exciting projects that require developing data engineering solutions with open-source or cloud technologies.

Main Responsibilities:

  • Design, implement, and optimize ETL pipelines using Apache Nifi, Talend, Pentaho, Airflow, and AWS Glue.
  • Write Python scripts to automate data extraction, transformation, and loading processes, ensuring optimized code for performance and scalability.
  • Work with Apache Spark and PySpark to process large datasets in a distributed computing environment, optimizing Spark jobs for performance and resource efficiency.
  • Use Apache Airflow or other orchestration tools to schedule, monitor, and automate data pipeline workflows.
  • Design and implement real-time data streaming solutions using technologies like Apache Kafka or AWS Kinesis for high-throughput, low-latency data processing.
  • Manage relational databases such as PostgreSQL, MySQL, and SQL Server for data storage, management, and optimization, understanding database concepts like normalization, indexing, and query optimization.
  • Write and optimize complex SQL queries for data extraction, transformations, and aggregation across large datasets, ensuring efficient and scalable queries.

Key Skills & Qualifications:

  • ETL Tools: Experience working with open-source ETL tools such as Apache Nifi, Talend, or Pentaho. Cloud-based tools like AWS Glue or Azure Data Factory are beneficial.
  • Python Scripting: Proficiency in Python for automating data processing tasks, writing data pipelines, and working with libraries such as Pandas, Dask, PySpark, etc.
  • Big Data Technologies: Experience with Apache Spark and PySpark for distributed data processing, along with optimization techniques.
  • Data Orchestration: Experience using Apache Airflow or similar tools for scheduling and automating data pipelines.
  • Data Streaming: Experience with Apache Kafka or AWS Kinesis for building and managing real-time data pipelines.
  • Open-Source File Formats: Knowledge of Apache Parquet, Apache Avro, Delta Lake, or similar open-source table formats for efficient data storage and retrieval.
  • Relational Databases: Strong experience with at least one relational database and a solid understanding of database concepts like indexing, normalization, and query optimization.
  • SQL Expertise: Strong skills in writing and optimizing complex SQL queries.

Benefits:

  • A competitive salary and benefits package.
  • The opportunity to work with cutting-edge technology and innovative projects.
  • Ongoing training and professional development opportunities.

Requirements:

  • Bachelor's or Master's degree in Computer Science, Engineering, Information Systems, or a related field (or equivalent work experience).

Additional Qualities:

  • Excellent problem-solving and troubleshooting skills.
  • Ability to work both independently and in a collaborative environment.
  • Strong communication skills, both written and verbal.
  • Detail-oriented with a focus on data quality and performance optimization.
  • Proactive attitude and the ability to take ownership of projects.

  • Data Engineer Leadership Position

    2 days ago

    Tiruchi, Tamil Nadu, India beBeeDatabase Full time ₹ 1,80,00,000 - ₹ 2,50,00,000

    Chief Data Engineering ArchitectThis is a senior leadership position within our data engineering team. The successful candidate will be responsible for architecting and scaling our core data foundation, which supports our enterprise SaaS and AI-driven products.Data Model Development: Design and develop a robust semantic layer using dbt pipelines and...

  • Senior Data Engineer

    4 days ago

    Tiruchi, Tamil Nadu, India beBeeData Full time ₹ 12,00,000 - ₹ 19,00,000

    Job OverviewA senior data engineer position is available, focusing on managing and analyzing complex data sets.Key Responsibilities:Manage large-scale data systems using AirflowDevelop and implement data pipelines in PythonCollaborate with cross-functional teams to drive business insightsRequirements:Senior data engineers should possess excellent...

  • Senior Data Engineering Specialist

    1 week ago

    Tiruchi, Tamil Nadu, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    About UsWe are seeking a highly skilled Senior Data Engineer to join our team. This is a critical role where you will create scalable pipelines that process clinical encounter data, retrieve associated medical documents, and deliver validated information to our machine learning systems for inference.This position involves working closely with software...

  • Transformative Data Engineer Opportunity

    6 days ago

    Tiruchi, Tamil Nadu, India beBeeDataEngineer Full time ₹ 30,00,000 - ₹ 40,00,000

    Senior Data Engineer RoleThis is a senior data engineer position that involves designing, developing, and maintaining scalable data pipelines and data warehouse solutions on Google Cloud Platform.

  • Senior Data Engineer

    2 days ago

    Tiruchi, Tamil Nadu, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 20,00,000

    Senior Data EngineerWe are seeking a highly skilled Senior Data Engineer to join our team.This is a unique opportunity to work with cutting-edge technology and collaborate with experienced professionals in the field of data engineering.Job Description:In this role, you will be responsible for designing, developing, and deploying scalable data pipelines and...

  • Senior Data Engineering Leader

    2 days ago

    Tiruchi, Tamil Nadu, India beBeeDataEngineering Full time ₹ 1,20,00,000 - ₹ 1,50,00,000

    As a senior data engineering leader, you will spearhead the design and implementation of scalable, high-performance data solutions.Key ResponsibilitiesYou will lead the development of complex data pipelines using Databricks, Azure Data Services, Python, PySpark, and distributed computing frameworks.The ideal candidate will have hands-on experience with data...

  • Senior Data Engineer Position

    2 days ago

    Tiruchi, Tamil Nadu, India beBeeData Full time ₹ 15,00,000 - ₹ 30,00,000

    Key Responsibilities:We seek an experienced Data Engineer to contribute to the planning, design, development, testing, debugging, and documentation of software solutions that meet our standards and policies.The ideal candidate will have a strong background in data engineering, with hands-on experience in Python, Spark, and big query.Collaborate with...

  • Senior Data Engineer Position

    4 days ago

    Tiruchi, Tamil Nadu, India beBeeDataEngineer Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Job OverviewWe are seeking a highly skilled Data Engineering Expert to design, develop, and maintain scalable data pipelines and ETL processes.The ideal candidate will have experience working with cloud environments (AWS, Azure, GCP) and Databricks, as well as a strong proficiency in SQL, Python, and other relevant programming languages.

  • Data Architect Position

    1 week ago

    Tiruchi, Tamil Nadu, India beBeeRedshift Full time ₹ 23,29,278 - ₹ 30,13,499

    Senior Redshift Data Architect RoleThe ideal candidate will possess extensive experience in Database platforms including AWS Redshift, RDS, MySQL and Snowflake.Key Responsibilities:Design and maintain scalable data architectures for cloud environments and applications.Develop and implement software, services, and applications by identifying system...

  • Senior Data Engineer Lead

    2 days ago

    Tiruchi, Tamil Nadu, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    High-End Engineering Manager WantedWe are seeking a senior data engineer to lead and mentor a team of engineers to deliver high-quality data solutions for a highly scalable data platform in clinical oncology or the healthcare domain.The successful candidate will oversee the design and maintenance of data systems, manage and mentor a team, and align technical...