Senior Big Data Developer

3 days ago

Madurai, Tamil Nadu, India beBeeDataEngineer Full time ₹ 1,20,00,000 - ₹ 2,50,00,000

Data Engineering Position

We are seeking an accomplished Data Engineer to play a pivotal role in designing, developing, and maintaining efficient ETL processes using Databricks PySpark. This includes extracting, transforming, and loading data from various sources into our data lake and warehouse.

Key Responsibilities:

  1. ETL Development:
  • Design and develop ETL processes using Databricks PySpark for efficient data extraction, transformation, and loading.
  • Optimize ETL workflows for performance and scalability, leveraging Databricks PySpark and Spark SQL.
  • Implement robust error handling and monitoring mechanisms to proactively detect and resolve issues within ETL processes.
  • Design and implement data solutions following the Medallion Architecture principles, organizing data into Bronze, Silver, and Gold layers.
  • Ensure data is appropriately cleansed, enriched, and optimized at each stage to support robust analytics and reporting.
Data Pipeline Management:
  • Create advanced data pipelines using Databricks workflows.
  • Develop and maintain data pipelines using Databricks PySpark, ensuring data quality, integrity, and reliability throughout the ETL lifecycle.
  • Collaborate with data engineering, data science, and business intelligence teams to translate data requirements into efficient ETL workflows and pipelines.
Data Analysis and Query Optimization:
  • Write and optimize complex SQL queries for data manipulation, aggregation, and analysis within Databricks PySpark applications.
Project Coordination and Continuous Improvement:
  • Participate in project planning and coordination activities to ensure timely delivery of ETL solutions.
  • Stay updated on the latest developments in Databricks PySpark, Spark SQL, and related technologies, recommending and implementing best practices and optimizations.
  • Document ETL processes, data lineage, and metadata to facilitate knowledge sharing and ensure compliance with data governance standards.

Required Skills and Qualifications:

  1. Bachelor's degree in Computer Science, Engineering, or a related field.
  2. Minimum 3 years of experience as an ETL developer with a strong focus on Databricks PySpark development.
  3. Proficiency in Python programming with extensive experience in developing and debugging Databricks PySpark applications.
  4. In-depth understanding of Spark architecture and internals, including hands-on experience with Spark RDDs, DataFrames, and Spark SQL.
  5. Expertise in writing and optimizing complex SQL queries for data manipulation, aggregation, and analysis.
  6. Proven experience working with large-scale data warehousing and ETL frameworks.
  7. Strong problem-solving skills and ability to troubleshoot and resolve ETL process issues.
  8. Excellent communication and collaboration skills with the ability to work effectively in a team environment.

Preferred Skills and Qualifications:

  1. Experience with cloud platforms, preferably AWS.
  2. Experience with data platform tools such as DataBricks, Snowflake, and Tableau.
  3. Demonstrated ability to implement best practices for ETL processes and data management.
  4. Strong understanding of data governance and data quality principles.
  5. Relevant certifications in Databricks PySpark, Spark SQL, or related technologies.

  • Senior Big Data Engineer

    3 days ago

    Madurai, Tamil Nadu, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Big Data Job OpportunitySkill: Big Data Hadoop SpecialistYears of Experience: 6 - 12 yearsLocation: IndiaNotice Period: FlexibleJob Description:Key Responsibilities:Develop and implement big data solutions using Spark, Hive, and HBase.Design and deploy scalable data pipelines for efficient data ingestion from disparate sources onto the Big Data...

  • Big Data Specialist

    16 hours ago

    Madurai, Tamil Nadu, India beBeeDataEngineering Full time ₹ 18,00,000 - ₹ 24,00,000

    Big Data SpecialistWe are seeking a highly skilled professional to design, develop and maintain data pipelines for large-scale data ingestion, transformation and storage.About the RoleThe successful candidate will be responsible for designing efficient data pipelines using Big Data ecosystems and working closely with cross-functional teams to deliver...

  • Big Data Engineer

    2 days ago

    Madurai, Tamil Nadu, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 25,00,000

    Data Solutions ArchitectWe are seeking a skilled professional to design and develop large-scale data processing systems using cutting-edge technologies.Key Responsibilities:Design and implement big data architectures using PySpark, Hive, and SparkDevelop and maintain efficient data processing pipelinesCollaborate with cross-functional teams to integrate big...

  • Senior Big Data Analyst

    5 days ago

    Madurai, Tamil Nadu, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Big Data Expert WantedWe are seeking a skilled expert in big data to play a critical role in data extraction, integration, and advanced analysis by leveraging traditional programming and emerging technologies.Key Responsibilities:Design and develop complex SQL queries for data extraction, manipulation, and reporting.Builddata pipelines for seamless...

  • Senior Big Data Specialist

    16 hours ago

    Madurai, Tamil Nadu, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Lead Data ArchitectJob OverviewDevelop and maintain large-scale data architectures that handle structured and unstructured data.Design and implement ETL/ELT processes for data pipelines, ensuring reliability, scalability, and high availability.Collaborate with cross-functional teams to leverage AWS services and power business-critical data workflows.Work...

  • Big Data Infrastructure Specialist

    2 days ago

    Madurai, Tamil Nadu, India beBeeInfrastructure Full time ₹ 20,00,000 - ₹ 25,00,000

    As a Big Data Infrastructure Specialist, you will be responsible for designing and optimizing large-scale data processing environments.Key Responsibilities:Designing and implementing high-performance data clustersMigrating data between different systems using efficient tools and techniquesPerforming performance tuning and optimization to ensure seamless data...

  • Senior Data Engineering Specialist

    5 days ago

    Madurai, Tamil Nadu, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Job Title: Senior Data EngineerWe are seeking an experienced Senior Data Engineer with expertise in Python, Pyspark, and SQL to join our team. This role requires a deep understanding of big data concepts, hands-on coding experience, and the ability to implement streaming analytics and data processing.Key Requirements:5+ years of experience as a Senior Data...

  • Senior Data Architect

    2 days ago

    Madurai, Tamil Nadu, India beBeeData Full time ₹ 1,80,00,000 - ₹ 2,50,00,000

    Job Title: Informatica ConsultantThis is a full-time on-site role for an ETL Developer, based in Bengaluru.This role involves understanding requirements, designing, developing and testing of ETL architecture, transformation jobs and workflow schedules.Key Responsibilities:Design and develop scalable ETL workflows using Informatica BDM/DEI.Integrate data from...

  • Lead Data Architect

    7 days ago

    Madurai, Tamil Nadu, India beBeeDataEngineer Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Senior Data EngineerWe are seeking a highly skilled Senior Data Engineer to lead our team of data engineers in designing and implementing scalable big data architectures.The ideal candidate will have expertise in cloud infrastructure, data pipelines, and streaming data processing.Lead a team of data engineers to design and implement big data...

  • Senior Data Specialist

    3 days ago

    Madurai, Tamil Nadu, India beBeeDataSpecialist Full time US$ 1,00,000 - US$ 1,20,000

    Senior Data SpecialistThe role of a Senior Data Specialist revolves around developing and implementing data-driven solutions to drive business growth. We are seeking an experienced professional to join our team and leverage their skills in data analysis, machine learning, and optimization methodologies.Key Responsibilities:Develop and maintain advanced data...