Big Data Engineering Specialist

6 days ago

Thāne, Maharashtra, India beBeeData Full time ₹ 10,00,000 - ₹ 16,00,000

Job Description

We are seeking a skilled Big Data Engineer to join our team. As a key member of our Professional Services, you will play a vital role in designing, building, and maintaining high-performance data pipelines that process massive amounts of data.

Key Responsibilities:

  • Design and implement scalable batch processing systems using Python and big data technologies.
  • Optimize database performance, focusing on slow-running queries and latency improvements.
  • Use Python profilers and performance monitoring tools to identify bottlenecks.
  • Reduce P95 and P99 latency metrics across our data platform.
  • Build efficient ETL pipelines that can handle large-scale data processing.
  • Collaborate with data scientists and product teams to understand data requirements.
  • Monitor and troubleshoot data pipeline issues in production.
  • Implement data quality checks and validation mechanisms.
  • Document data architecture and engineering processes.
  • Stay current with emerging big data technologies and best practices.

Qualifications:

  • Bachelor's degree in Computer Science, Engineering, or related technical field.
  • 4+ years of experience in data engineering roles.
  • Strong Python programming skills with focus on data processing libraries.
  • Experience with big data technologies (Spark, Hadoop, etc.).
  • Proven experience optimizing database performance (SQL or NoSQL).
  • Knowledge of data pipeline orchestration tools (Airflow, Luigi, etc.).
  • Understanding of performance optimization techniques and profiling tools.
  • Experience with cloud platforms (AWS, GCP, or Azure).
  • Ability to translate business requirements into technical specifications.
  • Ability to work in UK timezone (12 PM IST to 9:30 PM IST).

Preferred Qualifications:

  • Master's degree in Computer Science or related field.
  • Experience with SEO data or web crawling systems.
  • Experience with Clickhouse Database.
  • Knowledge of distributed systems and microservices architecture.
  • Familiarity with container orchestration (Kubernetes, Docker).
  • Experience with real-time data processing.
  • Contributions to open-source projects.
  • Experience with machine learning operations.

Skills and Abilities:

  • Strong analytical thinking and problem-solving skills.
  • Excellent attention to detail and commitment to data quality.
  • Ability to work effectively in a collaborative team environment.
  • Good communication skills to explain complex technical concepts.
  • Self-motivated with the ability to work independently.
  • Passion for performance optimization and efficiency.

  • Big Data Integration Specialist

    3 days ago

    Thāne, Maharashtra, India beBeeDataIntegration Full time ₹ 20,00,000 - ₹ 30,00,000

    Big Data Integration SpecialistWe are seeking a seasoned professional to join our team in the field of big data integration. As a key member, you will be responsible for validating data transformations, ingestion, and processing logic in Databricks tables across various layers.Key Responsibilities:Design and execute test cases to validate Databricks...

  • Big Data Professional

    4 days ago

    Thāne, Maharashtra, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    We are looking for a skilled Big Data professional to join our team as a Big Data Engineer"> Big Data Engineer ", we require a candidate with strong skills in data engineering and analytics. The ideal candidate will have hands-on experience with Big Data technologies, including data ingestion, processing, and storage.">Key Responsibilities:Data Ingestion:...

  • Big Data Specialist

    6 days ago

    Thāne, Maharashtra, India beBeeDataEngineering Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Databricks Data Engineer OpportunityKey Responsibilities:Develop and maintain scalable data pipelines and ETL processes on Databricks.Lead the architecture and implementation of high-performance data solutions.Collaborate with stakeholders to deliver actionable data insights.Optimize data workflows for performance, cost-efficiency, and reliability.Integrate...

  • Big Data Developer

    4 weeks ago

    Thāne, Maharashtra, India Tata Consultancy Services Full time

    Role :Data Engineering/Big Data Development Location: HyderabadExperience: 4 to 10 YearsMust Have skills :Bigdata Hadoop Platform -Cloudera

  • Senior Big Data Specialist

    4 hours ago

    Thāne, Maharashtra, India beBeeData Full time US$ 90,000 - US$ 1,60,000

    Data Engineering RoleWe are seeking a skilled Data Engineer to join our team.About the Role:This is an exciting opportunity for a seasoned professional with experience in designing and executing large-scale distributed data systems.The ideal candidate will have expertise in big data technologies, including Apache Hadoop, relational databases (e.g., MS SQL...

  • Principal Big Data Specialist

    4 hours ago

    Thāne, Maharashtra, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 25,00,000

    Job Title: Data EngineerWe are seeking a skilled and experienced Data Engineer to join our team. As a Data Engineer, you will be responsible for designing, implementing, and maintaining large-scale data processing systems.The ideal candidate will have strong knowledge of data engineering concepts, including data pipelines, distributed systems, and big data...

  • Big Data Engineer: Optimizing Scalable Data Pipelines

    4 days ago

    Thāne, Maharashtra, India beBeeData Full time ₹ 15,00,000 - ₹ 30,00,000

    Big Data Engineer PositionThe role is for a highly skilled Big Data Engineer with strong expertise in Scala, Spark, Python, NiFi, and Apache Kafka. The ideal candidate will have hands-on experience in building, scaling, and optimizing big data pipelines.We are seeking a talented individual who can design, develop, and optimize large-scale data pipelines and...

  • Lead Big Data Specialist

    4 hours ago

    Thāne, Maharashtra, India beBeeDataEngineer Full time ₹ 1,76,40,000 - ₹ 2,21,80,000

    Senior Data EngineerWe are seeking a Senior Data Engineer to join our team. The ideal candidate will have extensive experience in designing, implementing, and maintaining large-scale data processing systems using big data technologies.Job DescriptionStudy existing technology landscape and understand current data integration framework and do impact assessment...

  • Big Data Leader – Cloud Architect

    6 days ago

    Thāne, Maharashtra, India beBeeCloud Full time ₹ 15,00,000 - ₹ 25,00,000

    Job Title: Data Engineering LeadAbout the RoleWe are seeking an experienced and strategic data engineering leader with expertise in Big Data platforms, Microsoft Azure, and Databricks. The ideal candidate will lead a team of data engineers to design and deliver scalable data pipelines and analytics platforms.This role is perfect for individuals who enjoy...

  • Big Data Expert Wanted

    4 days ago

    Thāne, Maharashtra, India beBeeData Full time ₹ 15,00,000 - ₹ 25,00,000

    Big Data Professional OpportunityKey Responsibilities:Greetings from a top IT services company.PySparkHiveTo succeed in this role, you should have experience with Big Data platforms and data ingestion from disparate sources. You will work closely with cross-functional teams to drive business outcomes.We offer competitive compensation and benefits packages to...