Big Data Specialist

4 days ago

Allahabad, Uttar Pradesh, India beBeeDataEngineer Full time US$ 90,000 - US$ 1,20,000
Data Engineering Role

Job Title: Data Engineer

We are seeking a skilled Data Engineer to support data-driven initiatives.

This role involves designing scalable pipelines, working with streaming data, and collaborating with data scientists and stakeholders to deliver high-quality solutions.

Key Responsibilities:

  • Design, build, and optimize data pipelines and ETL workflows.
  • Manage and process large datasets using Spark, PySpark, and SQL.
  • Build and maintain real-time streaming applications with Spark Streaming/Kafka.
  • Collaborate with data scientists and product teams to integrate AI/ML models into production.
  • Ensure data quality, scalability, and performance in all pipelines.
  • Deploy and manage Spark workloads on cloud platforms.
  • Automate testing and deployment of Spark jobs via CI/CD pipelines.

Required Skills and Qualifications:

  • Bachelor's/Master's degree in Computer Science, Computer Engineering, or related field.
  • Minimum 4 years of professional experience as a Data Engineer.
  • Strong expertise in Python, Spark, PySpark.
  • Hands-on experience with Spark SQL, DataFrames, UDFs, DAG execution.
  • Knowledge of data ingestion tools (Kafka, Flume, Kinesis) and data formats (Parquet, Avro, ORC).
  • Proficiency in Airflow/Prefect for scheduling and orchestration.
  • Familiarity with performance tuning in Spark (partitioning, caching, broadcast joins).
  • Experience deploying on Databricks, AWS EMR, Azure HDInsight, or GCP Dataproc.
  • Exposure to testing and CI/CD for data pipelines.

Benefits:

  • Diverse project portfolio.
  • Opportunities for growth and learning.
  • Collaborative team environment.

Others:

  • Please note that the company is committed to providing equal opportunities to all applicants.
  • The company values diversity and inclusion in the workplace.

  • Big Data Expert

    2 days ago

    Allahabad, Uttar Pradesh, India beBeeData Full time ₹ 20,00,000 - ₹ 40,00,000

    Greetings from our organizationOur Team is Looking for a Talented Big Data ProfessionalYears of Experience: 6 - 12 yearsLocation: Pune/ Mumbai/ ChennaiNotice Period: 0 - 45 daysJob Description:We are Seeking Highly Skilled Professionals to Join Our TeamPySparkHiveKey Skills:SparkHBaseData Quality toolsAgile Scrum methodologiesData ingestion from multiple...

  • Data Engineering Specialist

    15 hours ago

    Allahabad, Uttar Pradesh, India beBeeBigData Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Job Title: Data Engineering SpecialistJob Overview:As a Data Engineering Specialist, you will play a pivotal role in designing and developing large-scale data processing systems using cutting-edge technologies.This challenging position requires expertise in big data technologies, including PySpark and Hive. Familiarity with Spark, HBase, and data quality...

  • Big Data Solutions Specialist

    1 day ago

    Allahabad, Uttar Pradesh, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 25,00,000

    Job Title: Big Data EngineerWe are seeking a highly skilled professional with expertise in Scala, Spark, Python, NiFi, and Apache Kafka to join our data engineering team.Key Responsibilities:Design, develop, and optimize large-scale data pipelines and distributed data processing systems.Work extensively with Scala, Spark (PySpark), and Python for data...

  • Freelance Big Data Specialist

    19 hours ago

    Allahabad, Uttar Pradesh, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Job TitleWe are seeking a highly skilled Freelance Data Engineer with expertise in designing, building, and optimizing large-scale data pipelines using Python, SQL, and cloud-based technologies.Data Engineering Expertise: Develop and maintain ETL/ELT pipelines using PySpark, AWS, and Databricks to extract, transform, and load data from various sources.Big...

  • Lead Big Data Engineer

    5 days ago

    Allahabad, Uttar Pradesh, India beBeeDataEngineer Full time ₹ 20,00,000 - ₹ 25,00,000

    Job DescriptionWe are seeking a senior data engineer to lead our big data engineering team. As a key member of the team, you will be responsible for developing architectures that meet business needs and providing technical leadership.The role involves setting up cloud-based data lakes and implementing standardized data models. You will develop data pipelines...

  • Big Data Developer

    19 hours ago

    Allahabad, Uttar Pradesh, India beBeeDataInfrastructure Full time ₹ 12,50,000 - ₹ 17,50,000

    Job Title: Data Infrastructure SpecialistSeeking a seasoned data architect to design and build scalable data systems that handle high-volume financial information.This role involves leading the development of robust data pipelines that ensure data integrity, reliability, and real-time availability across our platforms.The ideal candidate will have expertise...

  • Data Engineer

    2 days ago

    Allahabad, Uttar Pradesh, India beBeeInformatica Full time ₹ 1,80,00,000 - ₹ 2,40,00,000

    Job Title: Data Engineer - Informatica SpecialistAbout the RoleThe successful candidate will be responsible for designing and developing scalable ETL workflows using Informatica BDM/DEI. This will involve integrating data from structured and unstructured sources, including big data platforms.The ideal candidate will have strong data warehousing concepts to...

  • Senior Big Data Specialist

    1 day ago

    Allahabad, Uttar Pradesh, India beBeeDataEngineer Full time ₹ 1,60,00,000 - ₹ 2,40,00,000

    Job OverviewWe are seeking an experienced data professional to join our team as a Data Engineer. The ideal candidate will have strong expertise in PySpark and hands-on experience in building scalable data pipelines.The successful candidate will be responsible for designing, developing, and maintaining batch and streaming data pipelines. They will also ensure...

  • Lead Data Engineering Specialist

    6 days ago

    Allahabad, Uttar Pradesh, India beBeeData Full time ₹ 18,00,000 - ₹ 25,00,000

    Job OpportunityWe are seeking an experienced Senior Data Engineer to lead our data engineering efforts. As a Senior Data Engineer, you will be responsible for designing and building robust data pipelines for ingestion, transformation, and analytics using the best of modern data engineering tools and practices on Google Cloud (GCP).The ideal candidate will...

  • Chief Data Processing Specialist

    3 days ago

    Allahabad, Uttar Pradesh, India beBeeData Full time ₹ 18,00,000 - ₹ 22,00,000

    Big Data Engineer PositionWe're looking for a skilled data engineer to join our professional services team.The ideal candidate will have experience designing and implementing scalable batch processing systems, optimizing database performance, and reducing latency metrics. They should also be proficient in Python programming, big data technologies, and cloud...