Senior Big Data Engineer

20 hours ago

Kolkata, West Bengal, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

Key Results and Activities:

  • Study the existing data landscape and understand the current integration framework, performing impact assessments as required.
  • Develop Spark jobs using Scala for new project requirements, enhancing efficiency and productivity.
  • Improve existing Spark jobs for ongoing product enhancements, ensuring optimal performance and results.
  • Tune Spark job performance, conduct stress testing, and optimize outcomes.
  • Create new data pipelines for developed/enhanced Spark jobs using AWS Lambda or Apache Airflow, streamlining data processing and analysis.
  • Design databases, logical design, physical design, star schema, snowflake schema, and ensure seamless data integration and sharing.
  • Analyze data processing, integration, modeling, and reporting requirements, defining data loading strategies considering volume, data types, frequency, and analytics specifications.
  • Maintain a balance between cost and performance, adhering to quality guidelines and schedules.

Work Experience and Qualifications:

Must-Have:

  • 7-9 years of IT experience with a strong background in data engineering.
  • 5+ years of AWS related project experience with a focus on big data and cloud computing.
  • Associate Level and Professional Level AWS Certification (good to have).
  • In-depth knowledge of AWS Services: S3, EC2, EMR, Serverless, Athena, AWS Glue, Lambda, Step Functions.
  • Cloud Databases: AWS Aurora, Singlestore, RedShift, Snowflake.
  • Big Data: Hadoop, Hive, Spark, YARN.
  • Programming Language: Scala, Python, Shell Scripts, PySpark.
  • Operating System: Any flavor of Linux, Windows.
  • Strong SQL Skills.
  • Orchestration Tools: Apache Airflow.
  • Expertise in developing ETL workflows with complex transformations like SCD, deduplications, aggregations.
  • Thorough understanding of AWS VPC, Subnets, Security Groups, & Route Tables.
  • Quick learner, adaptable to new technologies.

Education and Certifications:

  • Bachelor's degree in computer science, engineering, or related field (Master's degree is a plus).
  • Demonstrated continued learning through technical certifications or related methods.
  • Minimum 5 years of Cloud related project experience.

Desirable Traits:

  • Hold strong technical knowledge and experience.
  • Capability to deep dive and research in various technical fields.
  • Self-motivated and focused on delivering outcomes.
  • Able to communicate persuasively through speaking, writing, and presentations.
  • Prior experience of working in a large media company would be an added advantage.

  • Big Data Platform Engineer

    4 weeks ago

    Kolkata, West Bengal, India Peoplemint Consultant LLP Full time

    Job Summary :We are seeking an experienced Big Data Platform Engineer to join our banking technology team. The ideal candidate will be responsible for designing, building, and maintaining scalable big data platforms to support advanced analytics, reporting, and regulatory compliance. This role demands strong technical acumen in distributed systems, data...

  • Senior Big Data Solutions Architect

    1 day ago

    Kolkata, West Bengal, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    **Job Title:** Senior Data EngineerWe are seeking an experienced Senior Data Engineer to join our team.Bachelor's or Master's degree in Computer Science, Information Technology, or related field5+ years of experience in data analytics and engineeringProficiency in big data tools: Hadoop, Spark, Kafka, Hive, and relational SQL and NoSQL databasesExperience...

  • Senior Big Data Engineer

    7 days ago

    Kolkata, West Bengal, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Job Opportunity: Senior Data EngineerWe are seeking a highly skilled Data Engineer to join our team and contribute to the development of a cutting-edge application for a leading retailer. As a key member of our engineering team, you will be responsible for designing, implementing, and maintaining large-scale data systems.Key Responsibilities:Design and...

  • Mastering Big Data Solutions

    5 days ago

    Kolkata, West Bengal, India beBeeData Full time ₹ 0 - ₹ 1

    Big Data Specialist\We are seeking a skilled Big Data Specialist to drive data innovation and growth across our global operations.\The ideal candidate will have at least 5-8 years of experience in big data engineering with strong expertise in:\Key Technologies:\Palantir Foundry / WorkshopPySparkTypescript\About the Role:\You will leverage your expertise in...

  • Big Data Specialist

    20 hours ago

    Kolkata, West Bengal, India beBeeDataEngineer Full time US$ 8,00,000 - US$ 12,00,000

    Job Title: Big Data Specialist About the Role:We are seeking a highly skilled professional to join our team as a Data Engineer. As a Data Engineer, you will be responsible for designing and developing scalable data architectures, ensuring high-performance data processing, and optimizing data quality. Key Responsibilities:Design and develop scalable data...

  • Strategic Data Engineering Lead

    20 hours ago

    Kolkata, West Bengal, India beBeeBigData Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Job OverviewAs a Senior Big Data Engineer, you will be responsible for leading big data initiatives and driving business outcomes through technical innovation. The ideal candidate will have experience in managing large-scale data systems, designing and implementing efficient ingestion, transformation, and enrichment pipelines."],

  • Technical Specialist – Big Data Engineer

    5 days ago

    Kolkata, West Bengal, India beBeeData Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Job Title: Software Engineering ProfessionalWe are looking for an experienced Software Engineering Professional to join our team.This is a highly demanding role that requires the ability to design and implement scalable data architectures, develop and optimize ETL/ELT pipelines, and perform data ingestion, transformation, and orchestration across structured...

  • High-Performance Big Data Solutions Specialist

    19 hours ago

    Kolkata, West Bengal, India beBeeBigDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    We are seeking a highly skilled Big Data Engineer to design, develop, and deploy scalable big data solutions.Key Responsibilities:Designing and developing big data architectures using Hadoop and Spark technologiesDeveloping and deploying data pipelines using Scala programming languageFamiliarity with AWS cloud platform and experience working with large...

  • Big Data Systems Developer

    5 days ago

    Kolkata, West Bengal, India beBeeData Full time ₹ 18,00,000 - ₹ 24,00,000

    Job DescriptionWe are seeking a skilled Data Engineer to join our team. As a Data Engineer, you will play a critical role in building and maintaining complex data systems.Your primary responsibility will be to work closely with our data science team to design, develop, and deploy data pipelines that collect, transform, and analyze large datasets. You will...

  • Big Data Platform Leader

    5 days ago

    Kolkata, West Bengal, India beBeeTechnical Full time US$ 1,80,000 - US$ 2,50,000

    Job DescriptionWe are seeking a seasoned professional to lead our team of engineers in building a scalable and extensible big data platform.The ideal candidate will have a strong track record of relevant work experience and a computer science or related technical discipline.Responsibilities include:Aligning the organization with key client...