Big Data Technical Lead

4 days ago

Kottayam, Kerala, India beBeeData Full time ₹ 90,00,000 - ₹ 1,50,00,000
Senior Data Architect

Job Title: Senior Data Architect

The role involves designing scalable data pipelines, managing large datasets and collaborating with stakeholders to deliver high-quality solutions.

Key Responsibilities:

  • Design, build and optimize complex data systems.
  • Manage large datasets using Spark, PySpark and SQL.
  • Build and maintain real-time applications with Spark Streaming/Kafka.
  • Collaborate with stakeholders to integrate AI/ML models into production.
  • Ensure data quality, scalability and performance in all systems.
  • Deploy and manage Spark workloads on cloud platforms.
  • Automate testing and deployment of Spark jobs via CI/CD pipelines.

Requirements:

  • Bachelor's/Master's degree in Computer Science or related field.
  • Minimum 4 years of professional experience as a Senior Data Architect.
  • Strong expertise in Python, Spark, PySpark.
  • Hands-on experience with Spark SQL, DataFrames, UDFs, DAG execution.
  • Knowledge of data ingestion tools (Kafka, Flume, Kinesis) and data formats (Parquet, Avro, ORC).
  • Proficiency in Airflow/Prefect for scheduling and orchestration.
  • Familiarity with performance tuning in Spark (partitioning, caching, broadcast joins).
  • Experience deploying on Databricks, AWS EMR, Azure HDInsight, or GCP Dataproc.
  • Exposure to testing and CI/CD for data pipelines (pytest, Jenkins, GitHub Actions).

Technical Skills:

  • Spark Core API: RDDs, transformations/actions, DAG execution.
  • Spark SQL: DataFrames, schema optimization, UDFs.
  • Streaming: Structured Streaming, Kafka integration.
  • Data Handling: S3, HDFS, JDBC, Parquet, Avro, ORC.
  • Orchestration: Airflow/Prefect.
  • Performance Tuning: Partitioning, caching, broadcast joins.
  • Cloud Deployment: Databricks, AWS EMR, Azure HDInsight, GCP Dataproc.
  • CI/CD: Pytest/unittest for Spark jobs, Jenkins, GitHub Actions.

  • Building Scalable Big Data Platforms

    2 days ago

    Kottayam, Kerala, India beBeeTechnical Full time ₹ 20,00,000 - ₹ 30,00,000

    Big Data Platform EngineerSigmoid empowers enterprises to make data-driven decisions by integrating advanced data engineering with AI consulting. We collaborate with leading organizations across sectors such as CPG-retail, BFSI, life sciences, manufacturing, and more to solve complex business challenges.As a Technical Lead, you will build a highly scalable...

  • Senior Data Scientist

    1 day ago

    Kottayam, Kerala, India beBeeData Full time ₹ 20,00,000 - ₹ 25,00,000

    Big Data Engineer PositionWe are seeking a skilled Big Data Engineer to lead the design, development, and maintenance of scalable data pipelines and ETL processes.About the RoleThe ideal candidate will have a strong background in data engineering and experience working with big data ecosystems. They will collaborate with stakeholders to gather requirements...

  • Big Data Expertise

    3 days ago

    Kottayam, Kerala, India beBeeDataEngineer Full time ₹ 18,00,000 - ₹ 25,00,000

    Job Title: Data Engineering ExpertiseOverviewWe are seeking a skilled Data Engineer to design and build high-performance data pipelines that process massive amounts of data. The ideal candidate will have expertise in big data technologies, Python programming skills, and experience with database optimization.Key ResponsibilitiesDesign and implement scalable...

  • Big Data Specialist

    2 days ago

    Kottayam, Kerala, India beBeeData Full time ₹ 20,00,000 - ₹ 25,00,000

    Big Data ProfessionalWe are seeking a highly skilled professional to work with large datasets and possess expertise in data analysis, visualization, and processing.The ideal candidate will have experience working with big data technologies and skills in:PySpark: To analyze and process large datasetsHive: For storing and querying large datasetsIn addition to...

  • Big Data Architect

    2 days ago

    Kottayam, Kerala, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Job Role: Big Data ArchitectWe are seeking a seasoned data expert to drive the development and implementation of our big data infrastructure.The ideal candidate will have a strong background in designing, building, and maintaining large-scale data systems, including data warehousing, ETL/ELT pipelines, and data governance.Key Responsibilities:* Design and...

  • Principal Big Data Specialist

    1 day ago

    Kottayam, Kerala, India beBeeDataEngineering Full time ₹ 20,00,000 - ₹ 25,00,000

    Job Title: Data EngineerOverview:We are looking for a skilled professional to design, develop and maintain large-scale data systems.About the Role:This is a key position within our organization, responsible for creating, implementing and maintaining scalable data solutions.The ideal candidate will have strong technical skills, excellent problem-solving...

  • Senior Big Data Specialist

    3 days ago

    Kottayam, Kerala, India beBeeData Full time ₹ 1,68,00,000 - ₹ 2,03,20,000

    Job OverviewDevelop high-performance data pipelines using AWS and PythonCollaborate with cross-functional teams to implement scalable big data solutionsRequired Skills & QualificationsProficiency in Python and PySparkHands-on experience with version control tools like GitFamiliarity with Amazon's Analytics services such as AWS Glue, Amazon Athena, and...

  • Practice Lead Data

    3 days ago

    Kottayam, Kerala, India Coforge Full time

    Practice Lead Data & AnalyticsJob Location: Greater NoidaExperience Required: 20+ YearsMandatory Skills: Data & Analytics, Data Management, SSIS, SSRS, SQL, Snowflake, Azure Data Factory, Big Data, Hadoop, Cloud data migrations, Data Quality and Governance, Data Engineering, ETL /ELT, Data Ops, BI and Analytics, Data Science and Machine Learning, or AI / Gen...

  • Lead Data Analyst

    2 days ago

    Kottayam, Kerala, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Lead Data Analyst - Financial Modeling ExpertThe ideal candidate will be responsible for creating and implementing data-driven solutions to drive business growth.They will work closely with cross-functional teams to develop predictive models, leveraging expertise in machine learning, forecasting, and data analysis.Key responsibilities include:Developing and...

  • AI Technical Lead

    2 days ago

    Kottayam, Kerala, India beBeeGenerative Full time ₹ 30,00,000 - ₹ 40,00,000

    About the RoleWe are seeking a seasoned AI leader to drive the development and deployment of Large Language Models (LLMs) and Generative AI applications. This is an exceptional opportunity to leverage your expertise in traditional machine learning, deep neural networks, and Azure AI services to build and deploy cutting-edge AI solutions.The successful...