Optimizing Data Pipelines

5 days ago

Ellore, Andhra Pradesh, India beBeeData Full time ₹ 12,60,000 - ₹ 20,16,000
Job Role: Senior Data Engineer

The role of a Senior Data Engineer is to design and implement data pipeline architecture that is optimal.

  • Key Responsibilities:
  • Create and maintain data pipeline architecture that is efficient and scalable.
  • Design and implement events/streaming based architecture/design.
  • Identify and implement internal process improvements.
  • Build the infrastructure required for extraction, transformation, and loading of data from various sources.
  • Develop data tools for analytics and data scientist team members.

Requirements:

  • Proficiency in either Scala or Python, with expertise in Spark, and experience with performance tuning and optimization.
  • Experience with big data tools: HDFS/S3, Spark/Flink,Hive,Hbase, Kafka/Kinesis, etc.
  • Experience with relational SQL and NoSQL databases, including Elasticsearch and Cassandra/Mongodb.
  • Experience with data pipeline and workflow management tools: Azkaban, Luigi, Airflow, etc.
  • Experience with AWS /GCP cloud services.
  • Experience with stream-processing systems: Spark-Streaming/Flink etc.
  • Experience with object-oriented/object function scripting languages: Java, Scala, etc.
  • Strong analytic skills related to working with structured/unstructured datasets.

Desirable Skills:

  • A few weekend side projects up on GitHub.
  • Contributed to an open-source project.
  • Worked at a product company.
  • Working knowledge of a backend programming language.

  • Lead Data Pipeline Architect

    2 weeks ago

    Ellore, Andhra Pradesh, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 20,00,000

    **Data Engineer Position Overview**Seeking a skilled Data Engineer to design, build, and optimize scalable data pipelines. The ideal candidate will enable seamless data integration, transformation, and analytics to support business intelligence and advanced analytics initiatives.Design, build, and maintain ETL/ELT pipelines using AWS Glue, Lambda, and...

  • Data Engineer for Advanced Data Pipelines

    2 weeks ago

    Ellore, Andhra Pradesh, India beBeeData Full time ₹ 15,00,000 - ₹ 25,00,000

    Job Title: Data Ingestion SpecialistWe are seeking a skilled engineer to fill the role of Data Ingestion Specialist.Key Responsibilities:Design, develop and optimize data ingestion pipelines for integrating multiple sources into Databricks.Implement and maintain Continuous Integration/Continuous Deployment (CI/CD) pipelines for data workflows.Deploy and...

  • Data Pipeline Specialist

    3 days ago

    Ellore, Andhra Pradesh, India beBeeAbInitioDeveloper Full time ₹ 18,00,000 - ₹ 25,00,000

    Job Opportunity: Ab Initio DeveloperAbout the RoleWe are seeking a skilled Ab Initio Developer to join our team. As an Ab Initio Developer, you will be responsible for designing and implementing data pipelines using Ab Initio.Main Responsibilities:Design and implement efficient data pipelines using Ab Initio, ensuring seamless integration with existing...

  • Senior Data Pipeline Specialist

    2 weeks ago

    Ellore, Andhra Pradesh, India beBeeData Full time ₹ 15,00,000 - ₹ 25,00,000

    Job OverviewWe are seeking a skilled Data Transformation Specialist to join our team.The ideal candidate will have a strong background in developing and maintaining data pipelines using various frameworks.They will be responsible for implementing client-specific data transformations, supporting data ingestion and quality checks during onboarding, and...

  • A Pioneer in Scalable Data Pipelines

    1 week ago

    Ellore, Andhra Pradesh, India beBeeDataEngineer Full time ₹ 1,29,74,400 - ₹ 2,03,60,600

    Data Engineering ProfessionalAs a skilled Data Engineer, you will play a pivotal role in designing and developing scalable data pipelines that drive business growth. Our innovative platform harnesses the power of big data to deliver real-time insights and strategic recommendations.Key Responsibilities:Design and build production-level data pipelines that...

  • Data Pipeline Architect

    4 days ago

    Ellore, Andhra Pradesh, India beBeeData Full time ₹ 12,00,000 - ₹ 20,00,000

    Job Title: Data Pipeline ArchitectAs a key member of our data team, you will be responsible for designing and implementing scalable data pipelines to ensure seamless data flow and accessibility.Create interactive dashboards using Bold BI, Power BI, or Superset to provide actionable insights to stakeholders.Develop data pipelines and automation scripts using...

  • Senior Data Pipeline Developer

    4 days ago

    Ellore, Andhra Pradesh, India beBeeDataEngineering Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Job Opportunity: Data Engineering SpecialistWe are seeking a highly skilled Data Engineering Specialist to lead the design, development, and deployment of scalable data pipelines and ETL processes using Databricks.Responsibilities:Collaborate with stakeholders to understand data requirements and develop technical solutions that meet their needs.Design,...

  • Data Pipeline Architect

    6 days ago

    Ellore, Andhra Pradesh, India beBeeDataFlowConsultant Full time ₹ 15,00,000 - ₹ 25,50,000

    Data Flow ConsultantAs a Data Flow Consultant, you will design and develop scalable data pipelines using Oracle Data Flow or Apache Spark (PySpark) with expertise in data patterns, Python, and SQL.The ideal candidate will have 3-12 years of experience in building and managing Data Lakes, Data Warehouses, and Data Pipelines for seamless integration and...

  • Data Pipeline Architect

    2 weeks ago

    Ellore, Andhra Pradesh, India beBeeExpertise Full time ₹ 15,00,000 - ₹ 25,00,000

    Cloud Data SpecialistDesign and implement scalable data pipelines using Azure services such as Data Factory, Databricks, and Data Lake to deliver high-quality business insights and visualization solutions.The ideal candidate will have a strong expertise in building end-to-end data pipelines and delivering efficient data solutions on Azure.Key...

  • Senior Data Pipeline Developer

    2 days ago

    Ellore, Andhra Pradesh, India beBeeDataEngineer Full time ₹ 20,00,000 - ₹ 25,00,000

    Job SummaryWe are seeking an experienced Data Engineer to design, develop, and manage scalable data pipelines and transformation frameworks.Key Responsibilities:Design and build large-scale ETL/ELT pipelines using Azure Data Factory (ADF), Azure Databricks (Spark), and Azure Synapse Analytics.Develop and optimize PySpark/Spark SQL jobs for batch and...