Large-Scale Data Solutions Engineer

1 day ago

Salem, Tamil Nadu, India beBeeDataProcessing Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

Job Title: Large-Scale Data Processing Specialist

 

**Job Overview**

This contract remote role seeks a highly motivated data processing specialist with 3–4 years of experience in building, optimizing, and maintaining large-scale data solutions. The ideal candidate will bring hands-on expertise in PySpark, Python, SQL, and real-time data streaming, contributing to the design and implementation of scalable data pipelines in cloud environments.

 

**Key Responsibilities**

  • Design, develop, and optimize data pipelines using PySpark, Spark, and Python.
  • Implement ETL/ELT solutions to integrate structured and unstructured data from multiple sources.
  • Develop and maintain Hive queries for seamless data transformations.
  • Write and optimize SQL queries for large datasets.
  • Build, monitor, and support real-time streaming solutions using Apache Kafka.
  • Work on Databricks to create, manage, and optimize data workflows.
  • Deploy and manage solutions in Azure cloud services.
  • Debug, troubleshoot, and optimize code for performance and scalability.
  • Collaborate with cross-functional teams to deliver data-driven solutions.

 

**Required Skills & Qualifications**

  • 3–4 years of experience in data engineering with a focus on PySpark and Python.
  • Strong knowledge of Spark, Hive, SQL, and Unix shell scripting.
  • Experience with Databricks in a production environment.
  • Hands-on experience with Azure cloud services.
  • Experience working with Apache Kafka or other real-time streaming technologies.
  • Strong problem-solving and debugging skills.
  • Excellent written and verbal communication skills.
  • Ability to work independently in a fast-paced environment and adapt to new technologies quickly.

 

**Nice-to-Have Skills**

  • Exposure to CI/CD pipelines for data solutions.
  • Knowledge of data governance and security best practices.
  • Experience with Agile/Scrum methodology.

  • Solution Architect for Large Scale Data Pipelines

    2 weeks ago

    Salem, Tamil Nadu, India beBeeData Full time ₹ 20,00,000 - ₹ 25,00,000

    Job Title: Solution Architect for Large Scale Data Pipelines

  • Large Scale Data Processing Expert

    7 days ago

    Salem, Tamil Nadu, India beBeeDataEngineering Full time ₹ 12,00,000 - ₹ 24,00,000

    Data Engineering RoleAt Incept Labs, we are dedicated to building domain-specific AI solutions to enhance education and research. We believe in the power of human-AI collaboration, where each partner strengthens the other.We are seeking a skilled Software Engineer with expertise in large-scale data processing to join our high-impact team responsible for both...

  • Large Scale Data Processing Specialist

    3 days ago

    Salem, Tamil Nadu, India beBeeDataEngineer Full time ₹ 20,00,000 - ₹ 25,00,000

    Our company is seeking a highly skilled Senior Data Engineer to drive the design and implementation of large-scale data processing systems on Google Cloud Platform (GCP). As a key member of our team, you will be responsible for creating efficient data pipelines using GCP services such as GCS, PubSub, Dataflow, and BigQuery.About the RoleThe ideal candidate...

  • Large Scale Business Intelligence Solutions Specialist

    7 days ago

    Salem, Tamil Nadu, India beBeeBusiness Full time ₹ 10,00,000 - ₹ 20,00,000

    Job Title: Business Intelligence DeveloperKey Responsibilities:Develop and maintain large-scale business intelligence applications.Experience with data visualization, ETL processes, and database administration.About the Role:This role requires a skilled professional to design and implement business intelligence solutions. The successful candidate will have...

  • Data Engineer

    5 days ago

    Salem, Tamil Nadu, India beBeeDataEngineer Full time ₹ 10,00,000 - ₹ 20,00,000

    Data Engineer RoleAs a Data Engineer, you will be responsible for designing and developing scalable data pipelines using Apache Spark and Azure services.Design and develop large-scale data processing systems to handle complex data flowsCollaborate with cross-functional teams to integrate data engineering workflows into business applicationsThe ideal...

  • Expertise in Large Scale Data Engineering

    2 weeks ago

    Salem, Tamil Nadu, India beBeeDataEngineer Full time ₹ 18,00,000 - ₹ 20,00,000

    Senior Data EngineerWe are seeking a highly skilled Senior Data Engineer to join our team. The ideal candidate will have 4–5 years of experience in designing and implementing high-quality, secure, and scalable data pipelines and systems that power reporting, analytics, and operational use cases.This is a remote-first opportunity with a strong preference...

  • HR Manager

    2 weeks ago

    Salem, Tamil Nadu, India beBeeOperations Full time ₹ 1,20,00,000 - ₹ 1,40,00,000

    Job Opportunity:We are seeking a highly skilled professional to oversee and streamline HR functions for a large-scale workforce.Key ResponsibilitiesManage HR operations processes across the employee life cycle, ensuring seamless integration of new hires.Oversee onboarding, induction, and employee ID creation, guaranteeing efficient workflows.Drive compliance...

  • Engineering Team Lead

    5 days ago

    Salem, Tamil Nadu, India beBeeSoftware Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Senior Software Engineer - Distributed SystemsWe are seeking a highly experienced Senior Software Engineer with 10+ years of experience in designing and developing large-scale distributed applications.Key Responsibilities:Design and develop scalable, fault-tolerant distributed systems using industry-standard technologies.Collaborate with cross-functional...

  • Transformative Big Data Solutions Architect

    5 days ago

    Salem, Tamil Nadu, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Job Opportunity for a Data EngineerWe are seeking a skilled data engineer with 3–4 years of experience in building, designing and maintaining large-scale data processing systems.The ideal candidate will bring expertise in PySpark, Python, SQL and real-time data streaming, contributing to the design and implementation of scalable data pipelines in cloud...

  • Data Engineering Specialist

    7 days ago

    Salem, Tamil Nadu, India beBeeDataEngineering Full time ₹ 15,00,000 - ₹ 25,00,000

    About This RoleAs a Data Engineering Specialist, you will design, develop, and maintain large-scale data systems to process and analyze massive datasets. You will work with distributed data processing and storage technologies like Apache Spark, Hadoop, and cloud platforms such as AWS/GCP/Azure.This role requires expertise in developing efficient ETL...