Senior Big Data Specialist

3 days ago

Guntur, Andhra Pradesh, India beBeeDataEngineer Full time ₹ 80,00,000 - ₹ 1,50,00,000

We are seeking a skilled Data Engineer to design and implement data pipelines, automate deployments, and work with modern DevOps tools.

About the Role
  • CICD Pipeline Design: Implement CI/CD pipelines using GitHub Actions or equivalent tools, enabling seamless integration of code changes into production environments.
  • Deployment Management: Configure and maintain GitHub branch protection rules and workflows for smooth code deployment and minimize errors.
  • Spark Job Optimization: Build, optimize, and debug Spark jobs in Databricks for large-scale data processing, ensuring high performance and reliability.
  • Automation and Deployment: Automate Databricks deployments using Databricks Asset Bundles (DABs) or similar frameworks, streamlining the deployment process and reducing manual effort.
  • Data Pipeline Development: Develop, test, and maintain robust data pipelines using Python and SQL, providing accurate and timely insights to support business decision-making.
  • Quality Assurance: Implement unit testing, validation, and quality checks for data workflows, ensuring data accuracy and consistency.
  • Multi-Environment Deployments: Manage multi-environment deployments (Dev, Stage, Prod) ensuring reliability and consistency across all environments.
  • Configuration Management: Write and maintain configuration files (YAML, JSON) for workflows, pipelines, and automation, ensuring standardized and consistent configurations.
  • Troubleshooting: Troubleshoot and resolve configuration or syntax issues in YAML and related files, minimizing downtime and ensuring rapid resolution.
  • Business Continuity: Ensure critical Databricks notebooks run reliably, maintaining business continuity and minimizing disruptions.
  • Data Migration: Migrate legacy reports into modern, self-serve dashboards (Snowflake, Tableau, Power BI), improving user experience and accessibility.
  • Collaboration: Partner with cross-functional teams to understand needs and deliver data models, insights, and reporting they can trust.
  • Reliability: Establish guardrails with data contracts, quality checks, and documentation to ensure long-term reliability and data integrity.
Required Skills and Qualifications
  • 3+ years of experience in data engineering, with expertise in designing and implementing data pipelines and automating deployments.
  • Strong understanding of CI/CD pipelines, including GitHub Actions and equivalent tools.
  • Familiarity with Spark job optimization and automation using Databricks.
  • Experience with data pipeline development using Python and SQL, including unit testing and validation.
  • Excellent problem-solving skills, with ability to troubleshoot configuration or syntax issues in YAML and related files.
  • Strong collaboration and communication skills, with ability to partner with cross-functional teams.
Benefits
  • Opportunity to work on cutting-edge data engineering projects and contribute to the development of innovative solutions.
  • Collaborative and dynamic work environment, with opportunities for professional growth and development.
  • Competitive salary and benefits package, including comprehensive health insurance and retirement plan.
Others
  • Why choose us? Our company values innovation, collaboration, and customer satisfaction. We offer a unique opportunity to join a team of talented professionals who share your passion for data engineering and analytics.

  • Big Data Specialist

    2 days ago

    Guntur, Andhra Pradesh, India beBeeBigData Full time ₹ 15,00,000 - ₹ 20,00,000

    Job Posting for Big Data SpecialistGreetings from our organization.We are seeking a seasoned Big Data Engineer to join our team. As a Big Data Specialist, you will be responsible for designing, developing, and implementing large-scale data processing systems using technologies such as PySpark and Hive.The ideal candidate should have at least 6 years of...

  • Senior Big Data Processing Specialist

    7 days ago

    Guntur, Andhra Pradesh, India beBeeData Full time ₹ 25,00,000 - ₹ 30,00,000

    We are seeking a highly skilled professional to join our team as a Senior Big Data Processing Specialist. The ideal candidate will have hands-on experience designing, building, and optimizing scalable data pipelines using Azure technologies.Key ResponsibilitiesDesign and develop efficient ETL workflows for batch data ingestion and egress using Azure Data...

  • Big Data Architect

    3 days ago

    Guntur, Andhra Pradesh, India beBeeData Full time ₹ 15,00,000 - ₹ 20,00,000

    Job Title: Big Data Architect We are seeking a skilled Big Data Architect to join our team. The ideal candidate will have experience with designing and implementing large-scale data systems. Key Responsibilities:Design and implement big data solutions using Apache Druid, Kafka, Hadoop, Spark, and Flink.Develop and maintain scalable data pipelines that meet...

  • Big Data Professional

    3 days ago

    Guntur, Andhra Pradesh, India beBeeDataScience Full time ₹ 20,00,000 - ₹ 25,00,000

    Greetings from our Data Science teamKey Responsibilities:Leading big data projects with diverse skill sets.Requirements:Expertise in PySpark, Hive and Spark.Familiarity with HBase, DQ tool, Agile Scrum experience.Benefits:Exposure in data ingestion from disparate sources onto big data platform.Preferred Qualifications:5+ years of experience in data science...

  • Chief Big Data Architect

    3 days ago

    Guntur, Andhra Pradesh, India beBeeDataEngineer Full time US$ 1,80,000 - US$ 2,00,000

    Job Role: Senior Data EngineerOverview:We are seeking a skilled Data Engineer to design, build and maintain high-performance data pipelines that process large amounts of data. The ideal candidate will have strong expertise in big data technologies and experience working with data processing systems.Key Responsibilities:Data Pipeline Design: Develop scalable...

  • Big Data Professional

    2 days ago

    Guntur, Andhra Pradesh, India beBeeData Full time ₹ 1,00,00,000 - ₹ 2,00,00,000

    Key Challenges in Data Engineering">Our team is searching for a highly skilled Data Engineer with expertise in designing and building robust data pipelines. If you are passionate about handling high-volume financial data, we encourage you to apply.">Responsibilities:">Develop and Manage Data Pipelines: Design, build, and maintain end-to-end data pipelines...

  • Senior Data Systems Architect

    5 days ago

    Guntur, Andhra Pradesh, India beBeeDataEngineer Full time ₹ 1,00,00,000 - ₹ 2,00,00,000

    Job Description:As a Senior Data Engineer, you will be responsible for designing and implementing large-scale data processing systems using Python and Pyspark.The ideal candidate will have hands-on coding expertise in Python and Pyspark, as well as strong experience with Telemetry, Big Data, and backend engineering.Domain knowledge of IoT systems,...

  • Data Architect

    3 days ago

    Guntur, Andhra Pradesh, India beBeeSoftwareEngineer Full time ₹ 18,49,298 - ₹ 25,13,056

    Job Overview:As a Senior Software Engineer/Technical Specialist, you will play a pivotal role in designing and implementing scalable data architectures following Medallion Architecture and best practices.Develop ETL/ELT pipelines using Azure Data Factory (ADF), Databricks, and Synapse Analytics to streamline data processing.Design and implement robust data...

  • Engineering Manager – Big Data, Azure

    5 days ago

    Guntur, Andhra Pradesh, India DataE2E Technologies Full time

    Technical Manager – AzureLocation: 100% RemoteEmployment Type: Full-timeShift Timing : 3 pm – 11 pm ISTJob Summary:About the Role:We are seeking a highly experienced and strategic Engineering Manager with deep expertise in Big Data platforms, Microsoft Azure, and Databricks. The ideal candidate will lead a team of talented data engineers to design and...

  • Senior Data Specialist

    2 days ago

    Guntur, Andhra Pradesh, India beBeeData Full time ₹ 1,00,00,000 - ₹ 1,50,00,000

    Senior Data SpecialistThis is a challenging opportunity to work as a skilled data specialist in an industry leading organization.We are seeking a detail-oriented individual with a strong technical background to drive the development and enhancement of our SQL programs.Key Responsibilities:Design, develop, stabilize, and enhance complex SQL programs to meet...