Freelance Senior Databricks Data Engineer

4 weeks ago

Kottayam, Kerala, India ThreatXIntel Full time

Company Description

ThreatXIntel is a startup cyber security company that specializes in protecting businesses and organizations from cyber threats. Our team offers services in cloud security, web and mobile security testing, cloud security assessment, and DevSecOps. We provide customized and affordable solutions to meet the specific needs of our clients, focusing on continuous monitoring and testing to identify vulnerabilities proactively.

Role Description

We are looking for a highly experienced Freelance Senior Data Engineer with 8+ years of expertise in building, optimizing, and maintaining large-scale data pipelines using Databricks and Apache Spark. You will play a key role in architecting data solutions that drive business insights and support advanced analytics initiatives.

Key Responsibilities

  • Architect, develop, and maintain scalable and efficient ETL/ELT pipelines using Databricks and Spark
  • Lead optimization efforts for complex data workflows and Spark job performance tuning
  • Collaborate closely with data scientists, analysts, and architects to deliver high-quality data solutions
  • Design and implement robust data ingestion processes across diverse structured and unstructured data sources
  • Implement data governance, security, and compliance measures within data pipelines
  • Develop automation frameworks for workflow orchestration using Databricks jobs and/or tools like Airflow
  • Mentor junior engineers and provide technical guidance on best practices and architecture decisions
  • Support cloud migration, integration, and modernization projects involving Databricks on AWS, Azure, or GCP
  • Ensure high data quality, reliability, and observability across all pipelines

Required Skills & Experience

  • 8+ years of experience in Data Engineering or related roles
  • Proven expertise with Databricks and Apache Spark (PySpark, Scala, SQL) at an enterprise scale
  • Deep knowledge of cloud data platforms: AWS, Azure, or Google Cloud Platform
  • Extensive experience designing and optimizing large-scale ETL/ELT pipelines
  • Strong skills in SQL and Python scripting
  • Hands-on experience with data warehousing solutions such as Delta Lake, Snowflake, or Redshift
  • Familiarity with orchestration tools like Airflow, Prefect, or native Databricks workflows
  • Solid understanding of data modeling, data governance, and best practices for secure data management
  • Excellent problem-solving skills and ability to work independently in a remote environment
  • Strong communication and collaboration skills with cross-functional teams

Preferred Qualifications

  • Experience with streaming data platforms (Kafka, Kinesis) and real-time processing
  • Knowledge of CI/CD processes for data engineering pipelines
  • Exposure to machine learning workflows and MLOps on Databricks
  • Previous experience working in freelance or contract roles with global clients

  • Freelance Data Engineering Professional

    23 hours ago

    Kottayam, Kerala, India beBeeData Full time ₹ 12,00,000 - ₹ 15,00,000

    Job OverviewWe are seeking a skilled Freelance Data Engineer to join our team.The ideal candidate will have expertise in designing, building, and optimizing large-scale data pipelines using Python, SQL, PySpark, AWS, and Databricks.Key Responsibilities:Design and develop ETL/ELT pipelines using PySpark, AWS, and Databricks.Perform data profiling, validation,...

  • Senior Cloud Data Engineer

    2 days ago

    Kottayam, Kerala, India beBeeData Full time ₹ 15,00,000 - ₹ 25,00,000

    We are looking for a highly skilled senior data engineer to design and develop scalable and high-performance data pipelines using Databricks and PySpark.Key Responsibilities:Develop, implement, and maintain complex data solutions on AWS using services such as S3, Glue, Lambda, EMR, and Redshift.Use automated deployment and testing tools for data workflows to...

  • Senior Data Analyst

    19 hours ago

    Kottayam, Kerala, India beBeeData Full time ₹ 30,00,000 - ₹ 50,00,000

    Job DescriptionKey Responsibilities:We are seeking an ETL Tester with strong knowledge on data validation and experience with Databricks to join our organization.Data Validation Specialist: Design and execute test cases to validate Databricks tables across various layers.Test Automation Expert: Proficiency in validating data transformations, ingestion, and...

  • Data Engineering Lead

    23 hours ago

    Kottayam, Kerala, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Job Overview We are seeking an experienced data engineer to design and implement scalable, high-performance data pipelines using AWS cloud services and Databricks. Design and develop data pipelines and workflows using AWS and Databricks.Build and optimize ETL/ELT processes for large-scale structured and unstructured datasets.Leverage Databricks for data...

  • Senior Cloud Data Architect

    4 days ago

    Kottayam, Kerala, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    We are seeking an experienced Data Engineer to lead our team in delivering high-quality data solutions.The ideal candidate is a hands-on problem solver with deep expertise in Azure and Databricks.This senior technical role requires the ability to design, develop, and optimize data pipelines using Azure and Databricks.Key Responsibilities:Design and implement...

  • Master Data Engineering Specialist

    7 days ago

    Kottayam, Kerala, India beBeeDataEngineer Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Senior Data Engineer Job DescriptionWe are seeking a seasoned data professional to lead our Global Intelligence Practice.The ideal candidate will have strong foundational skills in data warehousing and engineering, with exposure to various data architecture types.The successful candidate will work closely with client stakeholders to capture requirements,...

  • Senior Data Solutions Architect

    1 day ago

    Kottayam, Kerala, India beBeeData Full time ₹ 1,20,00,000 - ₹ 2,00,00,000

    Job Opportunity: Senior Software Engineer/Technical SpecialistWe are seeking a highly skilled professional to develop and optimize ETL/ELT pipelines using Azure Data Factory (ADF), Databricks, and Synapse Analytics.Key Responsibilities:Design and implement scalable data architectures following Medallion Architecture principles.Perform data ingestion,...

  • Data Systems Architect

    2 days ago

    Kottayam, Kerala, India beBeeDatabricks Full time ₹ 1,00,00,000 - ₹ 2,00,00,000

    Senior Databricks DeveloperWe are seeking an accomplished Senior Databricks Developer to join our team. This role involves designing, developing, and maintaining large-scale data processing systems using Databricks PySpark.Key Responsibilities:Design and develop ETL processes using Databricks PySpark to extract, transform, and load data from heterogeneous...

  • Data Engineer

    4 weeks ago

    Kottayam, Kerala, India VicTree Solutions Full time

    VicTree Solutions is hiring for a Data Engineer for a contract role.Job Title: Data EngineerLocation: RemoteExperience: 5-9 YearsEmployment Type: Contract (6 months renewable)Budget: 10-12 LPAShift – Australian shiftPositions - 2Requirements:Google Cloud Platform experienceAzure Databricks experienceExperience in DBTExperience with secure coding...

  • Senior Data Scientist

    1 day ago

    Kottayam, Kerala, India beBeeData Full time ₹ 20,00,000 - ₹ 25,00,000

    Big Data Engineer PositionWe are seeking a skilled Big Data Engineer to lead the design, development, and maintenance of scalable data pipelines and ETL processes.About the RoleThe ideal candidate will have a strong background in data engineering and experience working with big data ecosystems. They will collaborate with stakeholders to gather requirements...