Freelance Senior Databricks Data Engineer

1 week ago

Jaipur, Rajasthan, India ThreatXIntel Full time

Company Description

ThreatXIntel is a startup cyber security company that specializes in protecting businesses and organizations from cyber threats. Our team offers services in cloud security, web and mobile security testing, cloud security assessment, and DevSecOps. We provide customized and affordable solutions to meet the specific needs of our clients, focusing on continuous monitoring and testing to identify vulnerabilities proactively.

Role Description

We are looking for a highly experienced Freelance Senior Data Engineer with 8+ years of expertise in building, optimizing, and maintaining large-scale data pipelines using Databricks and Apache Spark. You will play a key role in architecting data solutions that drive business insights and support advanced analytics initiatives.

Key Responsibilities

  • Architect, develop, and maintain scalable and efficient ETL/ELT pipelines using Databricks and Spark
  • Lead optimization efforts for complex data workflows and Spark job performance tuning
  • Collaborate closely with data scientists, analysts, and architects to deliver high-quality data solutions
  • Design and implement robust data ingestion processes across diverse structured and unstructured data sources
  • Implement data governance, security, and compliance measures within data pipelines
  • Develop automation frameworks for workflow orchestration using Databricks jobs and/or tools like Airflow
  • Mentor junior engineers and provide technical guidance on best practices and architecture decisions
  • Support cloud migration, integration, and modernization projects involving Databricks on AWS, Azure, or GCP
  • Ensure high data quality, reliability, and observability across all pipelines

Required Skills & Experience

  • 8+ years of experience in Data Engineering or related roles
  • Proven expertise with Databricks and Apache Spark (PySpark, Scala, SQL) at an enterprise scale
  • Deep knowledge of cloud data platforms: AWS, Azure, or Google Cloud Platform
  • Extensive experience designing and optimizing large-scale ETL/ELT pipelines
  • Strong skills in SQL and Python scripting
  • Hands-on experience with data warehousing solutions such as Delta Lake, Snowflake, or Redshift
  • Familiarity with orchestration tools like Airflow, Prefect, or native Databricks workflows
  • Solid understanding of data modeling, data governance, and best practices for secure data management
  • Excellent problem-solving skills and ability to work independently in a remote environment
  • Strong communication and collaboration skills with cross-functional teams

Preferred Qualifications

  • Experience with streaming data platforms (Kafka, Kinesis) and real-time processing
  • Knowledge of CI/CD processes for data engineering pipelines
  • Exposure to machine learning workflows and MLOps on Databricks
  • Previous experience working in freelance or contract roles with global clients

  • Senior Data Solutions Engineer

    1 day ago

    Jaipur, Rajasthan, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Senior Data Solutions EngineerJob Description We are seeking an experienced Senior Data Solutions Engineer to lead the architecture and implementation of scalable, secure, and high-performance data solutions on Databricks. The ideal candidate will have a strong background in data engineering, with expertise in designing, developing, and maintaining data...

  • Databricks Solutions Architect

    3 days ago

    Jaipur, Rajasthan, India beBeeDatabricks Full time US$ 1,20,000 - US$ 2,00,000

    Job SummaryWe are seeking a highly skilled Databricks Engineer to design, develop and maintain data pipelines and ETL processes on Databricks.About the Role:The ideal candidate will have 7+ years of experience in Data Engineering and proven subject matter expertise in Databricks (including Databricks SQL, Delta Lake and MLflow).They will be responsible for...

  • Chief Data Architect

    1 day ago

    Jaipur, Rajasthan, India beBeeLead Full time ₹ 30,00,000 - ₹ 40,00,000

    Job Title: Lead Azure Data EngineerImmediate hiring for a highly skilled and experienced lead data engineer to join our team.We're looking for a senior technical expert with deep hands-on expertise in Azure and Databricks. This is a critical role with high expectations. The consultant must be extremely hands-on, technically strong in Azure and Databricks,...

  • GCP Data Engineer

    4 weeks ago

    Jaipur, Rajasthan, India Ciphercru Innovation Pvt Ltd Full time

    Job Title: GCP Data Engineer Location: Remote ( Contractual) Shift Timing: Australian ShiftExperience Required: 5 to 9 Years Job Description:We are looking for a highly skilled and motivated GCP Data Engineer to join our team immediately. This is a remote opportunity working in the Australian time zone, supporting one of our key clients in the retail...

  • Chief Data Strategist

    1 day ago

    Jaipur, Rajasthan, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Unlock Business PotentialOur team is seeking an experienced Data Architect to join our ranks. This role involves designing and implementing scalable, secure, and efficient data solutions using Databricks.Key Responsibilities Include:Developing data pipelines, architecting data lakes, and implementing data warehousing solutions using Databricks.Collaborating...

  • Data Engineer

    3 days ago

    Jaipur, Rajasthan, India Celebal Technologies Full time

    Job Role – Data EngineerJob Location – JaipurJob Type – PermanentExperience Required YearsAs a Data Engineer, you will play a critical role in designing, developing, and maintaining our data pipelines and infrastructure. You will work closely with our data scientists, analysts, and other stakeholders to ensure data is accurate, timely, and accessible....

  • Data Engineer

    4 days ago

    Jaipur, Rajasthan, India Celebal Technologies Full time

    Job Role – Data Engineer Job Location – Jaipur Job Type – Permanent Experience Required- (2-4) Years As a Data Engineer, you will play a critical role in designing, developing, and maintaining our data pipelines and infrastructure. You will work closely with our data scientists, analysts, and other stakeholders to ensure data is accurate, timely,...

  • Data Ingestion Engineer

    1 day ago

    Jaipur, Rajasthan, India beBeeEngineer Full time ₹ 1,00,00,000 - ₹ 1,50,00,000

    We are seeking an experienced professional to lead the development of data ingestion pipelines into Databricks.This role requires a strong understanding of data integration processes, focusing on integrating various data sources into Databricks. We are looking for someone with hands-on experience in designing and implementing efficient data workflows.The...

  • Senior Data Platform Architect

    23 hours ago

    Jaipur, Rajasthan, India beBeeDataArchitect Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Seeking a seasoned professional to lead data platform architecture and implementation.We require an individual with 10+ years of experience in data engineering and architecture roles, boasting expertise in building Data Lakes, Data Warehouses, Lakehouses, and Data Marts.The ideal candidate will possess strong knowledge of data modeling, database design, and...

  • Visionary Data Engineer

    5 days ago

    Jaipur, Rajasthan, India beBeeDataEngineer Full time ₹ 10,00,000 - ₹ 20,00,000

    Data Engineers are key players in shaping our clients' data strategies and driving innovation. The role involves consulting clients on cloud-first strategies for core business data initiatives, providing thought leadership on both process and technical matters.Strong knowledge of securing Azure environment, such as RBAC, Key Vault, and Azure Security Center...