Highly Skilled Data Engineer

7 days ago

Bengaluru, Karnataka, India beBeeDataEngineer Full time US$ 1,20,000 - US$ 2,00,000
Job Description

This role is responsible for designing and implementing data engineering solutions that enable business transformation projects using technology, review of data to enable process, systems and tool re-engineering in customer support and services.

  • Develop, maintain, and optimize data pipelines and workflows using Databricks Job workflows and Feature Store to ensure seamless data ingestion and transformation as a scalable data solution.
  • Architect Lakehouse Solutions: Design, implement, and architect Lakehouse solutions on Databricks, leveraging Delta Lake, and Feature Store to enhance data storage and processing.
  • In-depth Databricks Expertise: Demonstrate a deep understanding of Databricks platform features, including Spark SQL, Delta Lake, Feature Store and Databricks notebooks, to optimise data engineering processes.
  • Data Transformation: Implement advanced data transformations and quality checks within the Lakehouse architecture to ensure data accuracy, completeness, and consistency.
  • Data Integration: Seamlessly integrate data from diverse sources, aligning with Lakehouse principles for data ingestion and storage, leveraging AWS S3 Storage and possibly Snowflake as a SQL Data Warehouse.
  • Performance Optimisation: Architect data pipelines for performance and scalability, making efficient use of Databricks clusters and AWS storage resources through S3 Life cycle.
  • Documentation: Create detailed documentation for data pipelines, data models, Lakehouse architecture configurations, AWS integrations, and data migration plans.
  • Troubleshooting: Proactively identify and resolve complex data pipeline and architecture issues, ensuring data integrity and availability, with a focus on Databricks/AWS monitoring and diagnostics.
Qualification and Desired Experiences:
  • 7+ years of data analysis and engineering experience. Bachelor's degree in Computer Science, Statistics, Informatics, Information Systems or another quantitative field.
  • Experience with big data tools: Hadoop, Spark, Kafka, Spark Kafka Streaming, Python.
  • Familiarity with Snowflake environment.
  • Advanced working SQL experience working with relational databases, query authoring (SQL) and working familiarity with a variety of databases.
  • Hands-on experience in Databricks for data pipeline development and AWS services like S3, Glue, EMR, EC2.
  • Experience building 'big data' data pipelines, architectures and data sets.
  • Working knowledge of API or Stream-based data extraction processes like Salesforce API and Bulk API and have hands-on experience in web crawling.
Personal Skills:
  • Ability to collaborate cross-functionally in a fast-paced environment and build sound working relationships within all levels of the organization.
  • Ability to handle sensitive information with keen attention to detail and accuracy.
  • Passion for data handling ethics.
  • Ability to solve complex, technical problems with creative solutions while anticipating stakeholder needs and providing assistance to meet or exceed expectations.
  • Proven ability to work independently with minimal supervision and prioritize tasks effectively to meet deadlines.
Benefits:

This role offers a unique opportunity to work with cutting-edge technology and be part of a dynamic team that is shaping the future of data engineering.

  • Highly Skilled Data Engineer Wanted

    1 week ago

    Bengaluru, Karnataka, India beBeeDataEngineer Full time US$ 8,00,000 - US$ 15,00,000

    About this Role:We are seeking a highly skilled Senior Data Engineer with strong expertise in data engineering, modeling, and visualization to lead our data initiatives.This role demands a hands-on technologist who can design, develop, and optimize large-scale data solutions ensuring high availability, performance, and scalability.The candidate will work...

  • Highly Skilled Data Engineer

    7 days ago

    Bengaluru, Karnataka, India beBeeDataEngineer Full time ₹ 5,00,000 - ₹ 8,00,000

    Data EngineerWe are seeking a highly skilled Data Engineer to join our team. As a key member of the team, you will be responsible for designing and implementing data pipelines in Azure Synapse to transform and load data into ADLS in Delta format.Our ideal candidate has hands-on experience with Azure Synapse, ADLS, Delta format, and metadata-driven data...

  • Highly Skilled Data Engineering Expert

    2 weeks ago

    Bengaluru, Karnataka, India beBeeDataEngineer Full time ₹ 18,00,000 - ₹ 30,00,000

    Senior Data Engineering ProfessionalWe are seeking a highly skilled Senior Data Engineering Professional to join our team. This individual will be responsible for managing and developing tax data pipelines, supporting our Tax Data Warehouse and Data Lake, and delivering business intelligence solutions.The successful candidate will lead data engineering...

  • Highly Skilled Data Professional

    2 weeks ago

    Bengaluru, Karnataka, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 20,00,000

    Job Title: Data EngineerJob DescriptionWe are seeking a highly skilled and motivated Data Engineer to join our organization.

  • Highly Skilled Data Engineering Professional

    2 weeks ago

    Bengaluru, Karnataka, India beBeeDataEngineering Full time ₹ 1,20,00,000 - ₹ 2,00,00,000

    Lead Data EngineerWe are seeking a highly skilled and innovative Data Engineer to join our team. As a key member, you will play a vital role in designing and developing cutting-edge data products that drive business growth.Key responsibilities include:Designing and implementing scalable data architecturesDeveloping efficient data pipelines using Apache...

  • Highly Skilled Data Expert

    1 week ago

    Bengaluru, Karnataka, India beBeeDataEngineer Full time US$ 1,20,000 - US$ 1,40,000

    Job SummaryWe are seeking a highly skilled Data Engineer to join our Global Business Analytics team. As a key member of this team, you will be responsible for designing, developing, and maintaining large-scale data systems.

  • Highly Skilled Data Engineer Wanted

    2 weeks ago

    Bengaluru, Karnataka, India beBeeDataEngineer Full time ₹ 20,00,000 - ₹ 40,00,000

    Data Engineering ProfessionalImpetus is seeking a highly skilled data engineer to design and implement robust and scalable data pipelines using cloud services. The ideal candidate will have 4-7 years of experience in data engineering and ETL/ELT processes, with expertise in designing and implementing solutions using pySpark and Spark SQL.Main...

  • Highly Skilled Data Engineer Wanted

    7 days ago

    Bengaluru, Karnataka, India beBeeDataEngineering Full time ₹ 15,00,000 - ₹ 28,00,000

    Job Description">Are you an experienced professional looking for a challenging role as a data engineer? We are seeking highly skilled individuals to join our data and AI delivery team. In this capacity, you will design and develop scalable data engineering solutions using Scala and Spark. Your strong technical skills in HDFS, Hive, Oozie, and Yarn will...

  • Highly Skilled Data Engineer Wanted

    2 weeks ago

    Bengaluru, Karnataka, India beBeeDataEngineer Full time ₹ 12,00,000 - ₹ 15,00,000

    Job OverviewWe are seeking a highly skilled Data Engineer to spearhead our team's data initiatives. The ideal candidate will have 4-6 years of experience with Big Data technologies, including pySpark, Hadoop, and Hive.Main ResponsibilitiesCreate efficient ETL pipelines using AWS services such as S3, IAM, Glue, EMR, and Redshift.Develop comprehensive...

  • Highly Skilled Data Professional

    1 week ago

    Bengaluru, Karnataka, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Azure Data EngineerWe are looking for a highly skilled data professional to join our team.Key Responsibilities:Design and develop scalable data pipelines on the Azure cloud platform.Work closely with cross-functional teams to ensure seamless integration of data engineering solutions.Maintain and document deliverables as needed.Participate in peer reviews for...