Senior Data Engineer

2 days ago

Hubli, Karnataka, India Deltacubes Full time

Job description:


• Develop, construct, test and maintain data acquisition pipelines for large volumes of structed and unstructured data. This includes batch and real-time processing (in google cloud)


• Build large and complex datasets based on business requirements


• Construct 'big data' pipeline architecture


• Identify opportunities for data acquisition via working with stakeholders and business clients


• Translate business needs to technical requirements


• Leverage a variety of tools in the Google Cloud Ecosystem such as Python, Data Flow, DataStream, CDC (Change Data Capture), Cloud Functions, Cloud Run, Pub Sub, BigQuery, Cloud Storage to integrate systems and data pipelines


• Use logs & alerts to effectively monitor pipelines


• Use SAP SLT to replicate SAP tables to Google Cloud using SLT


• Develop JSON messaging structure for integrating with various applications


• Leverage DevOps and CI/CD practices (GitHub, Terraform) to ensure the reliability and scalability of data pipelines


• Partition/Cluster and retrieve content in Big Query and use IAM roles & Policy Tags to secure the data


• Use roles to secure access to datasets, authorized views to share data between projects


• Design and build an ingestion pipeline using Rest API


• Recommends ways to improve data quality, reliability, and efficiency

Experience:


• Requires


• Junior DE: 2 to 3 years of experience in a GCP data engineering role (offshore)


• Senior DE: 4+ years of experience in a GCP data engineering role (offshore)


• Atleast 1 year of hands on experience in GCP tools highlighted(Python, SQL, Data Flow, DataStream, CDC (Change Data Capture), Cloud Functions, Cloud Run, Pub Sub, BigQuery, Cloud Storage)

  • Senior Data Engineer

    2 weeks ago

    Hubli, Karnataka, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,20,00,000

    As a senior data engineer, you will be responsible for designing and implementing scalable, secure, and efficient solutions on the Databricks platform.The ideal candidate will have experience working with Databricks, including platform architecture, data engineering, and data analytics. Proficiency in languages such as Python, Scala, or Java is also...

  • Senior Data Engineer

    4 days ago

    Hubli, Karnataka, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Job Title: Senior Data Engineer">Overview:">We are seeking a highly skilled and experienced data engineer to design and implement scalable IoT data solutions utilizing AWS services.">Key Responsibilities:">">Design, develop, and deploy large-scale IoT data pipelines using AWS services.">Write efficient, reusable, and maintainable Python code for data...

  • Senior Cloud Data Engineer

    5 hours ago

    Hubli, Karnataka, India beBeeCloudEngineer Full time ₹ 1,20,00,000 - ₹ 2,00,00,000

    Senior Data Engineer - Cloud InfrastructureWe are seeking a seasoned Senior Data Engineer to lead our cloud infrastructure initiatives on AWS. As a key member of our team, you will be responsible for designing, developing, and deploying end-to-end data pipelines using AWS services.This is an exceptional opportunity to leverage your technical expertise in...

  • Senior Data Specialist

    1 week ago

    Hubli, Karnataka, India beBeeData Full time US$ 1,00,000 - US$ 1,20,000

    Job Title:Key Responsibilities of a Senior Data SpecialistOverview:Design and implement data pipelines using Python and AWS services such as S3, Redshift, Lambda, and Glue.As a senior data specialist, you will be responsible for developing and maintaining large-scale data systems that support business analytics and reporting requirements.Requirements:7+...

  • Senior Data Engineer

    6 days ago

    Hubli, Karnataka, India beBeeDataEngineering Full time ₹ 18,00,000 - ₹ 22,00,000

    Job Title: Data Engineering Scrum MasterA senior data engineer with experience in scrum methodology is required to lead agile teams and ensure data quality.Responsibilities:Lead agile teams using scrum methodologyEnsure data integrity through documentation and processingCollaborate with cross-functional teams to drive business outcomesRequirements:6-8 years...

  • Senior Data Engineer

    4 days ago

    Hubli, Karnataka, India beBeeDataEngineer Full time ₹ 20,00,000 - ₹ 25,00,000

    Key Data Engineer PositionThis is a crucial data engineer role for an organization. The ideal candidate will have a deep understanding of data engineering principles and be able to apply them to real-world problems.The successful applicant will design and develop data pipelines using Python and cloud-based tools, implement data warehousing and ETL solutions,...

  • Data Engineer

    1 week ago

    Hubli, Karnataka, India Lenity Health Full time

    Role Overview Location: Hebbal, Bengaluru Work Mode: Monday to Friday | On-site Work Time: 2pm - 11pm Compensation: ₹15-20 LPA (negotiable based on experience) Experience Level: 3+ years About Lenity Health The healthcare industry operates on systems so outdated, that it clogs the flow of innovation and efficient care coordination. These...

  • Data Engineering Expertise Sought

    3 days ago

    Hubli, Karnataka, India beBeeDataEngineer Full time ₹ 17,50,000 - ₹ 25,75,000

    Senior Data Engineer OpportunityWe are seeking an experienced Senior Data Engineer to lead the development of our self-service data platform for reporting and analytics.Key Responsibilities:Design and implement scalable technologies such as Databricks, SQL, Python, Spark, and Delta Lake on Azure.Ensure adherence to best practices for Extract, Transform, Load...

  • Chief Data Engineering Architect

    4 days ago

    Hubli, Karnataka, India beBeeDataEngineering Full time ₹ 15,00,000 - ₹ 25,00,000

    Job OverviewWe are seeking a visionary Senior Data Engineering Lead to lead our data engineering team.This leadership role requires a technical expert who can design, develop, and manage scalable data infrastructure and pipelines, while mentoring and inspiring a team of data engineers.Key Responsibilities:Lead and manage a team of data engineers, providing...

  • Principal Data Engineering Position

    5 days ago

    Hubli, Karnataka, India beBeeDataEngineer Full time ₹ 20,00,000 - ₹ 25,00,000

    Senior Data Engineer OpportunityWe are seeking a highly skilled Senior Data Engineer to join our team. This is a senior-level position that requires expertise in designing and implementing data load processes using Azure Data Factory, Databricks, Delta Lake, Azure Delta Lake Storage, and Python/PySpark.The ideal candidate will have proficiency with...