Data Engineer

4 weeks ago

Bangalore, Karnataka, India Scaling Theory Full time

Requirements :


Mandatory Skills :


Primary skills (Must Have) :


- Azure data factory, azure data lake, java, sql, python, and r.


- Deep knowledge of azure data lake, azure data factory, and other azure cloud services.

- Proficiency in programming and query languages like python, sql, java, u-sql, and r.

- Familiarity with data warehousing concepts, etl processes, and big data technologies.

- Skills in data modeling and database systems.

- Understanding data security and privacy best practices.

- Ability to optimize data queries and etl jobs for performance.

- Experience in devops for continuous integration and delivery.

- Excellent problem-solving and communication skills to explain complex data processes.

- Integration with source systems technical implementation of required pipelines for bronze, silver, gold, and diamond layers.

Roles and Responsibilities :


1. Data Collection and Integration : Data engineers collect data from various sources, including databases, APIs, external data providers, and streaming sources. They must design and implement efficient data pipelines to ensure a smooth flow of information into the data warehouse or storage system.


2. Data Storage and Management : Once the data is collected, data engineers are responsible for its storage and management. This involves choosing appropriate database systems, optimizing data schemas, and ensuring data quality and integrity. They also must consider scalability and performance to handle large volumes of data.


3. ETL (Extract, Transform, Load) : Processes ETL is a fundamental process in data engineering. Data engineers design ETL pipelines to transform raw data into a format suitable for analysis. This involves data cleansing, aggregation, and enrichment, ensuring the data is usable for data scientists and analysts.


4. Big Data Technologies : In today's data landscape, dealing with big data is the norm rather than the exception. Data engineers work with big data technologies such as Hadoop and Spark to efficiently process and analyze massive datasets.


5. NoSQL Databases : In addition to traditional relational databases, data engineers often work with NoSQL databases like MongoDB and Cassandra, which are well-suited for handling unstructured or semi-structured data.


6. Cloud Computing : Cloud platforms like AWS, Azure, and Google Cloud have become the backbone of modern data infrastructure. Data engineers leverage these platforms to build scalable and cost-effective data solutions.


7. Distributed Systems : Data engineering often involves distributed systems architecture to handle huge data volumes and ensure fault tolerance. Understanding how distributed systems work is essential for data engineers.


8. Streaming : Data Real-time data processing is crucial in many industries. Data engineers work with streaming technologies like Apache Kafka to handle and analyze data as it flows in.

(ref:hirist.tech)

  • Aptus Data Labs

    4 weeks ago

    Bangalore, Karnataka, India Aptus Data Labs Full time

    Aptus Data Labs is an innovative Data Engineering and Advanced Analytics company offering analytical solutions and consulting services.Role - Machine Learning EngineerExp- 3-5 YrsLocation - RemoteSkill - AWS Sagemaker , ML Pipeline, Machine Learning, AI, Python, NLP, , CI/CD , AWS Services - Lambda, Step functions and other AWS ServicesThe ideal...

  • Data Engineer

    4 weeks ago

    Bangalore, Karnataka, India Magna Hire Full time

    Responsibilities :Support the data engineering team in implementing analytics solutions, ensuring adherence to project timelines and quality standards.Participate in at least one full implementation lifecycle of dbt (data build tool), contributing to design, development, and deployment phases.Assist senior team members in optimizing dbt models and...

  • Aptus Data Labs

    4 weeks ago

    Bangalore, Karnataka, India Aptus Data Labs Full time

    Aptus Data Labs is a Bangalore based an innovative Data Engineering and Advanced Analytics company offering analytical solutions and consulting servicesRole : Fresher- Frontend : BangaloreJob Description :Qualifications :- Bachelor's degree in computer science, MIS, or related discipline required (2023/2024 passed out)- knowledge in developing modular...

  • Staff Engineer

    3 weeks ago

    Bangalore, Karnataka, India Prudential Manpower Pvt.lTD Full time

    We are urgently hiring for our client for the position of Staff Engineer - Data Engineering. Location : Anywhere in India.Np : max 30 days.What you'll do : As the Staff Engineer of the data engineering team, you will be entrusted with the delivery of scalable services that empower our customers to maximize return on brand outreach and recruitment. Staff...

  • Kafka Data Engineer

    2 weeks ago

    Bangalore, Karnataka, India Codersbrain technology pvt ltd Full time

    Mandatory skills : - Azure, Kafka, Data Engineering or any Data Platforms background- Experience with Azure/MS Data Platforms: Postgres Single and Flex, Azure SQL, MS SQL server, Kubernetes- Terraform (Preferred not Compulsory), Bicep or other IaC.- Helm (helm deployments as well writing charts).- Azure DevOps pipelines.- Engineer, Data Engineer, or similar...

  • Senior Data Engineer

    4 weeks ago

    Bangalore, Karnataka, India Squareroot Consulting Pvt Ltd Full time

    We are seeking a highly skilled and experienced Senior Data Engineer to join our team.Location : Bangalore (Work from Office all 5 days)Experience : 6+ YearsRole : Senior Data EngineerThe ideal candidate will have a strong background in big data technologies and extensive hands-on experience with Cloudera, Spark, Hive, HDFS, Flink, Data Lake, and...

  • Data Engineer

    4 weeks ago

    Bangalore, Karnataka, India SKEP IT SYSTEMS Full time

    Role : Data EngineerMobilution IT Systems is a pioneering organization at the forefront of technological innovation, specializing in cutting-edge Tech solutions. With a strong commitment to leveraging advanced technologies for scientific advancements, Mobilution IT Systems is dedicated to revolutionizing the way data is analysed and utilized across various...

  • Data Engineer

    4 weeks ago

    Bangalore, Karnataka, India CYBERSAPIENT TECHNOLOGIES PRIVATE LIMITED Full time

    Company Description : CST Cyber Sapient is a company dedicated to helping clients achieve advocacy through exceptional product and service experiences. We work closely with our clients to drive change within their organization and provide support in areas such as strategic vision, product innovation, user experience, talent support, and technical delivery....

  • Senior Data Engineer

    4 weeks ago

    Bangalore, Karnataka, India Recrud Full time

    Recrud is proud to partner with a dynamic SaaS startup to scout for a skilled Data Engineer, adept in crafting robust data pipelines.This role is integral to the client's team, demanding a candidate with a minimum of 5 years of dedicated experience in the field.Position Overview :The Data Engineer will be a crucial part of the startup's technical...

  • Azure Data Engineer

    4 weeks ago

    Bangalore, Karnataka, India Orange Shark Full time

    Position : Azure Data EngineerExperience : 6-10 YearsNotice Period : Immediate to 15 DaysLocation : Bangalore.Primary Skills : Azure, DevOps, Azure Data Factory (Min. 4 Years ), Azure SQL/MS SQL.Job Description :We are seeking a skilled Azure Data Engineer to join our dynamic team. The ideal candidate will have hands-on experience designing, implementing,...

  • Python Data Engineer

    3 weeks ago

    Bangalore, Karnataka, India Cyitechsearch Full time

    A Python Data Engineer job description typically includes the following responsibilities, qualifications, and skills:Job Description :Responsibilities :- Develop and maintain robust data pipelines to extract, transform, and load (ETL) data from various sources into data lakes, warehouses, or databases.- Design, build, and optimize data models and schemas to...

  • Senior Engineer

    3 weeks ago

    Bangalore, Karnataka, India Prudential Manpower Pvt.lTD Full time

    Immediately hiring for our client for the role of Senior Engineer Data & Analytics. Total Exp : 8+ years. Location: Anywhere in India. What you'll do:As the senior manager of the data engineering team, you will be entrusted with the delivery of scalable services that empower our customers to maximize return on brand outreach and recruitment.- Data...

  • Director - Data Engineering

    4 weeks ago

    Bangalore, Karnataka, India Ethos HR Management & Projects Pvt. Ltd Full time

    Job Description (Minimum 100 Characters) Your Responsibility :- Lead efforts in building and delivering the data strategy for data engineering. - Manage development and support of data fabric platform that caters to needs of real time events and batch data processing. - Manage end to end development of data products that include traditional RDBMS and big...

  • Senior Data Engineer

    4 weeks ago

    Bangalore, Karnataka, India Growel Softech Pvt. Ltd. Full time

    Job Title: Azure Data Engineer. Location: Bangalore, India. Experience Level: 4-8 years. Job Description:. We are looking for an experienced Azure Data Engineer to join our team in Bangalore. As an Azure Data Engineer, you will be responsible for designing, developing, and maintaining scalable data pipelines and solutions using Azure Data Factory,...

  • Software Engineer

    2 weeks ago

    Bangalore, Karnataka, India TalentXo Full time

    We are looking for a talented and motivated Software Engineer to join our growing Data Engineering & Analytics team. In this role, you will play a pivotal role in developing, maintaining, and improving our reporting products and data pipelines.Responsibilities :- Design and develop high-impact, user-facing features for our reporting products, leveraging your...

  • Senior Data Engineer

    4 weeks ago

    Bangalore, Karnataka, India Muoro Full time

    Job Description :About the role :We are looking for a Senior Data Engineer to join our growing engineering team. As a member of the team:- You will work on enterprise data platforms, architect and implement data lakes both on-prem and in the cloud.- You will be responsible for evolving technical architecture, design and implementation of data solutions using...

  • Senior Data Engineer

    4 weeks ago

    Bangalore, Karnataka, India Amicon Hub Full time

    Data Engineer role.Location : BangaloreNeed only Immediate Joiners and candidates from Product based/Start up Companies.Responsibilities :- Analyze and organize raw data - Build data systems and pipelines- Evaluate business needs and objectives- Interpret trends and patterns- Conduct complex data analysis and report on results - Prepare data for prescriptive...

  • Data Engineer

    3 weeks ago

    Bangalore, Karnataka, India Excellence and Eminence LLP Full time

    About the Role:We are seeking a talented and motivated Data Engineer to join our team and play a crucial role in building and maintaining our data infrastructure. This role is ideal for a candidate with a strong foundation in data engineering principles and experience working with data pipelines and Design, develop, and implement data pipelines for efficient...

  • Azure Data Engineer

    4 weeks ago

    Bangalore, Karnataka, India Connectio IT Pvt Ltd Full time

    Role Description :This is a full-time Azure Data Engineer role at Connect io IT Pvt Ltd located in Bengaluru. As an Azure Data Engineer, you will be responsible for tasks such as data engineering, data modeling, Extract Transform Load (ETL), data warehousing, and data analytics. You will work on-site in Bengaluru.Qualifications :- Data Engineering, Data...

  • Data Engineer

    4 weeks ago

    Bangalore, Karnataka, India Mobile Programming LLC Full time

    Job Description :Please note : This position is focused on development rather than migration. We are seeking an experienced Data Engineer to join our team. As a Data Engineer, you will play a crucial role in developing and maintaining our data infrastructure and ensuring the smooth operation of our data platforms. The ideal candidate should have a strong...