Data Engineer

3 weeks ago


Hyderabad, Telangana, India Risk Resources LLP Full time
Job Title: Data Engineer - Pyspark Databricks Specialist

About the Role

Risk Resources LLP is seeking a skilled Data Engineer to join our team. As a Data Engineer - Pyspark Databricks Specialist, you will be responsible for developing and implementing data processing pipelines using Pyspark on the Databricks platform.



Key Responsibilities

  • Design and Develop Data Processing Pipelines: Utilize Pyspark and Databricks to design and develop efficient data processing pipelines that ensure scalable data processing and analysis.
  • Optimize Spark Jobs and Clusters: Optimize and tune the performance of Spark jobs and clusters to ensure efficient data processing.
  • Implement Data Transformations and Aggregations: Implement data transformations and aggregations for analysis and reporting.
  • Collaborate with Cross-Functional Teams: Collaborate with data engineers and data scientists to integrate data from various sources into the pipelines.
  • Maintain ETL Processes: Develop and maintain ETL processes to extract, transform, and load data into the data lake.
  • Monitor and Troubleshoot Data Processing Jobs: Monitor and troubleshoot data processing jobs to ensure reliability and stability.
  • Provide Technical Guidance: Conduct code reviews and provide technical guidance to junior team members.
  • Implement Best Practices: Implement best practices for data pipeline development and documentation.


Requirements

  • Bachelor's Degree: Bachelor's degree in Computer Science, Engineering, or a related field.
  • Proven Experience: Proven experience in developing data processing pipelines using Pyspark and Databricks.
  • Python Proficiency: Strong proficiency in Python programming language.
  • Big Data Technologies: Hands-on experience with Big Data technologies and distributed computing.
  • Expertise in Optimizing Spark Jobs: Expertise in optimizing Spark jobs and clusters for performance.
  • Knowledge of ETL Processes: Knowledge of ETL processes and data warehousing concepts.
  • Experience with Data Integration and Modeling: Experience with data integration and data modeling.
  • Ability to Troubleshoot Complex Issues: Ability to troubleshoot and debug complex data processing issues.
  • SQL and NoSQL Databases: Excellent understanding of SQL and NoSQL databases.
  • Strong Communication Skills: Strong communication and collaboration skills to work effectively in a team environment.
  • Certifications: Certifications in Pyspark Databricks or related technologies are a plus.
  • Cloud Platforms: Familiarity with cloud platforms such as AWS, Azure, or Google Cloud is beneficial.


Data Engineer - Pyspark Databricks Specialist, Risk Resources LLP, Pyspark, Databricks, Data Engineer


  • Big Data Engineer

    3 weeks ago


    Hyderabad, Telangana, India data Full time

    Overview: We're seeking a skilled Big Data Engineer to join our team at data.ai. As a key member of our engineering team, you will be responsible for designing and implementing innovative data solutions that drive business growth. If you're passionate about working with large data sets, have experience with big data technologies, and are eager to...


  • Hyderabad, Telangana, India data Full time

    Unlock the Power of Data with UsWe're seeking an experienced Big Data Engineer to join our team and contribute to the development of innovative analytics and data products. As a key member of our engineering team, you will work closely with Data Scientists, Data Engineers, Product Managers, and Software Architects to design and implement complex data product...


  • Hyderabad, Telangana, India data Full time

    About the RoleWe are seeking an experienced Big Data Engineer to join our team at data.ai. As a key member of our engineering team, you will be responsible for designing and implementing complex data product components based on requirements with possible technical solutions.Key ResponsibilitiesDesign and implement data processing programs using Python (e.g.,...

  • Big Data Engineer

    1 month ago


    Hyderabad, Telangana, India data Full time

    About the RoleWe are seeking an experienced Big Data Engineer to join our team in Hyderabad, India. As a key member of our engineering team, you will be responsible for designing and implementing complex data product components based on requirements with possible technical solutions.Key ResponsibilitiesDesign and implement data processing programs using...

  • Data Engineer

    1 month ago


    Hyderabad, Telangana, India NTT DATA Services Full time

    Job Title: Data EngineerNTT DATA Services is seeking a highly skilled Data Engineer to join our team. As a Data Engineer, you will be responsible for designing, building, and testing scalable Big Data ecosystems for our clients.Key Responsibilities:Analyze and understand data sources and APIsDesign and develop methods to connect and collect data from...

  • Big Data Engineer

    2 weeks ago


    Hyderabad, Telangana, India NTT DATA Services Full time

    Data Engineer RoleData engineers at NTT DATA Services are responsible for building and testing scalable Big Data ecosystems for our clients. They work closely with data scientists to ensure the source data is aggregated and cleansed, and with product managers to understand the business objectives. Data engineers must have expertise in all phases of data...

  • Data Engineer

    4 weeks ago


    Hyderabad, Telangana, India NTT DATA Services Full time

    Job Title: Data Engineer - Big Data SpecialistAt NTT DATA Services, we are seeking a skilled Data Engineer to join our team. As a Data Engineer, you will be responsible for designing and developing scalable Big Data ecosystems for our clients. Your expertise will help us build stable and highly optimized data systems, ensuring seamless data processing and...


  • Hyderabad, Telangana, India NTT DATA Full time

    Job Title: Data EngineerJob Summary:We are seeking a skilled Data Engineer to design, build, and maintain large-scale data systems. The ideal candidate will have expertise in big data ecosystems, data processing, and data storage.Key Responsibilities:Design and develop scalable data systems for business applications.Work closely with data scientists to...

  • Data Analyst Position

    3 weeks ago


    Hyderabad, Telangana, India data Full time

    A Career Defining Role in Data IntelligenceAs a Data Analyst, you will play a pivotal role in helping us uncover hidden insights and trends within our vast dataset. Your expertise will be instrumental in ensuring the accuracy and quality of our data, enabling us to make informed decisions that drive business growth.ResponsibilitiesTake ownership of data.ai's...


  • Hyderabad, Telangana, India data Full time

    Data Quality and InsightsWe are seeking a skilled Data Insights Analyst to join our team at data.ai. As a Data Insights Analyst, you will be responsible for analyzing, monitoring, and delivering comprehensive and high-quality app market data to app professionals.Key Responsibilities:Take full responsibility for data.ai core intelligence data qualityDeep...


  • Hyderabad, Telangana, India data Full time

    About the RoleWe are seeking a highly skilled Data Intelligence Specialist to join our team at data.ai. As a Data Intelligence Specialist, you will be responsible for analyzing, monitoring, and delivering comprehensive and high-quality app market data to app professionals.Key ResponsibilitiesTake full responsibility for data.ai core intelligence data...

  • Data Modeling Expert

    3 weeks ago


    Hyderabad, Telangana, India data Full time

    What drives innovation in data-driven products?We are looking for an experienced data engineer to join our team and contribute to the development of innovative analytics and data products. As a big data engineer, you will be part of a collaborative team that uses agile methodologies and open-source technologies to build cutting-edge data processing programs....


  • Hyderabad, Telangana, India Statistics & Data Corporation (SDC) Full time

    Data Engineering SpecialistStatistics & Data Corporation (SDC), a specialized contract research organization (CRO), is seeking a skilled Data Engineering Specialist to join our team. With a strong focus on clinical trials, we provide top-tier services to pharmaceutical, biologic, and medical device/diagnostic companies.Job SummaryAs a Data Engineering...

  • Data Scientist

    3 weeks ago


    Hyderabad, Telangana, India data Full time

    About the RoleWe are seeking a highly skilled Data Analyst to join our engineering team at data.ai. As a Data Analyst, you will play a vital role in delivering high-quality app market data to app professionals.The successful candidate will be responsible for data profiling, data cleansing, data quality monitoring, and root cause analysis. You will work...

  • Senior Data Engineer

    4 weeks ago


    Hyderabad, Telangana, India NTT DATA Business Solutions Full time

    A Senior Data Engineer plays a pivotal role in technology-driven organizations, responsible for designing, building, and managing the infrastructure and tools that enable efficient processing and analysis of large data sets.Responsibilities:Designing and Implementing ETL Processes:Develop and maintain scalable data pipelines.Create efficient Extract,...


  • Hyderabad, Telangana, India NTT DATA Full time

    Job RoleData engineers play a vital role in building and testing scalable Big Data ecosystems for businesses. They design and develop methods to connect and collect data from various data sources, working closely with data scientists to ensure data quality and with product managers to understand business objectives.Key Responsibilities• Analyze and...

  • SAP Data Engineer

    2 weeks ago


    Hyderabad, Telangana, India NTT DATA Services Full time

    About the RoleWe are seeking an experienced SAP Data Engineer to join our team in Hyderabad, India. The successful candidate will play a crucial role in designing and implementing scalable data systems and pipelines.Key ResponsibilitiesBuild a deep understanding of client needs to identify the best data solution across various industriesCollaborate with data...

  • Data Scientist

    4 weeks ago


    Hyderabad, Telangana, India NTT DATA Full time

    Job Title: Data ScientistJob Summary:We are seeking a highly skilled Data Scientist to join our team in Hyderabad, Telangana, India. The successful candidate will be responsible for creating and maintaining test deliverables and artifacts to test complex applications and system integrations for Finance applications.Key Responsibilities:Programming Languages:...

  • Data Scientist

    4 weeks ago


    Hyderabad, Telangana, India NTT DATA Full time

    Job DescriptionReq ID: 276502NTT DATA Services is a recognized leader in IT and business services, including cloud, data, and applications. We are seeking a Data Scientist to join our team in Hyderabad, Telangana, India.Data Scientist ResponsibilitiesProgramming Languages (Python, R), Data Manipulation and Analysis Tools, Machine Learning Libraries, Data...

  • Senior Data Engineer

    3 weeks ago


    Hyderabad, Telangana, India The Modern Data Company Full time

    Job Description :The Senior Data Engineer will provide technical expertise in needs identification, data modeling, data movement, and transformation mapping (source to target), automation and testing strategies, translating business needs into technical solutions with adherence to established data guidelines and approaches from a business unit or project...