Data Engineer for Big Data Technologies

4 days ago


Hyderabad, Telangana, India SOULPAGE IT SOLUTIONS PRIVATE LIMITED Full time
About Us
Soulpage IT Solutions is a leading AI/ML technology company based in Hyderabad. We specialize in cutting-edge solutions for data engineering, AI, and cloud-based analytics.

The Role
We are looking for an experienced Data Engineer with expertise in Big Data technologies, Python, and PySpark to join our team. The ideal candidate will have strong hands-on experience in developing and optimizing PySpark applications using Spark DataFrames in Python. Additionally, the candidate should be able to work on large-scale data processing, ensuring performance optimization for high-volume data pipelines.

Key Responsibilities:
  • Develop and optimize PySpark applications using Spark DataFrames in Python.
  • Work on large-scale data processing, ensuring performance optimization for high-volume data pipelines.
  • Implement best practices for version control using Git.
  • Work with Amazon Analytics services such as Amazon EMR, Amazon Athena, and AWS Glue.
  • Utilize Amazon Compute services including AWS Lambda, EC2, and storage solutions like S3, SNS.
  • Work with columnar storage formats like Parquet, Avro, ORC, and apply compression techniques (Snappy, Gzip).
  • Good to have experience with data warehousing concepts (dimensions, facts, schemas - Star, Snowflake).
  • Good to have experience with AWS databases such as Aurora, RDS, Redshift, ElastiCache, or DynamoDB.

Requirements:
  • 4 years of experience in IT, with strong hands-on expertise in Big Data technologies.
  • Hands-on experience in Python and PySpark.
  • Experience working on AWS ecosystem, including EMR, Athena, Glue, Lambda, EC2, S3, and SNS.
  • Strong understanding of data modeling and warehousing concepts.
  • Proficiency in working with structured and semi-structured data.
  • Ability to optimize Spark jobs for better performance and efficiency.


  • Hyderabad, Telangana, India data Remote Work Freelance Full time

    What can you tell your friends when they ask you what you do? We're looking for an experienced Big Data Engineer who can create innovative new products in the analytics and data space. You will participate in the development that creates the world's #1 mobile app analytics service. Together with the team, you will build out new product features and...


  • Hyderabad, Telangana, India data Remote Work Freelance Full time

    What can you tell your friends when they ask you what you do? "As a Data Analyst at data.ai, I help analyze, monitor, and deliver comprehensive and high-quality app market data to app professionals. I am empowered to work independently with the support of team members spread across APAC, AMER, and EMEA. I am the key contributor to the new and existing...

  • Data Engineer

    4 days ago


    Hyderabad, Telangana, India Astrosoft technologies Full time

    Unlock the Power of Data with Astrosoft TechnologiesWe're seeking an experienced Data Engineer - Streaming and Big Data to join our team! As a key member of our data engineering team, you'll play a crucial role in designing, building, and maintaining our large-scale data processing systems.About the Role:Work closely with our data science team to develop and...


  • Hyderabad, Telangana, India Whopper Technologies Full time

    Job DescriptionAs a Senior Big Data Engineer Leader at Whopper Technologies, you will be responsible for leading the design and implementation of scalable data infrastructure on cloud platforms. This is an exciting opportunity to work with a talented team of engineers and technologists who are passionate about delivering innovative solutions that drive...

  • Big Data Engineer

    2 weeks ago


    Hyderabad, Telangana, India SMARTWORK IT SERVICES Full time

    Job Title : Big Data Engineer (Dynatrace)Location : Chennai, Bangalore, HyderabadExperience : 6-10 YearsRequired skills : Big Data, DynatraceJob Description : We are looking for an experienced Big Data Engineer (Dynatrace) with building scalable data pipelines and ensuring performance optimization through Dynatrace monitoring. This role requires a seasoned...

  • Senior Data Engineer

    3 weeks ago


    Hyderabad, Telangana, India Enable Data Incorporated Full time

    Enable Data Incorporated is seeking a highly skilled Senior Data Engineer with expertise in cloud technologies, Spark, and Databricks to join our dynamic team. As a leading provider of advanced application, data, and cloud engineering services, we leverage modern solutions to drive increased value across our clients' business ecosystem.As a Senior Data...


  • Hyderabad, Telangana, India Folks Palette LLP Full time

    Key ResponsibilitiesData PipelinesWe are looking for a skilled engineer to design, develop, and implement data pipelines for processing large datasets using PySpark. The ideal candidate will have a strong background in big data technologies, particularly PySpark, and a deep understanding of data processing, transformation, and optimization.Scalable Data...

  • Big Data Engineer

    1 day ago


    Hyderabad, Telangana, India Folks Palette LLP Full time

    Job Description : We are seeking an experienced Big Data Engineer with PySpark Developer to join our dynamic team. As a key member of our data engineering team, you will be responsible for designing, developing, and maintaining scalable data solutions in a big data ecosystem. The ideal candidate will have a strong background in big data technologies,...

  • Big Data Engineer

    3 days ago


    Hyderabad, Telangana, India Tek Ninjas Full time

    Job Title : Big Data Engineer Location : Hyderabad Work Mode : Hybrid Notice Period : 15 - 20 days of serving notice Job Description : We are looking for an experienced Big Data Engineer to design, build, and maintain scalable data ingestion pipelines and transformations at scale. In this role, you will work closely with Data Science and other data teams to...

  • Big Data Architect

    4 days ago


    Hyderabad, Telangana, India BlueByte Technologies Full time

    Job Description:At BlueByte Technologies, we are seeking a highly skilled Big Data Architect to join our team. As a key member of our data engineering group, you will be responsible for designing, developing, and maintaining large-scale distributed data processing systems using Apache Spark and Apache Hadoop ecosystem tools.Key Responsibilities:Design and...


  • Hyderabad, Telangana, India Resource Tree Global Services Full time

    Key responsibilities : - You will engage in a diverse range of impactful customer technical Big Data projects, including the development of reference architectures, how-to guides, and Minimal Viable Products (MVPs). - Lead strategic initiatives that encompass the end-to-end design, build, and deployment of industry-leading big data and AI applications. -...

  • AWS Big Data Engineer

    3 weeks ago


    Hyderabad, Telangana, India Tiger Analytics Full time

    Job Description We are looking for a Senior Data Engineer to be based out of our Chennai, Bangalore & Hyderabad offices. This role involves a combination of hands-on contribution, customer engagement, and technical team management. As a Senior Data Engineer, you will Design and build solutions for near real-time stream processing as well as batch...


  • Hyderabad, Telangana, India Resource Tree Global Services Full time

    We are seeking a highly skilled Big Data Technology Solutions Architect to join our team at Resource Tree Global Services. As a key member of our organization, you will be responsible for designing and implementing cutting-edge big data solutions that drive business growth and innovation.The ideal candidate will have extensive experience in big data...


  • Hyderabad, Telangana, India Enable Data Incorporated Full time

    Enable Data Incorporated is seeking a highly skilled Senior Data Engineer with expertise in cloud technologies, Spark, and Databricks to join our dynamic team. As a leading provider of advanced application, data, and cloud engineering services, we leverage modern solutions to drive increased value across our clients' business ecosystem.As a Senior Data...


  • Hyderabad, Telangana, India Virtusa Full time

    Big Data Python Engineer - CREQ191701 Description Role GDT WS Big Data Python Engineer Exp 5+ Location Pune/Hyderabad MUST Have Skills Design and implement data processing pipelines using Python + SCALA programming and big data technologies such as Hadoop, MapReduce, Hive, and Spark. Airflow Jenkins Understand business requirements to code up data assets...


  • Hyderabad, Telangana, India Programmers Full time

    Job Summary:We are seeking an experienced Big Data Engineer Lead to join our team. The successful candidate will have a strong background in designing and developing Big Data solutions, leveraging technologies such as Apache Spark and Data Bricks.Responsibilities:Lead the design and development of Big Data solutionsLeverage expertise in Big Data Platforms,...

  • Big Data Engineer

    4 days ago


    Hyderabad, Telangana, India Questsphere Private ltd. Full time

    Our client is looking to deploy big data engineer at one of the worlds top product company R&D centre based out of Hyderabad. As a Big Data Engineer, you are going to work with cross-functional collaboration and communication with members from other teams. As an Engineer, you are responsible to analyze large data sets to identify errors . You are responsible...

  • Big Data Engineer

    4 weeks ago


    Hyderabad, Telangana, India TechSophy Full time

    Job Description: Roles&Responsibilities : 1. As an Engineer, you are responsible to analyze large data sets to identify errors in the map. 2. As an Engineer, you are responsible to design and implement a complex algorithm for resolving the issue, review the solution with a team of engineers and analysts, and integrate the resulting solution into the...


  • Hyderabad, Telangana, India Bright Vision Technologies Full time

    **Job Summary**Bright Vision Technologies is a leading provider of innovative data solutions. We are seeking a highly motivated and experienced Senior Lead Data Engineer to join our team and contribute to the success of our data-driven projects.As a Senior Data Engineer at Bright Vision Technologies, you will be responsible for designing, developing, and...


  • Hyderabad, Telangana, India Tata Consultancy Services Full time

    At Tata Consultancy Services, we are seeking an experienced Azure Data Engineer to join our team. As a key member of our data engineering group, you will be responsible for designing, building, and maintaining large-scale data architectures on the Microsoft Azure platform.The ideal candidate will have a strong background in data engineering, with experience...