Draup - Big Data Engineer - Spark/Python

2 months ago

Bangalore, India Draup Full time

Role : Big Data Engineer

Job Description :

The Big Data Engineer at Draup is responsible for building scalable techniques and processes for data storage, transformations, and analysis. The role includes designing, and implementation of the optimal, generic, and reusable data-platforms and pipelines. You will work with a very proficient, inquisitive, enthusiastic, and experienced team of developers, engineers, and product team. You will have ability to shape up the data products.

What You Will Do :

- Build scalable architectures for data storage, transformations, and analysis.

- Work on data engineering projects to ensure pipelines are reliable, efficient, testable, & maintainable.

- Design and develop solutions which are scalable, generic, and reusable.

- Build and execute data warehousing, data mining and data modelling activities using agile development techniques.

- Leading and taking ownership of big data projects successfully from scratch to production.

- Bring the problem-solving attitude with focus on understanding the basics and implementing the use cases of organizational impact.

- Collaborate with various teams including data science, backend, data harvesting and product teams.

What You'll Need :

- Proficient understanding of big data and distributed systems principles.

- Must have good programming experience in Python.

- Proficiency in Apache Spark (PySpark) is a must.

- Good work experience in developing ETL and ELT solutions in a scalable way from multiple data sources.

- Understanding in technologies like Relational and NoSQL datastores.

- Working and conceptual Knowledge of MapReduce, HDFS, Amazon S3.

- Ability to code and think in functional programming paradigm.

- Enthusiastic about optimizing the code performance and resources of the system.

- Ability to communicate complex technical concepts to both technical and non-technical audiences.

- Takes ownership of all technical aspects of software development for assigned projects.

What Will Give You an Advantage :

- Expertise in big data infrastructure, distributed systems, data modelling, query processing and relational.

- Involved in the design of big data solutions with Spark/HDFS/MapReduce/Flink.

- Worked with different types of file-storage formats like Parquet, ORC, Avro, Sequence files etc.

- Experience and understanding of cluster managers like YARN, Spark Standalone, Mesos or Kubernetes, etc.

- Strong knowledge of data structures and algorithms.

- Understands how to apply technologies to solve big data problems and to develop innovative big data solutions.

- Someone with problem solving mind-set with good design and architectural patterns will be preferred.

- Experience in working with AWS tools like EMR, Lambda, Glue or equivalent tools on other cloud systems.

- Knowledge of workflow orchestration tools like Airflow, Jenkins.

Who You Are :

- B.E / B.Tech / M.E / M.Tech / M.S in Computer Science or software engineering.

- Experience of 2-6 Years working with Big Data technologies.

- Open to embrace the challenge of dealing with terabytes and petabytes of data on daily basis. If you can think out of the box have good code discipline, then you fit right in

(ref:hirist.tech)

  • Draup - Python Developer - Backend Architecture

    2 months ago

    Bangalore, India Draup Full time

    Designation : Python Developer Work Experience : 2-5 YearsEducational Qualification : B.Tech/M.Tech/MSPrimary Responsibilities : - Leverage your expertise in Python to design and implement distributed systems for web scraping, ensuring robust data extraction from diverse web sources.- Develop and optimize scalable, self-healing scraping frameworks,...

  • Big data developer

    2 weeks ago

    Bangalore, India Recro Full time

    Job Title: Data Engineer – Python, Py Spark, Big Data Location: (Onsite) Experience: (5+ years) Job Description: Seeking an experienced Data Engineer with expertise in Python, Py Spark, and Big Data technologies to design and build scalable data solutions on the Google Cloud Platform (GCP). Responsibilities: Develop/optimize data...

  • Cloud big data engineer

    4 weeks ago

    Bangalore, India Lakshya Software Technologies Pvt. Ltd. Full time

    Title: Cloud Big Data Engineer Exp - 5+ years Location: Anywhere in India Work Mode: HYBRID Mandatory Skills required: Big Data (Hadoop, Hive), Python, Spark, Py Spark, Cloud Platform, Azure Job description: Understanding of Big data Ecosystem with hands on experience working on Apache Spark, Hadoop (HDFS , Hive), Parquet, Elastic Search ...

  • Big data developer

    2 weeks ago

    Bangalore, India Recro Full time

    Job Title : Big Data Engineer (Python & Py Spark) Location : Bangalore Experience : 5+ years Key Responsibilities : Develop scalable data pipelines using Python and Py Spark . Build and optimize Apache Spark jobs for data transformation and aggregation. Work with Big Data tools: Hadoop , HDFS , Hive , Kafka , and large...

  • Cloud Big Data Engineer

    4 weeks ago

    Bangalore, India Lakshya Software Technologies Pvt. Ltd. Full time

    Title: Cloud Big Data Engineer Exp - 5+ years Location: Anywhere in India Work Mode: HYBRID Mandatory Skills required: Big Data (Hadoop, Hive), Python, Spark, PySpark, Cloud Platform, Azure Job description: Understanding of Big data Ecosystem with hands on experience working on Apache Spark, Hadoop (HDFS , Hive), Parquet, Elastic Search ...

  • Recro | Big Data Developer

    5 days ago

    bangalore, India Recro Full time

    Job Title : Big Data Engineer (Python & PySpark)Location : BangaloreExperience : 5+ yearsKey Responsibilities :Develop scalable data pipelines using Python and PySpark .Build and optimize Apache Spark jobs for data transformation and aggregation.Work with Big Data tools: Hadoop , HDFS , Hive , Kafka , and large datasets.Implement cloud-based...

  • Data Engineer

    2 months ago

    Bangalore, India Coders Brain Technology Private Limited Full time

    Job Position : Data EngineerLocation : BangaloreExperience : 5 to 10 yearsNotice Period : immediate to 15 days Mandatory skills : Cloud Platform,Snowflake, Python,Pyspark Job Description : - 5-7 years of data engineering experience with focus on designing and implementing scalable big data transformations and pipelines- Proficiency with Cloud Platforms like...

  • Recro | Big Data Developer

    2 weeks ago

    bangalore, India Recro Full time

    Job Title: Data Engineer – Python, PySpark, Big DataLocation: (Onsite)Experience: (5+ years)Job Description:Seeking an experienced Data Engineer with expertise in Python, PySpark, and Big Data technologies to design and build scalable data solutions on the Google Cloud Platform (GCP).Responsibilities:Develop/optimize data pipelines using Python and...

  • Data Engineer

    2 months ago

    Mumbai/Bangalore, India Arting Digital Full time

    Job Title : Data EngineerExperience : 5-7yrLocations : BangaloreSkills : Data Engineer, Snowflake, Python, DBT AND Spark or Hadoop AND Machine Learning.Notice period :1 Monthly - Immediate joinerRoles and Responsibility :- Migration of machine learning models from Spark to Snowflake.- Bachelor's/ master's degree in computer science, information...

  • Big Data Developer

    2 weeks ago

    Bangalore, India Recro Full time

    Job Title : Big Data Engineer (Python & PySpark) Location : Bangalore Experience : 5+ years Key Responsibilities : Develop scalable data pipelines using Python and PySpark . Build and optimize Apache Spark jobs for data transformation and aggregation. Work with Big Data tools: Hadoop , HDFS , Hive , Kafka , and large...

  • Senior Data Engineer

    2 months ago

    Bangalore, India Visionyle Solutions Full time

    Title : Senior Data EngineerExperience : 6+ YearsNotice Period : Immediate to 15 Days (Serving NP candidates)Work Location - BangaloreMandatory skills : Big data applications, Spark, python, scala , AWS S3,Airflow,Data bricks, data platforms, Jenkins, CICD PipelinesRequirements :- 6+ years of experience in developing scalable Big Data applications or...

  • Big data + gcp data engineer

    2 days ago

    Bangalore, India Deloitte Full time

    Key Skills : GCP, Spark, SQL, Scala , AIRFLOW, AUTOMIC , CICD , BIG QUERY , Py Spark, Unix , Python , Hive , Hadoop, Kafka, Spark Streaming GCP : Big Query, Cloud Storage, Dataproc, Dataflow, Pub/Sub, and Bigtable ETL : Experience in designing, building, and managing ETL/ELT pipelines Hadoop & Spark : Proficiency with Hadoop ecosystem tools and...

  • Recro | Big Data Developer

    2 weeks ago

    bangalore, India Recro Full time

    Job Title: Data Engineer – Python, PySpark, Big Data Location: (Onsite) Experience: (5+ years) Job Description: Seeking an experienced Data Engineer with expertise in Python, PySpark, and Big Data technologies to design and build scalable data solutions on the Google Cloud Platform (GCP). Responsibilities: Develop/optimize data pipelines using Python...

  • Recro | Big Data Developer

    4 weeks ago

    bangalore, India Recro Full time

    Job Title: Data Engineer – Python, PySpark, Big DataLocation: [Onsite]Experience: [5+ years]Job Description:Seeking an experienced Data Engineer with expertise in Python, PySpark, and Big Data technologies to design and build scalable data solutions on the Google Cloud Platform (GCP).Responsibilities:Develop/optimize data pipelines using Python and...

  • Big Data Developer

    4 weeks ago

    Bangalore Urban, India Recro Full time

    Job Title : Big Data Engineer (Python & PySpark) Location : Bangalore Experience : 5+ years Key Responsibilities : Develop scalable data pipelines using Python and PySpark . Build and optimize Apache Spark jobs for data transformation and aggregation. Work with Big Data tools: Hadoop , HDFS , Hive , Kafka , and large datasets. Implement cloud-based...

  • Big Data Developer

    4 weeks ago

    Bangalore Urban, India Recro Full time

    Job Title: Big Data Engineer (Python & PySpark)Location: BangaloreExperience: 5+ yearsKey Responsibilities:Develop scalable data pipelines using Python and PySpark.Build and optimize Apache Spark jobs for data transformation and aggregation.Work with Big Data tools: Hadoop, HDFS, Hive, Kafka, and large datasets.Implement cloud-based solutions on Google Cloud...

  • Data Engineer

    3 weeks ago

    Bangalore, India Coders Brain Technology Private Limited Full time

    Job Description : Key Responsibilities :- Design and implement scalable data pipelines to support big data transformations and analytics.- Work with cloud platforms such as Azure (preferred), AWS, or Google Cloud to deploy and manage data systems.- Utilize orchestration tools like Azure Data Factory and other cloud-native services for data pipeline...

  • Lead Big Data Engineer

    1 month ago

    Bangalore, India TekPillar Full time

    Job Title : Senior Data Engineer (Big Data) Experience : 4-9 YearsLocation : BangaloreJob Overview : We are seeking a skilled Senior Data Engineer to coordinate with cross-functional teams, manage and prioritize projects, and drive key data engineering initiatives. The successful candidate will have a deep understanding of big data tools and the ability...

  • Data Engineer for Cloud Computing and Big Data

    2 weeks ago

    Chennai/Bangalore, India Globalemployees Full time

    About UsGlobalemployees is a cutting-edge technology company that leverages the power of big data to drive innovation.Job DescriptionWe are seeking a highly skilled Data Engineer to join our team. As a key member of our engineering team, you will design, build, and maintain large-scale data processing systems using cloud-based technologies like...

  • Data Engineer

    2 months ago

    Bangalore, India NAM Info Inc Full time

    Job Title : Data Engineer. Location : Bangalore. Experience Level : 6- 8 years. Employment Type : Full-time with NAM INFO PVT LTD. Shift timings : Rotational Shift (should be able to support 24 x 7). Workdays : 5 days a week. Job Description : We are looking for experienced and talented Senior Data Engineers to join our team in Bangalore. As a Senior Data...