Studios - Data Engineer - Hive/Spark

6 days ago

Bengaluru, India CAW Full time

Job Description
We are looking for a savvy Data Engineer to join our growing team, responsible for expanding and optimizing our data and data pipeline architecture, as well as optimizing data flow. The ideal candidate is an experienced data pipeline builder and data wrangler who enjoys optimizing data systems and building them from the ground up. The right candidate will be excited by the prospect of optimizing or even re-designing our company's data architecture to support our next generation of products and data :

  • Create and maintain optimal data pipeline architecture.
  • Create and maintain events/streaming-based architecture/design.
  • Identify, design, and implement internal process improvements : automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
  • Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and AWS big data technologies.
  • Create data tools for analytics and data scientist team members that assist them in building

and optimizing our product into an innovative industry leader.

  • Work with data scientists to strive for greater functionality in our data systems.

Requirements

  • We are looking for a candidate with 2+ years of experience in a Data Engineer role.
  • Experience with big data tools : HDFS/S3 Spark/Flink, Hive, Hbase, Kafka/Kinesis, etc.
  • Experience with relational SQL and NoSQL databases, including Elasticsearch and Cassandra/Mongodb.
  • Experience with data pipeline and workflow management tools : Azkaban, Luigi, Airflow, etc.
  • Experience with AWS /GCP cloud services.
  • Experience with stream-processing systems : Spark-Streaming/Flink, etc.
  • Experience with object-oriented/object function scripting languages : Java, Scala, etc.
  • Experience building and optimizing big data pipelines, architectures, and data sets.
  • Strong analytic skills related to working with structured/unstructured datasets.
  • Build processes supporting data transformation, data structures, dimensional modeling,

metadata, dependency, schema registration/evolution, and workload management.

  • Working knowledge of message queuing, stream processing, and highly scalable big data data

stores.

  • Experience supporting and working with cross-functional teams in a dynamic environment.
  • Have a few weekend side projects up on GitHub.
  • Have contributed to an open-source project.
  • Have worked at a product company.
  • Have a working knowledge of a backend programming language.

)

  • CAW Studios

    1 week ago

    Bengaluru, India CAW Studios Pvt Ltd Full time

    Job Description : We are looking for a savvy Data Engineer to join our growing team, responsible for expanding and optimizing our data and data pipeline architecture, as well as optimizing data flow. The ideal candidate is an experienced data pipeline builder and data wrangler who enjoys optimizing data systems and building them from the ground up. The right...

  • Big Data Engineer

    4 days ago

    Bengaluru, India True Talents Full time

    Job Title : Big Data EngineerLocation : Bangalore / PuneEmployment Type : Full TimeExperience : 4 to 8 yearsNotice Period : 15 daysAbout the Role :We are looking for a skilled Big Data Engineer to design, develop, and maintain large-scale data processing systems. The ideal candidate will have hands-on experience with big data technologies and a strong...

  • Principal Apache Hive Engineer

    24 hours ago

    Bengaluru, Karnataka, India BairesDev Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    At BairesDev, we've been leading the way in technology projects for over 15 years. We deliver cutting-edge solutions to giants like Google and the most innovative startups in Silicon Valley. Our diverse 4,000+ team, composed of the world's Top 1% of tech talent, works on roles that drive significant impact worldwide.When you apply for this position, you're...

  • Data Engineer

    3 days ago

    Bengaluru, India Tata Consultancy Services Full time

    Dear CandidateGreetings from TATA Consultancy ServicesJob Openings at TCSSkill - Data Engineer ( Spark Scala)Exp range – 5 yrs – 10 yrsLocation- Chennai /Pune/Bangalore/Hyderabad /DelhiNotice period – 30 daysPls find the Job Description below.Big Data Hadoop - Hive and Spark/Scala solid experienceSQL advance knowledge - Been able to test changes and...

  • Data Engineer

    2 days ago

    Bengaluru, India Tata Consultancy Services Full time

    Dear CandidateGreetings from TATA Consultancy ServicesJob Openings at TCSSkill - Data Engineer (Spark Scala)Exp range – 5 yrs – 10 yrsLocation- Chennai /Pune/Bangalore/Hyderabad /DelhiNotice period – 30 daysPls find the Job Description below.- Big Data Hadoop - Hive and Spark/Scala solid experience- SQL advance knowledge - Been able to test changes and...

  • Data Engineer

    19 hours ago

    Bengaluru, India Tata Consultancy Services Full time

    Dear CandidateGreetings from TATA Consultancy ServicesJob Openings at TCSSkill - Data Engineer (Spark Scala)Exp range – 5 yrs – 10 yrsLocation- Chennai /Pune/Bangalore/Hyderabad /DelhiNotice period – 30 daysPls find the Job Description below.- Big Data Hadoop - Hive and Spark/Scala solid experience- SQL advance knowledge - Been able to test changes and...

  • ▷ 15h Left: Senior Data Engineer

    1 week ago

    Bengaluru, India Axheleon IT Services Private Limited Full time

    Job Description About the Role We are looking for a highly skilled Senior Data Engineer with a strong background in Python, PySpark, Hive, and distributed data processing frameworks. The ideal candidate will have 5+ years of hands-on experience in building and optimizing big data pipelines, with expertise in both batch and streaming data processing using...

  • Big Data Engineer – Spark

    3 weeks ago

    Bengaluru, Karnataka, India POWER IT SERVICES Full time

    Big Data Engineer – Spark & Analytics Job Description • Gather operational intel on business processes and policies from multiple sources • Prepare periodical and ad-hoc reports using operational data • Develop semantic core to align data with business processes • Support operations team's work streams for data processing, analysis and reporting...

  • Big Data Engineer – Spark

    3 weeks ago

    Bengaluru, Karnataka, India POWER IT SERVICES Full time

    Big Data Engineer – Spark & AnalyticsJob Description• Gather operational intel on business processes and policies from multiple sources• Prepare periodical and ad-hoc reports using operational data• Develop semantic core to align data with business processes• Support operations team's work streams for data processing, analysis and reporting•...

  • Big Data Engineer – Spark

    6 days ago

    Bengaluru, India POWER IT SERVICES Full time

    Big Data Engineer – Spark & Analytics Location- ( Bangalore, Maharashtra) Job Description •Gather operational intel on business processes and policies from multiple sources •Prepare periodical and ad-hoc reports using operational data •Develop semantic core to align data with business processes •Support operations team's work streams for data...