Big Data Engineer

2 weeks ago


india ATech Full time

Designation: BIG DATA ENGINEER

Job Description:

Your Role and Responsibilities:

- Understand a data warehousing solution and able to work independently in such an environment

- Responsible in Project development and delivery experience of a few good size projects

- Design, build, optimize and support new and existing data models and ETL processes based on our clients business requirements.

- Build, deploy and manage data infrastructure that can adequately handle the needs of a rapidly growing data driven organization.

- Coordinate data access and security to enable data scientists and analysts to easily access to data whenever they need too.

- Experiencing developing scalable Big Data applications or solutions on distributed platforms

- Able to partner with others in solving complex problems by taking a broad perspective to identify innovative solutions

- Strong skills building positive relationships across Product and Engineering.

- Able to influence and communicate effectively, both verbally and written, with team members and business stakeholders

- Able to quickly pick up new programming languages, technologies, and frameworks

- Experience working in Agile and Scrum development process

- Experience working in a fast-paced, results-oriented environment

- Experience in Amazon Web Services (AWS) or other cloud platform tools

- Experience working with Data warehousing tools, including Dynamo DB, SQL, Amazon Redshift, and Snowflake

- Experience architecting data product in Streaming, Serverless and Microservices Architecture and platform.

- Experience working with Data platforms, including EMR, Data Bricks etc

- Experience working with distributed technology tools, including Spark, Presto, Scala, Python, Databricks, Airflow

- Developed the Pysprk code for AWS Glue jobs and for EMR.. Worked on scalable distributed data system using Hadoop ecosystem in AWS EMR, MapR distribution..

- Developed Python and pyspark programs for data analysis.. Good working experience with python to develop Custom Framework for generating of rules (just like rules engine).

- Developed Hadoop streaming Jobs using python for integrating python API supported applications..

- Developed Python code to gather the data from HBase and designs the solution to implement using Pyspark. Apache Spark DataFrames/RDD's were used to apply business transformations and utilized Hive Context objects to perform read/write operations..

- Re - write some Hive queries to Spark SQL to reduce the overall batch time

Required Technical and Professional Expertise:

- First and most important- Sound understanding of data structures & SQL concepts and experience in writing complex SQL especially around OLAP systems

- Sound knowledge of the ETL tool like informatica, 5+ years of experience, Big Data technologies'' like Hadoop ecosystem, its various components, along with different tools including Spark, Hive, Sqoop,etc..

- In-depth knowledge of MPP/distributed systems

Preferred Technical and Professional Expertise:

- The ability to write precise, scalable, and high-performance code

- The ability to write precise, scalable, and high-performance code

- Knowledge/Exposure in data modeling with OLAP (Optional)

(ref:hirist.tech)
  • Big Data Engineer

    4 weeks ago


    india Awign Expert Full time

    Job Description This is a remote position. Job Title: Big Data Solutions Engineer Reporting to: Director, Product Management Location: India (Remote) Our Ideal Candidate Our company is seeking a talented Big Data Solutions Engineer to join our team. The ideal candidate will be responsible for designing, developing, and...

  • Big Data Engineer

    2 weeks ago


    India Careerfit Full time

    Job Title: Big Data Solutions EngineerAbout the Company: Our company is a leading tech firm specializing in innovative data solutions that empower businesses to make informed decisions.Responsibilities :Solution Design : Design, develop, and maintain scalable big data solutions and data pipelines for efficient data ingestion, processing, storage, and...

  • Big Data Engineer

    7 days ago


    india Cielo Talent Full time

    Client: IBM, This represents a direct and enduring opportunity within IBM, with Cielo as the primary recruitment partner. Title: Big Data Engineer Location: Hyderabad (Hybrid) Experience: 5 to 10 Years Must have skills: Big data, Scala(Mandatory), Hive, Hadoop, Spark and PySpark Required Technical and Professional Expertise: First and most important...


  • India Grownex HR Solution Full time

    Role : Lead Big Data Engineer Work Mode : Remote(5 Days working) The Opportunity - Development team is establishing a new offshore product development team for its partner , that is building next-generation Big Data, Cloud-Based Business Operation Support technology for utilities, retail energy suppliers and Community Choice Aggregators (CCA). The candidate...


  • India Gethyr Full time

    Our company is seeking a talented Big Data Solutions Engineer to join our team. The ideal candidate will be responsible for designing, developing, and implementing our proprietary Bigdata-powered Fin Crime solution for our clients. The candidate will work closely with the software development team, data scientists, and product managers to understand...

  • Big Data Engineer

    3 weeks ago


    india Evnek Technologies Pvt Ltd Full time

    Job Description This is a remote position. Duration:  6 MonthLocation:  RemoteTimings: Full Time (As per company timings)Notice Period: within 15 days or immediate joinerExperience: 4-6 Years   Responsibilities Create and maintain data pipelines that transfer data from select locations (maintaining security protocols) into our Big Data financial...

  • Big Data Engineer

    2 weeks ago


    India ATech Full time

    Designation:BIG DATA ENGINEERJob Description:Your Role and Responsibilities: Understand a data warehousing solution and able to work independently in such an environment Responsible in Project development and delivery experience of a few good size projects Design, build, optimize and support new and existing data models and ETL processes based on our clients...

  • Big Data Engineer

    2 weeks ago


    Anywhere in India/Multiple Locations D-TechWorks Pvt Ltd Full time

    We are hiring for Big Data AI/ML Engineer - Remote , India Position : Big Data AI/ML EngineerExperience : 6 to 9 YrsLocation : Remote We are seeking a talented Data Engineer with AI & ML knowledge to join our team. As Data Engineer or ML Ops Engineer, your primary responsibility will be to develop & integrate ML solutions that focus on technology...

  • Data Engineer

    4 weeks ago


    india Evnek Technologies Pvt Ltd Full time

    Job Description This is a remote position. Role: Data Engineer  Experience: 2+Years Notice period: Immediate Location: Anywhere from India Job Type: Full Time Responsibilities: Designing & building data architectures including the design & set-up of scalable (big) data lakes (full cloud or hybrid cloud), scheduled transformations &...

  • Big Data Architect

    2 weeks ago


    India ARR Recruitment Solutions Full time

    About the Role : We are seeking a highly experienced Big Data Architect to join our team and lead the design, development, and implementation of our big data solutions. This role is ideal for a candidate with a deep understanding of big data technologies and a proven track record of delivering scalable and robust big data architectures.Responsibilities :...

  • Big Data Engineer

    2 weeks ago


    india Programming.com Full time

    Job Title : Big Data EngineerLocation : RemoteAbout Us :We are a leading specializing in software development . Our team is dedicated to leveraging cutting-edge technologies to deliver scalable and efficient solutions to our clients globally.Role Overview :We are seeking a highly skilled Big Data Engineer with a strong background in designing and developing...

  • GCP Big data Engineer

    2 weeks ago


    india IKAI Technology Solutions Full time

    We are seeking a talented Big Data + GCP Engineer to join our dynamic team. In this role, you will be responsible for designing, implementing, and maintaining scalable data pipelines and infrastructure on Google Cloud Platform (GCP). You will collaborate closely with cross-functional teams to ensure that our data platform meets the needs of our business...

  • GCP Big data Engineer

    2 weeks ago


    India IKAI Technology Solutions Full time

    We are seeking a talented Big Data + GCP Engineer to join our dynamic team. In this role, you will be responsible for designing, implementing, and maintaining scalable data pipelines and infrastructure on Google Cloud Platform (GCP). You will collaborate closely with cross-functional teams to ensure that our data platform meets the needs of our business...

  • GCP Big data Engineer

    2 weeks ago


    India IKAI Technology Solutions Full time

    We are seeking a talented Big Data + GCP Engineer to join our dynamic team. In this role, you will be responsible for designing, implementing, and maintaining scalable data pipelines and infrastructure on Google Cloud Platform (GCP). You will collaborate closely with cross-functional teams to ensure that our data platform meets the needs of our business...


  • India Leadway Resource Full time

    We are seeking a highly skilled and experienced Big Data QA Engineer to join our team. You will play a critical role in ensuring the quality of our big data platform and data solutions. You will leverage your expertise in Big Data testing methodologies, automation frameworks, and scripting languages to design, develop, and execute comprehensive test...

  • Big Data Architect

    2 weeks ago


    India Swift Strategic Staff Solutions INC Full time

    Responsibilities :Data Ingestion & Processing : Design and build robust data pipelines using tools like Apache Spark, Flink, or Kafka to ingest, transform, and process massive datasets from various sources.Big Data Technologies : Utilize your expertise in big data frameworks like Hadoop, YARN, and HDFS to manage, store, and analyze large volumes of data...


  • india TechTiv Software Solutions Full time

    The Opportunity : Digit88 development team is establishing a new offshore product development team for its partner , that is building next-generation Big Data, Cloud-Based Business Operation Support technology for utilities, retail energy suppliers and Community Choice Aggregators (CCA). The candidate would be joining an existing team of outstanding data...


  • India Leading MNC Full time

    Looking for a Freelance Big Data Solutions Engineer to join a team of rockstar developers. The candidate should have a minimum of 4+experience. There are multiple openings. If you're looking for freelance/ part time opportunity (along with your day job) & a chance to work with the top 0.1% of developers in the industry, this one is for you You will report...

  • Big Data Engineer

    2 weeks ago


    india Taiyō.AI Full time

    We empower the people who build the world. Taiyō.AI is the world's largest infrastructure construction data-mesh technology and the first AI platform for the global infrastructure construction industry. Our clients include some of the largest construction firms, suppliers, and government. About The Team We are looking for Big Data Engineer to help support...

  • Big Data Engineer

    2 weeks ago


    India Taiyō.AI Full time

    We empower the people who build the world.Taiyō.AI is the world's largest infrastructure construction data-mesh technology and the first AI platform for the global infrastructure construction industry. Our clients include some of the largest construction firms, suppliers, and government. About The TeamWe are looking for Big Data Engineer to help support and...