Big Data Engineer

2 weeks ago

GoaMumbaiJammu KashmirJammuSrinagarPondicherryJaipurLucknowVaranasiBanarasPatnaRanchi, IN ATech Full time

Designation: BIG DATA ENGINEER

Job Description:

Your Role and Responsibilities:

- Understand a data warehousing solution and able to work independently in such an environment

- Responsible in Project development and delivery experience of a few good size projects

- Design, build, optimize and support new and existing data models and ETL processes based on our clients business requirements.

- Build, deploy and manage data infrastructure that can adequately handle the needs of a rapidly growing data driven organization.

- Coordinate data access and security to enable data scientists and analysts to easily access to data whenever they need too.

- Experiencing developing scalable Big Data applications or solutions on distributed platforms

- Able to partner with others in solving complex problems by taking a broad perspective to identify innovative solutions

- Strong skills building positive relationships across Product and Engineering.

- Able to influence and communicate effectively, both verbally and written, with team members and business stakeholders

- Able to quickly pick up new programming languages, technologies, and frameworks

- Experience working in Agile and Scrum development process

- Experience working in a fast-paced, results-oriented environment

- Experience in Amazon Web Services (AWS) or other cloud platform tools

- Experience working with Data warehousing tools, including Dynamo DB, SQL, Amazon Redshift, and Snowflake

- Experience architecting data product in Streaming, Serverless and Microservices Architecture and platform.

- Experience working with Data platforms, including EMR, Data Bricks etc

- Experience working with distributed technology tools, including Spark, Presto, Scala, Python, Databricks, Airflow

- Developed the Pysprk code for AWS Glue jobs and for EMR.. Worked on scalable distributed data system using Hadoop ecosystem in AWS EMR, MapR distribution..

- Developed Python and pyspark programs for data analysis.. Good working experience with python to develop Custom Framework for generating of rules (just like rules engine).

- Developed Hadoop streaming Jobs using python for integrating python API supported applications..

- Developed Python code to gather the data from HBase and designs the solution to implement using Pyspark. Apache Spark DataFrames/RDD's were used to apply business transformations and utilized Hive Context objects to perform read/write operations..

- Re - write some Hive queries to Spark SQL to reduce the overall batch time

Required Technical and Professional Expertise:

- First and most important- Sound understanding of data structures & SQL concepts and experience in writing complex SQL especially around OLAP systems

- Sound knowledge of the ETL tool like informatica, 5+ years of experience, Big Data technologies'' like Hadoop ecosystem, its various components, along with different tools including Spark, Hive, Sqoop,etc..

- In-depth knowledge of MPP/distributed systems

Preferred Technical and Professional Expertise:

- The ability to write precise, scalable, and high-performance code

- The ability to write precise, scalable, and high-performance code

- Knowledge/Exposure in data modeling with OLAP (Optional)

(ref:hirist.tech)

  • Big Data Engineer

    2 months ago

    Goa/Mumbai/Jammu & Kashmir/Jammu/Srinagar/Pondicherry/Jaipur/Lucknow/Varanasi/Banaras/Patna/Ranchi, IN ATech Full time

    Designation: BIG DATA ENGINEERJob Description:Your Role and Responsibilities:- Understand a data warehousing solution and able to work independently in such an environment- Responsible in Project development and delivery experience of a few good size projects- Design, build, optimize and support new and existing data models and ETL processes based on our...

  • Big Data Engineer

    2 days ago

    Mumbai/Bangalore/Pune, IN Techno Wise Full time

    Job Role : Big Data EngineerRelevant Experience : 3+ yearsLocation : Navi Mumbai /Bengaluru/PuneNotice Period : Immediate or serving Notice PeriodPrimary Skills : Big Data, PySpark, Cloud- Azure or AWS, Databricks, SQL, PythonOverview : We are seeking a talented Big Data Engineer proficient in PySpark to join our dynamic team. The ideal candidate will play a...

  • Data Engineer

    3 weeks ago

    Navi Mumbai/Varanasi/Banaras, IN Connexions Full time

    Job Title : Data Engineer-BankLocation : Navi : 5 years +Function : Data Platform : ITEducation / Certification (If any) : Graduate- AWS / GCP / Data platform & Power BIGrade : Chief ManagerRole : Data Platform SUMMARY :Handle end-to-end implementation & Support of the Bank's data platformThe company is a rapidly growing financial institution that is...

  • Data Engineer

    2 weeks ago

    Navi Mumbai/Varanasi/Banaras, IN Connexions Full time

    Job Title : Data Engineer-BankLocation : Navi : 5 years +Function : Data Platform : ITEducation / Certification (If any) : Graduate- AWS / GCP / Data platform & Power BIGrade : Chief ManagerRole : Data Platform SUMMARY :Handle end-to-end implementation & Support of the Bank's data platformThe company is a rapidly growing financial institution that is...

  • Big Data Engineer

    2 days ago

    Mumbai/Pune/Bangalore, IN Techno Wise Full time

    Job Description : We are seeking a talented Big Data Engineer proficient in PySpark to join our dynamic team. The ideal candidate will play a pivotal role in designing, implementing, and maintaining scalable data solutions leveraging the Big Data technologies like PySpark. This role requires a strong understanding of big data technologies, data engineering...

  • Big Data Engineer

    2 months ago

    Rajasthan/Karnataka/Metros/Jaipur/Udaipur/Bangalore/Raipur, IN ATech Full time

    Hiring For Big Data Engineer (Spark, Scala, AWS)Job description :- Spark, SCALA, AWS OR Spark, python with AWS lambda exposure.- Design, build, optimize and support new and existing data models and ETL processes based on our clients business requirements.- Build, deploy and manage data infrastructure that can adequately handle the needs of a rapidly growing...

  • Big Data Engineer

    2 weeks ago

    Rajasthan/Karnataka/Metros/Jaipur/Udaipur/Bangalore/Raipur, IN ATech Full time

    Hiring For Big Data Engineer (Spark, Scala, AWS)Job description :- Spark, SCALA, AWS OR Spark, python with AWS lambda exposure.- Design, build, optimize and support new and existing data models and ETL processes based on our clients business requirements.- Build, deploy and manage data infrastructure that can adequately handle the needs of a rapidly growing...

  • Valiance Solutions

    2 months ago

    Noida/Bangalore/Mumbai, IN Valiance Analytics Pvt. Ltd. Full time

    About Us : Valiance is a global AI & Data analytics firm helping clients build cutting-edge technology solutions for digital transformation. We work with some of the marquee brands across India, US and APAC to build transformative solutions for Credit Risk, Fraud, Predictive Maintenance, Quality Inspection, Data lake, IOT analytics etc. Our team comprises...

  • Valiance Solutions

    2 weeks ago

    Noida/Bangalore/Mumbai, IN Valiance Analytics Pvt. Ltd. Full time

    About Us : Valiance is a global AI & Data analytics firm helping clients build cutting-edge technology solutions for digital transformation. We work with some of the marquee brands across India, US and APAC to build transformative solutions for Credit Risk, Fraud, Predictive Maintenance, Quality Inspection, Data lake, IOT analytics etc. Our team comprises...

  • PySpark/Databricks Engineer

    3 weeks ago

    Anywhere in India/Multiple Locations/Hyderabad/Srinagar/Jaipur, IN Aricent Full time

    Job : PySpark/Databricks EngineerOpen for Multiple Locations with WFO and WFHJob Description :We are looking for a PySpark solutions developer and data engineer that is able to design and build solutions for one of our Fortune 500 Client programs, which aims to build a data standardized and curation-based Hadoop clusterThis high visibility, fast-paced key...

  • PySpark/Databricks Engineer

    2 weeks ago

    Anywhere in India/Multiple Locations/Hyderabad/Srinagar/Jaipur, IN Aricent Full time

    Job : PySpark/Databricks EngineerOpen for Multiple Locations with WFO and WFHJob Description :We are looking for a PySpark solutions developer and data engineer that is able to design and build solutions for one of our Fortune 500 Client programs, which aims to build a data standardized and curation-based Hadoop clusterThis high visibility, fast-paced key...

  • IT Manager

    2 months ago

    Gujarat/Ahmedabad/Rajkot/Surat/Vadodara/Baroda/Guwahati/Srinagar/Jammu & Kashmir/Jammu/Hyderabad/Tel, IN Aricent Full time

    Role : IT Data Engineer - ManagerJob description :- The Data Engineer Manager role will be an integral part of an Agile team focused on delivering best in class business intelligence- Under the general supervision of senior management and the Data Integration Chapter Lead in the Enterprise Data Product Portfolio, you will be working with key customers to...

  • IT Manager

    2 weeks ago

    Gujarat/Ahmedabad/Rajkot/Surat/Vadodara/Baroda/Guwahati/Srinagar/Jammu & Kashmir/Jammu/Hyderabad/Tel, IN Aricent Full time

    Role : IT Data Engineer - ManagerJob description :- The Data Engineer Manager role will be an integral part of an Agile team focused on delivering best in class business intelligence- Under the general supervision of senior management and the Data Integration Chapter Lead in the Enterprise Data Product Portfolio, you will be working with key customers to...

  • Data Engineer

    3 weeks ago

    Mohali/Jaipur/Pune, IN closeloop Technologies private Limited Full time

    Requirements:- Snowflake : Proven experience in using Snowflake for data warehousing solutions.- Databricks: Strong hands-on experience with Databricks for big data processing and analytics using Machine Learning.- Redshift: Expertise in managing and optimizing Amazon Redshift data warehouses.- Spark: Advanced knowledge of Apache Spark for large-scale data...

  • Data Engineer

    2 weeks ago

    Mohali/Jaipur/Pune, IN closeloop Technologies private Limited Full time

    Requirements:- Snowflake : Proven experience in using Snowflake for data warehousing solutions.- Databricks: Strong hands-on experience with Databricks for big data processing and analytics using Machine Learning.- Redshift: Expertise in managing and optimizing Amazon Redshift data warehouses.- Spark: Advanced knowledge of Apache Spark for large-scale data...

  • Hadoop Administrator

    3 weeks ago

    Jaipur/Ahmedabad/Anywhere in India/Multiple Locations, IN ATech Full time

    Job Profile : Hadoop AdministratorWFH and WFO (Hiring Office Jaipur and Ahmedabad)Role: Big Data EngineerIndustry Type: IT Services & ConsultingJob description :- Good understanding of SDLC and agile methodologies- Installation and configuration of Hadoop clusters, including HDFS, MapReduce, Hive, Pig, HBase, and other related tools- Managing and monitoring...

  • Hadoop Administrator

    2 weeks ago

    Jaipur/Ahmedabad/Anywhere in India/Multiple Locations, IN ATech Full time

    Job Profile : Hadoop AdministratorWFH and WFO (Hiring Office Jaipur and Ahmedabad)Role: Big Data EngineerIndustry Type: IT Services & ConsultingJob description :- Good understanding of SDLC and agile methodologies- Installation and configuration of Hadoop clusters, including HDFS, MapReduce, Hive, Pig, HBase, and other related tools- Managing and monitoring...

  • Big Data Developer

    2 months ago

    Bangalore/Gurgaon/Gurugram/Pune/Chennai/Coimbatore/Mumbai/Cochin/Kochi/Noida, IN SP Software Pvt. Ltd. Full time

    Job Description :- Should have hands on experience in Software development- Responsible for designing, deploying, and managing high quality data solutions in the AWS cloud ecosystem.- Create and maintain optimal data architecture pipeline.- Collaborate with various onsite and client stakeholders in Agile environment to identify data engineering requirements...

  • Big Data Developer

    2 weeks ago

    Bangalore/Gurgaon/Gurugram/Pune/Chennai/Coimbatore/Mumbai/Cochin/Kochi/Noida, IN SP Software Pvt. Ltd. Full time

    Job Description :- Should have hands on experience in Software development- Responsible for designing, deploying, and managing high quality data solutions in the AWS cloud ecosystem.- Create and maintain optimal data architecture pipeline.- Collaborate with various onsite and client stakeholders in Agile environment to identify data engineering requirements...

  • Lead Full Stack Developer

    2 months ago

    Andhra Pradesh/Vijayawada/Vishakhapatnam/Vizag/Jammu & Kashmir/Jammu/Srinagar, IN ATS India Full time

    Role : Lead Full Stack DeveloperIndustry Type : IT Services Department : Engineering Employment Type : Full Time, PermanentJob description :- Work with distributed development teams and product managers to ideate software solutions.- Developing front end website architecture.- Designing user interactions on web pages.- Developing back-end website...