Big Data Engineer: Scalable Solutions Developer
14 hours ago
We are seeking an experienced PySpark Data Engineer to join our team at ITI Data. As a key member of our team, you will design and build scalable data solutions for one of our Fortune 500 Client programs.
About the Role:
The ideal candidate will have extensive experience in working with Hadoop, NoSQL databases, and Spark framework on Python. You will be responsible for building PySpark based applications for both batch and streaming requirements, developing and executing data pipeline testing processes, and optimizing performance of built Spark applications in Hadoop.
Key Responsibilities:
- Design, build, and unit test applications on Spark framework on Python.
- Build PySpark based applications for both batch and streaming requirements, which will require in-depth knowledge on majority of Hadoop and NoSQL databases as well.
- Develop and execute data pipeline testing processes and validate business rules and policies.
- Optimize performance of the built Spark applications in Hadoop using configurations around Spark Context, Spark-SQL, Data Frame, and Pair RDD's.
- Experience in skilled in Hadoop, Kafka, Scala, Spark.
- Optimize performance for data access requirements by choosing the appropriate native Hadoop file formats (Avro, Parquet, ORC etc) and compression codec respectively.
- Ability to design & build real-time applications using Apache Kafka & Spark Streaming.
- Build integrated solutions leveraging Unix shell scripting, RDBMS, Hive, HDFS File System, HDFS File Types, HDFS compression codec.
- Build data tokenization libraries and integrate with Hive & Spark for column-level obfuscation.
- Experience in processing large amounts of structured and unstructured data, including integrating data from multiple sources.
- Create and maintain integration and regression testing framework on Jenkins integrated with BitBucket and/or GIT repositories.
- Participate in the agile development process, and document and communicate issues and bugs relative to data standards in scrum meetings.
- Work collaboratively with onsite and offshore team. Develop & review technical documentation for artifacts delivered.
- Ability to solve complex data-driven scenarios and triage towards defects and production issues.
- Ability to learn-unlearn-relearn concepts with an open and analytical mindset.
- Participate in code release and production deployment.
What We Offer:
We offer a competitive salary range of $120,000 - $180,000 per annum, depending on experience, and a comprehensive benefits package, including health insurance, retirement plan, and paid time off.
Requirements:
To be successful in this role, you will need to have:
- Bachelor's degree in Computer Science or related field.
- Minimum 10 years of experience in working with big data technologies, including Hadoop, Spark, and NoSQL databases.
- Strong understanding of data engineering principles and practices.
- Excellent problem-solving skills and ability to work independently.
- Strong communication and collaboration skills.
About ITI Data:
ITI Data is a leading provider of data solutions and services. We help our clients unlock the value of their data and make informed business decisions. We are committed to delivering high-quality solutions and exceptional customer service.
-
India ANSR Full timeAbout ANSR: A Leading Life Science Tools and Integrated Systems ProviderAt ANSR, we are dedicated to making genomics useful for all. Our global hub in Bengaluru focuses on technology activities that will help usher in the next wave of innovation in healthcare and medical research. If you're eager to work on challenging projects that use cutting-edge...
-
Data Engineering Expert for Big Data Solutions
15 hours ago
India Cynosure Corporate Solutions Full timeCynosure Corporate Solutions is seeking a highly skilled Data Engineer to lead our big data solutions team. This role requires expertise in designing, building, and optimizing complex data pipelines using Hadoop/Spark ecosystem.About the Role:We are looking for an experienced professional who can work collaboratively with cross-functional teams to deliver...
-
Big Data Developer
2 days ago
India Impetus Full timeLead Software Engineer Locations- Indore, Noida, Hyderabad Job Description We are looking for a Lead Big Data Developer to design and oversee the implementation of large-scale data solutions, lead a talented team, and ensure the efficient processing and analysis of massive datasets. Roles & Responsibilities Key Responsibilities ...
-
India Aidetic Full timeWe are seeking an experienced Lead Data Architect to join our team at Aidetic. The successful candidate will have a proven track record of designing and implementing scalable data engineering solutions using Databricks or Snowflake.This role will focus on leading a team of data engineers who are building, maintaining, and optimizing large-scale data...
-
Big Data Engineer
3 weeks ago
India Pert Telecom Solutions Full timeJob Description : We are seeking a skilled Big Data Engineer with deep expertise in PySpark, Python, MySQL, Hadoop and Linux. The ideal candidate should have extensive experience deploying and managing these tools within on-premises environments (no public cloud). Proficiency in cluster management, performance optimization, and monitoring is essential...
-
Big Data Architect and Developer
2 weeks ago
India Atyeti Inc Full timeAbout Atyeti IncWe are a leading tech company that thrives on innovation. Our mission is to push the boundaries of what is possible in big data technology.Job Title: Big Data Architect and DeveloperThe successful candidate will work closely with our team to design and develop scalable big data solutions using Spark and Hadoop.Salary:$120,000 - $180,000 per...
-
Big Data Solutions Architect Leader
4 days ago
India bluCognition Full timeAbout bluCognition : We are an innovative AI/ML start-up focused on developing data-driven products using alternative data sources.Our clients are prominent names in the financial and telecom sectors, and we're seeking a Big Data Solutions Architect to lead our data integration, transformation, and distribution efforts.The successful candidate will design...
-
india Wavicle Data Solutions Full timeJob Description: We are seeking a highly experienced Lead Data Engineer with over 8 years of expertise in data engineering. As a Lead Data Engineer, you will play a pivotal role in architecting and implementing data solutions. Your proficiency in Python, PySpark, AWS, Databricks, SQL, and leadership skills will be crucial for success. Key Responsibilities:...
-
india Wavicle Data Solutions Full timeJob Description:We are seeking a highly experienced Lead Data Engineer with over 8 years of expertise in data engineering.As a Lead Data Engineer, you will play a pivotal role in architecting and implementing data solutions.Your proficiency in Python, PySpark, AWS, Databricks, SQL, and leadership skills will be crucial for success.Key Responsibilities:Lead...
-
india Wavicle Data Solutions Full timeJob Description: We are seeking a highly experienced Lead Data Engineer with over 8 years of expertise in data engineering. As a Lead Data Engineer, you will play a pivotal role in architecting and implementing data solutions. Your proficiency in Python, PySpark, AWS, Databricks, SQL, and leadership skills will be crucial for success. Key Responsibilities:...
-
Big Data Developer
6 days ago
India Luxoft Full timeSenior Data Engineer Job DescriptionWe are seeking a highly skilled Senior Data Engineer to join our team at Luxoft. As a key member of our Data Engineering & Analytics team, you will play a crucial role in developing data-driven solutions that drive business growth.About the RoleThis is an exciting opportunity for a seasoned data engineer to leverage their...
-
india Pert Telecom Solutions Full timeJob Description : We are seeking a skilled Big Data Engineer with deep expertise in PySpark, Python, MySQL, Hadoop and Linux. The ideal candidate should have extensive experience deploying and managing these tools within on-premises environments (no public cloud). Proficiency in cluster management, performance optimization, and monitoring is essential for...
-
Big Data Developer
6 days ago
India Luxoft Full timeProject Description: As a Senior Engineer in the Data Engineering & Analytics team, you will develop data & analytics solutions that sit atop vast datasets gathered by retail stores, restaurants, banks, and other consumer-focused companies. The challenge will be to create high-performance algorithms, cutting-edge analytical techniques including machine...
-
Big Data Developer
7 days ago
India Luxoft Full timeProject Description:As a Senior Engineer in the Data Engineering & Analytics team, you will develop data & analytics solutions that sit atop vast datasets gathered by retail stores, restaurants, banks, and other consumer-focused companies. The challenge will be to create high-performance algorithms, cutting-edge analytical techniques including machine...
-
Big Data
3 months ago
india Myananta Technologies Full timeAt Myananta Technologies, we’re looking for a Big Data (Hadoop). The ideal candidate is excited by experimentation and looking for a new challenge that stretches their talents. Job descriptionFull-Time Big Data (Hadoop) DeveloperMyananta Technologies | Gurugram is hiring for “Big Data (Hadoop) Developer” Experience: 3 - 5 Years.Salary: 5.0 - 12.0...
-
Big Data Expert for IoT Analytics Solutions
1 week ago
India NCR Voyix Full timeSenior Big Data Engineer PositionWe are seeking an experienced professional to join our team at NCR Voyix as a Senior Big Data Engineer. This is an exciting opportunity to work with cutting-edge technology and contribute to the development of innovative data products that drive business growth.About the RoleThe successful candidate will be responsible for...
-
Big Data Engineer for Innovative Team
14 hours ago
India Coffeee Full timeWe are seeking an experienced Big Data Engineer to join our innovative team at Coffeee. As a key member of our team, you will be responsible for developing and implementing Hadoop-based solutions that drive business growth.The ideal candidate will have hands-on experience with the Hadoop ecosystem, including HDFS, Hive, Pig, and Spark. You will also be...
-
Big Data Architect
8 months ago
India Princeton IT Services Full time**Job Title**: Big Data Architect **Location**:Kakinada (work from office) or Bangalore (work from office) **Shift**: UK shift preferred **Job Overview**: We are seeking a highly skilled and motivated Big Data Architect to join our team. This individual will play a crucial role in designing and implementing scalable and efficient big data solutions,...
-
Data Architect for Big Data Solutions
2 days ago
India Priceline Full timeUnlock Your Potential in a Fast-Paced Fintech EnvironmentWe are seeking an experienced Data Architect to join our Technology team, where you will play a key role in designing and implementing big data solutions that drive business growth.About the Role:Job Summary:This is a high-level position responsible for leading the design, development, and...
-
Senior Data Engineering Expert
1 week ago
Anywhere in India/Multiple Locations Crescendo Global Full timeCrescendo Global is Hiring a Senior Data EngineerWe are seeking an experienced Senior Data Engineer to join our team at Crescendo Global, a leading technology firm in India. As a key member of our data engineering team, you will be responsible for designing and developing scalable data pipelines using Python, Pyspark, and AWS services.About the RoleThis is a...
