AWS Data Engineer
4 weeks ago
Job Description
We are looking for an AWS Data with primary skills on PySpark development who will be able to design and build solutions for one of our Fortune 500 Client programs, which aims towards building an Enterprise Data Lake on AWS Cloud platform, build Data pipelines by developing several AWS Data Integration, Engineering & Analytics resources. There is no requirement for Machine Learning skills. This is a high visibility, fast-paced key initiative will integrate data across internal and external sources, provide analytical insights, and integrate with the customer’s critical systems.
Key Responsibilities
• Design, build and unit test applications on Spark framework on Python.
• Build Python and PySpark based applications based on data in both Relational databases (e.g. Oracle), NoSQL databases (e.g. DynamoDB, MongoDB) and filesystems (e.g. S3, HDFS)
• Build AWS Lambda functions on Python runtime leveraging pandas, json, boto3, requests, avro libraries
• Build PySpark based data pipeline jobs on AWS Glue ETL requiring in-depth knowledge on AWS Glue Dynamic Frames and Options
• Build Python based event-driven integration with Kafka Topics, leveraging Confluent Kafka libraries
• Design and Build Generic, Reusable utility applications in Python
• Build the Python programs across Glue ETL jobs and Lambda functions
• Optimize performance for data access requirements by choosing the appropriate native Hadoop file formats (Avro, Parquet, ORC etc) and compression codec respectively.
• Design & Build S3 buckets, tiers, lifecycle policies, as strategic storage layer for "Data Lake"
• Optimize performance of Spark applications in Hadoop using configurations around Spark Context, Spark-SQL, Data Frame, and Pair RDD's
• Setup the Glue crawlers in order to catalog OracleDB tables, MongoDB collections and S3 objects
• Configure Athena tables and SQL views based on Glue Cataloged datasets
• Ability to monitor, troubleshoot and debug failures using AWS CloudWatch and Datadog
• Ability to solve complex data-driven scenarios and triage towards defects and production issues
• Participate in code release and production deployment.
-
AWS Data Engineer
6 days ago
India ITI Data Full timeJob Description We are looking for an AWS Data with primary skills on PySpark development who will be able to design and build solutions for one of our Fortune 500 Client programs, which aims towards building an Enterprise Data Lake on AWS Cloud platform, build Data pipelines by developing several AWS Data Integration, Engineering & Analytics resources....
-
AWS Data Engineer
3 weeks ago
India ITI Data Full timeJob Description We are looking for an AWS Data with primary skills on PySpark development who will be able to design and build solutions for one of our Fortune 500 Client programs, which aims towards building an Enterprise Data Lake on AWS Cloud platform, build Data pipelines by developing several AWS Data Integration, Engineering & Analytics resources....
-
AWS Data Engineer
18 hours ago
India FLYONIT Full timeJob Summary: We are looking for a highly skilled AWS Data Engineer with 5-6 years of experience to join our dynamic team. The ideal candidate will have hands-on experience with AWS services, data engineering best practices, and a strong understanding of building, deploying, and maintaining data pipelines in the cloud. You will be responsible for designing...
-
AWS Data Engineer
18 hours ago
India Mactores Full timeMactores is a trusted leader among businesses in providing modern data platform solutions. Since 2008, Mactores has been enabling businesses to accelerate their value through automation by providing automated, agile, and secure end-to-end data solutions. We collaborate with customers to strategize, navigate, and accelerate an ideal path forward with digital...
-
Lead Data Engineer
3 months ago
india TIU Consulting Full timeJob Description : - Design, build and maintain highly scalable and available big data solutions on AWS cloud - Develop and manage AWS Glue data catalog, crawlers, and ETL jobs- Design and implement data synchronization solutions using AWS Datasync- Manage and optimize S3 data storage and access patterns- Develop and maintain Infrastructure as Code (IaC)...
-
Freelance Opportunity: AWS Data Engineer
3 weeks ago
India ThreatXIntel Full timeCompany Description ThreatXIntel is a startup cyber security company that specializes in protecting businesses and organizations from cyber threats. Our services include cloud security, web and mobile security testing, cloud security assessment, and DevSecOps. We provide customized and affordable solutions to meet the specific needs of our clients,...
-
Freelance Opportunity: AWS Data Engineer
4 weeks ago
India ThreatXIntel Full timeCompany DescriptionThreatXIntel is a startup cyber security company that specializes in protecting businesses and organizations from cyber threats. Our services include cloud security, web and mobile security testing, cloud security assessment, and DevSecOps. We provide customized and affordable solutions to meet the specific needs of our clients, regardless...
-
Senior AWS Data Engineer
1 week ago
India UsefulBI Corporation Full timeOverview:UsefulBI is looking for highly skilled candidates, with expertise in generating power business insights from very large datasets, in this space where the primary aim would be to enable needle moving business impacts through cutting edge statistical analysis. We are looking for passionate data engineers who can envision the design and development of...
-
Senior AWS Data Engineer
7 days ago
India UsefulBI Corporation Full timeOverview: UsefulBI is looking for highly skilled candidates, with expertise in generating power business insights from very large datasets, in this space where the primary aim would be to enable needle moving business impacts through cutting edge statistical analysis. We are looking for passionate data engineers who can envision the design and development...
-
Lead /Data Architect Aws Devops Engineer
5 days ago
India People Tech Group Inc Full timeHello Linkedin connections We are Hiring '' Lead /Data Architect Aws Devops Engineer '' and below are the requirements. This will be through ''People Tech Group Company''. Position : Aws Lead Devops Engineer/Data Architect Company : People Tech Group ( Mnc) Mode of Work: WFO Experience: 6+yrs Location : Bangalore Only ...
-
AWS DevOps Engineer
5 days ago
India Space Multimedia Full timeSpace is an innovative and dynamic company that specializes in leading businesses through transformative journeys in the digital era. They are pioneers in driving innovation and helping organizations transition into digitally mature entities. With a wide range of cutting-edge services, including web enterprise application development, AI & ML development,...
-
Data Engineer
3 weeks ago
India Insight Global Full timeJob Title: Senior Data Engineer Location: Remote (India) – Required to work from 1 PM IST to 12 AM IST Pay Rate: Total compensation ranges from 22 to 35 lakhs per annum, based on experience. Duration: This is a contract position. Only apply if you are comfortable with contract work. The position is approved for full-time work up to a year and is likely...
-
Data Engineer
4 weeks ago
India Insight Global Full timeJob Title: Senior Data EngineerLocation: Remote (India) – Required to work from 1 PM IST to 12 AM ISTPay Rate: Total compensation ranges from 22 to 35 lakhs per annum, based on experience.Duration: This is a contract position. Only apply if you are comfortable with contract work. The position is approved for full-time work up to a year and is likely to...
-
AWS Network Engineer
2 weeks ago
India System Soft Technologies Full timeJob Summary:The client is looking for an AWS Network Engineer:We are seeking an experienced AWS Network Engineer with a strong background in networking and hands-on experience with Amazon Web Services (AWS) infrastructure and services. As an AWS Network Engineer, he/she will be responsible for implementing and maintaining our AWS network infrastructure to...
-
AWS Network Engineer
1 week ago
India System Soft Technologies Full timeJob Summary: The client is looking for an AWS Network Engineer: We are seeking an experienced AWS Network Engineer with a strong background in networking and hands-on experience with Amazon Web Services (AWS) infrastructure and services. As an AWS Network Engineer, he/she will be responsible for implementing and maintaining our AWS network infrastructure...
-
Data Engineer
2 weeks ago
India Techiebutler Full timeJob Title: Data EngineerLocation: AhmedabadEmployment Type: Full-TimeExperience Required: 4+ YearsJob DescriptionTechiebutler is looking for an experienced Data Engineer to develop and maintain scalable, secure data solutions. You will collaborate closely with data science, business analytics, and product development teams, deploying cutting-edge...
-
Data Engineer
1 week ago
India Techiebutler Full timeJob Title: Data Engineer Location: Ahmedabad Employment Type: Full-Time Experience Required: 4+ Years Job Description Techiebutler is looking for an experienced Data Engineer to develop and maintain scalable, secure data solutions. You will collaborate closely with data science, business analytics, and product development teams, deploying cutting-edge...
-
Data Engineer
2 weeks ago
India Perficient India Full timeSnowflake Data EngineerLocation - RemoteRole - FulltimeYears of Experience - 5+Implement and manage data transformation in Snowflake and Aurora with DBT.Build and maintain data pipelines to extract, transform and load data effectively.Work with architect and data analysts to understand data models and requirements.Create technical documents and unit test...
-
Data Engineer
1 week ago
India Perficient India Full timeSnowflake Data Engineer Location - Remote Role - Fulltime Years of Experience - 5+ Implement and manage data transformation in Snowflake and Aurora with DBT. Build and maintain data pipelines to extract, transform and load data effectively. Work with architect and data analysts to understand data models and requirements. Create technical documents and unit...
-
Data Engineer
3 months ago
india Suitable.AI Full timeRole Objective : Data Engineer will be responsible for expanding and optimizing our data and database architecture, as well as optimizing data flow and collection for cross functional teams. The ideal candidate is an experienced data pipeline builder and data wrangler who enjoys optimizing data systems and building. The Data Engineer will support our...