Digi Upaay

Experience : 6 to 9 years

Location : Hyderabad

Job Role : Data Engineer

Description :

Data engineers implement methods to improve data reliability and quality. They combine raw information from different sources to create consistent and machine-readable formats. They also develop and test architectures that enable data extraction and transformation for predictive or prescriptive modeling.

Senior Data Engineer is responsible for creating effective technological solutions for work with big data. They are also responsible for managing a team of specialists in their field and related IT professions. In addition, their roles include quality control of the work performed.

Preferred Skills : AWS Glue PySpark and Python

Qualifications :

- Data Cloud (AWS) - Engineer

- AWS Redshift, Aurora, AWS Glue, AWS Lambda, etc.

- Extensive experience with Data gathering and ingestion (from multiple sources), and manipulation, orchestration and optimization on AWS Cloud

- PySpark / Python

- Developing RESTful APIs

- Skilled in using ETL tools

- Proven track record of leading multi-shore teams on sizable Cloud + Data initiatives

Data Engineer Job Responsibilities :

- Develops and maintains scalable data pipelines and builds out new API integrations to support continuing increases in data volume and complexity.

- Collaborates with analytics and business teams to improve data models that feed business intelligence tools, increasing data accessibility and fostering data-driven decision making across the organization.

- Implements processes and systems to monitor data quality, ensuring production data is always accurate and available for key stakeholders and business processes that depend on it.

- Writes unit/integration tests, contributes to engineering wiki, and documents work.

- Performs data analysis required to troubleshoot data related issues and assist in the resolution of data issues.

- Works closely with a team of frontend and backend engineers, product managers, and analysts.

- Defines company data assets (data models), spark, sparkSQL, and hiveSQL jobs to populate data models.

- Designs data integrations and data quality framework.

- Designs and evaluates open source and vendor tools for data lineage.

- Works closely with all business units and engineering teams to develop strategy for long term data platform architecture.

Data Engineer Qualifications/Skills :

- Knowledge of best practices and IT operations in an always-up, always-available service

- Experience with or knowledge of Agile Software Development methodologies

- Excellent problem solving and troubleshooting skills

- Process oriented with great documentation skills

- Excellent oral and written communication skills with a keen sense of customer service