Arche Softronix

Primary skills expected :

- Data engineering problem solving.

- Able to understand requirement and to provide detailed implementation steps considering dependencies with - (upstream systems, downstream systems, and other modules).

- PySpark data ingestion and transformation Batch jobs development and optimization.

- Job orchestration using Python scripts or tools like Apache Airflow.

- Implementation of CI/CD pipeline and Data Quality validation control jobs.

- Strong HIVE and Impala SQL implementation and optimization.

- Implementation experience of CI/CD, Regression testing, Code promotion from lower environment to Prod environment.

- Handling data source types like Relational databases, Files, APIs

- Analyze, identify performance bottlenecks, and optimize by applying concurrency, parallelism, and other techniques on large scale data processing jobs.

- Understanding of data compliance features like PII, RBAC, Data masking.

- Implementation of Logging (Splunk), Alerts and Notifications.

Secondary skills :

- Real time data ingestion Implementation experience.

- Understanding of DR process.