View all jobs
RoC 20 - Data Flow Engineer in support of Integration of Data Platform with Frontex Applications (DESOPS)
Warsaw, PolandJob Description: Data Flow Engineer
DESCRIPTION OF THE TASKS
The external service provider will perform the following typical tasks and responsibilities:
KNOWLEDGE AND SKILLS
MANDATORY EXPERTISE
REQUIRED CERTIFICATES
At least one (1) certification from the following list (or equivalent recognized internationally) is required:
| Role: | Data Flow Engineer |
| Location: | Warsaw, Poland |
| Languages: | English B2 (Minimum level confirmed by CEFR) |
| Work Model: | Hybrid – 60% Onsite / 40% Remote |
| Context/Project: | Data Flow / Data integration / NiFi / Big Data Streaming & Flow Management |
| Education Requirement: | Minimum level of education: Level 6 - Bachelor |
| Security/Integrity: | Personal Security Clearance Required: RESTREINT UE/EU RESTRICTED (procedure must be initiated within first 45 days of assignment) |
DESCRIPTION OF THE TASKS
The external service provider will perform the following typical tasks and responsibilities:
- Pipeline Development: Design, implementation, testing, and maintenance of complex data flows in Cloudera DataFlow (Apache NiFi), including ingestion, transformation, enrichment, routing, and egress.
- CDC Optimization: Building and optimizing CDC-based pipelines (real-time / near-real-time) using NiFi, Kafka, and Debezium/SQL CDC connectors.
- System Integration: Integration with external systems via REST API, JDBC, Kafka, and other protocols.
- Governance & Metadata: Managing data schemas (Avro), metadata, and lineage in Apache Atlas.
- Security Configuration: Configuring security and governance using Ranger policies for data flows.
- Operations: Monitoring, alerting, and troubleshooting the performance and reliability of data pipelines.
- Collaboration: Collaborating with data engineers, architects, and business stakeholders to define requirements and architecture.
- Documentation: Creating and maintaining SOPs, technical documentation, and runbooks.
- Lifecycle Management: Participating in CDP / NiFi / Kafka upgrades and migrations.
KNOWLEDGE AND SKILLS
- Expert knowledge in defining, designing, and maintaining complex data flows in Apache NiFi (Cloudera DataFlow).
- Advanced Python programming skills for data processing, NiFi custom logic, flow automation, and integrations.
- Advanced skills in building integrations based on REST API (endpoints' calling, OAuth/JWT authentication, rate limiting, error recovery).
- Hands-on experience in building CDC-based data flows (Change Data Capture) using native NiFi connectors and SQL Builder.
- Good knowledge of Apache Iceberg (tables, schema evolution, partitioning).
- Knowledge of data governance and cataloging in CDP using Apache Atlas and Apache Ranger.
- Experience with Apache Kafka (message broker) and Apache Avro (serialization standard).
MANDATORY EXPERTISE
- IT Professional Experience: Minimum of 8 years of IT-relevant professional experience.
- Similar Position Experience: Minimum of 6 years of experience in a similar position.
- Technical Experience:
- Min. 2-3 years of hands-on experience in daily work with Apache NiFi (design, deployment, monitoring, troubleshooting).
- Documented experience in at least one big integration project using NiFi as the central tool.
- Practical knowledge of Apache Iceberg and implementation of CDC pipelines to/from relational databases.
- Experience managing governance/lineage in Apache Atlas + Ranger and working with Apache Kafka in the CDP ecosystem.
REQUIRED CERTIFICATES
At least one (1) certification from the following list (or equivalent recognized internationally) is required:
- Cloudera Certified Developer for Apache NiFi.
- Cloudera Data Flow (CFM) related certification.