Data Engineer / Senior Data Engineer

About the role

About the Role

Cyble's threat intelligence platform ingests cybercrime data at petabyte scale: stealer logs, breach dumps, dark web forums and marketplaces, Telegram and Discord channels, ransomware leak sites, and domain intelligence. That data feeds our AI agents and analyst tooling, and it is only useful when it is clean, fresh, deduplicated, and retrievable.

We're hiring a Data Engineer /Senior Data Engineer to build and run the Iceberg-based data lake that powers Cyble's AI agents. You'll own pipelines end to end, from raw scraped and leaked data landing in S3 to curated lake tables, OpenSearch indexes, and retrieval-ready datasets that agents and products query. You'll work in a small, low-process team where engineers own their domains, ship directly, and use AI coding tools as a normal part of the workflow.

What You’ll Do at Cyble:

- Build and operate the data lake powering Cyble's AI agents: ingestion, Data modeling, deduplication, freshness, lineage, and retrieval-ready outputs.

- Design and run PySpark pipelines on EMR over Apache Iceberg on S3 (Glue Data Catalog), including schema evolution, partitioning, compaction, snapshot expiry, and small-file control.

- Ingest scraped, crawled, and leaked data from heterogeneous sources (HTML, JSON, CSV, text dumps, archives, feeds, APIs) and handle the realities: encoding issues, schema drift, malformed records, duplicates, and re-crawled content.

- Build and tune OpenSearch/Elasticsearch indexing pipelines: mappings, analyzers, shard and index lifecycle strategy, bulk ingestion, and query performance.

- Build the data side of RAG and agent workloads: chunking, embedding generation, vector and hybrid (keyword + kNN) indexing, and datasets the agents can trust.

- Run streaming and event-driven pipelines on Kafka; orchestrate batch workloads with Airflow.

- Deploy and operate data workloads on Kubernetes (EKS) and ECS, including Spark jobs, Trino, workers, and supporting services.

- Own lake and pipeline reliability: monitoring, alerting, data quality checks, backfills and reprocessing, retention, and storage/compute cost control.

- Use AI coding assistants and agents (Claude Code, Cursor, Copilot, or similar) daily to build, refactor, test, debug, and document, and review AI-generated code critically before it reaches production.

- Work with security researchers, data scientists, and product engineers on data models for new intelligence products.

What You’ll Need:

- Bachelor's degree in computer science, engineering, or equivalent experience.

- 3+ years building production data pipelines in Python and Spark.

- Deep Spark knowledge on AWS EMR: shuffle, skew, partitioning, memory tuning, AQE, and cost-aware cluster sizing.

- Hands-on experience with Apache Iceberg (or Delta/Hudi with willingness to go deep on Iceberg): table design, partition evolution, compaction, snapshot management, time travel.