Transformers

Transformers are powerful middleware components that process, enrich, and modify DNS traffic data as it flows through your DNS-collector pipeline. They enable real-time data transformation, filtering, analysis, and privacy protection without requiring external processing tools.

Processing Pipeline Order

Transformers execute in a specific sequence to ensure data consistency and optimal performance. By default, the execution order is predefined, but it can be customized in the configuration file.

Customizing the Order

You can define a global execution order in the global section, which will be applied to all transformer pipelines by default.

global:
  transformers-order: [ "extract", "normalize", "filtering", "geoip", "atags", "suspicious", "user-privacy", "machine-learning", "rest", "relabeling", "latency", "rewrite", "new-domain-tracker", "unique-response-tracker", "reducer", "reordering", "frequency-filtering" ]

You can also override this order for a specific pipeline using the order key in the transformers section.

pipelines:
  - name: my-pipeline
    transforms:
      order: [ "geoip", "normalize" ]

If both are omitted, the default order is used.

Note: When defining a custom order, only the listed transformers will be initialized. Any transformer enabled in the configuration but missing from the order list will be ignored. If an unknown name is provided, an error will be logged during startup.

Default Order

The default logical processing order is:

extract - Data Extractor: Full DNS payload preservation.
normalize - Normalize: Standardizes DNS message format.
filtering - Traffic Filtering: Applies sampling and filtering rules.
geoip - GeoIP Metadata: Geographic traffic analysis.
bgp - BGP Metadata: BGP routing analysis (Origin ASN, AS-Path, Prefix).
atags - Additional Tags: Custom metadata.
suspicious - Suspicious Traffic Detector: Malformed packets, tunneling attempts, etc.
user-privacy - User Privacy: Masking or hashing components.
machine-learning - Traffic Prediction: ML-ready data preparation.
rest - REST Lookup: Custom data addition.
relabeling - JSON Relabeling: Standardize JSON keys.
latency - Latency Computing: Measure DNS resolution speed.
rewrite - DNS Message Rewrite: Change DNS record data.
new-domain-tracker - Newly Observed Domains: Track first-time domain appearances.
unique-response-tracker - Unique Domain Responses: Track newly observed response associations (QNAME, RRType, RDATA) with choice of LRU (fast) or Cuckoo Filter (memory-efficient) storage.
reducer - Traffic Reducer: Deduplicates repetitive queries.
reordering - Reordering: Sorts DNS messages by timestamp.
frequency-filtering - Frequency Filtering: Heavy-hitters detection and adaptive downsampling with Counting Cuckoo Filter.

Transformer Categories

Data Normalization & Standardization

Transformer	Capabilities	Impact
Normalize	• Convert domain names to lowercase • Extract TLD and TLD+1 components • Standardize text formatting • Clean malformed queries	Essential for consistent data analysis and storage
Reordering	• Sort DNS messages by timestamp • Handle out-of-order packet processing • Maintain chronological data flow	Critical for accurate time-series analysis

Traffic Management & Optimization

Transformer	Capabilities	Use Cases
Traffic Filtering	• Downsampling: Reduce data volume by percentage • Domain Filtering: Drop/allow specific domains • IP Filtering: Filter by client or server IP • Response Code Filtering: Filter by DNS response codes	• High-volume environment optimization • Focused monitoring on specific domains • Compliance and policy enforcement
Frequency Filtering	• Heavy-Hitters Detection: Identify high-frequency domains or client IPs • Tier Classification: Classify queries as rare, frequent, or heavy • Adaptive Actions: Drop, sample, or tag heavy traffic • Counting Cuckoo Filter: Memory-efficient frequency tracking with sliding window decay	• Protect SIEM/Loki from query floods • Preserve rare queries at 100% while downsampling noisy top talkers • High-volume DDoS and botnet traffic attenuation
Traffic Reducer	• Detect identical repeated queries • Log unique queries only once • Maintain occurrence counters • Reduce storage requirements	• Minimize storage costs • Focus on unique DNS patterns • Performance optimization

Security & Threat Detection

Transformer	Detection Capabilities	Security Benefits
Suspicious Traffic Detector	• Malformed Packets: Invalid DNS structure • Oversized Queries: Potential DDoS indicators • Uncommon Query Types: Rare or suspicious Qtypes • Invalid Characters: Malicious domain encoding • Excessive Labels: DNS tunneling attempts • Long Domain Names: Covert channel detection	• Early threat detection • DNS tunneling prevention • Malware C&C identification • DDoS attack mitigation
Newly Observed Domains	• Track first-time domain appearances • Identify domain generation algorithms (DGA) • Monitor new subdomain creation • Alert on suspicious registration patterns	• Zero-day domain detection • Brand protection monitoring • Typosquatting identification • Advanced persistent threat tracking
Unique Domain Responses	• Track newly observed (QNAME, RRType, RDATA) tuples • Detect unprecedented DNS answer associations • Dual storage engines: LRU (default, ~192 ns/op) or Cuckoo Filter (optional, 81% memory reduction) • Disk persistence support (LRU engine)	• DNS hijacking & cache poisoning detection • Fast-Flux C2 tracking • Unauthorized zone record changes

Privacy & Compliance

Transformer	Privacy Features	Compliance Support
User Privacy	• IP Anonymization: Hash or mask client IPs • Domain Minimization: Reduce domain specificity • SHA1 Hashing: Irreversible data protection • Configurable Privacy Levels: Granular control	• GDPR compliance • Internal privacy policies • Data sharing agreements • Research data anonymization

Performance Analysis & Monitoring

Transformer	Metrics & Analysis	Operational Value
Latency Computing	• Query-Response Matching: Correlate requests with responses • Round-Trip Time: Measure DNS resolution speed • Timeout Detection: Identify unanswered queries • Performance Trends: Track resolution performance	• SLA monitoring • Performance troubleshooting • Capacity planning • Service quality assurance
Traffic Prediction	• Feature Extraction: ML-ready data preparation • Pattern Recognition: Identify traffic patterns • Anomaly Scoring: Statistical deviation detection • Trend Analysis: Historical comparison	• Predictive scaling • Anomaly detection • Capacity forecasting • AI/ML model training

Data Enrichment & Intelligence

Transformer	Enrichment Capabilities	Enhanced Insights
GeoIP Metadata	• Country Identification: Client geolocation • City-Level Data: Detailed location information • ASN Mapping: Internet service provider data • IP Intelligence: Threat reputation scoring	• Geographic traffic analysis • Compliance monitoring • Threat intelligence correlation • Content delivery optimization
BGP Metadata	• Origin ASN: Real-time BGP Origin AS • AS-Path: Full BGP routing path • BGP Prefix: Longest matching route prefix	• Anycast routing troubleshooting • BGP hijack detection • Traffic engineering
Data Extractor	• Base64 Encoding: Full DNS payload preservation • Binary Data Handling: Raw packet analysis • Metadata Extraction: Protocol-level details • Custom Field Addition: Flexible data enhancement	• Deep packet inspection • Forensic analysis • Custom analytics • Advanced research
REST Lookup	• Custom Data Addition: Flexible data enhancement	• Business intelligence integration

Data Transformation & Formatting

Transformer	Transformation Features	Integration Benefits
Additional Tags	• Custom Metadata: Business-specific labels • Conditional Tagging: Rule-based classification • Dynamic Values: Runtime data injection • Multi-Tag Support: Complex categorization	• Business intelligence integration • Custom analytics dashboards • Automated workflows • Data organization
JSON Relabeling	• Field Renaming: Standardize JSON keys • Field Removal: Clean unnecessary data • Structure Modification: Reshape data format • Nested Object Handling: Deep JSON manipulation	• System integration • Data standardization • Storage optimization • API compatibility
DNS Message Rewrite	• Field Value Modification: Change DNS record data • Conditional Rewriting: Rule-based transformations • Pattern Matching: Regex-based modifications • Multi-Field Updates: Bulk data changes	• Data normalization • Privacy compliance • Testing scenarios • Data migration