Cloudflare Logpush Transformers: SQL via DataFusion Enables Schema Normalization and Redaction

Cloudflare has announced the general availability of Transformers for its Logpush service. This update allows teams to apply SQL-based transformations to log streams before they are exported to external destinations.

What happened

Cloudflare published details on how Transformers operate within the Logpush export pipeline. The feature enables users to filter, redact, and reshape log data using SQL queries. Powered by Apache DataFusion and Apache Arrow, the engine applies changes directly to log batches before they reach their final destination. According to the vendor, this eliminates the need for separate processing services that teams previously had to deploy and maintain.

Logpush delivers logs in batches with near real-time latency. During Cloudflare’s recent Birthday Week, the company made Logpush available on all self-serve plans. Transformers were also released as generally available during this period. These updates form part of eight major changes to Cloudflare Observability. The goal is to make advanced features accessible to all users.

A Transformer is defined as a SQL query specific to a Logpush dataset. The query determines which records and fields are exported. Logpush applies the query to each batch before sending it. Records that do not match the WHERE clause are removed. The SELECT statement output becomes the final exported record. Results are exported in NDJSON format.

Why it matters

Observability teams often export logs to SIEMs, analytics platforms, or object storage. Before export, they may need to remove health checks or omit sensitive fields. Previously, this required external processing infrastructure. With Transformers, this processing moves into the Logpush workflow. Teams can define SQL queries to filter records and reshape output by adding, renaming, or nesting fields.

Cloudflare’s data pipeline ingests over one billion events per second. The Bufstream delivery service handles more than 110 million Logpush records per second. This system runs across hundreds of Kubernetes pods. Job volumes follow a Pareto-like distribution, with some jobs processing millions of logs per second. Transformers meet these operational requirements for reliable delivery and predictable resource use.

Customers can manage Transformers via the Cloudflare dashboard or API. Each update creates a new version for review. An example use case involves reshaping HTTP request logs to match a team-defined schema. This includes redacting client IP addresses and filtering out internal traffic. Both input and output use NDJSON serialization.

Next steps

• Define SQL queries to filter specific log datasets.
• Attach Transformers to compatible NDJSON Logpush jobs.
• Monitor resource usage and performance metrics.
• Review version history for transformation changes.

What to do and how to stay safe: Cloudflare

• Audit current log export volumes to identify unnecessary data.
• Test SQL filters in a staging environment before production.
• Verify that redaction rules meet compliance requirements.
• Monitor export costs after implementing pre-export filtering.

Step-by-step guide: Wifi router security: Stop hidden admin access from leaking data

General security guidance from the Packet Tribune newsroom. It is not confirmed advice from the organisations named in this story.

Frequently asked questions

What technology powers Cloudflare Logpush Transformers?

Transformers are powered by Apache DataFusion and Apache Arrow, embedded in the export pipeline.

How does Logpush handle records that do not match the filter?

Records that do not match the WHERE clause in the SQL query are removed before export.

Can users manage Transformers programmatically?

Yes, customers can create and manage Transformers through the Cloudflare API or Transformer Studio.