Low-Latency Blockchain Event Streaming for Enterprises: SonarX Kafka Datastreams
How enterprise data teams evaluate real-time blockchain data providers, why Kafka is the transport capital markets settled on, and how SonarX Kafka Datastreams deliver decoded, normalised on-chain data across 130+ chains, Hyperliquid included.

SonarX Kafka Datastreams publish decoded, normalised blockchain events to Apache Kafka topics for enterprise data pipelines, across 130+ chains. A WebSocket that disconnects loses what happened while it was gone and does not tell you; a Kafka topic keeps the events and your place in them. That difference is the whole reason enterprise data teams move blockchain feeds onto Kafka, and most learn it the expensive way: the first pipeline is a socket subscription, it works in the demo, and the first production outage reveals that nothing was stored on the way in.
This piece is for the people who build and run those pipelines: heads of data engineering, platform leads, the quant and risk engineers who are told “we need this to be real-time” and then “we need this to be reliable” in the same quarter. It covers what to check in a provider, who needs the stream and why, what Kafka is and why capital markets standardised on it, where WebSockets and webhooks fail, what it takes to be on the receiving side, and what SonarX streams, with a section on Hyperliquid, the venue that settles on-chain in one block.
Key takeaways
- SonarX Kafka Datastreams publish decoded, normalised on-chain events to Kafka topics, one per chain and per dataset, across 130+ chains. For primitive datasets for delivery of Avro, P50 latency is under 1 second and P90 latency is under 2.1 seconds, measured from the block timestamp, across chains including Polygon, Monad and Arbitrum. Retention is configurable per topic from 1 hour to 1 day, and topics are filterable on content. Every dataset on the Hyperliquid docs page streams the same way.
- The latency clock starts at the block timestamp, the time the chain itself records for the block, and not at the moment a block reaches a node. A figure counted from node receipt comes out in milliseconds because it leaves out the time the block took to get there.
- Four teams drive the demand: trading and market-making, risk and surveillance, compliance and monitoring, and the data engineering team that feeds everyone else. They want different events but the same guarantees.
- Apache Kafka is a durable, partitioned, replicated commit log. Events are stored on the broker before anyone reads them, consumers track their own offset, and a consumer that goes offline resumes where it stopped, as long as retention still covers it. Transactional exactly-once processing has been part of Kafka since version 0.11 (2017).
- Capital markets reached this conclusion first. In July 2024 NYSE described publishing its real-time BQT feed to cloud customers as a Kafka-compatible stream; FactSet delivers its real-time price feed over Kafka with messages retained for several hours so consumers can recover. The vendor is the source of truth; Kafka is the distribution fabric.
- A WebSocket subscription dies with the connection and a re-subscribe starts “from now”, so the gap is silent unless you backfill it yourself. A webhook is retried on the sender’s schedule, in whatever order the retries land, and the receiver owns de-duplication.
- The receiving side is a Kafka client with credentials: twenty lines of Python, or a Kafka Connect sink into the warehouse you already run.
Blockchain event streaming over Kafka means on-chain events (blocks, transactions, logs, transfers, fills, funding, positions) are indexed, decoded and normalised by a data provider and written to Kafka topics, one per chain and dataset, from which any number of consumers read at their own pace and resume after a disconnect from the offset they last committed. The difference from a WebSocket or webhook is storage: the log holds the events whether or not anyone is listening.
The checklist
What to check in an enterprise blockchain data provider
| Factor | The question to ask |
|---|---|
| Latency | Is it stated with a percentile, for which datasets and chains, and where does the clock start: at the block timestamp, or when a block reaches a node? |
| Recovery | How long are events retained, and what happens when a consumer falls behind that window? |
| Delivery | What ordering applies, and who handles duplicates? |
| Coverage | Are the chains and datasets you need available, and is availability per chain documented? |
| Integration | Can standard Kafka clients, Kafka Connect sinks and stream processors consume it as-is? |
| Data handling | Are schemas registered and normalised across chains, and how do schema changes reach you? |
Every section below is one row of that table, in the order a data engineering team usually meets them.
The people
Who needs low-latency blockchain event streams
The request arrives in four shapes, and the shape tells you what to build.
A trading or market-making desk wants fills, funding payments, liquidations and large transfers the moment they land, because the model that consumes them is re-pricing on every tick. Their pain is the blind spot: a dropped socket during a volatile minute that nobody notices until the P&L does. The critical factors are latency with a percentile attached, replay after a disconnect, and coverage of the venue they trade.
A risk or surveillance team wants liquidity moves, concentration changes and suspicious patterns flagged in the same block they happen. Their pain is the polling interval that turns a 2-second event into a 30-second one, and the retry at the receiver that fires the same alert twice. They check ordering guarantees, duplicate handling and schema stability, because a field that changes name breaks a rule that was audited.
A compliance or monitoring function wants sanctioned-address touches, validator and network health, and an auditable record of what was seen and when. Their pain is the webhook receiver that was down for an hour and lost the hour. They ask about retention, about producing an offset as evidence, and about delivery semantics.
The data engineering and blockchain analytics team sits underneath all three. It wants warehouses, lakehouses and dashboards current without running nodes, and its pain is that RPC polling and WebSocket reconnect code become the permanent maintenance load of a team that was hired to do something else. It asks about Kafka Connect sinks, consumer groups for fan-out, and how many chains arrive through one integration.
The sequence is the same in every case. Version one polls an RPC. Version two subscribes to a WebSocket. Version three is the post-mortem that says “we need this to be reliable”, and reliable, for an event feed, has a specific meaning: the events are stored before they are delivered, and a consumer that fails can resume and replay.
The coverage
SonarX Kafka Datastreams: latency, coverage and delivery
SonarX Kafka Datastreams publish decoded, normalised on-chain events to Kafka topics, one topic per chain and per dataset, across the chains in the coverage table, 130-plus at the time of writing, with Kafka availability shown per chain. For primitive datasets (blocks, transactions, logs, transfers) for delivery of Avro, P50 latency is under 1 second and P90 latency is under 2.1 seconds from the block timestamp, across chains including Polygon, Monad and Arbitrum; derived datasets add their decoding time.
Where the clock starts matters as much as the number. SonarX measures from the block timestamp, the time the chain itself records for the block, and not from the moment a block reaches a node. Counted from node receipt, latency leaves out the time the block spent getting there, and comes out in milliseconds. Counted from the block timestamp, it covers everything that happens after the chain produces the block, which is the delay a pipeline actually experiences. Two latency figures are comparable only when they start from the same point.
Retention is configurable per topic, from 1 hour to 1 day, set to what the customer needs. A new consumer group can start from the beginning of retention, and an existing one can resume after an outage inside that window. History older than the window is in the historical datasets, which carry the same data.
Any dataset SonarX indexes on a supported chain can be a topic: raw blocks, transactions, receipts, logs and traces; decoded token transfers, priced transfers and net transfers; approvals, NFT transfers and token metadata on EVM chains; DEX trades, staking and validator events, and the chain-specific datasets where a chain has them. The data on the topic is the same normalised data SonarX delivers as shares into Snowflake, Databricks and BigQuery, so a field means the same thing on the stream as it does in the warehouse, and the same thing on Ethereum as it does on Solana.
Within a topic’s chain and dataset, the stream can be filtered on message content, so a desk receives transfers above a threshold for the two tokens it cares about, or every event touching a list of addresses, and nothing else. Filtered topics are provisioned by the SonarX team as part of the white-glove service every customer receives.
SonarX also delivers real-time data through low-latency APIs for request-response access to balances, holdings and transfers. An API answers a question; a Kafka topic delivers every event as it happens. Teams that need both use both.
The technology
What Apache Kafka is, in one page
Apache Kafka is an open-source distributed event-streaming platform, and the simplest accurate description is an append-only log that many programs can read. Producers write events to a topic. The topic is divided into partitions so that writes and reads run in parallel across a cluster of brokers, and each partition is replicated, typically three times, so that a single broker failure does not take the data with it. Consumers read from an offset, a position in the log, and commit that position as they go. A consumer group spreads one topic across several instances of the same service; several groups read the same topic independently, each at its own pace, without the producer knowing or caring.
Three properties follow, and they are the ones enterprise engineering teams pay for.
The log waits for you. Events sit on the broker for as long as the retention policy says. A consumer that disconnects reconnects and continues from the offset it last committed, provided that offset is still within retention. Catching up is the default behaviour, not a system you design.
One write, many readers. The trading engine, the risk service and the warehouse loader each read the same topic through their own consumer group. Adding a fourth reader is a configuration change at the reader, not a change to the feed.
Order and delivery are defined. Messages within a partition arrive in the order they were written. Delivery is at-least-once by default, which is what most pipelines want; transactional exactly-once processing was added in Kafka 0.11 (KIP-98, 2017) for the pipelines that need it, and end-to-end guarantees still depend on what the consumer does with the event.
The precedent
Capital markets answered this question first
The market-data industry went through this debate a decade ago. The question was: should a bank replace its vendor feed with Kafka? The answer the industry converged on is a hybrid. The vendor remains the source of truth, because the expensive part of a market-data feed is the normalisation, the symbology, the entitlements and the audit trail that come with it, and the transport is cheap by comparison. Kafka becomes the internal fabric that takes that trusted feed and fans it out to every desk, risk engine and analytics platform in the firm.
The exchanges and vendors then put the feed on Kafka themselves. In July 2024 the AWS Industries blog described how NYSE built its real-time market data platform for cloud customers: the BQT feed published as a Kafka-compatible stream across three availability zones. FactSet’s Kafka Real-Time Feed delivers exchange and contributor prices over Kafka, with messages retained for several hours so a consumer can recover from an outage.
Blockchain data has the same shape, and two complications on top: 130-plus heterogeneous chains rather than a few hundred exchanges on a shared protocol, and venues that settle on-chain in one block, so the gap between polling and streaming is no longer academic. The hybrid still holds: a trusted provider indexes, decodes and normalises the chain, and the normalised feed arrives on Kafka. What changes is that the provider delivers Kafka directly, so the customer does not build a WebSocket-to-Kafka bridge in front of it.
The failure modes
Where WebSockets and webhooks fail
Both are push transports, and both are fine for what they were designed for, which is a browser tab or a notification. Neither is storage, and that is where production pipelines break.
A WebSocket subscription lives exactly as long as the TCP connection. On an Ethereum node, eth_subscribe delivers new heads or logs while the socket is up; when the socket drops, the subscription is gone, and a re-subscribe delivers events from now. The blocks between the drop and the reconnect are not delivered, and no error says so. The standard mitigation, documented by several node operators in 2025 and 2026, is to persist the last block number and hash, resubscribe, backfill the gap with eth_getLogs, and de-duplicate the overlap. That is a small ingestion system, and teams that run a WebSocket feed in production end up writing one.
A webhook moves the storage problem to the sender, with a time limit. HTTP delivery is at-least-once: if the acknowledgement is lost, the event is sent again, so the receiver owns idempotency, usually a de-duplication key written in the same transaction as the side effect. If the receiver is down, the sender retries on a schedule and for a window that the sender chooses, and the events arrive in whatever order the retries land. The receiver also has to be reachable from the internet, which is its own conversation with the security team.
Kafka changes the recovery model. The consumer connects out to the broker, reads at its own pace, and commits its offset after processing. An outage on the consumer side costs nothing if it is shorter than the topic’s retention, and the consumer picks up in order from where it stopped. A slow consumer reads slowly, and only falls into trouble if its lag exceeds retention. Ordering is defined per partition. The client library manages offsets; the application still coordinates commits with successful processing and makes its downstream writes tolerate a replayed event. The receiver is never a server.
The receiving side
What it takes to consume
Four things from the provider: a bootstrap server address, SASL credentials, Schema Registry credentials, and the list of topics provisioned for you. Everything else is a standard Kafka client, in whichever language the team already uses. This is the minimal version; offsets are committed only after the event has been processed.
from confluent_kafka import Consumer
from confluent_kafka.schema_registry import SchemaRegistryClient
from confluent_kafka.schema_registry.avro import AvroDeserializer
from confluent_kafka.serialization import MessageField, SerializationContext
consumer = Consumer({
"bootstrap.servers": "<bootstrap-server-url>",
"group.id": "risk-service",
"auto.offset.reset": "earliest",
"enable.auto.commit": False,
"security.protocol": "SASL_SSL",
"sasl.mechanisms": "PLAIN",
"sasl.username": "<API-KEY>",
"sasl.password": "<API-SECRET>",
})
deserializer = AvroDeserializer(SchemaRegistryClient({
"url": "<schema-registry-url>",
"basic.auth.user.info": "<SR-KEY>:<SR-SECRET>",
}))
consumer.subscribe(["<chain>.<dataset>"])
try:
while True:
msg = consumer.poll(1.0)
if msg is None:
continue
if msg.error():
raise RuntimeError(msg.error())
event = deserializer(msg.value(), SerializationContext(msg.topic(), MessageField.VALUE))
handle(event)
consumer.commit(message=msg, asynchronous=False)
finally:
consumer.close()That is the connection and consumption path. A production consumer adds error handling, monitoring and a downstream write that tolerates a replayed event. The same credentials work from Java, Go, Node, .NET or Rust, from Kafka Streams and Flink for in-flight processing, and from Kafka Connect sinks straight into Snowflake, Databricks, BigQuery or object storage. A team that already runs Kafka adds a topic; a team that does not starts with one consumer and the SonarX Kafka quick-start.
The venue
Hyperliquid, specifically
Hyperliquid runs its perpetual and spot order books on-chain, and its own documentation states that every order, cancel, trade and liquidation happens with one-block finality. A venue that settles that fast is where the gap between polling and streaming is widest.
Every dataset on the SonarX Hyperliquid page can be streamed as a Kafka topic, covering Hyperliquid Core and HyperEVM. On Hyperliquid Core that is orders, orders failed and order status updates; node fills; liquidations, which Hyperliquid records as fills carrying a marker with the liquidated user, the mark price and the method, and which SonarX exposes both inside the fills dataset and as their own dataset; funding rates, hourly and per user; open interest and perp positions at regular block intervals; ledger updates covering deposits, withdrawals and account transfers; transfers, transactions and failed transactions; events, staking events, and delegator and validator details. Full-depth L2 and L4 order-book data is offered separately for purchase. On HyperEVM it is the full EVM table set, the same one SonarX delivers on every EVM chain.
A perps desk subscribes to fills and funding. A risk team subscribes to open interest and positions. A validator monitor subscribes to staking and validator details. Each reads the events as they are indexed, and each resumes from its own offset if it ever has to. How liquidations are recorded on HyperCore, and how to read them out of the fills data, is covered in Hyperliquid Liquidations Data Explained.
The door
The topic list is one request away
The question a data engineering team should ask any real-time blockchain data provider is the one capital markets settled years ago: what happens to the events while we are not listening. If the answer is “they are retained, and you resume from your offset”, the rest of the evaluation is about coverage, latency and whether the data is normalised the same way on every chain. If the answer is anything else, the team will be writing the ingestion system itself.
The topics are per chain and per dataset, across 130-plus chains, and the full Hyperliquid set is on the docs. Send the chain, the dataset and the predicate, and the team will provision the topic. Not tax, legal or investment advice, but all of it verifiable.
Sources
- Apache Kafka documentation, Message Delivery Semantics: https://kafka.apache.org/documentation/#semantics
- KIP-98, Exactly Once Delivery and Transactional Messaging (2017): https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging
- AWS Industries blog, How the New York Stock Exchange built its real-time market data platform on AWS (8 July 2024): https://aws.amazon.com/blogs/industries/how-the-new-york-stock-exchange-built-its-real-time-market-data-platform-on-aws/
- FactSet Kafka Real-Time Feed Service: https://www.factset.com/marketplace/catalog/product/factset-kafka-real-time-feed-service
- Hyperliquid documentation, About Hyperliquid: https://hyperliquid.gitbook.io/hyperliquid-docs/about-hyperliquid
- SonarX docs, Real-Time Streams overview: https://docs.sonarx.com/datasets/RTS/overview
- SonarX docs, Chain coverage: https://docs.sonarx.com/datasets/chaincoverage
- SonarX docs, Hyperliquid: https://docs.sonarx.com/datasets/HYPERLIQUID/Hyperliquid
- SonarX docs, Apache Kafka quick-start: https://docs.sonarx.com/api-reference/kafka
Frequently asked questions
Does SonarX offer real-time blockchain data over Kafka?
Yes. SonarX Kafka Datastreams publish decoded, normalised on-chain events to Kafka topics organised by chain and dataset, with retention configurable per topic from 1 hour to 1 day and content filtering provisioned by the team.
Can SonarX stream Hyperliquid data?
Yes. Every dataset listed on the SonarX Hyperliquid documentation page can be streamed, covering Hyperliquid Core and HyperEVM.
How does Kafka streaming differ from a low-latency data API?
An API answers a request for a balance, a holding or a set of transfers. A Kafka topic delivers every event in a dataset as it happens, and keeps it for replay within the topic's retention window. SonarX offers both.
How do I request a Kafka topic?
Name the chain, the dataset and any content filter, and the SonarX team provisions it. The Kafka quick-start has the connection example.
Up Next

SonarX Will Support Elysium From Day One
Elysium, Kinetiq's 100 ms Layer 2 for Hyperliquid, launches November 9. SonarX will index it from block one. Reserve data access now to query on launch day.

We Did It… Again! Five Questions From the KPMG Digital Assets Forum, Answered With Five Queries
Sharp questions from the people building institutional digital-asset infrastructure, hosted by KPMG and Hypernative. We took five of them back to the warehouse.