How to Stream SAP Data to Kafka: A Complete Walkthrough
Onibex Team
Last updated: August 2026
What is SAP to Kafka streaming
Streaming SAP data to Kafka means moving record changes from SAP ECC or SAP S/4HANA onto Apache Kafka topics as they happen, not on a batch schedule. Onibex OneConnect captures application-layer business events, pushes them out over an HTTP RFC destination to the Smart Gateway, and writes them to Kafka, creating topics and schema registry subjects on the fly.
This walkthrough takes you from an SAP object to a live, consumable Kafka topic without writing custom extraction code. The path is: configure the Smart Gateway and its Kafka or Confluent Cloud target first, model the object in the SAP Data Modeler, configure an outbound HTTP RFC destination that points at the Smart Gateway, then verify and consume the stream. The Smart Gateway comes first because creating the workspace is what generates the endpoint and credentials the SAP side needs. The architecture in one line: SAP business event, then HTTP RFC destination Type G, then Smart Gateway, then Apache Kafka with a schema registry subject created automatically.
Why do the usual SAP-to-Kafka approaches stall?
Most teams asked to stream SAP data to Kafka hit the same three walls.
The first is the ABAP dependency. Custom extraction usually means an ABAP developer writing and maintaining extractor programs, and that developer is rarely available on the timeline the data team is measured against.
The second is schema drift. Extraction built against physical tables breaks when an upstream structure changes, and the break is usually discovered by a business user looking at a wrong number, not by an alert.
The third is per-object toil. Every new SAP object becomes a new manual topic, a new schema definition, and a new ticket, so onboarding the tenth object costs the same as the first.
OneConnect removes all three by capturing at the application layer and creating topics and schema registry subjects on the fly. The rest of this guide shows exactly how.
Prerequisites
Before you start, confirm you have the following.
- An SAP system: SAP ECC or SAP S/4HANA. For S/4HANA Cloud, the supported target is S/4HANA Cloud Private Edition.
- Authorization to create an outbound RFC destination (an HTTP connection to an external server) and to use the SAP Data Modeler in your SAP environment.
- A Kafka target: an Apache Kafka cluster or a Confluent Cloud cluster, with a Schema Registry endpoint and credentials.
- The Smart Gateway deployed and reachable from SAP over HTTP or HTTPS.
If your Basis team needs the authorization objects, ports, and security model before approving the outbound destination, coordinate that with them up front. To try the end-to-end flow before configuring your own system, use the Onibex sandbox at onibex.cloud.
Step 1: configure the Smart Gateway and its Kafka target
Start on the Smart Gateway, not in SAP. This step comes first because it produces the two values the SAP side needs later: the inbound endpoint SAP will call, and the credentials SAP will authenticate with.
In the Smart Gateway, create the workspace and configure the target: the Kafka or Confluent Cloud bootstrap server, the security settings, and the Schema Registry endpoint and credentials. The Smart Gateway receives the business events from SAP and routes them to this target.
Creating the workspace generates the inbound endpoint (host, port, and path prefix) that SAP will call, plus the username and password SAP will use. Note both down; you will enter them in SM59 in Step 3.
You do not pre-create the topic or hand-register a schema. When events begin to flow, OneConnect creates the Kafka topic and the schema registry subject on the fly, and supports schema evolution as the entity changes over time.
Expected result: a Smart Gateway workspace pointed at your cluster, with the Schema Registry endpoint set, and a generated endpoint URL plus credentials in hand.
Step 2: model the object in the SAP Data Modeler
Now move to SAP. Open the SAP Data Modeler with transaction ZONT_ONECM, the main interface OneConnect delivers for designing and managing SAP data products. Define the entity you want to stream by selecting the SAP object, tables, and views that make up the entity and structuring them for extraction. This is a modeling step, not a development step, so no ABAP extractor is written.
Add the tables that make up the entity and join them with inner or left-outer joins. For example, add a table such as VBAK and continue.
Because OneConnect ships pre-packaged SAP data dictionary semantics, field-level meaning travels with the entity rather than being rebuilt downstream. You are defining what to expose, and the meaning of each field comes along with it.
For real-time streaming, the underlying BOR event linkage is activated and monitored with standard SAP transactions: SWETYPV to activate the event type linkage OneConnect requires, and SWEL to review the event log and confirm events are being raised. OneConnect configuration tables are maintained through SM30 on the ZONT* tables.
Expected result: a saved entity definition in the SAP Data Modeler that names the object and the fields you want on the stream, with the real-time event linkage active.
Step 3: configure the outbound HTTP RFC destination
Create the outbound RFC destination in transaction SM59, using the endpoint and credentials the Smart Gateway generated in Step 1. This is the channel SAP uses to push business events outward. Nothing is pulled from SAP and nothing reads the database layer; SAP initiates the call when a business event fires.
Under HTTP Connections to External Server, create a connection with these values:
| Field | Value |
|---|---|
| Connection Type | G, HTTP Connection to External Server |
| Destination Name | ONIBEX_ONECONNECT (or ONIBEX_POC) |
On the Technical Settings tab, set the Host, Port, and Path Prefix that make up the Smart Gateway endpoint. Split the endpoint URL into its three parts. For an endpoint of the form https://{DOMAIN}:5051/api/v1/saplistener/{CLIENT}:
| Field | Value |
|---|---|
| Host | {DOMAIN} |
| Port | 5051 |
| Path Prefix | /api/v1/saplistener/{CLIENT} |
On the Logon & Security tab, enter the username and password configured for the corresponding workspace in the Smart Gateway. The credentials in SAP must match the Smart Gateway user for that workspace.
Then click Connection Test and read the HTTP status:
| Status | Meaning | Action |
|---|---|---|
| 200 | Successful connection | None |
| 400 | Server not detected | Review host, port, and path prefix |
| 500 | Authentication failure | Review username and password in Logon & Security |
Expected result: a Type G RFC destination named ONIBEX_ONECONNECT that returns 200 on its connection test.
Step 4: verify the topic and schema registry entry
Trigger a change on the modeled object in SAP, or run the entity, so at least one business event is produced.
Then check two things on the Kafka side. First, confirm the topic now exists in your cluster. Second, confirm a subject for that topic appears in the Schema Registry. Seeing both is how you know the on-the-fly creation worked end to end.
Expected result: a new topic in the cluster and a matching subject in the Schema Registry, neither of which you created by hand.
What SAP sends: the event envelope
Each business event SAP pushes to the Smart Gateway is a JSON envelope with three parts: properties (what kind of change this is), metadata (the SAP data dictionary definition of every field), and body (the changed rows). The metadata travels with the data, which is what lets OneConnect build the Kafka schema on the fly rather than requiring a hand-registered schema per object.
The example below is sanitized and shows a single illustrative row. The real envelope carries the same shape for any modeled entity.
{
"oneconnect": {
"properties": {
"messageType": "TABLE",
"change": "X",
"delete": "",
"domain": "SALES",
"entity": "SALES_OFFICE_TEXT"
},
"metadata": [
{
"table": "TVKBT",
"metadata": [
{ "fieldname": "mandt", "type": "C", "length": "000003", "keyflag": "X", "decimals": "0", "fieldtext": "Client" },
{ "fieldname": "spras", "type": "C", "length": "000001", "keyflag": "X", "decimals": "0", "fieldtext": "Language Key" },
{ "fieldname": "vkbur", "type": "C", "length": "000004", "keyflag": "X", "decimals": "0", "fieldtext": "Sales Office" },
{ "fieldname": "bezei", "type": "C", "length": "000020", "keyflag": "", "decimals": "0", "fieldtext": "Description" }
]
}
],
"body": {
"table": "TVKBT",
"data": [
{ "mandt": "000", "spras": "E", "vkbur": "0000", "bezei": "Example Sales Office" }
]
}
}
}
Three fields in the envelope drive the streaming behavior:
keyflagmarks the fields that form the SAP primary key. Those fields become the Kafka message key, which is what gives you correct partitioning and upsert behavior downstream.deletecarries the change semantics. When SAP flags a record as deleted, the event marks it as a delete rather than silently dropping the row from a later extract.typeandlengthare the SAP data dictionary types (for exampleCfor character) that OneConnect maps to the schema registered for the topic.
Step 5: consume the stream
Consume the topic to confirm records are arriving and are decodable against the registered schema. Using the standard Confluent Avro console consumer, the shape is:
kafka-avro-console-consumer \
--bootstrap-server <broker-host>:9092 \
--topic <your-topic> \
--property schema.registry.url=<schema-registry-url> \
--from-beginning
Expected result: records printed to the console, deserialized against the schema in the registry. From here the stream is a normal Apache Kafka topic: any Kafka Connect sink connector, stream processor, or consumer application can read it.
Batch extraction versus SAP to Kafka event streaming
The dimensions below are the ones an engineer actually weighs when deciding whether to stream.
| Dimension | Batch extraction | SAP to Kafka event streaming with OneConnect |
|---|---|---|
| Latency | Data is as old as the last extract window | Events flow as the change happens in SAP |
| SAP load | Repeated full or delta reads compete with transactions | Application-layer events pushed outbound, no bulk table reads |
| Deletes | A removed row simply stops appearing | Deletes arrive as events, not as silent absence |
| Schema changes | Extractor built on tables breaks on refactor | Capture is application-layer, schema evolution supported at the registry |
| New object onboarding | New topic, new schema, new ticket each time | Topic and schema registry subject created on the fly |
| ABAP effort | Custom extractor programs to write and maintain | Configuration in the SAP Data Modeler, no custom extraction code |
On throughput, the pipeline is built for high-volume change streams: in Onibex testing it sustains on the order of 100,000 records per minute. Treat that as an order-of-magnitude reference rather than a guarantee for your landscape, since sustained rate depends on payload size, cluster sizing, and network path.
How is schema evolution handled?
Schema evolution is supported at the Schema Registry. Because capture happens at the application layer rather than against physical tables, the stream is less exposed to physical table refactoring than table-level replication is.
The operational detail an engineer needs before adopting is the compatibility mode. Set the compatibility mode on the subject to match how your consumers are written, and decide in advance what should happen when an incompatible change arrives, because that behavior is enforced by the registry, not by OneConnect. Document that choice next to the pipeline so the person paged at 3 a.m. does not have to rediscover it.
Troubleshooting
Work the pipeline in the same order you built it.
- No topic appears after a change. Confirm the Type G destination passes its connection test from SAP, and confirm the Smart Gateway target points at the right bootstrap server. A topic that never appears usually means the event never left SAP or never reached the cluster.
- Topic exists but no schema subject. Check the Schema Registry endpoint and credentials on the Smart Gateway target. The topic can be created while schema registration fails separately if the registry endpoint is wrong.
- Consumer cannot deserialize. Confirm the consumer points at the same Schema Registry URL used by the Smart Gateway, and that the consumer uses the matching deserializer for the format on the topic.
- Records stop arriving after an upstream change. Check the subject compatibility mode and the most recent registered version against the change that was made in SAP.
Frequently asked questions
How do I connect SAP to Kafka?
Model the objects you need in the SAP Data Modeler, configure an outbound HTTP RFC destination from SAP ECC or SAP S/4HANA to the OneConnect Smart Gateway, and point the Smart Gateway at Apache Kafka or Confluent Cloud. Topics and schema registry subjects are created on the fly, and schema evolution is supported.
Do I need ABAP development to extract SAP data?
No custom extraction development is required. OneConnect captures changes through application-layer business events using standard SAP frameworks (RAP, BOR, BTE, and PPF) and pushes them outbound over an HTTP RFC destination. Configuration replaces custom extractor programs, so there is nothing to rewrite at the next upgrade.
Are topics and schemas created automatically?
Yes. OneConnect creates the Kafka topic and the Schema Registry subject on the fly when events begin to flow, so you do not pre-create a topic or hand-register a schema for each SAP object.
How is schema evolution handled?
Schema evolution is supported at the Schema Registry. Because capture is application-layer rather than table-level, the stream is less exposed to physical schema refactoring. The compatibility mode you set on the subject determines what happens when a changed schema arrives.
What OneConnect does not do
OneConnect does not read the SAP database layer and does not use database triggers, log-based change data capture, or table replication. It does not use Operational Data Provisioning or change pointers. It captures application-layer business events and pushes them outbound over an HTTP RFC destination. Coverage is SAP ECC and SAP S/4HANA, with S/4HANA Cloud Private Edition as the supported cloud target, not S/4HANA Cloud Public Edition. The Schema Registry enforces compatibility; OneConnect creates and evolves subjects but does not override the registry's compatibility decision.
About Onibex
Onibex is an SAP Silver Partner and Confluent Gold Partner building real-time SAP data hyperconnectivity for the enterprise. Its flagship platform, Onibex OneConnect, streams SAP ECC and SAP S/4HANA business events to Apache Kafka, Confluent Cloud, Databricks, Snowflake, and other destinations in real time. Onibex also maintains ASK (Agentic Semantic Knowledge), a semantic-layer specification and platform that grounds AI agents in governed, business-ready SAP data. Onibex is headquartered in The Woodlands, Texas.