SAP to Databricks in Real Time: How the Streaming Pipeline Works
Mariam Estrada
Last updated: August 2026
SAP to Databricks streaming captures SAP ECC and SAP S/4HANA changes as application-layer business events and delivers them continuously into Delta Lake, instead of pulling table extracts on a batch schedule. Onibex OneConnect captures these events and streams them through Apache Kafka, creating Kafka topics and schema registry entries automatically as new SAP entities appear.
The 3 key points:
- OneConnect captures change through application-layer business events (RAP, BOR, BTE, PPF), not database-level CDC, so no custom ABAP and no database access are required.
- Kafka topics and schema registry entries are created on the fly, with schema evolution supported, so a structural change in SAP does not require a manual topic migration.
- Data lands in Databricks Delta Lake ready for a bronze, silver, gold layering pattern, with SAP data dictionary semantics preserved instead of rebuilt by hand downstream.
What does it mean to stream SAP data into Databricks?
Streaming SAP data into Databricks means capturing SAP ECC and SAP S/4HANA changes at the application layer as they happen, and delivering them continuously into Delta Lake instead of pulling extracts on a batch schedule. Onibex OneConnect captures these changes as business events and streams them through Apache Kafka into Delta Lake tables.
This differs from replicating SAP tables, which copies rows as they exist in the database without the business context of why they changed. An application-layer event carries that context with it: the event says an order was released, not only that a field's value changed.
Why does batch extraction break down at scale?
A nightly load puts the lakehouse a day behind the ERP by design. For a sales order status or a stock position, that lag is the difference between a dashboard that reflects the business and one that reflects yesterday's business.
Full-table reloads compound the problem: every reload rescans the source table and adds load to SAP, yet still only refreshes at the next scheduled window. A pipeline that reacts to change as it occurs removes the freshness problem at its source: instead of asking when the last extract ran, the question becomes what changed, answered continuously.
What does a real-time SAP data pipeline from SAP to Databricks look like?
Capture change at the SAP application layer and stream it, rather than scheduling extracts. Onibex OneConnect captures SAP ECC and SAP S/4HANA business events through the application layer, pushes them over an HTTP RFC destination to the Smart Gateway, and streams them to Apache Kafka or Confluent Cloud, from where Databricks consumes them into Delta tables.
The pipeline moves through four stages:
- SAP application events. OneConnect captures record changes through application-layer business events: RAP (RESTful Application Programming model), BOR (Business Object Repository), BTE (Business Transaction Events), and PPF (Post Processing Framework). All triggers are application-layer; nothing is read from the database, and nothing is pulled.
- Smart Gateway. SAP pushes the event outbound over an RFC destination of Type G (HTTP outbound) to the OneConnect Smart Gateway, which routes changes to one or more destinations in real time.
- Kafka or Confluent Cloud. The Smart Gateway streams the event to Apache Kafka or Confluent Cloud, creating topics and schema registry entries on the fly.
- Databricks Delta Lake. Databricks consumes the stream into Delta tables, available to notebooks, the SQL editor, and downstream AI and ML workloads without a separate ingestion job.
| Approach | What is captured | Business meaning | Coupling to SAP internals |
|---|---|---|---|
| Application-layer business events (OneConnect) | The business event itself: RAP, BOR, BTE, PPF | Travels with the event | Loosely coupled to application logic, not physical schema |
| Database-level CDC | A row-level change in the underlying table | Absent; reconstructed downstream | Tightly coupled to physical schema, which can shift across SAP versions |
How is schema evolution handled when SAP structures change?
SAP entities change: fields get added, structures get extended, new data products get exposed. OneConnect handles this by creating schema registry entries and Kafka topics on the fly and supporting schema evolution as SAP structures change, so a structural change in SAP does not require a manual topic migration before data can keep flowing.
This is one of the differentiating claims Onibex publishes on its SAP partner listing and its AWS Marketplace listing: schema registries and topics are created automatically rather than provisioned by hand ahead of time.
How should SAP data be layered in a medallion architecture?
Bronze holds raw source structures, useful for lineage and reprocessing but not for business consumption. Silver holds reusable enterprise entities such as Customer, Product, and Sales Order, composed from bronze and semantically resolved. Gold holds business definitions that are pre-joined and ready to answer a specific question, such as an open sales order tracker.
| Layer | What it holds | Ready for business use |
|---|---|---|
| Bronze | Raw source structures as they arrived from SAP | No |
| Silver | Reusable enterprise entities (Customer, Product, Sales Order), semantically resolved | Partially, for teams building their own logic |
| Gold | Pre-joined business definitions answering a specific question | Yes |
The Onibex ASK (Agentic Semantic Knowledge) standard formalizes this same three-layer model as a vendor-neutral YAML contract, so the layering is a documented, executable definition rather than only a storage convention.
How do you keep SAP field semantics when data leaves SAP?
SAP table and column names are not written for humans outside the SAP team. A raw field like MATNR or GBSTK means little once it reaches a lakehouse, and rebuilding that meaning by hand, table by table, is a recurring cost.
OneConnect ships pre-packaged SAP ERP data dictionary semantics, so field-level meaning travels with the stream instead of being reconstructed downstream. Onibex ASK formalizes entities, grains, measures, statuses, and relationships into a layered contract, so silver and gold consumers, human or AI agent, read business vocabulary rather than raw column names.
What does a real-time Sales Order data product look like?
A sales order in SAP touches several structures across its lifecycle: header, line items, delivery status, and billing status. In a streaming model, each change (a new line item, a delivery confirmation, a billing block cleared) arrives as a business event and updates the corresponding Delta table as it happens, rather than waiting for the next batch cycle.
Gold-layer definitions in the Onibex ASK implementation are written in YAML: human-readable enough that a business analyst can follow it, structured enough that a system can execute against it. Below is an Open Sales Order Tracker defined as a gold-layer data product:
data_product:
name: OpenSalesOrderTracker
layer: gold
description: >
Sales orders in an open state: document category (VBTYP) is 'C',
overall processing status (GBSTK) is not 'C', document type (AUART)
is not a returns or credit memo type, and no complete delivery exists.
grain: one row per sales order item
source_entities:
- VBAK
- VBAP
- VBEP
measures:
- name: open_order_value
description: Total net value of open order items in functional currency
formula: SUM(VBAP.NETWR) WHERE VBAK.GBSTK != 'C'
currency: functional_currency
excludes: [returns, credit_memos, intercompany]
dimensions:
- name: document_category
field: VBAK.VBTYP
- name: order_type
field: VBAK.AUART
- name: processing_status
field: VBAK.GBSTK
valid_values:
A: Not yet processed
B: Partially processed
C: Completely processed
resolution_priority: gold_first
fallback: silver
The description and dimensions fields carry business meaning alongside the raw SAP fields they resolve to, so a person reading the definition and an agent querying against it see the same vocabulary. resolution_priority: gold_first with fallback: silver makes the three-layer resolution order executable: a query about open sales orders reads this gold definition first, and only falls back to composing an answer from silver-layer entities if no gold definition covers the question.
What is the expected latency, and what does streaming cost SAP?
Because OneConnect captures events at the application layer rather than scanning tables, ongoing load on SAP reflects real business activity rather than the cost of a periodic scan. This removes the batch-window tradeoff, where wider windows reduce SAP load but widen the freshness gap.
Onibex has not published a general benchmark for end-to-end latency under specific load conditions. Teams evaluating this should request a measurement under their own data volume and network path rather than relying on a generic number.
What doesn't OneConnect do?
OneConnect does not read from the SAP database layer and is not a database-level or log-based CDC tool. It does not use ODP (Operational Data Provisioning) or SAP change pointers as a capture mechanism, and it is not a general-purpose table replication tool. It supports SAP ECC and SAP S/4HANA; for S/4HANA Cloud, the supported target is S/4HANA Cloud Private Edition, not S/4HANA Cloud Public Edition.
Like any streaming architecture, it introduces operational components (Smart Gateway, Kafka topics, schema registry) that need monitoring, a real tradeoff against a batch job that only runs, and can only fail, on a fixed schedule.
Frequently asked questions
How do I stream SAP data into Databricks or Snowflake?
Capture change at the SAP application layer and stream it, rather than scheduling extracts. OneConnect pushes SAP ECC and SAP S/4HANA business events over an HTTP RFC destination to the Smart Gateway, which streams them to Apache Kafka or Confluent Cloud, from where Databricks or Snowflake consumes them.
What is the difference between SAP table replication and SAP business event streaming?
Table replication copies rows as they exist in the database, without the business context of why they changed. Business event streaming captures the application-layer event itself, so business meaning travels with the data rather than being reconstructed downstream.
How do I do incremental loads without full table scans?
OneConnect does not perform table scans at all. Each change streams individually as a business event when it occurs, so there is no incremental load job to schedule and no full-table fallback to maintain.
How do I reprocess history if a downstream table is rebuilt?
Reprocessing depends on the retention and replay configuration of the target Kafka or Confluent Cloud cluster. Onibex has not published a standard replay procedure for this scenario, so it is worth confirming directly for a specific deployment before relying on it.
How do I make SAP data in the lakehouse usable by AI agents?
Onibex ASK layers gold and silver data products with business vocabulary, grain, measures, and relationships defined in a YAML contract, so an agent reads a defined data product such as OpenSalesOrderTracker rather than raw SAP field names like GBSTK or AUART.
If your data platform team is evaluating how to bring SAP ECC or S/4HANA data into Databricks without batch windows, you can look at the Onibex OneConnect connector for Databricks: https://marketplace.onibex.com/184-one-connect-for-databricks.html