Redpanda

Availability: Airbyte Cloud: ❌, Airbyte OSS: ✅
Support Level: Community
Latest Version: 0.1.0
Definition Id: 825c5ee3-ed9a-4dd1-a2b6-79ed722f7b13

The Airbyte Redpanda destination connector allows you to sync data to Redpada. Each stream is written to the corresponding Redpanda topic.

Sync overview

Output schema

Each stream will be output into a Redpanda topic.

The Redpanda topic will be created with the following format {namespace}_{stream}

Currently, this connector only writes data with JSON format. More formats (e.g. Apache Avro) will be supported in the future.

Each record will contain in its key the uuid assigned by Airbyte, and in the value these 3 fields:

• _airbyte_ab_id: a uuid assigned by Airbyte to each event that is processed.
• _airbyte_emitted_at: a timestamp representing when the event was pulled from the data source.
• _airbyte_data: a json blob representing with the event data.

Data type mapping

Integration Type	Airbyte Type	Notes

Features

This section should contain a table with the following format:

Feature	Supported?(Yes/No)	Notes
Full Refresh Sync	No
Incremental - Append Sync	Yes
Incremental - Append + Deduped	No
Namespaces	Yes

Performance considerations

Granted you have enough Redpanda nodes/partitions the cluster should be able to handle any type of load you throw at it from the connector.

Getting started

Requirements

• The connector should be able to create topics using the AdminClient
• Configuration options
  • Bootstrap servers
  • Buffer Memory
  • Compression Type
  • Batch Size
  • Retries
  • Number of topic partitions
  • Topic replication factor
  • Socket Connection Setup Timeout
  • Socket Connection Setup Max Timeout

More info about this can be found in the Redpanda producer configs documentation site.

NOTE: Configurations for SSL are not available yet.

CHANGELOG

Version	Date	Pull Request	Subject
0.1.0	2022-08-05	18884	Initial commit