Databridge - Data Pipeline System

Databridge is a flexible data pipeline system for processing and transferring data between various sources and destinations. It is designed to run on Kubernetes and supports multiple data processing pipelines.

Features

• DBF to PostgreSQL: Import data from DBF files to PostgreSQL
• CSV Export: Export data from PostgreSQL to CSV files
• Kubernetes Native: Designed to run as Kubernetes Jobs
• ZFS Storage: Supports ZFS persistent storage
• Parameterized Pipelines: Flexible configuration via environment variables

Getting Started

Prerequisites

• Kubernetes cluster
• ZFS storage provisioner
• PostgreSQL database

Installation

1. Deploy Storage Infrastructure:

   kubectl apply -f k8s/pv.yaml
   kubectl apply -f k8s/pvc.yaml
   kubectl apply -f k8s/rbac.yaml