Trending

Branch, Test, Deploy: A Git-Inspired Approach for Data

(2018) How I created a database of all interesting Rush Hour configurations

The end of the road for Kafka-delta-ingest

Super fast aggregations in PostgreSQL 19

Show HN: Parqeye – A CLI tool to visualize and inspect Parquet files

Listen to Database Changes Through the Postgres WAL

650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark

Using the expand and contract pattern for schema changes

Moving tables across PostgreSQL instances

Where did the Epstein emails dataset go

SynthDB - A Zero-Config Database Seeder Written in Rust 🦀 (Seeking Contributors!)

Indexing, Partitioning, Sharding - it is all about reducing the search space

Spring Batch Concepts Tutorial to handle large-scale data processing with ease using Spring: Defining Jobs, Steps, Chunk processing, flow control, and workflows etc.

Creating a PostgreSQL Extension: Walk through how to do it from start to finish

[D] Name and describe a data processing technique you use that is not very well known.

Has anyone tried selling data on Opendatabay or similar platforms?

How Transaction Isolation Provides Data Integrity in Databases

What’s the biggest data governance challenge you face when building cross-agent pipelines?

apache

Save item

airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

apache

Save item

arrow

Apache Arrow is the universal columnar format and multi-language toolbox for fast data interchange and in-memory analytics

dlt-hub

Save item

dlt

data load tool (dlt) is an open source Python library that makes data loading easy 🛠️

pathwaycom

Save item

pathway

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

Unstructured-IO

Save item

unstructured

Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.

dbt-labs

Save item

dbt-utils

Utility functions for dbt projects.

apache

Save item

dolphinscheduler

Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code

apache

Save item

seatunnel

SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.