croft

Product Introduction

1. Definition: Croft is an open-source, TypeScript-native data pipeline tool. Technically, it is a CLI application that orchestrates data ingestion, transformation, and scheduling into a single, local DuckDB database file.
2. Core Value Proposition: Croft exists to consolidate the fragmented modern data stack (ELT, transformation, orchestration) into a single, type-safe tool for developers who work primarily in TypeScript/JavaScript ecosystems. Its primary value is enabling end-to-end, scheduled data workflows with robust type safety from the source to the application code, all without requiring external database servers or complex Python-based infrastructure.

Main Features

1. Unified Ingestion & Transformation Pipeline: Croft defines data assets as .ts or .sql files within a project folder. An ingest() function in TypeScript handles pulling data from APIs (like Stripe, GitHub) or files with built-in support for incremental updates. Transformation is done either via raw SQL queries on ingested tables or transform() functions in TypeScript. The tool manages the execution graph and runs these assets in the correct dependency order.
2. End-to-End Type Safety with Automatic Generation: This is Croft's flagship feature. It automatically generates TypeScript type definitions (.d.ts files) for every table in the pipeline, derived from the actual DuckDB schema. This creates a compile-time safety net: renaming a column in a SQL transformation will cause type errors in any downstream TypeScript code that references the old column name before the pipeline even runs, catching breaking changes early.
3. Declarative Scheduling & Local-First Storage: Pipelines are scheduled using plain English phrases like schedule: "every hour" directly in the asset definition. The orchestration is handled by the Croft CLI. All data persists in a single DuckDB file on the local machine, eliminating the need for a separate database server or cloud data warehouse for storage. The @zabaca/croft/read package provides a typed client for applications to query this local DuckDB file.

Problems Solved

1. Pain Point: It solves the complexity and context-switching overhead of managing a multi-tool data stack (e.g., Airbyte/Fivetran for ingestion, dbt for transformation, Airflow for orchestration, a cloud warehouse for storage). It specifically addresses the lack of type safety between SQL transformations and application business logic, a common source of runtime errors.
2. Target Audience: The primary personas are full-stack or backend developers in TypeScript/Node.js/Bun environments (e.g., SaaS startup teams, web agencies) who need to build internal data pipelines, product analytics, or sync external API data into their applications. It's also highly relevant for data-savvy frontend developers and teams prioritizing a simple, self-contained tool over a scalable but complex warehouse-centric setup.
3. Use Cases: Essential for building internal dashboards that pull from third-party APIs, syncing operational data (e.g., Stripe transactions, GitHub issues) into an application's admin panel, creating scheduled reports from log files, and prototyping data products quickly without infrastructure commitment.

Unique Advantages

1. Strengths & Limitations (Pros & Cons):

   • Pros: Exceptional developer experience with full-type safety; drastically simplified architecture (one tool, one file); no external infrastructure or ongoing costs; seamless integration with TypeScript/Node.js/Bun ecosystems; built-in safety features like atomic updates and a trash system; agent-friendly design for AI-assisted development.
   • Cons: It is inherently limited by the local DuckDB file, making it unsuitable for team collaboration on the same data or big data workloads that exceed local machine capacity. It does not integrate with or migrate existing dbt or Airflow projects. Currently limited to macOS and Linux, excluding Windows users.

2. Key Alternatives & Differentiation:

   • dbt + Cloud Warehouse (Snowflake/BigQuery): The industry standard. dbt is more powerful for complex SQL transformations and is built for team collaboration on a central cloud data warehouse. Croft differentiates by being free, local-first, and providing type safety that bridges SQL and application code, which dbt does not.
   • Airbyte / Meltano + Dagster/Airflow: A common orchestration-heavy stack. These tools are more focused on robust, scalable ingestion and orchestration. Croft differentiates by bundling ingestion, transformation, and scheduling into a vastly simpler, developer-centric tool with no YAML configuration and a focus on type-safe application integration.
   • Direct Scripting (Custom Python/Node.js scripts): Many developers write one-off scripts. Croft differentiates by providing a structured framework with scheduling, dependency management, type generation, and data integrity guarantees that ad-hoc scripts lack.

Frequently Asked Questions (FAQ)

1. Can Croft replace my existing data warehouse like Snowflake? No, Croft is not a direct replacement for a cloud data warehouse. It is designed for local, single-machine data pipelines using DuckDB. It is ideal for individual developers or small teams building internal tools, whereas Snowflake is built for petabyte-scale data and large-team collaboration.
2. How does Croft handle data validation and testing? Croft incorporates checks that run before data is saved. In SQL assets, you can add -- check: comments with conditions (e.g., -- check: revenue >= 0). Furthermore, its core type-generation system acts as a powerful compile-time validation layer, ensuring column names and types are consistent across the entire pipeline.
3. Is Croft suitable for production use in a business application? Croft is suitable for production in specific scenarios: powering internal admin tools, dashboards, or background jobs within a Node.js/Bun application where the data volume fits comfortably on a single server. It is not suitable for customer-facing analytics features at scale or for centralizing data across a large organization.
4. What happens if my local DuckDB file gets corrupted or deleted? Since Croft is local-first, data safety is your responsibility. The tool's design encourages treating the pipeline definitions (the .ts/.sql files) as the source of truth. For irreplaceable data, you must implement your own backup strategy for the DuckDB file. Croft's incremental ingestion and trash system help mitigate accidental data loss during pipeline operations.
5. Does Croft support connecting to databases as a source or destination? Based on its documentation, Croft's primary ingestion sources are HTTP APIs and local files (like CSV). Its destination is strictly its internal DuckDB file. It is not an ETL tool for syncing data between different SQL databases. You would use its ingest() function to pull from a database's API, if available.