Seeknal

Product Introduction

1. Definition: Seeknal is an open-source, all-in-one Command Line Interface (CLI) and orchestration platform specifically engineered for data and AI/ML engineering. It functions as a unified framework for building data pipelines, managing feature stores, and deploying AI agents that interact directly with structured data. Technically, it bridges the gap between raw data infrastructure (PostgreSQL, Apache Iceberg) and the semantic layer, utilizing DuckDB for local processing and LLMs (Gemini, OpenAI, Ollama) for natural language reasoning.

2. Core Value Proposition: Seeknal exists to eliminate the fragmentation in modern data stacks by unifying transformation, feature serving, and insight generation into a single toolset. Its primary value lies in the "Organize, Expose, Action" framework, which allows engineers to define pipelines in YAML or Python, materialize data into high-performance formats, and immediately query that data using natural language. It is built for the "agent world," ensuring that data is not just stored but is "agent-ready" with built-in lineage, metadata, and semantic definitions.

Main Features

1. Dual Pipeline Authoring & Safe Workflow: Seeknal allows developers to define data transformations using a hybrid approach: declarative YAML for clarity and version control, or imperative Python decorators for complex logic. These definitions are compiled into a single execution graph. The platform enforces a rigorous four-stage lifecycle: Draft (scaffolding), Dry-run (plan compilation and conflict detection), Apply (incremental execution and state management), and Run (scheduling and execution). This "git-like" workflow ensures no silent failures and provides a 0-wasted-run environment by using watermark-based detection and snapshot-based incrementals.

2. Integrated ML Feature Store & Semantic Layer: The platform features a built-in feature store capable of managing both offline batch processing and online real-time serving via a TTL flag. It supports critical ML engineering requirements such as point-in-time joins, entity consolidation, and automatic versioning. The semantic layer allows for the definition of entities, metrics, dimensions, and time grains as reusable concepts, providing a single source of truth that the AI agent uses to understand the underlying data schema.

3. Agentic Natural Language Query & Ingest (Seeknal Ask): Seeknal integrates a sophisticated AI agent that utilizes Gemini, OpenAI, or local Ollama instances to perform "grounded" data analysis. Users can query data in natural language, and the agent utilizes the platform's schema knowledge and tools to plan joins, rank data, and generate reports. Furthermore, it supports "conversational ingest," where users can drop .xlsx, .csv, .json files, or even screenshots of documents into the chat. The agent automatically previews the schema, identifies business keys, and appends the data to the appropriate ingest tables with full provenance tracking.

Problems Solved

1. Pain Point: Tool Sprawl and Pipeline Fragility. Traditional data stacks require separate tools for orchestration, transformation, feature stores, and BI. This leads to complex integrations and "silent failures" where data moves through pipelines without proper validation. Seeknal solves this by providing a unified CLI that handles everything from raw ingestion to API delivery with built-in data checks and column-level lineage.

2. Target Audience: The primary users are Data Engineers, AI/ML Engineers, and Analytics Engineers who need to bridge the gap between data warehousing and AI application development. It also serves Data Scientists who require a streamlined way to move from local Python experimentation to production-grade materialized features.

3. Use Cases:

• Building retail analytics pipelines that require joining disparate PostgreSQL and Iceberg tables with point-in-time accuracy.
• Creating an "Ask Your Data" interface for internal teams where the AI agent is restricted to a sandboxed Python environment for safety.
• Automating insight-to-action workflows, such as triggering Telegram alerts or publishing REST API endpoints based on specific data triggers or anomalies detected in the pipeline.

Unique Advantages

1. Differentiation: Unlike traditional transformation tools like dbt, Seeknal is natively designed for the AI agent era. It doesn't stop at materializing tables; it exposes those tables via a gateway as APIs, WebSocket streams, or interactive reports. Its ability to handle both YAML and Python within the same graph provides flexibility that purely SQL-based or purely code-based tools lack.

2. Key Innovation: The "Safe Workflow" (Draft -> Dry-run -> Apply) combined with "Agentic Ingest" is its most significant innovation. It treats data engineering as a high-fidelity software engineering discipline, providing a plan-first execution model that shows exactly which rows will move and which conflicts exist before execution. The automated generation of "SKILL.md" files during ingest allows the AI agent to learn and repeat complex data handling tasks autonomously.

Frequently Asked Questions (FAQ)

1. How does Seeknal ensure data security when using LLMs for queries? Seeknal is designed with a "Private by Default" architecture. Users can run the platform fully local using Ollama, ensuring data never leaves the machine. When using cloud providers like Gemini or OpenAI, Seeknal uses sandboxed execution for Python and restricted tool access. Every write and query emits a provenance JSON sidecar with SHA-256 hashes and drift decisions for auditability.

2. Does Seeknal support incremental data loading and materialized views? Yes. Seeknal is "incremental by default." It uses watermark-based detection for relational databases like PostgreSQL and snapshot-based detection for Apache Iceberg. If the data fingerprints match, the execution engine skips the step entirely. It supports materialization to both PostgreSQL (for operational use) and Iceberg (for analytical scale) simultaneously.

3. Can I use Seeknal with my existing Python and SQL scripts? Absolutely. Seeknal's dual pipeline authoring supports standard SQL via the DuckDB engine and imperative Python via decorators. It can auto-register Parquet files, PostgreSQL tables, and Iceberg catalogs, allowing you to use the Interactive SQL REPL to iterate on existing data without leaving the terminal.