Data Sources

Read from databases, APIs, local files, or scrape the web to pull data into the pipeline.

Overview

The Data Sources group holds nodes that pull data into a workflow from databases, external APIs, local files, code repositories, and the web. Unlike Manual Input, these nodes fetch live data each time the workflow runs.

This group spans three underlying node categories — Source, Storage, and Repository — each shown with its own color-coded label in the palette.

Nodes in This Group

Source

Node	Description
Relational DB (SQL)	Query a relational database (e.g. PostgreSQL, MySQL) and output the result set.
Document DB (NoSQL)	Query a document-oriented database (e.g. MongoDB) and output matching documents.
Key-Value DB	Read one or more values from a key-value store.
Vector DB	Query a vector database by similarity search and output matching records.
API Source	Fetch data from an external HTTP endpoint; output is the response body (string or JSON).
Local Drive	Read a file from the server's local/persistent disk storage.
Web Scraper	Fetch and extract content from a public web page URL.
Define Schema	Declares an expected data shape for validation or downstream mapping, without fetching data itself.

Storage

Node	Description
File Read	Reads the contents of an uploaded or referenced file.
PDF Generator	Generates a PDF document from upstream content.
Image Processor	Loads and prepares an image file for downstream processing.

Repository

Node	Description
GitHub	Reads repository data (files, issues, pull requests) from a connected GitHub account.
Bitbucket	Reads repository data from a connected Bitbucket account.
GitLab	Reads repository data from a connected GitLab account.

Usage Tips

Pair with Validation & Security

When pulling data from an external API or database, connect the output to a Validation node (Schema Validation, Integrity Check) before it reaches business logic — this catches shape or quality issues early in the pipeline.

Credentials

Database and repository nodes require a connection configured in the node's settings panel (host, credentials, or an OAuth connection for GitHub/GitLab/Bitbucket). Connections are stored per tenant and reused across workflows.

Frequently Asked Questions

What is the difference between API Source and a node in the Integrations group?

API Source is a generic read-only HTTP fetch meant to pull data at the start of a pipeline. Integrations group nodes (Slack, Email, HTTP Request, etc.) are typically used mid-pipeline or at the end to send data out, though some can also read data from a specific third-party service.

Does Web Scraper respect robots.txt?

Web Scraper is intended for scraping pages you have the right to access programmatically. Always confirm you have permission to scrape a target site before using this node in a production workflow.