Data
Data collections turn files and records from external systems into reusable, searchable knowledge for Siesta AI agents. This section explains the supported Data sources in the Web App and what belongs in each field.
The Data Model
- A connection stores authentication for an external system.
- A data collection is the access-controlled business container.
- A data source defines exactly what to import through a connection.
- Documents and chunks are the processed content that retrieval can search.
- An agent retrieves from one or more assigned collections.
Credentials belong in Connections. Folder IDs, paths, blob selectors, project or space keys, and URLs belong in the Data-source form.
Start Here
| Goal | Documentation |
|---|---|
| Compare all sources and decide where content belongs | Choose a Data Source |
| Review discovered files before indexing | Discovery mode |
| Upload a controlled file snapshot | Manual Upload |
| Set the default or user-specific Data upload allowance | Data Upload Limits |
| Synchronize Google folders or Shared Drives | Google Drive |
| Synchronize a user’s OneDrive folders | OneDrive Data Sources |
| Synchronize a team’s SharePoint site or library | SharePoint Data Sources |
| Read blobs or Azure file shares | Azure Storage |
| Configure an Azure File Share source | Azure File Share |
| Use files from a Windows server | Azure File Share Data Sources |
| Import Jira projects or Confluence spaces | Atlassian |
| Scrape one page or crawl a bounded website | Firecrawl |
| Configure processing, retrieval, sync, and diagnose failures | Processing, Sync, and Troubleshooting |
When To Use Data
Use a collection when several users or agents should reuse the same approved source material, content should synchronize, processing and logs must be inspectable, or the knowledge needs a private, team, or organization access boundary.
For a single temporary file, a chat attachment can be simpler. For durable knowledge, use a collection.
Create a Collection
Open Data and select Create collection.
- Name: use a durable business name such as
Customer Support — Approved Knowledge. - Description: record content scope, owner, exclusions, and intended agents.
- Visibility: choose one of three access scopes:
- Private: only you and people you explicitly grant access to can use the collection.
- Entire organization: all organization members receive the permission selected under Organization permission.
- Selected teams: only members of the teams added under Team Access receive access.
The collection is the access boundary. Do not mix confidential HR documents and public product documentation merely because both are stored in Google Drive.
Add A Data Source
Open a collection and select Add data source. Manual Upload is independent of Connections. Every integration-backed source first needs a compatible entry under Connections.
All connected-source forms share:
- Name identifying this specific import,
- optional Description with owner, scope, and exclusions,
- Connection ID selecting stored authentication,
- Sync frequency: On Demand, Daily, Weekly, or Monthly.
If Connection ID is empty, create or request access to the correct connection and return to the collection. Never paste a secret into this selector.
Choose the source, then configure its scope and processing profile. Discovery mode describes processing profiles, update states, estimates, and approval before indexing.
The default Quality profile enables Advanced content extraction for Manual Upload, Google Drive, OneDrive, SharePoint, Azure File Share, and Azure Storage Account. It uses Azure Document Intelligence, billed per analyzed PDF page. Economy disables it; Custom lets you choose. Review the technical extraction path and analyzed-page allowance before applying it to a large source.
Review Discovered Files
Source creation starts discovery. In the Web App, Auto-process discovered files is off by default, so the update waits for approval before its eligible documents are indexed. Discovery and token/page estimates can read source content; they are not completed indexing.
Open Overview → Updates, review the summary and Skipped Data, then Approve or Reject the selected update. With automatic processing enabled, eligible files continue after discovery without that manual decision. The profile, auto-processing setting, and sync schedule are separate choices.
After approval, use Files to inspect document status, extracted chunks, and original previews when available. The detail tabs are Overview, Configuration, and Files; update history and approval actions are under Overview. The Discovery mode reference explains update states, estimates, and approval behavior.
Store Source Files
Store source files controls whether Siesta AI keeps a copy of each original remote file in addition to the representations required for retrieval. Configure it when you create the source; it is not the same as uploading a file to a chat or publishing an artifact.
| Source | Creation behavior |
|---|---|
| Manual Upload | Original uploaded files are stored. The creation request always enables source-file storage and does not show an optional switch. |
| Google Drive | Optional Store source files switch under Processing; off by default. |
| OneDrive | Optional switch under Processing; off by default. |
| SharePoint | Optional switch under Processing; off by default. |
| Azure File Share | Optional switch under Processing; off by default. |
| Azure Storage Account | Optional switch under Processing; off by default. |
These three representations serve different purposes:
- the original source file preserves the downloaded binary object when storage is enabled;
- extracted text and processing metadata make the document readable and record what the parser produced;
- chunks and the retrieval index are the searchable units used to retrieve relevant context for an agent.
Turning the option off for a remote source does not mean that the source is ignored. Siesta AI can still discover, parse, chunk, embed, and index supported content. It means that a separate retained copy of the remote original is not requested. Turning it on therefore adds storage for the original binaries on top of the processed retrieval representation; estimate the impact from the size and change rate of the selected folders, libraries, shares, or blobs.
After creation, the source Overview shows Source files stored when a non-manual source was created with the option enabled. The badge reports the stored configuration; use Files, document status, and Overview → Updates to prove that individual documents were successfully processed.
If creation fails after the preliminary data-source record is made, the service schedules deletion of the corresponding RAG source and rolls back the preliminary record. This is creation-failure cleanup, not a published retention schedule. Do not infer how long successful originals are retained, or when remote deletions remove stored copies, unless your organization's deployed policy states it separately.
Choose the option deliberately:
- enable it when workflows need the original binary, an exact downloadable source copy is operationally required, or a retained source copy is needed for document previews;
- leave it off when retrieval is the only requirement and duplicating a large remote corpus would create unnecessary storage, governance, or residency exposure.
Because the setting is part of source creation rather than the editable Configuration tab, confirm it before creating a large source. If a different policy is required later, validate the migration or replacement procedure in the deployed environment before changing production ingestion.
Inspect Before Agents Use It
Check Overview → Updates, representative files, Indexed and Readable state, and extracted chunks. A waiting update needs an approval decision before its files can proceed. A successful run proves that processing completed; it does not prove that every expected document is present, useful, current, or safe for the collection audience.
For an organization-wide inventory view, open Analytics > Data. It can expose an unexpectedly small collection, missing source type, dominant source, or unusual file mix. Treat RAG Data Ingested Over Time by User as the current indexed footprint grouped by first discovery date and source creator—not as a count of sync runs or reprocessed bytes. Return to the collection, document status, and Overview → Updates for the processing-level explanation.