Govern Data Collections and Sources

Administrators use Data to govern the boundary between external systems, stored credentials, indexed content, collection access, and the agents that use that content. A source working technically is not enough: it must also have a clear owner, approved scope, refresh policy, and retirement path.

Build the First Pilot Data Pack

Start with one coherent set of documents for one use case. For example, a Tender Review — Pilot Pack should contain only the approved tender documents that the pilot agent is expected to review, not every document the bid team can access.

1. In Data, create a collection named for the use case and keep it Private. Record the business owner, intended agent, audience, content version, and retirement or replacement rule in the description.
2. Choose Manual Upload for a controlled snapshot, or a supported synchronized source when the authoritative content changes regularly. Do not assume that an internal or observed endpoint is a supported upload API; use API ingestion only when the deployed version has a published REST contract for the endpoint, authentication, limits, and failure behavior.
3. Add only documents with the same purpose, audience, and classification. Remove credentials, obsolete versions, and unrelated material before ingestion.
4. For the initial run, leave Advanced content extraction off. Enable it only when representative scanned, table-heavy, or figure-heavy documents fail normal extraction and the owner accepts the analyzed-page cost and allowance.
5. Before a large first ingestion, name an ingestion owner. That person owns the expected file count, upload limits, processing window, cost-sensitive settings, failed-file review, and coordination with the internal platform team or Siesta AI delivery contact.
6. Wait for processing, then review Overview, Files, representative extracted chunks, and Logs. Confirm the expected files are present and have usable Indexed and Readable states; investigate skipped, unreadable, unindexed, or Failed files.
7. After the processing checks pass, attach the collection to the private pilot agent through its Data connection configuration.
8. Through that agent, test one known answer, one answer that is absent, and one stale or contradictory case. Require the result to point back to a traceable source document rather than accepting a plausible answer.

For onboarding, prefer one agent with one coherent pilot pack because the purpose, owner, and access boundary are easy to inspect. This is an operating recommendation, not a platform invariant. Split material into separate collections when ownership, confidentiality, refresh, or retention differs; attach multiple collections only when the agent genuinely needs them and their audiences are compatible.

Ingestion creates processed, searchable content and is not a realtime query. If the use case needs the current state of Power BI or another live system, continue with Tools, APIs, and MCP access instead of treating a data collection as a live integration.

For supported remote sources, decide during creation whether original binaries must also be retained. The Store source files option is separate from parsing and indexing and is off by default for Google Drive, OneDrive, SharePoint, Azure File Share, and Azure Storage Account; Manual Upload stores originals. Record the business reason, storage/residency impact, and owner for any enabled source. The exact behavior and creation-failure cleanup boundary are documented in Store Source Files.

The Data Guide

1. Configure data upload limits to establish sensible defaults and per-user exceptions before ingestion begins.
2. Configure enabled connections to control which integrations are approved for your organization.
3. Choose the best data source for your use case before storing or synchronizing business content.
4. Design a data collection architecture for your teams so ownership and access match how your organization operates.
5. Create and populate your first data collection using the approved scope and access model.
6. Monitor data-storage utilization to track consumption and address capacity risks early.

Use these pages as the implementation checklist; use this Admin Guide to define identity ownership, least privilege, collection access, review, monitoring, and retirement.

For every source, keep these objects separate:

• Connection: authentication and provider identity.
• Data source: the folder, path, blob selector, share, project, space, or URL being ingested.
• Collection: the access-controlled business grouping.
• Agent assignment: which production behaviors can retrieve from the collection.

Choose Direct Context, RAG, or a Hybrid Pipeline

Use the following ranges as operating heuristics, not product limits. Count the usable parsed content, not only the source files: fewer than 20 long documents can exceed 100k tokens and require retrieval, while many short records may fit comfortably. Confirm the choice with an evaluation set before production.

Corpus guideline	Starting approach	Why, tradeoff, and administrator action
Approximately 1–20 files and no more than 100k tokens	Direct context	Best for close reading, complete-document comparison, and reasoning that depends on seeing the whole small corpus. It avoids retrieval misses, but context cost and latency grow with every document. Verify the model context window and load only approved content.
Approximately 21–49 files	Decide by task	Use direct context when the task needs most documents at once and the token total fits. Use RAG or a hybrid approach for pinpoint questions, sparse evidence, frequent updates, or mixed access scopes. Test both on representative questions rather than choosing by file count alone.
Approximately 50–199 files	Start using RAG	Retrieval reduces context size and cost, but can omit evidence needed for broad comparisons. Define metadata, build known-answer and missing-answer tests, and inspect retrieved chunks before rollout.
Approximately 200–999 files	Prefer RAG	Loading the corpus directly is usually inefficient and increases noise. Use metadata filtering and hybrid lexical/vector retrieval with reranking; monitor relevance, latency, and stale or duplicate content.
1,000 or more files	RAG is practically necessary	The corpus is too large for routine direct loading. Partition it by ownership and access, require metadata quality, evaluate retrieval continuously, and set operational alerts for ingestion and retrieval failures.

Task shape can override the starting range. For comparisons or reasoning across many documents, retrieve candidates first and then load the complete relevant documents when they fit the context window. For exact identifiers and semantically phrased questions in the same collection, use the recommended hybrid retrieval pipeline.

Plan Synchronization and Failure Handling

Choose a synchronization frequency from the maximum acceptable staleness of the content, not from corpus size alone. Daily or Weekly is the frequency at which a run starts; it is not a request to parse and reindex every file. A run enumerates the complete source scope for discovery, then processes only new, changed, and previously Failed files. Successfully processed unchanged files and files marked IsUserDeleted are skipped. The incremental reingestion reference defines the complete state matrix.

Before production, record the source owner, schedule, expected document-count range, acceptable staleness, and escalation owner. After each planned run, review Overview for last and next synchronization, Logs for discovered and processed counts, and Files for Failed, skipped, unreadable, or unindexed documents.

A previously Failed file is retried on the next run. Repeated failure is not a reason to shorten the schedule: inspect provider access, the selector, supported format, extraction, size, and logs, correct the cause, and verify the file after the next run. Faster schedules increase provider enumeration and processing load and can expose unreviewed upstream changes sooner. Follow Processing, Sync, and Troubleshooting for detailed inspection and incident response.

Admin Design Checklist

Before creating a production source, decide:

• Who owns the upstream content?
• Which service or user identity will authenticate?
• Which exact folders/projects/spaces/paths are in scope?
• Who may use the resulting collection?
• Who may edit, move, synchronize, or delete its sources?
• How quickly must upstream changes appear?
• How will you test retrieval quality and missing-answer behavior?
• What happens when the credential owner leaves or the source is retired?