Govern Data Collections and Sources

Administrators use Data to govern the boundary between external systems, stored credentials, indexed content, collection access, and the agents that use that content. A source working technically is not enough: it must also have a clear owner, approved scope, refresh policy, and retirement path.

Build the First Pilot Data Pack

Start with one coherent set of documents for one use case. For example, a Tender Review — Pilot Pack should contain only the approved tender documents that the pilot agent is expected to review, not every document the bid team can access.

1. In Data, create a collection named for the use case and keep it Private. Record the business owner, intended agent, audience, content version, and retirement or replacement rule in the description.
2. Choose Manual Upload for a controlled snapshot, or a supported synchronized source when the authoritative content changes regularly. Do not assume that an internal or observed endpoint is a supported upload API; use API ingestion only when the deployed version has a published REST contract for the endpoint, authentication, limits, and failure behavior.
3. Add only documents with the same purpose, audience, and classification. Remove credentials, obsolete versions, and unrelated material before ingestion.
4. Review the Processing profile. The app defaults to Quality, including advanced PDF extraction for supported file sources. Choose Economy for a compact pilot, or Custom to disable extraction while retaining larger vectors. Keep Auto-process discovered files off so the ingestion owner can review the first update.
5. Before a large first ingestion, name an ingestion owner. That person owns the expected file count, upload limits, processing window, cost-sensitive settings, failed-file review, and coordination with the internal platform team or Siesta AI delivery contact.
6. Review Overview → Updates and Skipped Data, compare counts and estimates with the approved scope, then approve the selected update. After processing, inspect Files and representative extracted chunks. Confirm usable Indexed and Readable states and investigate skipped, unreadable, unindexed, or failed files. Follow Discovery mode for the review steps.
7. After the processing checks pass, attach the collection to the private pilot agent through its Data connection configuration.
8. Through that agent, test one known answer, one answer that is absent, and one stale or contradictory case. Require the result to point back to a traceable source document rather than accepting a plausible answer.

For onboarding, prefer one agent with one coherent pilot pack because the purpose, owner, and access boundary are easy to inspect. This is an operating recommendation, not a platform invariant. Split material into separate collections when ownership, confidentiality, refresh, or retention differs; attach multiple collections only when the agent genuinely needs them and their audiences are compatible.

Ingestion creates processed, searchable content and is not a realtime query. If the use case needs the current state of Power BI or another live system, continue with Tools, APIs, and MCP access instead of treating a data collection as a live integration.

For supported remote sources, decide during creation whether original binaries must also be retained. The Store source files option is separate from parsing and indexing and is off by default for Google Drive, OneDrive, SharePoint, Azure File Share, and Azure Storage Account; Manual Upload stores originals. Record the business reason, storage/residency impact, and owner for any enabled source. The exact behavior and creation-failure cleanup boundary are documented in Store Source Files.

The Data Guide

1. Configure data upload limits to establish sensible defaults and per-user exceptions before ingestion begins.
2. Configure enabled connections to control which integrations are approved for your organization.
3. Choose the best data source for your use case before storing or synchronizing business content.
4. Design a data collection architecture for your teams so ownership and access match how your organization operates.
5. Create and populate your first data collection using the approved scope and access model.
6. Monitor data-storage utilization to track consumption and address capacity risks early.

For Microsoft content, use the separate OneDrive and SharePoint source guides. For Windows folders copied through Azure, follow Azure File Share Data Sources.

Use these pages as the implementation checklist; use this Admin Guide to define identity ownership, least privilege, collection access, review, monitoring, and retirement.

For every source, keep these objects separate:

• Connection: authentication and provider identity.
• Data source: the folder, path, blob selector, share, project, space, or URL being ingested.
• Collection: the access-controlled business grouping.
• Agent assignment: which production behaviors can retrieve from the collection.

Source architecture and credential governance

Choose the system containing the authoritative version, not merely an existing Connection. Use Manual Upload for a controlled snapshot; Google Drive for shared Google folders; OneDrive for user-owned files; SharePoint for team libraries; Azure Storage Account for blob exports; Azure File Share for SMB-style repositories; Jira for project issues; Confluence for maintained spaces; and Firecrawl for approved websites.

For Google and Microsoft OAuth sources, use a dedicated or team identity for shared production data, limit its access to the necessary folders or libraries, record its owner and recovery process, and test reauthorization. For Azure, keep connection strings, account keys and SAS secrets in the Connection, not collection descriptions or selectors. Rotate credentials in the Connection and verify synchronization afterward. Blob and File Share sources remain distinct. The Azure Storage form checks that selectors are present, so inspect the selected files and results after creation.

For Jira and Confluence, use a service account and store its URL, identity and API token in Connections; the source receives a project or space key. For Firecrawl, store the API endpoint and key in Connections and approve each domain, starting URL, limits and include/exclude rules.

Collection access and structure

Collections provide the access boundary after ingestion. Private, Entire organization, and Selected teams offer different sharing scopes. Creating, editing, moving, or deleting a source requires write access to its parent collection. Separate Can Use from edit access and review both provider and Siesta AI access.

Assign business and technical owners, grant team use access to consumers, reserve editing for trained administrators, and share organization-wide only when appropriate. Split collections when audience, confidentiality, ownership, source system, schedule, retention, agents, or retrieval needs differ. Choose durable collection names based on the business purpose; source names can describe their implementation.

Source-specific checks

• Google Drive: verify folder IDs with the connected account, review recursive scope and Shared Drives, and remove unrelated shortcuts or duplicate versions.
• OneDrive and SharePoint: use OneDrive for user-owned content and SharePoint for team libraries; record exact paths and give the service identity direct access.
• Azure Storage: document the account, container and selector; use blob selectors in Blobs, inspect Files and Overview → Updates, and separate sources with different access or retention.
• Azure File Share: specify exact share names, use a valid connection secret, and verify actual files. See Azure File Share Data Sources for Windows-to-Azure transfer.
• Jira and Confluence: each source uses one project or space key. Split sources by team, sensitivity or retention and retest after service-account permissions change.
• Firecrawl: approve domain and content rights; use Scrape for one page or restrict Crawl scope with conservative limits and include/exclude rules.

Upload limits and processing approval

Manage default and individual upload limits under Admin → Limits → Data limits. Review exceptions separately from collection sharing and source scope.

The Quality profile starts with 3072 dimensions and advanced PDF extraction for six supported file sources; Economy uses 1536 dimensions, Int8 and no advanced extraction. Custom allows other combinations. Review the processing profiles and PDF page costs for the deployment and use case.

For the first run, leave automatic processing off and have the owner review Overview → Updates, including skipped files, before approval. Use relevant file types and path exclusions. Some integrations pass ChunkSize: 700 and ChunkOverlap: 200; actual chunking depends on the deployed RAG pipeline. Changes to vectors or chunking may require reprocessing.

Acceptance, monitoring and retirement

For production acceptance, verify approved discovery scope, document counts and names, readable chunks and Indexed/Readable states. Test present and absent answers, citations, unauthorized access, and a controlled upstream change. Review failed and skipped entries, then record the owner, schedule, review date and retirement procedure. For Azure File Share test new, modified and deleted upstream files; deleting an Azure file alone does not establish removal from the collection.

Monitor collection health, Files, Overview → Updates, agent conversations and audit records. For a material incident, detach the collection from production agents if needed, preserve non-sensitive diagnostic context, distinguish provider access, selector, sync, parsing, indexing and retrieval failures, and test the fix before restoring service.

Deleting a document removes its indexed content; deleting a source schedules retrieval-data removal; deleting a collection removes its sources and agent links. Before deletion identify dependent agents, preserve required audit evidence, confirm replacement content, notify the business owner and test the agent without the source.

Choose Direct Context, RAG, or a Hybrid Pipeline

Use the following ranges as operating heuristics, not product limits. Count the usable parsed content, not only the source files: fewer than 20 long documents can exceed 100k tokens and require retrieval, while many short records may fit comfortably. Confirm the choice with an evaluation set before production.

Corpus guideline	Starting approach	Why, tradeoff, and administrator action
Approximately 1–20 files and no more than 100k tokens	Direct context	Best for close reading, complete-document comparison, and reasoning that depends on seeing the whole small corpus. It avoids retrieval misses, but context cost and latency grow with every document. Verify the model context window and load only approved content.
Approximately 21–49 files	Decide by task	Use direct context when the task needs most documents at once and the token total fits. Use RAG or a hybrid approach for pinpoint questions, sparse evidence, frequent updates, or mixed access scopes. Test both on representative questions rather than choosing by file count alone.
Approximately 50–199 files	Start using RAG	Retrieval reduces context size and cost, but can omit evidence needed for broad comparisons. Define metadata, build known-answer and missing-answer tests, and inspect retrieved chunks before rollout.
Approximately 200–999 files	Prefer RAG	Loading the corpus directly is usually inefficient and increases noise. Use metadata filtering and hybrid lexical/vector retrieval with reranking; monitor relevance, latency, and stale or duplicate content.
1,000 or more files	RAG is practically necessary	The corpus is too large for routine direct loading. Partition it by ownership and access, require metadata quality, evaluate retrieval continuously, and set operational alerts for ingestion and retrieval failures.

Task shape can override the starting range. For comparisons or reasoning across many documents, retrieve candidates first and then load the complete relevant documents when they fit the context window. For exact identifiers and semantically phrased questions in the same collection, use the recommended hybrid retrieval pipeline.

Plan Synchronization and Failure Handling

Choose a synchronization frequency from the maximum acceptable staleness of the content, not from corpus size alone. Daily or Weekly is the frequency at which a run starts; it is not a request to parse and reindex every file. A run enumerates the configured scope, estimates new, changed, and previously failed files, and waits for approval unless automatic processing is enabled. Rejected files and files released from a removed path exclusion can also return in a later discovery. Successfully processed unchanged files and files marked IsUserDeleted are skipped. The incremental reingestion reference defines the complete state matrix.

Before production, record the source owner, schedule, expected document-count range, acceptable staleness, and escalation owner. After each planned run, review Overview → Updates for discovery, approval, and processing outcomes, and Files for failed, skipped, unreadable, or unindexed documents. Assign someone to handle waiting updates so a scheduled discovery does not remain unapproved.

A previously failed file can be retried through the next discovery and approval flow. Repeated failure is not a reason to shorten the schedule: inspect provider access, the selector, supported format, extraction, size, and logs, correct the cause, and verify the file after the next run. Faster schedules increase provider enumeration and processing load and can expose unreviewed upstream changes sooner. Follow Processing, Sync, and Troubleshooting for detailed inspection and incident response.

Admin Design Checklist

Before creating a production source, decide:

• Who owns the upstream content?
• Which service or user identity will authenticate?
• Which exact folders/projects/spaces/paths are in scope?
• Who may use the resulting collection?
• Who may edit, move, synchronize, or delete its sources?
• How quickly must upstream changes appear, and who approves discovered updates?
• Which processing profile and PDF-page allowance are approved?
• Which file path exclusions prevent obsolete material from entering future runs?
• How will you test retrieval quality and missing-answer behavior?
• What happens when the credential owner leaves or the source is retired?