Processing, Sync, and Troubleshooting
Source scope decides what enters the collection. Processing decides how it becomes searchable. Retrieval decides which chunks reach an agent. Review the processing profile before creating a source, then validate it with representative files and questions. Discovery mode describes source settings, update states, estimates, and approval behavior.
When search succeeds, internal chat can preserve structured links to the returned Data documents. These references identify the collection, source, and document but do not bypass permissions or replace processing evidence. See Documents Used in an Answer, then return here when a referenced document is missing, stale, unreadable, or retrieves poorly.
Advanced PDF Content Extraction
Advanced content extraction is the PDF-specific processing path for documents whose meaning depends on page layout, tables, or figures. When the switch is enabled and the discovered document is a PDF, Siesta AI sends it to Azure Document Intelligence with the prebuilt-layout model and requests Markdown output plus figure data. Other supported file types continue through their normal readers; enabling the switch does not send Word, Excel, PowerPoint, JSON, or text files through this PDF analysis path.
The analysis produces three kinds of retrieval input:
- the document's layout-aware text in Markdown;
- extracted table elements, including their table markup and header context;
- extracted figure elements with image data when the service returns a matching figure.
During chunking, Siesta AI places tables and figures near their original position in the document, generates a textual description for each supported element, and embeds those descriptions as searchable chunks. This can improve retrieval for a value stored in a complex table or for meaning carried by a diagram. It does not guarantee that every visual detail is interpreted correctly, so inspect representative chunks and test questions whose answers depend on both tables and figures.
Supported sources and operating cost
The current PDF extraction path is implemented for Manual Upload, Google Drive, OneDrive, SharePoint, Azure Storage Account, and Azure File Share. Discovery estimates PDF pages locally without calling paid Document Intelligence. After approval, the whole PDF must fit the remaining analyzed-page allowance before paid analysis starts. A connector must still be able to discover and read the PDF; the switch does not bypass provider permissions, source scope, file filters, or connection failures.
Azure Document Intelligence is billed per analyzed page. The analyzed-page counter increases when analysis returns, even if the document is later skipped because of a filter, an unchanged hash, empty content, or another ingestion decision. Repeated synchronization of a PDF can therefore add analyzed pages without producing a new indexed version. Use Overview → Processed Data and Data Upload Limits to monitor this separately from stored megabytes.
When to enable it
The default Quality profile enables advanced extraction for all six supported file sources. Economy disables it. Turning it off while keeping Quality's vector settings makes the profile Custom.
Enable advanced extraction for scanned PDFs, layout-heavy reports, multi-column documents, complex tables, or diagrams that are important to retrieval. Turn it off for ordinary text PDFs when the standard reader already produces clean, complete text; advanced extraction adds processing time and per-page cost.
Pilot before enabling it for a large synchronized source:
- Process a representative text PDF, scanned PDF, table-heavy report, and figure-heavy report.
- Open Files and confirm
IndexedandReadablestate. - Inspect extracted chunks for reading order, headings, table values, and figure descriptions.
- Ask questions that require exact table values and visual context, then compare the answers with the source pages.
- Record the analyzed-page increase and estimate the cost of the planned synchronization frequency.
Troubleshooting advanced extraction
| Symptom | Check first |
|---|---|
| PDF is skipped with User analyzed pages limit exceeded | The owner's effective analyzed-page allowance and total analyzed pages across all of their Data sources |
| PDF is readable but tables or figures retrieve poorly | Extracted chunks, generated element descriptions, reading order, and a representative retrieval test |
| Analyzed pages increase but no new indexed version appears | Whether analysis completed before a later hash, filter, or content decision skipped the document |
| Only non-PDF files are present | Use their standard readers; Advanced content extraction is not applied to them |
| A synchronized PDF cannot be analyzed | Provider permission, selector or path, connection health, file readability, source Overview → Updates, and Azure processing availability |
If the analyzed-page allowance is exhausted, increasing the storage allowance alone does not help. The two quotas are enforced independently.
JSON Features and JSON Metadata Definitions
When a Data source imports JSON, these two settings describe the role of named fields:
| Setting | Practical question | Recommended use | Typical keys |
|---|---|---|---|
| JSON Features | From which content should the AI draw? | Meaningful text that should support retrieval and answers | title, description, summary, content, normalized comment text |
| JSON Metadata Definitions | How should the system identify, classify, or potentially filter the document? | Stable identifiers and controlled classification values | id, category, author, language, createdAt, customerId |
The fields are available in the current creation forms for Manual Upload, Google Drive, OneDrive, SharePoint, Azure Storage Account, and Azure File Share. Do not put the same key in both lists without a specific reason and a retrieval test.
Prepare the JSON contract
Before configuring either list, collect:
- one to three representative JSON objects,
- examples of questions that users will ask,
- the classifications or filters they expect to use,
- examples of nested, optional, or structurally inconsistent objects.
Prefer concise text with business meaning for features and stable, normalized scalar values for metadata. Exclude binary content, access tokens or signed URLs, internal processing values, duplicated text, and large objects without a clear retrieval benefit.
Nested objects, arrays, missing values, and a key that changes data type between records require a stable upstream contract. Flatten nested values into documented keys, normalize dates and identifiers, and combine useful arrays or comments into a deliberate text field when necessary. Do not assume that an untested nested path will be interpreted as intended.
Example configuration
Consider this representative object:
{
"id": "policy-1042",
"title": "Password reset policy",
"summary": "Rules for resetting an employee account password.",
"content": "After a reset, the employee must create a new password and confirm multifactor authentication.",
"category": "Identity",
"author": "Security Operations",
"language": "en",
"createdAt": "2026-08-01T09:30:00Z",
"_etag": "internal-revision-value"
}
Recommended values:
JSON Features: title, summary, content
JSON Metadata Definitions: id, category, author, language, createdAt
| Key | List | Reason and expected effect |
|---|---|---|
title | Features | Adds the document's subject and terminology to searchable content. |
summary | Features | Supplies a concise description that can improve retrieval for broad questions. |
content | Features | Contains the facts from which answers should be produced. |
id | Metadata | Provides a stable identifier for the logical record. |
category | Metadata | Classifies the record for review and intended category filtering. |
author | Metadata | Records the accountable source or team. |
language | Metadata | Provides a normalized language classification for intended filtering. |
createdAt | Metadata | Preserves a sortable, normalized timestamp when the RAG service supports it. |
_etag | Neither | It is an internal processing value with no user-facing retrieval benefit. |
This split should support a content question such as “What must an employee do after a password reset?” and is intended to support a filter such as “Identity documents in English.” Test both behaviors against the deployed RAG service before relying on them.
The frontend presents both settings as repeatable lists of field names and removes entries whose content is only whitespace before creating the Data source. The application backend does not verify that a name exists in the imported JSON and passes the received lists unchanged to the RAG client's JsonFeatures and MetadataDefinitions properties.
The application backend sends these values during Data-source creation. It does not store them separately in its own database, return them in the current Data-source detail, or accept them through the current update endpoint. Define and test the lists before creating a production source.
The application and backend do not establish exactly how the separate RAG service indexes selected features, applies metadata filters, resolves nested paths, handles missing or mixed-type values, or resolves a key present in both lists. Treat the recommendations above as a qualified starting point. Pilot representative records, inspect extracted documents and chunks, and test at least one content question and one intended metadata filter.
Shared Connected-Source Fields
Every integration-backed source requires Name, optional Description, Connection ID, and a sync frequency. Connection ID is a selector for stored authentication; it is never a field for a secret.
Sync Frequency
| Frequency | Use for |
|---|---|
| On Demand | Signed policies, reviewed releases, quarterly exports |
| Daily | Operational documentation, active projects, changing shared folders |
| Weekly | Maintained but non-urgent knowledge |
| Monthly | Slow-moving archives and reference material |
Manual Upload has no upstream system to synchronize. Faster is not automatically better: it uses provider and processing capacity and makes unreviewed upstream changes available sooner.
Processing Profiles
Every source-creation form starts with Quality. The profile sets vector storage and, for supported file sources, advanced PDF extraction. It does not change collection access, synchronization frequency, or the Auto-process discovered files switch.
| Profile | Vector size | Quantization | Advanced PDF extraction |
|---|---|---|---|
| Economy | 1536 dimensions | Int8 | Off for supported file sources |
| Quality | 3072 dimensions | None | On for supported file sources |
| Custom | Your selected values | Your selected value | Your selected value, where supported |
Manual Upload, Google Drive, OneDrive, SharePoint, Azure Storage Account, and Azure File Share support advanced PDF extraction. Jira, Confluence, and Firecrawl do not expose that switch. Selecting Quality for those three sources still selects 3072 dimensions and no quantization.
Custom opens Advanced settings for manual tuning. Switching only advanced extraction off while retaining 3072 dimensions and no quantization also produces a Custom configuration. Review the final values after changing profiles; choosing Economy or Quality replaces the settings controlled by that profile.
Use Economy when compact vectors and lower storage usage suit the tested content. Use Quality when the larger vectors and supported PDF extraction are useful. Neither profile guarantees a better answer for every document. Compare representative questions before changing a production ingestion design.
Other processing settings
Depending on the source, Processing also offers Allowed file types, Store source files, file path exclusions, and JSON Features and JSON Metadata Definitions. Advanced settings exposes the available vector and chunking controls. Supported remote file sources default Store source files to off; Manual Upload keeps originals.
Agent collection assignment controls which content the agent may use. Processing profiles do not replace retrieval-quality testing or grant access to another collection.
File Path Exclusions
Add one expression per row under File path exclusions (regex) when creating the source, or update it in Configuration. Patterns are .NET regular expressions matched anywhere in the full source path, including folders. Use / between folders. Matching is case-sensitive; add (?i) to ignore case.
| Example | Excludes |
|---|---|
(^|/)archive/ | Files under a folder named archive |
(?i)\.tmp$ | Paths ending in .tmp, regardless of case |
Leave the list empty to exclude nothing. These are regular expressions, not wildcard patterns such as *.tmp. An invalid expression cannot be saved. Matches are skipped before estimation and processing; inspect their reason in Files.
Changes apply to future runs. Removing an exclusion allows an unchanged file previously skipped by that expression to be discovered on the next run. Editing a filter is not a command to delete already indexed content.
Inspect Before Production
The source detail has Overview, Configuration, and Files tabs. In Overview, inspect Updates for run outcomes and approval actions; View details opens the update summary. Check Processed Data for indexed content metrics, and Skipped Data for the discovery exclusions. Inspect skipped rows under Files and open a row with Show. Use Configuration for editable source settings and Files for document status, original previews where available, and extracted chunks.
A Waiting for confirmation update needs approval unless automatic processing was enabled. Discovered and Estimated files are not evidence of completed indexing. Pending means processing is in progress; inspect both the final update and individual files.
Test:
- a known present answer,
- a known absent answer,
- conflicting versions,
- a recently changed item,
- an expected exclusion.
Troubleshooting Matrix
| Symptom | Check first |
|---|---|
| Waiting for confirmation | Review Overview → Updates and approve or reject the selected update |
| Discovery numbers differ from processed numbers | Discovery estimates, skips, later quota/content decisions, and the selected update |
| Connection ID is empty | Compatible connection exists and is visible or shared correctly |
| Successful source contains zero files | Selector/path/key spelling, account permission, supported file types |
| Google folder missing | Folder ID, Shared Drive/subfolder switches, connected account access |
| Microsoft folder missing | Exact relative path, correct drive/site/library, account access |
| Azure source empty | Blob selector or exact share name, credential validity |
| Jira or Confluence empty | Project/Space Key and service-account permission |
| Firecrawl contains irrelevant pages | Narrow include regex, add exclusions, lower limit |
| Document exists but is not retrieved | Indexed/Readable state, chunks, agent assignment, retrieval settings |
| Scheduled refresh stopped | Connection health, token expiry, Overview → Updates, schedule, provider quota |
Incident Response
If incorrect content creates material risk, detach or disable the collection from production agents, preserve non-secret diagnostic evidence, identify whether the cause is permission, selector, sync, extraction, indexing, or retrieval, fix it on a test agent, and re-run the evaluation set before restoration.