Automated File Ingestion with Azure File Share
Use this pattern when approved documents originate on a local or on-premises file server and must reach Siesta AI automatically. A synchronization service publishes a controlled copy to Azure File Share. Siesta AI then reads that share into a Data collection for an agent.
This is an integration pattern, not a new API. The exact metadata-to-document joining behavior must be confirmed in a pilot with the deployed ingestion service.
Architecture
+------------------+
| Internal share |
| Source of truth |
+--------+---------+
|
v
+------------------+
| Sync service |
| Filter and copy |
+--------+---------+
|
v
+------------------+
| Azure File Share |
| Landing zone |
+--------+---------+
|
v
+------------------+
| Siesta AI |
| Data collection |
+--------+---------+
|
v
+------------------+
| Agent |
| Answers + sources|
+------------------+
The synchronization service and Siesta AI use separate access paths:
- The synchronization identity writes approved files and metadata to Azure File Share.
- The Siesta AI Connection reads the same share for ingestion.
Do not give Siesta AI write access merely because the uploader requires it. Do not place the uploader credential in the Siesta AI Data-source fields.
Responsibilities
| Role | Responsibility |
|---|---|
| Document owner | Approves folders, file types, exclusions, status values, retention, and deletion behavior |
| Azure administrator | Creates the Storage Account and share, configures networking, identity-based SMB access, RBAC, ACLs, logging, and credential rotation |
| Synchronization operator | Runs the trusted sync service, applies filters, publishes files and metadata atomically, and monitors transfer failures |
| Siesta AI administrator | Creates the Connection, Data collection, and Azure File Share source; sets access and sync frequency; validates ingestion |
| Agent owner | Assigns the collection, defines citation and absent-answer behavior, runs evaluation tests, and monitors answer quality |
One person may hold several roles, but each responsibility still needs a named owner and a backup.
Phase 1: Prepare the Azure Landing Zone
- Create or select an Azure Storage Account that supports Azure Files.
- Create a dedicated file share for the ingestion boundary.
- Choose a stable share name, such as
approved-documents. - Restrict network access to the trusted sync host and the approved Siesta AI ingestion path.
- Enable Azure diagnostic logging and define retention.
- Define separate uploader and reader identities.
Prefer identity-based SMB authentication. Grant the synchronization identity the Storage File Data SMB Share Contributor role at the narrowest practical scope and apply matching file and directory ACLs. The role allows share-level access; ACLs control access inside the share. Follow the Microsoft guidance for Azure Files share-level permissions.
The Siesta AI reader should receive only the access required by the deployed Connection design. If the current integration uses a Storage Account connection string, treat it as a broad secret, store it only in Connections, rotate it, and never copy it into a source description or selector.
Phase 2: Mount the Share on a Trusted Windows Server
Run the sync process on a managed server that can read the internal repository and reach Azure Files. Azure Files SMB access requires outbound TCP port 445. Test the route before troubleshooting credentials.
Portal-assisted mount
- Sign in to the Azure portal.
- Open Storage accounts, select the target account, then open Data storage > File shares.
- Select the ingestion share and choose Connect.
- Select Windows, choose a drive letter such as
Z:, and select the approved authentication method. - Copy the generated PowerShell command.
- Open PowerShell as Administrator on the trusted server and run the command.
- Confirm that the share is available in File Explorer and that the sync identity can create, replace, and remove a test file within its approved folder.
See Microsoft's Windows Azure Files mount guide for current prerequisites and commands.
Do not confuse these values
| Value | Example | Where it is used |
|---|---|---|
| Azure File Share name | approved-documents | File Shares in the Siesta AI Data source |
| UNC path | \\\\storageaccount.file.core.windows.net\\approved-documents | Windows SMB access and mount commands |
| Mapped drive | Z:\\ | Local path used by the sync service on that server |
| Internal source path | \\\\fileserver\\departments\\manuals | Upstream path read by the sync service |
Enter only the exact share name in the Siesta AI File Shares field. Do not enter Z:\\, a UNC path, a folder inside the share, or a Blob Storage URL.
Account-key fallback
A Storage Account key can be used when identity-based SMB access is not available, but it is a less secure fallback. The key grants broad access as the Storage Account identity, bypasses individual user authorization, and must be protected and rotated. Do not interpret a user's Azure RBAC assignment as effective SMB access when the mount authenticates with an account key.
Phase 3: Select and Synchronize Files
The sync service must publish a curated dataset, not mirror every accessible file.
Selection policy
- Allow only approved root folders and file types.
- Exclude drafts, temporary files, archives, obsolete folders, backups, and unsupported formats.
- Reject files outside the approved audience or confidentiality boundary.
- Preserve stable relative paths or stable document IDs across updates.
- Update an existing document instead of creating a duplicate for every run.
- Define whether a removed upstream file is deleted, marked deprecated, or retained for a fixed period.
- Write an operational log with run ID, start/end time, counts, bytes, skipped items, removals, and errors. Never log secrets or document contents.
Start with a whitelist, for example:
Approved roots: \\fileserver\departments\manuals\approved
File types: .pdf, .docx, .xlsx, .pptx, .md, .txt
Excluded names: ~$*, *.tmp, *.bak
Excluded paths: drafts, archive, obsolete, temp
Atomic publishing
Siesta AI must not ingest a half-written batch. Upload each file under a temporary name or staging path, validate its size or checksum, then atomically rename or move it to the final path. Publish the completed metadata manifest only after all documents in the batch are ready.
If atomic rename is not available in the selected implementation, publish to a versioned staging folder and switch a tested release pointer or equivalent deployment boundary only after validation.
Metadata Contract
Use JSON as the canonical metadata format because the Azure File Share form exposes JSON Metadata Definitions. CSV may exist upstream, but the synchronization process must convert it to the supported JSON contract unless the deployment explicitly supports CSV.
The following sidecar manifest is an integration contract. Confirm during the pilot how the deployed ingestion service discovers the manifest, joins entries to files or chunks, validates unknown fields, and handles missing entries.
{
"schema_version": "1.0",
"generated_at": "2026-07-21T08:30:00Z",
"documents": [
{
"document_id": "operations-boiler-startup-v3",
"file_path": "manuals/operations/boiler-startup.pdf",
"source_path": "\\\\fileserver\\operations\\approved\\boiler-startup.pdf",
"title": "Boiler Startup Procedure",
"category": "Operations",
"owner": "Operations Engineering",
"version": "3.0",
"status": "approved",
"last_modified": "2026-07-20T14:05:31Z",
"confidentiality": "internal",
"tags": ["boiler", "startup", "safety"]
}
]
}
Field definitions
| Field | Requirement |
|---|---|
document_id | Required, stable, unique identifier. Keep it unchanged when the same logical document receives a new version. Do not derive it from a temporary filename. |
file_path | Required path relative to the root of the Azure File Share, using / separators. It must resolve to exactly one published file. |
source_path | Original internal path for audit and troubleshooting. Do not expose it in user-facing citations unless approved. |
title | Human-readable document title used for review and, when supported, citations. |
category | Controlled business category, not an arbitrary folder dump. |
owner | Team or role accountable for content accuracy. Prefer a durable group name over a personal email. |
version | Source-controlled version string. Use one convention consistently. |
status | Lifecycle state. Recommended values: approved, deprecated, archived, draft. Only explicitly approved values should be ingestible. |
last_modified | Source modification time in RFC 3339 format, including timezone, such as 2026-07-20T14:05:31Z. |
confidentiality | Controlled classification such as public, internal, confidential, or an organization-approved value. |
tags | JSON array of normalized search and governance labels. Use stable terms and avoid duplicates that differ only by case. |
Validate the manifest before publication. Reject duplicate document_id values, missing files, absolute file_path values, invalid timestamps, unsupported statuses, and classifications that do not match the target collection.
Phase 4: Configure Siesta AI
- Ask an admin to create an Azure Storage Account Connection with read access to the landing zone.
- Create a collection with a business name, owner, audience, and retention policy.
- Add Azure File Share as the Data source.
- Select the prepared Connection.
- Enter the exact Azure File Share name under File Shares.
- Choose an initial On Demand sync frequency for the pilot.
- Add and test JSON Metadata Definitions only according to the deployed ingestion contract.
- Run ingestion, then inspect Files, chunks, status, and Logs.
- Assign the collection to a test agent before any production agent.
The local Siesta AI application and backend confirm that JSON Metadata Definitions are passed into Azure File Share source creation. They do not by themselves prove how an external RAG service joins a sidecar manifest to every chunk. Treat that behavior as unverified until the pilot demonstrates it.
Phase 5: Pilot, Monitor, and Release
Run the following tests in an isolated collection:
| Test | Expected result |
|---|---|
| New approved file | One readable, indexed document appears with the expected title and source |
Updated file with the same document_id | The new content replaces or versions the logical document without an unintended duplicate |
| Draft or excluded path | The file is skipped and recorded in the sync log |
| Removed source file | The documented delete, deprecate, or retain policy is applied |
| Malformed metadata | The batch or entry fails safely with a useful non-secret error |
| Empty share | The run completes safely and does not erase valid production content unless explicitly designed to do so |
| Known-answer question | The agent answers from the collection and cites a traceable document |
| Absent-answer question | The agent states that the answer is not in the collection instead of guessing |
| New source version | After synchronization, the agent uses the new approved version |
Before production, record the baseline document count, excluded count, last successful run, expected schedule, owners, alert recipients, recovery procedure, credential rotation procedure, and deletion semantics.
Monitor both halves of the pipeline:
- Sync service: source scan failures, rejected files, transfer errors, manifest validation, duration, and last successful publication.
- Siesta AI: source status, last and next sync, indexed/readable files, processing failures, retrieval quality, citations, and access/audit events.
Pause or detach the collection if the landing zone contains unapproved data, metadata no longer matches files, a deletion causes unsafe answers, or the agent cites an obsolete version.