# fess-ds-microsoft365

**Repository Path**: mirrors_codelibs/fess-ds-microsoft365

## Basic Information

- **Project Name**: fess-ds-microsoft365
- **Description**: No description available
- **Primary Language**: Unknown
- **License**: Apache-2.0
- **Default Branch**: main
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2025-08-22
- **Last Updated**: 2026-10-28

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# Microsoft365 Data Store for Fess

[![Java CI with Maven](https://github.com/codelibs/fess-ds-microsoft365/actions/workflows/maven.yml/badge.svg)](https://github.com/codelibs/fess-ds-microsoft365/actions/workflows/maven.yml)
[![Maven Central](https://img.shields.io/maven-central/v/org.codelibs.fess/fess-ds-microsoft365.svg?label=Maven%20Central)](https://search.maven.org/search?q=g:%22org.codelibs.fess%22%20AND%20a:%22fess-ds-microsoft365%22)
[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)

A powerful Fess Data Store plugin that enables enterprise search across Microsoft 365 services including OneDrive, OneNote, Teams, SharePoint Document Libraries, SharePoint Lists, and SharePoint Pages via Microsoft Graph API v6.

> **Upgrading from an earlier version?** See [UPGRADING.md](UPGRADING.md).

## Overview

This plugin extends [Fess](https://fess.codelibs.org/) enterprise search capabilities to comprehensively index Microsoft 365 content, providing unified search across your organization's cloud documents, conversations, and data with role-based access control integration.

## ✨ Key Features

### 📁 **Comprehensive Content Crawling**
- **OneDrive**: User drives, group drives, shared documents, and specific drives with metadata extraction
- **OneNote**: Complete notebooks with aggregated content from all sections and pages, supporting site, user, and group notebooks
- **Teams**: Channels, messages, chats with conversation context
- **SharePoint Document Libraries**: Document library metadata indexing (libraries crawled as searchable entities, not individual files)
- **SharePoint Lists**: Custom lists and list items with dynamic field mapping
- **SharePoint Pages**: Site pages and news articles with full content extraction

### 🔐 **Enterprise-Grade Security**
- **Role-based Access Control**: Seamless integration with Fess security model
- **Azure AD Authentication**: Client credentials flow with automatic token refresh
- **Permission Mapping**: Where Microsoft Graph exposes an ACL - OneDrive and SharePoint document library items, and Teams channel/chat membership - it is mapped onto Fess roles. SharePoint lists, SharePoint pages and OneNote site notebooks get `default_permissions` and the data config's Permissions field only; see [What ACL Each DataStore Can Produce](#what-acl-each-datastore-can-produce)
- **Sensitivity Labels**: OneDriveDataStore can read each file's Microsoft Purview sensitivity labels, index them, and skip, index without content, or narrow the ACL of files by label; see [Sensitivity labels](#sensitivity-labels)

### ⚡ **Performance & Reliability**
- **Microsoft Graph SDK v6**: Latest API with efficient pagination and caching
- **Multi-threaded Processing**: Configurable thread pools for optimal performance
- **Smart Caching**: User type, group ID, UPN and group name caching to reduce API calls, bounded by `cache_size`
- **Robust Error Handling**: Comprehensive error tracking with configurable failure recovery
- **Content Filtering**: Advanced include/exclude patterns with system content filtering

### 🛠 **Developer-Friendly**
- **Maven Integration**: Clean build process with dependency shading
- **Tested against mock Graph responses**: UTFlute plus an OkHttp MockWebServer harness, so Graph paths, paging and retry behaviour are exercised without a live tenant
- **Configurable Field Mapping**: Customizable data extraction scripts for each service

## 🚀 Quick Start

### Prerequisites

- **Java**: 21 or higher
- **Fess**: 15.2.0 or higher
- **Azure AD**: App registration with Microsoft Graph API permissions

### Installation

#### Option 1: Download Pre-built JAR
1. Download the latest `fess-ds-microsoft365-X.X.X.jar` from [Maven Central](https://repo1.maven.org/maven2/org/codelibs/fess/fess-ds-microsoft365/)
2. Copy the JAR file to your Fess installation:
   ```bash
   # For standard installation
   cp fess-ds-microsoft365-X.X.X.jar $FESS_HOME/app/WEB-INF/lib/
   
   # For system-wide installation
   sudo cp fess-ds-microsoft365-X.X.X.jar /usr/share/fess/app/WEB-INF/lib/
   ```
3. Restart Fess service

#### Option 2: Build from Source
```bash
# Clone the repository
git clone https://github.com/codelibs/fess-ds-microsoft365.git
cd fess-ds-microsoft365

# Build the plugin
mvn clean package

# Copy to Fess installation
cp target/fess-ds-microsoft365-*.jar $FESS_HOME/app/WEB-INF/lib/
```

### Azure App Registration Setup

Before using this plugin, create an Azure App registration with the required permissions:

1. **Register Application** in Azure Portal
2. **Add API Permissions** (Microsoft Graph) - see [Required Permissions by DataStore](#required-permissions-by-datastore) below
3. **Grant Admin Consent** for the permissions
4. **Create Client Secret** and note the values
5. **For OneNote only**: also enable *Allow public client flows* and grant the permissions as
   **delegated** rather than application ones - see
   [OneNote requires delegated authentication](#onenote-requires-delegated-authentication)

### Required Permissions by DataStore

Each DataStore requires specific Microsoft Graph API permissions. Grant only the permissions needed for your use case.

| DataStore | Required Permissions | Conditional Permissions |
|-----------|---------------------|------------------------|
| OneDriveDataStore | Files.Read.All, User.Read.All, Group.Read.All | Sites.Read.All (*1), SensitivityLabel.Read (*9) |
| OneNoteDataStore (*8) | Notes.Read.All | User.Read.All (*2), Group.Read.All (*3), Sites.Read.All (*4) |
| TeamsDataStore | Team.ReadBasic.All, Channel.ReadBasic.All, ChannelMessage.Read.All, ChannelMember.Read.All, Group.Read.All, User.Read.All | Chat.Read.All (*5), attachment permission (*6) |
| SharePointDocLibDataStore | Sites.Read.All, User.Read.All, Group.Read.All | - |
| SharePointListDataStore | Sites.Read.All (*7) | - |
| SharePointPageDataStore | Sites.Read.All (*7) | - |

`User.Read.All` and `Group.Read.All` are unconditional for OneDriveDataStore and
SharePointDocLibDataStore: every drive-item ACL entry resolves its grantee's UPN or group name via
`GET /users/{id}` / `GET /groups/{id}` (`processDriveItem` builds this ACL on every crawling mode,
not just user/group drives), regardless of which crawling mode found the item. `Files.Read.All` is
not listed for SharePointDocLibDataStore: the two endpoints it needs beyond site enumeration -
`GET /sites/{id}/drives` and `GET /drives/{id}/items/root/permissions` - both accept `Sites.Read.All`,
which the DataStore already needs to enumerate sites, so a separate `Files.Read.All` grant would buy
nothing.

**Conditional Permission Notes:**
- (*1) Required when `shared_documents_drive_crawler=true` (default: true). In that mode
  OneDriveDataStore enumerates every SharePoint site (`GET /sites`) and every subsite (`GET
  /sites/{site-id}/sites`, see the subsite note below) to find their document libraries; neither
  call accepts `Files.Read.All` as an alternative.
- (*2) Required when `user_note_crawler=true` (default: true)
- (*3) Required when `group_note_crawler=true` (default: true)
- (*4) Required when `site_note_crawler=true` (default: true), to enumerate the sites
  (`GET /sites`, or `GET /sites/{site-id}` when `site_id` is set) that site notebooks are
  enumerated under
- (*5) Required when `chat_id` is specified
- (*6) TeamsDataStore fetches attachment content via `GET /shares/{id}/driveItem/content`, whose
  Microsoft-documented Application permissions are `Files.ReadWrite.All` (least privileged) or
  `Sites.ReadWrite.All` (higher) - a write-class grant, even though this plugin only reads the
  attachment. `Files.Read.All` is not listed as acceptable for this call at all. `append_attachment`
  defaults to `true`, so this permission is required unless you set `append_attachment=false`,
  which skips attachment content entirely and keeps the grant read-only.
- (*7) Can be replaced with `Sites.Selected` when `site_id` is specified - see
  [Using Sites.Selected Permission](#using-sitesselected-permission) below.
- (*8) These must be granted as **delegated** permissions, not application ones, and the data
  config must set `username`/`password`: Microsoft retired app-only tokens for the Microsoft Graph
  OneNote API on 2025-03-31 and answers every `/onenote/` request made with one with a 401. Every
  other DataStore listed here keeps using application permissions. See
  [OneNote requires delegated authentication](#onenote-requires-delegated-authentication).
- (*9) Recommended when `sensitivity_label_enabled=true`. Reading a file's labels
  (`POST /drives/{drive-id}/items/{item-id}/extractSensitivityLabels`) needs only `Files.Read.All`,
  but loading the label definitions - names, parent labels and encryption settings
  (`GET /security/dataSecurityAndGovernance/sensitivityLabels` and `.../{id}/sublabels`) - needs
  `SensitivityLabel.Read`. Without it, labels are reported by ID only and only their own ID and `*`
  rules can match them - see
  [Sensitivity labels](#sensitivity-labels).

**Subsites:** `GET /sites/{site-id}/sites` recursion is used by OneDriveDataStore (in
shared-documents mode), SharePointDocLibDataStore, SharePointListDataStore, and
SharePointPageDataStore to discover every child site under a parent site. Each subsite it returns
is treated as an ordinary site - itself enumerated for content, and itself capable of having
further subsites - under the same `Sites.Read.All` (or `Sites.Selected`) grant.

#### Using Sites.Selected Permission

When `site_id` is specified, `Sites.Selected` is a genuine least-privilege alternative to
`Sites.Read.All` for SharePointDocLibDataStore, SharePointListDataStore and SharePointPageDataStore:

1. Grant `Sites.Selected` permission to your app registration in Azure Portal
2. Use Microsoft Graph PowerShell or API to grant access to specific sites
3. Explicitly allow access for each target site

`Sites.Selected` cannot serve `GET /sites` (listing every site in the tenant), which is why it only
helps when `site_id` restricts the crawl to sites you have explicitly granted the app access to;
without `site_id`, these three DataStores still need `Sites.Read.All`.

Reference: https://learn.microsoft.com/en-us/graph/permissions-reference#sitesselected

> **No individual Microsoft Graph API reference page lists `Sites.Selected`** in its per-endpoint
> permissions table - not `site-get`, not `list-list`, not `driveitem-list-permissions`, none of the
> endpoints these three DataStores call. The `Sites.Selected` model is documented only in the
> [permissions reference](https://learn.microsoft.com/en-us/graph/permissions-reference#sitesselected)
> and the site-level access-grant APIs, not as a row in any individual endpoint's table. Treat it as
> Microsoft's documented site-access model rather than a per-endpoint guarantee.

#### What ACL Each DataStore Can Produce

An administrator needs to predict what roles a crawled document will carry, not just what
permissions it took to crawl it:

| DataStore | Roles on each document |
|-----------|------------------------|
| OneDriveDataStore | `GET /drives/.../permissions` (per-user/per-group grantees and organization-scope sharing links) plus `default_permissions` |
| SharePointDocLibDataStore | `GET /drives/.../permissions` (per-user/per-group grantees and organization-scope sharing links) plus `default_permissions` |
| SharePointListDataStore | `default_permissions` only |
| SharePointPageDataStore | `default_permissions` only |
| OneNoteDataStore | User/group notebooks: a role synthesized from the owner's id, plus `default_permissions`. Site notebooks: `default_permissions` only |
| TeamsDataStore | Channel or chat membership plus `default_permissions` |

**Every row above also carries the data config's own Permissions field**, on top of what the row
names. That is the field on the Fess data config itself, which is a separate setting from the
`default_permissions` crawl parameter; both end up in the same role list, de-duplicated. This is
worth stating for all six rows because it was not true of all six until this release:
TeamsDataStore and OneNoteDataStore silently discarded it, and both apply it now - see
[Teams and OneNote: all-teams crawl and the Permissions field](UPGRADING.md#teams-and-onenote-all-teams-crawl-and-the-permissions-field)
before re-crawling a Teams **or OneNote** config.

Microsoft Graph exposes no app-only way to read SharePoint's own user and group role assignments
for a site, list, list item or page. Microsoft's own Azure AI Search SharePoint ACL indexer
requires `Sites.FullControl.All` or a full-control-class `Sites.Selected` grant to do the
equivalent lookup, in every documented scenario. This plugin does not attempt it: for
SharePointListDataStore, SharePointPageDataStore, and OneNoteDataStore's site notebooks,
`default_permissions` is the only way those documents get an audience - leave it unset and they
are indexed with no role at all, which means they are findable by **nobody**, not by everybody.

> **Upgrading from 15.8.x?** SharePoint lists, pages and OneNote site notebooks no longer need `Sites.FullControl.All`, and some ACLs changed; see [UPGRADING.md](UPGRADING.md#sharepoint-lists-pages-and-onenote-site-notebooks-sitesfullcontrolall-no-longer-needed) and [UPGRADING.md](UPGRADING.md#onedrive-sharepoint-document-libraries-and-onenote-acl-correction).

As always: if `default_permissions` is left unset on SharePointListDataStore,
SharePointPageDataStore, or OneNoteDataStore's site notebooks, those documents carry no role at
all and are searchable by nobody, and nothing in the logs flags it - only this note does.

**Organization-shared OneDrive files:** a document whose only Graph permission is an
organization-scope sharing link (`link.scope == "organization"`, i.e. "Anyone in your organization
with the link") previously carried no roles at all on the OneDrive path, so it was findable by
nobody. It now carries a single role, the group named `EVERYONE_IN_TENANT`, which also matches
nobody unless the operator creates an actual group named `EVERYONE_IN_TENANT` and maps it to a
Fess role. This is not a loss of visibility (those files were already unfindable), but it is the
only way organization-shared OneDrive files become searchable, and the sentinel group name appears
nowhere else in this document.

**Precedence between a named grantee and a link's scope:** when one permission carries both a
named grantee (`grantedToV2`, or an entry in `grantedToIdentitiesV2`) and a `link`, only the
grantee's role is added; the permission does not also widen the ACL to the link's scope. This is
deliberate and fail-closed - a permission that already names who it is for should not also be
read as "anyone in the organization" - but it narrows one ACL shape SharePointDocLibDataStore
already produced before this branch. Its root-drive permission lookup has always run through this
same code, and the previous version of that code ignored `grantedToIdentitiesV2` entirely, so a root
permission shaped like `{link: {scope: "organization"}, grantedToIdentitiesV2: [user A]}` used to
contribute `EVERYONE_IN_TENANT` in addition to A's role. Now it contributes only A's role.
SharePointDocLibDataStore indexes one document per document library (library metadata, not its
individual files - see the Data Store Types table below), and computes that document's entire ACL
once, from the drive's root permissions. So after upgrading, a document library whose root
permission names grantees alongside an organization-scope link loses `EVERYONE_IN_TENANT` from
that library's indexed document.

#### Minimum Permissions Examples

**OneDrive only (user drives):**
```
Files.Read.All, User.Read.All, Group.Read.All
```

**SharePoint Lists (specific site):**
```
Sites.Selected (with per-site configuration)
```

**Teams (channels only, no chat, read-only):**
```
Team.ReadBasic.All, Channel.ReadBasic.All, ChannelMessage.Read.All, ChannelMember.Read.All, Group.Read.All, User.Read.All
```
Requires `append_attachment=false` (default is `true`) to stay within this permission set - see
conditional note (*6) above.

### Basic Configuration

Configure the following authentication parameters in Fess:

```properties
# Required Azure AD credentials
tenant=********-****-****-****-************
client_id=********-****-****-****-************  
client_secret=***********************

# Optional performance settings
number_of_threads=1
ignore_error=false
```

`oneNoteDataStore` is the exception: it cannot use these credentials, because Microsoft retired
app-only tokens for the Microsoft Graph OneNote API on 2025-03-31. Its data config needs
`username`/`password` instead - see
[OneNote requires delegated authentication](#onenote-requires-delegated-authentication). Configure
that on the OneNote data config alone; leaving the other data configs on `client_secret` keeps
their crawls tenant-wide.

## 📊 Data Store Types

The plugin provides six specialized data store types, each optimized for different Microsoft 365 services:

| Data Store | Service | Content Types | Use Cases |
|------------|---------|---------------|----------|
| `oneDriveDataStore` | OneDrive | Files, Folders, Metadata | Document search, file discovery |
| `oneNoteDataStore` | OneNote | Notebooks (with sections & pages content) | Knowledge base search, note finding, documentation search |
| `teamsDataStore` | Teams | Channels, Messages, Chats | Conversation search, team communication |
| `sharePointDocLibDataStore` | SharePoint | Document Libraries (metadata only) | Document library discovery and metadata search |
| `sharePointListDataStore` | SharePoint | Lists, List Items, Custom Fields | Structured data search, business process content |
| `sharePointPageDataStore` | SharePoint | Modern Site Pages and News Posts (classic wiki pages are not indexed) | Web content search, intranet portal search |

### Configuration in Fess Admin UI

1. Navigate to **System > Data Store**
2. Click **Create New**
3. Select the desired data store type from the dropdown
4. Configure authentication and crawling parameters
5. Save and start crawling

### Scripts

#### OneDrive

```
title=file.name
content=file.description + "\n" + file.contents
mimetype=file.mimetype
created=file.created
last_modified=file.last_modified
url=file.web_url
role=file.roles
```

| Key | Value |
| --- | --- |
| file.name | The name of the file. |
| file.description | A short description of the file. |
| file.contents | The text contents of the file (extracted using Tika) |
| file.mimetype | The MIME type of the file. |
| file.filetype | The file type category determined by MIME type. |
| file.created | The time at which the file was created. |
| file.last_modified | The last time the file was modified by anyone. |
| file.size | The size of the file in bytes. |
| file.web_url | A link for opening the file in an editor or viewer in a browser. |
| file.url | The processed URL for the file (may differ from web_url for certain locations). |
| file.roles | Users/groups who can access the file. |
| file.ctag | Change tag for the file (used for change tracking). |
| file.etag | Entity tag for the file (used for caching). |
| file.id | The unique identifier of the file in OneDrive. |
| file.webdav_url | WebDAV URL for the file (if available). |
| file.location | Geographic location metadata (if available). |
| file.createdby_application | Application that created the file. |
| file.createdby_device | Device that created the file. |
| file.createdby_user | User who created the file. |
| file.deleted | Deletion information (if file was deleted). |
| file.hashes | File hash values for integrity checking. |
| file.last_modifiedby_application | Application that last modified the file. |
| file.last_modifiedby_device | Device that last modified the file. |
| file.last_modifiedby_user | User who last modified the file. |
| file.image | Image metadata (for image files). |
| file.parent | Parent reference information. |
| file.parent_id | ID of the parent folder. |
| file.parent_name | Name of the parent folder. |
| file.parent_path | Path to the parent folder. |
| file.photo | Photo metadata (for photo files). |
| file.publication | Publication information (if applicable). |
| file.search_result | Search result metadata (if file was found via search). |
| file.special_folder | Special folder name (if file is in a special folder). |
| file.video | Video metadata (for video files). |
| file.sensitivity_label_ids | IDs of the file's sensitivity labels. Set only when `sensitivity_label_enabled=true`; empty for an unlabeled file. |
| file.sensitivity_label_names | Display names of the file's sensitivity labels, a sublabel as `Parent\Sublabel`, or their IDs when the definitions cannot be read. Set only when `sensitivity_label_enabled=true`. |
| file.sensitivity_label_protected | Whether any of the file's sensitivity labels is known to apply encryption; `false` also when the definitions cannot be read. Set only when `sensitivity_label_enabled=true`. |

#### OneNote

```
title=notebook.name
content=notebook.contents
created=notebook.created
last_modified=notebook.last_modified
url=notebook.web_url
role=notebook.roles
size=notebook.size
```

| Key | Value |
| --- | --- |
| notebook.name | The name of the notebook. |
| notebook.contents | The extracted text contents from all sections and pages within the notebook. |
| notebook.size | The size of the notebook content in characters. |
| notebook.created | The time at which the notebook was created. |
| notebook.last_modified | The last time the notebook was modified by anyone. |
| notebook.web_url | A link for opening the notebook in OneNote web or desktop app. |
| notebook.roles | Users/groups who can access the notebook. |

#### Teams

```
title=message.title
content=message.content
created=message.created_date_time
last_modified=message.last_modified_date_time
url=message.web_url
role=message.roles
```

| Key | Value |
| --- | --- |
| message.title | The message title (sender name and timestamp). |
| message.content | The text contents of the message including attachments if configured. |
| message.created_date_time | The time at which the message was created. |
| message.last_modified_date_time | The last time the message was modified. |
| message.web_url | A link for opening the message in Teams. |
| message.roles | Users/groups who can access the team/channel/chat. |
| message.id | The unique identifier of the message. |
| message.from | The sender information. |
| message.subject | The subject of the message. |
| message.body | The body content with type information. |
| message.attachments | The raw Graph attachment objects. Populated for use by `message.content` (which appends the extracted attachment text when `append_attachment=true`); a script that maps it directly gets SDK objects, not text. |
| message.mentions | Users mentioned in the message. |
| message.channel_identity | The `channelIdentity` object (team ID and channel ID) for a channel message. |
| message.chat_id | The ID of the chat a chat message belongs to. |
| message.deleted_date_time | The time at which the message was deleted, if it was. |
| message.etag | The message's version tag. |
| message.hosted_contents | The raw Graph hosted-content objects (inline images and similar). |
| message.importance | The message importance (`normal`, `high`, `urgent`). |
| message.last_edited_date_time | The time at which the message was last edited by its sender. |
| message.locale | The message locale. |
| message.replies | The raw Graph reply objects. Replies are also indexed as documents of their own unless `ignore_replies=true`. |
| message.reply_to_id | The ID of the root message this message replies to. |
| message.summary | The message summary, when Graph supplies one. |
| team | The team object containing team information (when applicable). |
| channel | The channel object containing channel information (when applicable). |
| parent | The parent message for replies (when applicable). |
| messages | Only on a `chat_id` crawl: the raw Graph `ChatMessage` objects the consolidated document was built from. |

Every key populated by `TeamsDataStore` is listed above. There is no `message.reactions`: reactions
are read only to build the consolidated document for a `chat_id` crawl, and are never published as
a script key.

#### SharePoint Document Libraries

```
title=doclib.name
content=doclib.content
created=doclib.created
last_modified=doclib.modified
url=doclib.url
role=doclib.roles
```

| Key | Value |
| --- | --- |
| doclib.id | The unique identifier of the document library (Drive ID). |
| doclib.name | The name of the document library. |
| doclib.description | The description of the document library. |
| doclib.content | Rich content combining document library name, description, and site name for enhanced search. |
| doclib.web_url | The Microsoft Graph API web URL for the document library. |
| doclib.url | The URL that opens the document library in a browser. This is the same value as `doclib.web_url`; it differs only when Graph returns no `webUrl`, in which case a URL is composed from the site URL and the library name. The default script maps `url` to this key. |
| doclib.created | The time at which the document library was created. |
| doclib.modified | The last time the document library was modified. |
| doclib.type | The type of the drive (e.g., "documentLibrary"). |
| doclib.site_name | The display name of the SharePoint site containing this document library. |
| doclib.site_url | The web URL of the SharePoint site. |
| doclib.roles | Users/groups who can access the document library. |

**Important**: SharePointDocLibDataStore indexes document libraries themselves as searchable entities (not the files within them). Each document library becomes one search result containing aggregated metadata including library name, description, and parent site information. For individual file indexing within SharePoint document libraries, use the OneDriveDataStore which handles SharePoint document library files through the Microsoft Graph Drive API.

#### SharePoint Lists

```
title=item.title
content=item.content
created=item.created
last_modified=item.modified
url=item.url
role=item.roles
```

| Key | Value |
| --- | --- |
| item.title | The title of the list item (extracted from Title, LinkTitle, or FileLeafRef fields). A Links list item has none of them and takes the description of its URL field instead, or the address itself when the description is blank. |
| item.content | The text contents of the list item (the first of the Body, Description, Comments, or Notes fields that has a value). An item with none of them - a survey response, or an item whose text is in columns the list owner added - takes the text values of the list's columns that Graph reports as neither read-only nor hidden instead, one per line; `Title`, `ContentType`, `FileLeafRef` and date and time columns are left out. A rich text value, such as an Announcements list's `Body`, is indexed as its text with the HTML markup stripped. |
| item.id | The unique identifier of the list item |
| item.created | The time at which the list item was created. |
| item.modified | The last time the list item was modified. |
| item.url | A link for opening the list item in SharePoint - the list's `webUrl` plus `/DispForm.aspx?ID=<id>` (`/Forms/DispForm.aspx?ID=<id>` for a document library, picture library or site pages library), falling back to the item's own `webUrl` when the list has none. |
| item.web_url | The item's raw Graph `webUrl`, unmodified. Usually differs from `item.url`. |
| item.content_type | The name of the item's SharePoint content type, or an empty string when the item has none. |
| item.fields | All fields and values from the SharePoint list item as a map, exactly as Graph returns them - SharePoint's own internal fields included. |
| item.roles | The roles the list item is indexed with: `default_permissions` plus the data config's own Permissions field. No SharePoint user or group is resolved for a list item - see [What ACL Each DataStore Can Produce](#what-acl-each-datastore-can-produce). |
| item.site | Site information containing `id`, `name`, and `url` |
| item.list | List information containing `name`, `description`, `url`, and `template_type` |

**Data Structure**: The `item` object contains nested structures:
- `item.site` - Contains site metadata (site.id, site.name, site.url)
- `item.list` - Contains list metadata (list.name, list.description, list.url, list.template_type)
- `item.fields` - Dynamic map of all SharePoint list fields and their values

> **Upgrading from 15.8.x?** List items already in the index need a re-crawl to pick up these changes; see [UPGRADING.md](UPGRADING.md#sharepoint-list-items-links-titles-column-only-content-and-rich-text).

**Note**: The plugin automatically expands SharePoint list item fields to ensure content extraction. If fields are not initially available, it performs an individual API call with `$expand=fields` to retrieve the complete field data.

#### SharePoint Pages

```
title=page.title
content=page.content
created=page.created
last_modified=page.modified
url=page.url
role=page.roles
```

| Key | Value |
| --- | --- |
| page.title | The title of the SharePoint page. |
| page.content | The extracted text content from the page canvas layout including web parts. |
| page.id | The unique identifier of the page. |
| page.created | The time at which the page was created. |
| page.modified | The last time the page was modified. |
| page.author | The user who created the page. |
| page.type | The type of page (news, article, page). |
| page.description | The page description or summary. |
| page.url | A link for opening the page in SharePoint (the page's Graph `webUrl`). |
| page.web_url | The same value as `page.url`; both keys are populated from the page's Graph `webUrl`. |
| page.promotion_state | The promotion status of the page (for news pages). |
| page.site | A map describing the SharePoint site the page belongs to. Its keys are `site_name` (the site's display name), `site_url` (the site's web URL) and `id` (the site's Graph ID), so a script reads them as `page.site.site_name`, `page.site.site_url` and `page.site.id`. There is no top-level `page.site_name` or `page.site_url`. |
| page.roles | The roles the page is indexed with: `default_permissions` plus the data config's own Permissions field. No SharePoint user or group is resolved for a page - see [What ACL Each DataStore Can Produce](#what-acl-each-datastore-can-produce). |

**Content Extraction**: The SharePointPageDataStore extracts content from:
- **Page Title**: The main page title
- **Page Description**: Page description or summary text
- **Text web parts**: the web part's `innerHtml`, converted to plain text
- **Standard web parts** (everything else - Quick Links, Hero, Events, News and so on): the text
  the author entered, from SharePoint's own indexable projection of the web part -
  `serverProcessedContent.searchablePlainTexts` and `.htmlStrings`. The web part's `title` and
  `description` are **not** read: they are the web part type's toolbox name and description
  (for example "Quick links"), the same on every page. `serverProcessedContent.links` is not read
  either: it holds URLs (the site's base URL, link targets, image URLs) that are not shown as text
  on the page. The web part's free-form `properties` object is **not** read; Microsoft Graph
  exposes it as an untyped node with no published schema. Items a web part pulls in at render
  time - the posts a News web part lists, the events an Events web part lists - are not part of
  the page: news posts are crawled as pages of their own, and events are list items
  (SharePointListDataStore)

**Page Types**: The plugin classifies each page from the `promotionKind` and `pageLayout` properties Microsoft Graph returns for it:
- `news`: a news post (`promotionKind` is `newsPost`)
- `article`: any other page with the article layout (`pageLayout` is `article`)
- `page`: every other page, such as a site home page (`pageLayout` is `home`)

> **Upgrading from 15.8.x?** Page types and web-part text change for pages already in the index; see [UPGRADING.md](UPGRADING.md#sharepoint-pages-home-pages-are-now-typed-page) and [UPGRADING.md](UPGRADING.md#sharepoint-pages-standard-web-part-text).

## ⚙️ Configuration Reference

### Authentication Parameters (Required)

| Parameter | Description | Example |
|-----------|-------------|----------|
| `tenant` | Azure AD tenant ID | `contoso.onmicrosoft.com` or GUID |
| `client_id` | App registration client ID | `12345678-1234-1234-1234-123456789abc` |
| `client_secret` | App registration client secret. Required unless `username`/`password` are set | `abcdefghijk...` |
| `username` | User principal name of the account to sign in as. Set together with `password` to use delegated (app+user) authentication instead of app-only - **required by `oneNoteDataStore`**, see [OneNote requires delegated authentication](#onenote-requires-delegated-authentication) | `crawler@contoso.onmicrosoft.com` |
| `password` | Password for `username`. Set together with `username`; setting only one of the two is rejected | `********` |

### Common Crawling Parameters

| Parameter | Description | Default | Example |
|-----------|-------------|---------|----------|
| `number_of_threads` | Concurrent crawling threads | `1` | `3` |
| `executor_shutdown_timeout` | How long to wait for submitted crawling tasks to finish, in whole seconds. Whatever has not finished by then is cancelled and its documents are missing from the crawl, which is reported at `ERROR` - see below. Raise it for a large tenant. A value that is not a positive whole number keeps this default and logs a `WARN`. | `60` | `600` |
| `ignore_error` | Continue on errors. What "errors" means differs by DataStore - see below | `false` | `true` |
| `include_pattern` | Regex pattern for inclusion - semantics differ by DataStore, see below | - | `.*\.pdf$` |
| `exclude_pattern` | Regex pattern for exclusion - semantics differ by DataStore, see below | - | `.*temp.*` |
| `default_permissions` | Default role assignments | - | `{role}admin` |
| `permission_failure_policy` | What to do when a document's permissions cannot be retrieved | `skip` | `index_without_acl` |
| `connect_timeout` | Connect timeout for Microsoft Graph HTTP requests, in whole seconds - see below | `100` | `30` |
| `read_timeout` | Read timeout for Microsoft Graph HTTP requests, in whole seconds - see below | `100` | `30` |
| `access_timeout` | Overall timeout for a Microsoft Graph HTTP call, in whole seconds - see below | `100` | `120` |
| `max_retry_count` | Maximum automatic retries for a failed Graph request - see below | `3` | `5` |
| `retry_interval` | Delay between automatic retries, in whole seconds - see below | `3` | `10` |
| `cache_size` | Maximum number of entries in each of the client's four lookup caches (user type, group IDs by mail, user object ID to UPN, group object ID to name) - see below | `10000` | `50000` |
| `max_content_length` | Content length cap in bytes, applied at two different points - see [`max_content_length` is applied twice](#max_content_length-is-applied-twice) below | `-1` (defer to Fess's own limit, **not** unlimited - see below for which limit) | `10485760` |
| `proxy_host` | HTTP proxy host for both Azure AD token acquisition and Microsoft Graph calls - see below | - | `proxy.example.com` |
| `proxy_port` | HTTP proxy port. Required alongside `proxy_host`; neither takes effect on its own - see below | - | `8080` |
| `proxy_username` | Proxy user name. Only used when `proxy_password` is also set - see below | - | `crawler` |
| `proxy_password` | Proxy password. Only used when `proxy_username` is also set - see below | - | `********` |
| `additionally_allowed_tenants` | Tenant IDs the credential may also acquire tokens for, comma-separated, or `*` for any tenant - see below | - (only the configured `tenant`) | `*` |

#### `include_pattern` / `exclude_pattern` semantics differ by DataStore

`include_pattern` and `exclude_pattern` are accepted by several DataStores in this plugin, but
each one matches them against different content, using different regex semantics, and none of
that is visible from the table above:

| DataStore | Matched against | Match mechanism |
|-----------|------------------|------------------|
| `oneDriveDataStore` | the generated drive-item URL (indexed as `file.url`) - **not** the raw Graph `webUrl` (`file.web_url`); the two usually agree, but diverge for `/_layouts/` paths, which `getUrl()` rewrites | Fess `UrlFilter` - full match (`Matcher.matches()`) |
| `sharePointDocLibDataStore` | the document library's browser URL (indexed as `doclib.url`) - the drive's Graph `webUrl`, **not** the library's display name | Fess `UrlFilter` - full match (`Matcher.matches()`) |
| `sharePointListDataStore` | the list item's title (`Title`/`LinkTitle`/`FileLeafRef`, whichever resolves first, then the description or address in the `URL` field of a Links list item) | `Matcher.matches()` - full match |
| `sharePointPageDataStore` | the page's `webUrl` | `Matcher.find()` - **partial** match |
| `oneNoteDataStore` | the notebook's display name | `Matcher.matches()` - full match |
| `teamsDataStore` | not supported - both parameters are silently ignored | - |

Practically: a OneDrive or SharePoint document library pattern must match the *entire* URL, a
SharePoint List pattern must match the *entire* title, and a SharePoint Pages pattern only needs
to match *somewhere inside* the URL. A pattern written for one DataStore will not necessarily
behave the same way on another.

A OneNote pattern must also match the *entire* notebook display name: `exclude_pattern=Test.*`
removes a notebook called "Test Notebook" but keeps one called "Production Test Notes" - the name
contains "Test", but does not *start* with it, so the full match fails and the notebook is kept.
(A partial-match DataStore such as `sharePointPageDataStore` would drop it instead.)

The filter is applied once per notebook, in the `getNotebooks` callback for each of the SITE,
USER and GROUP scopes, before that notebook is handed off for processing. It therefore selects
whole notebooks only: there is no way to admit a notebook but exclude one of its sections or
pages, because `oneNoteDataStore` indexes one document per notebook and never inspects section or
page names to filter on. A notebook whose display name is missing, empty **or whitespace-only** is
matched as the empty string, like any other name - the three are treated as one thing, so a pattern
such as `.+` cannot admit one and reject another: with `include_pattern` set such a notebook is
excluded unless that pattern matches `""`, and with only `exclude_pattern` set it is kept unless
that pattern matches `""`. A name with any other character is matched verbatim, surrounding
whitespace included.

`sharePointListDataStore` answers the same question the other way: a list item with no title - its
`Title`, `LinkTitle` and `FileLeafRef` all blank, and no `URL` field description or address - skips
both patterns entirely and is crawled, even under an `include_pattern` that matches nothing about
it. Neither behaviour changed in this release; they are noted here because a pattern moved between
the two DataStores will treat unnamed content differently. What did change is that a Links list
item is no longer unnamed: its title is now the description or address in its `URL` field.

**A pattern that is not a valid regular expression aborts the crawl at its start.** Both
parameters are compiled once, before the first Microsoft Graph call, and a `PatternSyntaxException`
is reported as a `DataStoreException` naming the parameter and the syntax error - for example
`exclude_pattern is not a valid regular expression: .*temp.*[ (Unclosed character class near
index 8)`. Nothing is indexed by that crawl. This is deliberate and is not affected by
`ignore_error`: a filter that could not be compiled cannot be honoured, and an `exclude_pattern`
that quietly stopped filtering would index exactly the documents it was written to keep out. It
applies to all five DataStores that accept the two parameters, whether they compile the pattern
themselves or hand it to Fess's `UrlFilter`.

#### `ignore_error` scope differs by DataStore

`ignore_error` defaults to **`false`** in all six DataStores. It is not a blanket "keep going"
switch: each DataStore relaxes a specific, fixed set of failures, and the rest behave the same way
at either setting.

| DataStore | What `ignore_error=true` tolerates |
|-----------|------------------------------------|
| `oneDriveDataStore` | a failure extracting one file's content (the document is still indexed, without contents) |
| `sharePointDocLibDataStore` | a site whose document libraries cannot be listed while every site is crawled (`site_id` unset); the crawl moves on to the next site. Nothing else: a document library that fails to process is recorded as a failure URL and skipped at either setting, and with `site_id` set a failure to read that site or list its libraries aborts the crawl at either setting |
| `sharePointListDataStore` | a site that fails to process, a list that fails to process, a list item that fails to process, and a failure re-reading a list item's fields (the item is still indexed, with the fields already in hand) |
| `sharePointPageDataStore` | a site that fails to process and a page that fails to process |
| `oneNoteDataStore` | Microsoft Graph refusing the configured credentials (HTTP 401) while listing a site's, user's or group's notebooks. Every other listing failure - 403, 404, a server error - skips that one owner at either setting, logged at `WARN` (at `debug` for a user's 404, see "404 Visibility" below), because under delegated authentication "this owner is not visible to the signed-in account" is the normal case rather than a fault |
| `teamsDataStore` | an unresolvable `team_id`, an unresolvable `channel_id`, a failure listing an explicitly configured team's channels, a failure fetching a channel's messages, and a failure processing a configured `chat_id` |

`ignore_error` never widens a crawl. A `team_id` listed in `exclude_team_ids` that cannot be
resolved still aborts the crawl even with `ignore_error=true`, so that a team you asked to exclude
is never silently crawled.

#### `cache_size` and the client's lookup caches

`cache_size` bounds each of the four Guava caches the Graph client keeps, **individually** - it is
not a total across them. They are: user-type (is this object ID a user or a group?), group IDs by
mail address, user object ID to UPN, and group object ID to group name. The last two back the ACL
roles OneDriveDataStore and SharePointDocLibDataStore add alongside each raw object ID.

The default is `10000` entries per cache. A negative or non-numeric value logs a `WARN` and falls
back to that default; `0` is accepted as written and disables caching, so every lookup goes back to
Microsoft Graph. The caches are an optimisation, never a correctness requirement - a bad value
must not stop the client from being constructed - and they are dropped when the crawl closes the
client.

#### `max_content_length` is applied twice

The same parameter name is read by two different pieces of code, at two different points in a
crawl, with two different consequences. Both are **byte** counts, and neither truncates: content
over the cap is rejected, not shortened.

- **A pre-download size check, in OneDriveDataStore only.** Before a file is fetched, the drive
  item's Graph-reported `size` is compared against the cap. Over it, the item is **not indexed at
  all**: a `MaxLengthExceededException` is raised and the item is recorded in the failed-URL list.
  This is the only place `max_content_length` is checked *before* anything is downloaded; the
  extractor cap below can also stop a document being stored, but only after the fetch.
- **An extractor input cap, wherever text is extracted.** The value is handed to Fess's extractor
  as `maxContentLength(...)`, which measures the bytes it actually read and throws
  `MaxLengthExceededException` if they exceed it. Three call sites use it: OneDrive file contents,
  OneNote page contents, and **Teams attachment text** (reached whenever `append_attachment=true`,
  which is the default). Failing here does not necessarily drop the document - Fess's
  `crawler.ignore.content.exception` decides, and for OneDrive files `ignore_error` does too; when
  it is tolerated the document is indexed with empty contents for that source.

`-1` (the default) does **not** mean unlimited at either point. It means "defer to Fess", and the
effective cap becomes `ContentLengthHelper`'s limit for that MIME type, or its default limit when
the MIME type is unknown. No extractor call site passes a MIME type. OneDrive reaches a
per-MIME-type cap a different way: `OneDriveDataStore` resolves `-1` itself, calling
`ContentLengthHelper.getMaxLength(mimetype)` with the drive item's own MIME type before the file is
downloaded, and rejects an oversized item there. **OneNote page content and Teams attachments always
resolve to Fess's default limit**, because neither consults a MIME type at all - the Teams call
passes a filename, but a filename only steers which extractor is chosen and never reaches the length
lookup.

Two further asymmetries are worth knowing. OneDriveDataStore parses the value as a `long`, while
the Graph client parses it as an `int`: a value above `2147483647` is accepted by the first and
logged as unparseable by the second, which then keeps `-1`. And a Teams attachment or a OneNote
page is never skipped in advance on size - only its extraction can fail - because neither has a
Graph-reported size the way a drive item does.

#### Proxy parameters

`proxy_host` and `proxy_port` take effect only when **both** are set; either one alone is ignored
with no warning. When both are set, the proxy is applied to two independent HTTP stacks: the Azure
Identity client that acquires the OAuth token, and the OkHttp client the Microsoft Graph SDK uses
for every API call. A `proxy_port` that is not a number aborts the crawl with
`DataStoreException: Invalid proxy port`.

`proxy_username` and `proxy_password` are likewise applied only when **both** are non-empty; a user
name with no password (or the reverse) is silently ignored and the proxy is used unauthenticated.
Graph requests authenticate to the proxy with HTTP Basic via a `Proxy-Authorization` header.

See [`additionally_allowed_tenants` and the Graph host allowlist](#additionally_allowed_tenants-and-the-graph-host-allowlist)
below for a bearer-token leak that used to affect proxied deployments specifically.

#### Permission lookup failures

This applies to OneDriveDataStore and SharePointDocLibDataStore, where a document's ACL comes from
a separate Microsoft Graph call (`GET /drives/{id}/items/{id}/permissions`, listing its
permissions) that can fail on its own even when the document itself was read successfully -
typically a transient `429` or `503` under throttling.

SharePointListDataStore, SharePointPageDataStore, and OneNoteDataStore do not call this or any
other permission-fetch endpoint, so `permission_failure_policy` has no effect on any of them:
SharePointListDataStore and SharePointPageDataStore rely solely on `default_permissions` (see
[What ACL Each DataStore Can Produce](#what-acl-each-datastore-can-produce) above); OneNoteDataStore's
user and group notebooks resolve permissions from data already in hand while enumerating - the
same mechanism OneDriveDataStore uses for its own personal and group drives, synthesizing a role
from the owner's ID rather than calling Graph for it - and its site notebooks rely solely on
`default_permissions` too. TeamsDataStore does not call any of the permission-fetch methods this
parameter governs either, so this parameter has no effect for it too - but not because
TeamsDataStore has no permission-fetch failure path of its own. It makes a separate Graph call per
**channel** (listing the channel's members, to resolve its messages' roles), issued lazily on the
first message of that channel that needs it and reused for every later message and reply in it.
A failure there is not governed by `permission_failure_policy` at all: it propagates out of the
per-channel handler as a `DataStoreException`, which aborts the rest of that channel's messages
rather than skipping just the one message.

`permission_failure_policy` controls what happens when the permissions call fails for the
DataStores and paths it covers - OneDriveDataStore and SharePointDocLibDataStore - and takes one of
two values:

| Value | Behavior |
|-------|----------|
| `skip` (default) | The document is not indexed. The failure is recorded in the failed URL list and logged at `WARN`, so it is visible after the crawl. |
| `index_without_acl` | The document is indexed with whatever permissions were collected before the failure, possibly none. This was the only behavior before this parameter existed. |

An unrecognized value is logged at `WARN` and treated as `skip`. There is no value that aborts
the crawl: a permission lookup always runs inside a per-item handler that already catches every
exception for that item and moves on to the next one, so nothing thrown while resolving
permissions can stop the crawl.

Indexing a document with an incomplete ACL is not a neutral fallback. An empty ACL removes the
document from every user's search results until it is re-crawled successfully, and when
`default_permissions` is configured, `index_without_acl` instead publishes the document to
everyone that setting covers - more widely than intended, because the per-user/per-group roles
that should have narrowed access down were never added. Choose `index_without_acl` only if
surfacing a document without its full ACL is preferable to not surfacing it at all.

`ignore_error` and `permission_failure_policy` cover different failures. `ignore_error` governs
failures while enumerating and processing containers and their contents - sites, drives, lists,
list items, pages, and (since Teams started honouring the parameter) teams and channels; see
[`ignore_error` scope differs by DataStore](#ignore_error-scope-differs-by-datastore) above for the
exact set per DataStore. It has no effect on permission lookups for an individual document. Those
are governed solely by `permission_failure_policy`.

> **Upgrading from 15.8.x?** Documents indexed by an older version keep their old ACL until re-crawled; see [UPGRADING.md](UPGRADING.md#onedrive-sharepoint-document-libraries-and-onenote-acl-correction).

> **Upgrading from 15.8.x?** Several crawl filters now select different lists, libraries and drives, which a re-crawl applies; see [UPGRADING.md](UPGRADING.md#sharepoint-lists-sharepoint-document-libraries-and-onedrive-crawl-filter-fixes).

#### Graph client timeouts and retries

`connect_timeout`, `read_timeout`, and `access_timeout` (OkHttp's *call* timeout - the ceiling on
the whole request: DNS, connecting, writing, server processing, and reading the response, with
any redirects or retries all counted against the one period) configure the OkHttp client the
Microsoft Graph Java SDK builds internally. That client is built by the SDK's own
`GraphClientFactory`, which delegates to Kiota's `KiotaClientFactory` - and it is the latter that
hard-codes all three to **100 seconds**, regardless of whether this plugin passes it any options.
This is the Graph client library's own default, not OkHttp's raw
10-second connect/read default (OkHttp's call timeout has no default at all - `0`, meaning
unlimited). `0`, or leaving one of these parameters unset, keeps that 100-second default.

Setting one above `2147483` seconds - the largest whole-second value OkHttp's client builder
accepts - logs a `WARN` and uses `2147483` instead of failing. A non-numeric value also falls back
to the 100-second default, with a `WARN` logged. A negative value is treated the same as `0` (the
100-second default is kept), and now also logs a `WARN` naming the parameter and the value, so an
`access_timeout=-1` meant as "no timeout" is not silently ignored.

`max_retry_count` (default `3`, maximum `10`) and `retry_interval` (default `3` seconds, maximum
`180` seconds) configure the SDK's retry handler; which responses get retried is not
configurable. A value above the maximum, or a non-numeric value, logs a `WARN` and falls back to
the maximum or the default respectively. A negative value also logs a `WARN`, but falls back to
`0` - not to the parameter's own default of `3` - so a negative `max_retry_count` disables
retries entirely rather than reverting to three attempts, and a negative `retry_interval` removes
the delay between them.

#### `additionally_allowed_tenants` and the Graph host allowlist

The credential backing this client used to accept tokens for any Azure AD tenant
(`additionallyAllowedTenants("*")`, hard-coded). It now accepts tokens only for the configured
`tenant` unless `additionally_allowed_tenants` says otherwise. Set
`additionally_allowed_tenants=*` to restore the old behavior, or list specific tenant IDs
separated by commas (surrounding whitespace and empty entries are ignored) to allow just those.
No code path in this plugin was found that requests a token for a tenant other than the
configured one, which is why the default was changed - "no path was found", not "no path exists".
An installation that does something unusual with the underlying credential outside this plugin's
own code could still depend on the old, unrestricted behavior.

Separately, a deployment that configures `proxy_host`/`proxy_port` used to send the bearer token
to any host named by a Graph response's `@odata.nextLink`, even one outside the six Microsoft
Graph national-cloud hosts. A deployment without a proxy was never affected - it already carried
that same six-host restriction. The proxied path now gets it too: a `@odata.nextLink` naming a
host outside that list is still followed, but without the `Authorization` header, rather than
leaking the tenant's app-only token to it.

#### What the two crawl-completion `ERROR` messages mean

Every DataStore in this plugin now shuts down its crawling thread pool through one shared helper,
which can log two new kinds of `ERROR` on a code path that previously said nothing at all. In this
project, `ERROR` from `org.codelibs` is wired to notifications, so upgrading can mean a crawl
configuration that looked clean starts producing notifications - the failures documented below
were already happening; only the reporting is new. Both are bounded at one line per crawl.

- `<name>: N crawling task(s) were still running and M had not started after T seconds. They are
  about to be cancelled, and the documents they would have produced are missing from this crawl.
  Raise executor_shutdown_timeout for a large tenant.` - the wait configured by
  `executor_shutdown_timeout` expired while tasks were still queued or in flight; those tasks are
  cancelled and their documents never get produced. Before this release, the same expiry cancelled
  those tasks just as silently, and the crawl reported success anyway. Raising
  `executor_shutdown_timeout` is the direct fix; `number_of_threads` is the other lever, but note
  that it is capped (see below), and the effective value sets both the pool size and the queue
  capacity - so at most `2 x` the *effective* thread count can be running or queued when the wait
  begins.
- `<name>: N crawling task(s) failed; their documents are missing from this crawl. See the warnings
  above.` - a count of tasks that ended by throwing, logged once per crawl. Each individual failure
  it counts is logged separately at `WARN`, as `<name>: a crawling task failed.`, one line per
  failed task - `WARN` rather than `ERROR` because `SharePointListDataStore` and
  `SharePointPageDataStore` submit one task per document, so a bad crawl would otherwise raise one
  notification per document. This `ERROR` is the notification, and there is at most one of it per
  crawl. Its count is a snapshot taken as the shutdown ends, and on a timed-out shutdown the tasks
  being cancelled can still go on to fail after it is read, so it can undercount; the `WARN` lines
  are the complete record. Before this release, an escaped failure either vanished with no log line
  at all, or - when the pool's queue was already full - ran on the submitting thread instead and
  propagated out of the crawl itself, aborting it outright; which of the two happened depended on
  queue timing, not on anything the operator controlled.

Both messages, and the per-task failure line, name the concrete DataStore by its Java class name
(e.g. `OneDriveDataStore`), so an operator running several Microsoft 365 crawls in one Fess
instance can tell which one produced it.

#### `number_of_threads` is capped

The requested value is capped to `min(number_of_threads, availableProcessors() x 2)`, and the
capped value sets both the pool size and the queue capacity. On a 4-core host, `number_of_threads`
of 32 gives 16 threads and a 16-deep queue, not 32 and 32; raising the parameter past the cap has
no effect at all. The cap is logged at `DEBUG` when it applies.

### Teams-Specific Parameters

| Parameter | Description | Default | Notes |
|-----------|-------------|---------|-------|
| `team_id` | Specific team ID to crawl | All teams | Microsoft 365 group ID. Omit the line, or leave it empty (`team_id=`), to crawl all teams - both are treated the same |
| `exclude_team_ids` | Comma-separated team IDs to exclude | - | Multiple teams to skip |
| `include_visibility` | Team visibility levels to include | All | Comma-separated: `public`, `private` |
| `channel_id` | Specific channel ID to crawl | All channels | Within specified team |
| `chat_id` | Specific chat ID to crawl | - | For 1:1 or group chats |
| `ignore_replies` | Skip reply messages | `false` | Process only root messages |
| `append_attachment` | Include attachments in content | `true` | Append attachment text to message body |
| `max_content_length` | Byte cap on the text extracted from **each Teams attachment** | `-1` (defer to Fess's **default** content-length limit - the Teams extraction passes no MIME type, so a per-MIME-type limit can never apply here) | Only reached when `append_attachment=true`. An attachment over the cap contributes no text; whether that also fails the message depends on Fess's `crawler.ignore.content.exception`. See [`max_content_length` is applied twice](#max_content_length-is-applied-twice) |
| `ignore_system_events` | Skip system event messages | `true` | Filter out system notifications |
| `title_dateformat` | Date format for message titles | `yyyy/MM/dd'T'HH:mm:ss` | Java date pattern |
| `title_timezone_offset` | Timezone offset for message titles | `Z` | e.g., `Z`, `+09:00`, `-05:00` |
| `start_date` | Inclusive earliest message timestamp to index | - | `yyyy-MM-dd` (UTC start of day) or an ISO-8601 date-time **carrying an explicit offset or `Z`**, e.g. `2026-01-01` or `2026-01-01T09:00:00+09:00`. A zone-less `2026-01-01T09:00:00` is rejected. An unparseable value is logged and ignored |
| `end_date` | Inclusive latest message timestamp to index | - | `yyyy-MM-dd` (UTC end of day, `23:59:59.999999999Z`) or an ISO-8601 date-time carrying an explicit offset or `Z`. An unparseable value is logged and ignored |
| `number_of_threads` | Number of processing threads | `1` | Concurrent message processing |
| `default_permissions` | Default role assignments | - | Additional permissions for all messages |
| `ignore_error` | Continue crawling when a team, channel or chat cannot be processed | `false` | When `true`, an unresolvable `team_id` or `channel_id`, a team whose channels cannot be listed, a channel whose messages cannot be fetched, and a configured `chat_id` that fails to process are logged and skipped instead of aborting the crawl. Failures while enumerating **all** teams were already skipped and are unaffected; a `team_id` listed in `exclude_team_ids` that cannot be resolved still aborts, so that a team you asked to exclude is never silently crawled |

**How the Teams date range is applied**

**The filtering is client-side.** `start_date` and `end_date` are applied **after** the messages
are fetched. Microsoft's `channel: list messages` API supports only `$top` and `$expand` - "The
other OData query parameters aren't currently supported" - so there is no server-side date filter
to use. Every message in every crawled channel is still retrieved from Microsoft Graph and every
page is still walked; the range reduces what is **indexed**, not what is **transferred**. Setting
`start_date` to shorten a crawl does not work: it saves no Graph requests, no bandwidth and
(except for the reply listings noted below) no time.

The bound is compared against the message's `createdDateTime`, falling back to
`lastModifiedDateTime` when that is absent. A message with neither timestamp is indexed rather
than dropped: a missing timestamp must never be a reason to silently shrink the index. A timestamp
Microsoft Graph returns in a form the SDK cannot parse never reaches the range check at all - it
fails while the message list is being deserialized, which is a channel-level failure governed by
`ignore_error`, not by these two parameters.

Both root messages and replies are tested against the range. Because replies are only fetched for
a root message that was itself indexed, a root outside the range also excludes its replies - even
a reply that would have fallen inside it. This keeps a reply from ever being indexed with a
`parent` that was never processed, and it is the one place the range does save Graph traffic. A
reply is always at or after its root, so an `end_date` that excludes a root correctly excludes its
replies; only `start_date` can drop an in-range reply, and only from a conversation whose opening
message you asked not to index.

**`chat_id` is judged across the whole conversation.** A chat is consolidated into a single
document, so the range is evaluated once for the chat rather than per message, and the decision is
all-or-nothing: the entire conversation is indexed or none of it is, never a subset. The test is
whether **any** of the chat's messages falls inside the range - not the consolidated document's own
timestamp. That timestamp is inherited from whichever message Graph happened to return first (the
listing sets no `$orderby` and is not sorted afterwards), so judging by it would drop a chat
spanning years on the strength of one arbitrary message.

Two consequences follow from that shape:

- The indexed body is the **whole, unfiltered** conversation. Consolidating a chat into one
  document leaves no way to index part of it, so a range does not trim a chat's contents - it only
  decides whether the chat is indexed.
- `message.created_date_time` on that document is unchanged by the range. It is still the inherited
  timestamp described above, and it does **not** tell you which message put the chat in range.

A malformed `start_date` or `end_date` is logged once at `WARN` when the crawl starts and then
treated as unset, so that bound simply does not filter. The same treatment is applied to an
**inverted** range - a `start_date` later than the `end_date`, which would match no message at all:
one `WARN` at crawl start, and then **both** bounds are dropped, so the crawl runs completely
unfiltered rather than reporting success over an empty index. Equal bounds are a legitimate
one-instant range and do not warn, and a lone bound cannot be inverted, so it does not either.

A typo therefore gives you the unfiltered crawl you had before the parameter existed - it never
aborts the crawl and never silently empties the index. With both parameters unset - the default -
nothing is filtered and behaviour is unchanged.

> **Upgrading from 15.8.x?** An all-teams crawl now indexes documents and the data config's Permissions field now reaches Teams ACLs; review before re-crawling, see [UPGRADING.md](UPGRADING.md#teams-and-onenote-all-teams-crawl-and-the-permissions-field).

#### Teams Implementation Details

The TeamsDataStore provides comprehensive Microsoft Teams content crawling with the following capabilities:

**Core Functionality:**
- **Team-based Crawling**: Processes teams, channels, and messages hierarchically
- **Chat Support**: Crawls 1:1 and group chat conversations when chat_id is specified
- **Message Aggregation**: Consolidates chat messages into searchable conversation threads
- **Permission Mapping**: Extracts team/channel membership and maps to Fess role-based access control

**Crawling Modes:**
- **All Teams**: Leave `team_id` empty to crawl all accessible teams
- **Specific Team**: Set `team_id` to crawl only that team's channels and messages
- **Team Filtering**: Use `exclude_team_ids` to skip specific teams (comma-separated IDs)
- **Visibility Filtering**: Use `include_visibility` to filter by team visibility (public/private)
- **Specific Channel**: Set both `team_id` and `channel_id` to crawl a single channel
- **Chat Conversations**: Set `chat_id` to crawl specific chat conversations

**Content Processing:**
- **Message Title Generation**: Creates searchable titles using sender name and formatted timestamp
- **Content Extraction**: Extracts message body content (text/HTML) with proper formatting
- **Attachment Handling**: Optionally includes attachment information in message content
- **Reply Threading**: Supports crawling of reply messages with parent message context
- **System Event Filtering**: Automatically filters out system-generated messages

**Message Metadata Fields:**
The implementation extracts comprehensive message metadata including:
- Basic properties: id, subject, body, created/modified timestamps
- Sender information: from user/application details
- Conversation context: team, channel, parent message references
- Interaction data: mentions and importance level (reactions are **not** published as a script key)
- Rich content: attachments, hosted contents, web URLs
- Permission data: role-based access control from team/channel membership

See the [Teams script key table](#teams) above for the full list of `message.*` keys.

**Performance Optimizations:**
- **Multi-threaded Processing**: a configurable thread pool, whose unit of work is a **channel**.
  Channels are processed in parallel; the messages and replies within one channel are processed
  serially on that channel's thread
- **Pagination**: message, channel and chat listings follow Graph's `@odata.nextLink` directly.
  `$select` and `$orderby` are not supported by the channel-messages or chat-messages endpoints, so
  every property is returned and any ordering is applied client-side
- **Permission Caching**: a channel's member list is resolved once, on the first message that needs
  it, and reused for every later message and reply in that channel

**Error Handling & Resilience:**
- **Configurable Error Handling**: `ignore_error` relaxes the five failures that abort a Teams
  crawl today - an unresolvable `team_id`, an unresolvable `channel_id`, a failure listing an
  explicitly configured team's channels, a failure fetching a channel's messages, and a failure
  processing a configured `chat_id`. It does not change any other path: at the default `false` a
  Teams crawl behaves exactly as it did before the parameter was honoured
- **Always-tolerated failures**: when crawling **all** teams (no `team_id`), a team whose channels
  cannot be listed is skipped regardless of `ignore_error`. At normal log levels this is one `WARN`
  ("Skipping this team"). Note that enabling `DEBUG` for this class **replaces** that `WARN` with a
  `DEBUG` line carrying the stack trace - it does not add to it - so a log filter watching for the
  `WARN` stops matching once `DEBUG` is on
- **Never-tolerated failures**: an `exclude_team_ids` entry that cannot be resolved always aborts,
  even with `ignore_error=true`; ignoring it would crawl a team you asked to exclude
- **Per-message failures**: a message that fails to index is recorded in the failure-URL list and
  the crawl continues, independent of `ignore_error`
- **Comprehensive Logging**: Debug and info level logging for monitoring progress
- **Thread Pool Management**: Proper executor service shutdown and cleanup
- **Interruption Handling**: Graceful handling of thread interruption

**Content Filtering:**
- **Reply Message Filtering**: Option to skip reply messages and process only root messages
- **Date Range Filtering**: `start_date`/`end_date` bound which messages are indexed, applied
  client-side after fetching - see "How the Teams date range is applied" above
- **System Event Filtering**: Automatic detection and filtering of system-generated events
- **No pattern matching**: `include_pattern` and `exclude_pattern` are **not** supported by
  TeamsDataStore. It reads neither one, so setting either has no effect at all - not on message
  content, not on message URLs, and not on team or channel names. See the
  [semantics table](#include_pattern--exclude_pattern-semantics-differ-by-datastore) above. Use
  `team_id`, `exclude_team_ids`, `include_visibility`, `channel_id` and the date range to narrow a
  Teams crawl

**Use Cases:**
- **Team Communication Search**: Find conversations across teams and channels
- **Knowledge Discovery**: Search team discussions for solutions and decisions
- **Compliance Monitoring**: Index team communications for compliance requirements
- **Chat History Search**: Search through direct and group chat conversations

### OneNote-Specific Parameters

| Parameter | Description | Default | Notes |
|-----------|-------------|---------|-------|
| `site_note_crawler` | Enable crawling of site notebooks | `true` | Crawls notebooks in every SharePoint site in the tenant, or in `site_id` alone when that is set. Earlier releases crawled the root site and nothing else |
| `site_id` | Restrict site notebooks to one site | - | Site ID as `hostname,siteCollectionId,siteId`. Leave unset to enumerate every site, the same way `sharePointPageDataStore` reads this parameter. Under the delegated authentication OneNote requires, the sites listed are the ones the signed-in account can reach (`GET /sites?search=*`) |
| `user_note_crawler` | Enable crawling of user notebooks | `true` | Crawls personal OneNote notebooks for licensed users |
| `group_note_crawler` | Enable crawling of group notebooks | `true` | Crawls shared notebooks in Microsoft 365 groups |
| `include_pattern` | Regex a notebook name must fully match to be crawled | - | Matched against the notebook's display name with `Matcher.matches()` (full match). A regex that does not compile aborts the crawl at its start |
| `exclude_pattern` | Regex a notebook name must not fully match to be crawled | - | Matched against the notebook's display name with `Matcher.matches()` (full match). A regex that does not compile aborts the crawl at its start |
| `max_content_length` | Byte cap on the text extracted from **each OneNote page** | `-1` (defer to Fess's **default** content-length limit) | The extraction passes no MIME type or filename, so `-1` always resolves to Fess's *default* limit, never a per-MIME-type one. See [`max_content_length` is applied twice](#max_content_length-is-applied-twice) |
| `default_permissions` | Roles added to every notebook's ACL | - | For site notebooks this is the only role source besides the data config's own Permissions field; for user and group notebooks it is added on top of the owner-derived role |
| `number_of_threads` | Number of processing threads | `1` | Controls concurrent notebook processing |

If `include_pattern` or `exclude_pattern` is configured and, across all enabled scopes combined,
it admits zero of the notebooks the crawl actually saw, the crawl still finishes normally but logs
one `WARN` summarizing that - a hint that the pattern may be misconfigured, since the same crawl
otherwise reports success while indexing nothing. The warning names only the parameter (or
parameters) the configuration actually sets.

#### OneNote requires delegated authentication

Microsoft retired app-only tokens for the Microsoft Graph OneNote API on **2025-03-31**. Every
`/onenote/` request made with a client-credentials (app-only) token now comes back as HTTP 401
carrying "this API will no longer support app-only tokens", whatever application permissions the
app registration holds. No `Notes.Read.All` grant changes this. `oneNoteDataStore` therefore has to
sign in as a user; every other DataStore in this plugin is unaffected and keeps working app-only.

Set `username` and `password` on the **OneNote data config only**:

```properties
tenant=********-****-****-****-************
client_id=********-****-****-****-************
username=crawler@contoso.onmicrosoft.com
password=********
```

`client_secret` is not needed there (the delegated flow is a public-client one) and is ignored when
`username`/`password` are set. Setting only one of the two is rejected at startup rather than
falling back to app-only, so a typo cannot silently crawl as the wrong identity.

**App registration:** enable *Allow public client flows* and grant the **delegated** equivalents of
the permissions in [Required Permissions by DataStore](#required-permissions-by-datastore) -
`Notes.Read.All`, plus `User.Read.All`, `Group.Read.All` and `Sites.Read.All` for the scopes you
enable - with admin consent. This is the resource-owner-password-credentials flow, so the account
must not be subject to MFA or a conditional-access policy that blocks it. Microsoft has deprecated
this flow; it is the only unattended delegated flow a crawler can use, and Microsoft has not
announced a retirement date for it.

**What the crawl can reach.** A delegated token sees what the signed-in account sees, so coverage is
narrower than app-only was and is set by that account's memberships rather than by a tenant-wide
grant:

| Scope | Delegated token returns |
|-------|-------------------------|
| Site notebooks (`site_note_crawler`) | notebooks in the SharePoint sites the account can read |
| Group notebooks (`group_note_crawler`) | notebooks of the Microsoft 365 groups the account is a **member** of |
| User notebooks (`user_note_crawler`) | only what each user has **shared with** the account - `/users/{id}/onenote` is documented as returning the content that user shared with the signed-in user, so personal notebooks cannot be crawled exhaustively |

Recommended setup, which keeps site and group notebooks fully covered:

1. Create a dedicated service account, excluded from MFA and with a non-expiring password.
2. Add it as a **member** of every Microsoft 365 group whose notebooks should be indexed.
3. Grant it **read** on every SharePoint site whose notebooks should be indexed.
4. Accept that personal notebooks are only indexed where their owner shared them with the account.

Owners the account cannot see are skipped with a `WARN` and the crawl continues - under this setup
that is the normal case, not a fault. A 401, by contrast, means the credentials themselves were
refused and aborts the crawl unless `ignore_error=true`; before this was fixed such a crawl finished
reporting success with nothing indexed.

ACLs are unaffected: notebooks still carry the roles described above, so query-time permission
filtering behaves exactly as before. What changes is coverage, not access control.

> **Upgrading an existing OneNote data config?** It needs `username`/`password` and a re-crawl; see [UPGRADING.md](UPGRADING.md#onenote-delegated-authentication-and-site-notebooks-on-every-site) and [UPGRADING.md](UPGRADING.md#onenote-failure-url-rows-keyed-by-notebook-url).

#### OneNote Implementation Details

The OneNoteDataStore provides comprehensive OneNote notebook crawling with the following implementation features:

**Core Functionality:**
- **Multi-Source Notebook Crawling**: Processes notebooks from three distinct sources in a systematic order
- **Aggregated Content Extraction**: Consolidates all sections and pages within each notebook into searchable content
- **Permission Mapping**: User and group notebooks get a role synthesized from the owner's ID,
  plus `default_permissions` when configured. Site notebooks carry no per-user or per-group role
  at all - Microsoft Graph exposes no app-only way to read a SharePoint site's user and group role
  assignments (see [What ACL Each DataStore Can Produce](#what-acl-each-datastore-can-produce) in
  the permissions section) - so `default_permissions` is their only crawl-parameter role source.
  The data config's own Permissions field is folded in as well, for every notebook type; leave
  **both** unset and site notebooks are indexed but findable by nobody.

**Crawling Modes (Processing Order):**
1. **Site Notebooks**: Crawls notebooks in every SharePoint site (`/sites/{siteId}/onenote/notebooks`), or in `site_id` alone when that is set
2. **User Notebooks**: Iterates through all licensed users and crawls their personal notebooks (`/users/{userId}/onenote/notebooks`)
3. **Group Notebooks**: Crawls shared notebooks associated with Microsoft 365 groups (`/groups/{groupId}/onenote/notebooks`)

**Content Processing Pipeline:**
1. **Notebook Discovery**: Uses Microsoft Graph API to enumerate notebooks based on enabled crawling modes
2. **Section Traversal**: For each notebook, retrieves all sections within it
3. **Page Content Extraction**: For each section, fetches all pages and extracts their HTML content
4. **Content Aggregation**: Combines all page content using Tika to extract plain text from HTML
5. **Metadata Enrichment**: Captures notebook metadata including creation/modification times and access URLs

**Configuration Flexibility:**
- **Selective Crawling**: Enable/disable specific notebook sources independently
- **Boolean Parameter Handling**: Case-insensitive boolean values (`true`, `True`, `TRUE`, `false`, `False`, `FALSE`)
- **Invalid Value Handling**: Invalid boolean values default to `false` for safety
- **Null Value Handling**: Null or missing parameters use default values (all crawlers enabled by default)

**Performance Optimizations:**
- **Concurrent Processing**: Configurable thread pool for parallel notebook processing
- **Content Size Tracking**: Monitors and reports content size for each notebook

Notebooks, sections, pages and page content are each fetched with their own Graph request; the
plugin issues no `$batch` request anywhere.

**Error Handling & Resilience:**
- **Graceful Degradation**: Handles invalid parameter values by defaulting to safe configurations
- **Thread Pool Management**: Proper executor service lifecycle management with shutdown handling
- **Comprehensive Logging**: Debug-level logging for monitoring crawling progress and troubleshooting
- **404 Visibility**: A `404` when listing a site's or group's notebooks is logged at `WARN`, not
  `debug` - it usually means that site or group has no notebooks, but the same response is also
  what a request sent to the wrong Graph path would return, so it is worth seeing. A `404` for a
  *user's* notebooks stays at `debug`: an unprovisioned personal site 404s there routinely for any
  tenant with unlicensed-for-OneDrive or never-logged-in users, and the user path was never the
  one this fix repaired, so logging one `WARN` per such user would add volume without adding
  diagnostic value.
- **Site Enumeration Failures**: a failure listing the tenant's sites - or resolving the one named
  by `site_id` - only skips the site notebooks; user and group notebook crawling continues
  regardless. A single site that Graph returns without an id is skipped on its own, leaving the
  rest of the tenant to be crawled. `permission_failure_policy` is not involved: OneNoteDataStore
  never calls a Graph permission-fetch endpoint for any notebook type, so that parameter has no
  effect on it
- **Credential Rejection**: a 401 while listing notebooks aborts the crawl unless
  `ignore_error=true`, because the same credentials are used for every site, user and group and
  the crawl could only end with nothing indexed. When the credential is app-only, the failure
  names the OneNote app-only retirement and the `username`/`password` parameters that fix it

**Content Metadata Fields:**
The implementation extracts and indexes the following notebook metadata:
- `notebook.name`: The display name of the notebook
- `notebook.contents`: Aggregated text content from all sections and pages
- `notebook.size`: Total size of the extracted content in characters
- `notebook.created`: Notebook creation timestamp
- `notebook.last_modified`: Last modification timestamp
- `notebook.web_url`: Direct link to open the notebook in OneNote
- `notebook.roles`: the notebook's roles - the owner-derived role for user and group notebooks,
  plus `default_permissions` and the data config's Permissions field

**Use Cases:**
- **Knowledge Base Search**: Search across organizational OneNote documentation
- **Personal Note Discovery**: Find information in personal OneNote notebooks
- **Team Collaboration Search**: Search shared team notebooks for meeting notes and project documentation
- **Cross-Platform Content**: Index OneNote content created from web, desktop, and mobile applications

### OneDrive-Specific Parameters

| Parameter | Description | Default | Notes |
|-----------|-------------|---------|-------|
| `max_content_length` | Maximum file size in bytes. Checked against the drive item's Graph-reported size before download, and again by the extractor against the bytes read | `-1` (defer to Fess's per-MIME-type limit, **not** unlimited) | Files over the cap are not indexed at all - see [`max_content_length` is applied twice](#max_content_length-is-applied-twice) |
| `ignore_folder` | Skip folder documents | `true` | Process files only, ignore folders |
| `supported_mimetypes` | Supported MIME types pattern | `.*` | Regex pattern for supported file types |
| `drive_id` | Additional specific drive ID to crawl | - | Adds a fourth crawl of that one drive. It does **not** restrict the others: the three `*_drive_crawler` modes still run according to their own flags (all default to `true`), so setting only `drive_id` crawls that drive *in addition to* everything else. To crawl one drive and nothing else, also set `shared_documents_drive_crawler`, `user_drive_crawler` and `group_drive_crawler` to `false` |
| `shared_documents_drive_crawler` | Enable SharePoint document library crawling | `true` | Enumerates every SharePoint site and crawls the files in its document libraries (drives whose `driveType` is `documentLibrary`). Personal sites, which `GET /sites` also returns, are skipped: their drive is the user's OneDrive (`business`), crawled only by `user_drive_crawler`. It does **not** crawl the signed-in user's own OneDrive - no `/me/drive` request is ever issued by this DataStore |
| `user_drive_crawler` | Enable user drives crawling | `true` | Crawl all licensed users' drives |
| `group_drive_crawler` | Enable group drives crawling | `true` | Crawl Microsoft 365 group drives |
| `ignore_system_libraries` | Skip system libraries (Style Library, `FormServerTemplates`, and libraries under `_catalogs`) | `true` | Applies whenever `shared_documents_drive_crawler=true` (default), to the sub-mode that enumerates all SharePoint sites' document libraries (Crawling Mode 1 below) - independent of `drive_id`. Setting `drive_id` runs an additional, separate crawl (Crawling Mode 4) that does not go through this check; it does not turn off Mode 1. Has no effect on personal or group drives. Matched case-insensitively against the library's own URL segment, same as [SharePoint Document Library Parameters](#sharepoint-document-library-parameters) below. `false` also asks Graph for drives that carry the `system` facet, which it hides by default |
| `sensitivity_label_enabled` | Read each file's sensitivity labels | `false` | Costs one extra Graph request per targeted file. See [Sensitivity labels](#sensitivity-labels) |
| `sensitivity_label_policy.<label>` | The rule for one label: `<action>[;<action>...]` | - | One parameter per rule, e.g. `sensitivity_label_policy.Confidential=no_content`. Ignored, with a warning, unless `sensitivity_label_enabled=true`. A malformed rule stops the crawl |
| `sensitivity_label_failure_policy` | What to do with a file whose labels cannot be read | `skip` | `skip` (record a failure URL and do not index the file) or `index_without_label` (index it with what could be read) |
| `sensitivity_label_extensions` | Comma-separated file extensions whose labels are read | Office formats and `pdf` | The file types Microsoft Purview can label in SharePoint and OneDrive. Other files are treated as unlabeled without a request |

#### Sensitivity labels

With `sensitivity_label_enabled=true`, OneDriveDataStore reads the
[Microsoft Purview sensitivity labels](https://learn.microsoft.com/en-us/purview/sensitivity-labels)
of each file whose extension is in `sensitivity_label_extensions`, after the MIME type and
include/exclude filters and before the file is downloaded:

1. `POST /drives/{drive-id}/items/{item-id}/extractSensitivityLabels` returns the label IDs on the
   file (`Files.Read.All`). Microsoft documents that this call re-extracts the label from the file
   when SharePoint's stored copy is stale, and updates the item's metadata to match.
2. `GET /security/dataSecurityAndGovernance/sensitivityLabels` and
   `GET /security/dataSecurityAndGovernance/sensitivityLabels/{id}/sublabels` load the tenant's
   labels - name, display name, parent label and whether each applies encryption
   (`SensitivityLabel.Read`). They are loaded once per crawl, on the first labeled file.

The labels are exposed as `file.sensitivity_label_ids`, `file.sensitivity_label_names` and
`file.sensitivity_label_protected`. To index them, map them in the script, for example
`sensitivity_label=file.sensitivity_label_names`. Fess maps a new field dynamically; to return it
from the search API or filter on it, add it to `query.additional.response.fields`,
`query.additional.api.response.fields` and `query.additional.search.fields` in
`fess_config.properties`.

`sensitivity_label_policy.<label>=<action>[;<action>...]` parameters decide what happens to a
labeled file, one parameter per rule. (The data config's Parameters field is read one `key=value`
per line, so a single parameter cannot hold several rules; a bare `sensitivity_label_policy`
parameter stops the crawl.)

| `<label>` | Matches |
|-----------|---------|
| a label ID (GUID) | that label |
| a label name or display name | that label, case-insensitively. Needs `SensitivityLabel.Read` |
| a parent label's ID, name or display name | its sublabels. Needs `SensitivityLabel.Read` |
| `@protected` | any label that applies encryption. Needs `SensitivityLabel.Read` |
| `*` | any label not matched by a rule above |

| Action | Effect |
|--------|--------|
| `index` | Index the file as usual. Use it to exempt a label from `*` or `@protected` |
| `skip` | Do not index the file. It is discarded, not recorded as a failure |
| `no_content` | Index the file without downloading it: metadata only, `file.contents` is empty, and `max_content_length` does not apply |
| `restrict:<permissions>` | Keep only those roles of the file's ACL that are also listed, in the `default_permissions` syntax (`{user}`, `{group}`, `{role}`). The ACL is narrowed, never widened. A file left with no role is not indexed |

A file carries the sublabel it was labeled with (for example `Confidential\All Employees`), never
the parent label, so a rule for a parent label also applies to its sublabels. A label is matched by
its own ID, its own name, its parent's ID, its parent's name, `@protected` and `*`, in that order,
and only the first matching rule applies to it. When a file carries several labels, the result is
never less restrictive than any one of them: `skip` and `no_content` apply if any label asks for them, and
`restrict` lists are intersected.

For example, to keep `Highly Confidential` and its sublabels out of the index, index
`Confidential` and its sublabels without content for the legal and executive groups only, and
index other encrypted labels without content (which is also the default):

```
sensitivity_label_enabled=true
sensitivity_label_policy.Highly Confidential=skip
sensitivity_label_policy.Confidential=no_content;restrict:{group}legal@example.com,{group}executives@example.com
sensitivity_label_policy.@protected=no_content
```

**`restrict` and the file's ACL.** Microsoft Graph does not expose who a label's encryption
admits, so `restrict` cannot be derived from the label; it is the operator's statement of who may
see files with that label. It is intersected with the roles built from the file's ACL,
`default_permissions` and the data config's Permissions field, both in `file.roles` and in the role
field of the stored document, so no script can widen it. The comparison is by role string: list
groups and users in the form the ACL produces them - the object ID, or the UPN or group mail /
mail nickname / display name that is added next to it. A user granted access to the file directly,
rather than through a listed group, loses that access.

**Encrypted files.** Microsoft Graph returns a file protected by a label's encryption still
encrypted, so its content cannot be extracted. When no `@protected` rule is configured, a label
that applies encryption and has no rule of its own gets the `*` rule, if any, with `no_content`
added, so the file is searchable by name and metadata instead of failing extraction. This default needs the label definition; without
`SensitivityLabel.Read` such files are downloaded as before. Set `@protected=index` to download
them anyway.

**When labels cannot be read.** If `extractSensitivityLabels` fails - for example with `423 Locked`
for a double-key-encrypted file - `sensitivity_label_failure_policy` applies: `skip` (default)
records a failure URL and does not index the file; `index_without_label` indexes it as unlabeled.
The same policy applies when a label's definition cannot be read, no rule names the label's own ID,
and the policy has any rule other than `*`, because that rule might have matched it by name,
through its parent or by `@protected`. There `index_without_label` keeps the label by ID only: its
`*` rule and the rules of the file's other labels still apply. An `extractSensitivityLabels`
response without a `labels` property is treated as a failure, not as an unlabeled file.

> **Upgrading from 15.8.x?** Corrected drive-item URLs re-index as new documents, and the per-item failure log line changed; see [UPGRADING.md](UPGRADING.md#onedrive-drive-item-url-encoding-fix) and [UPGRADING.md](UPGRADING.md#onedrive-per-item-failure-log-line).

#### OneDrive Implementation Details

The OneDriveDataStore provides comprehensive Microsoft 365 file crawling capabilities with the following implementation features:

**Core Functionality:**
- **Multi-Drive Type Support**: Processes files from OneDrive personal drives, SharePoint document libraries (via Drive API), and Microsoft 365 group drives
- **Hierarchical File Traversal**: Recursively crawls drive items starting from root, handling both files and folders with proper parent-child relationships
- **Content Extraction & Indexing**: Each file becomes a searchable entity with extracted content, metadata, and permission information
- **Permission Integration**: Extracts and maps Microsoft 365 access permissions to Fess role-based access control

**Crawling Modes (Processing Order):**
1. **Shared Documents Drive**: Enumerates every SharePoint site (`GET /sites`) and crawls the files in each site's document libraries (drives whose `driveType` is `documentLibrary`, so personal sites' OneDrive drives are skipped; honoring `ignore_system_libraries`, see the parameters table above). Despite the mode's name it never touches the signed-in user's own OneDrive: the Graph client does have a `/me/drive` call path, but it is reachable only when a null drive ID is passed, and this DataStore's only caller is guarded to pass a non-blank one. Runs whenever `shared_documents_drive_crawler=true` (default), regardless of whether `drive_id` is also set for Mode 4 below - the two crawls run independently, not exclusively
2. **User Drives**: Iterates through all licensed users and crawls their personal OneDrive (`/users/{userId}/drive`)
3. **Group Drives**: Crawls Microsoft 365 group-associated drives (`/groups/{groupId}/drive`)
4. **Specific Drive**: Targets a single drive by ID when `drive_id` parameter is specified (`/drives/{driveId}`)

**Content Processing Pipeline:**
1. **Drive Discovery**: Uses Microsoft Graph API to enumerate drives based on enabled crawling modes and site/drive access
2. **Item Enumeration**: Retrieves drive items using pagination with `DriveItemCollectionResponse` and `@odata.nextLink` handling
3. **Content Filtering**: Applies MIME type filtering, file size limits, and include/exclude patterns before processing
4. **Content Extraction**: Uses Tika extractor with configurable name (`extractorName`, default: "tikaExtractor") for text extraction from supported file types
5. **Metadata Enrichment**: Extracts comprehensive file metadata including timestamps, permissions, and parent folder information
6. **URL Generation**: Creates user-friendly URLs based on crawler type and SharePoint/OneDrive location patterns

**Performance Optimizations:**
- **Concurrent Processing**: Configurable thread pool (`number_of_threads`) for parallel processing of multiple drives and files
- **Efficient Pagination**: Handles Microsoft Graph API pagination using `@odata.nextLink` with helper methods
- **Smart Filtering**: Pre-filters items by MIME type patterns and file size before expensive content extraction
- **Identity Caching**: The Graph client caches the user/group lookups this DataStore's ACL building repeats - see [`cache_size` and the client's lookup caches](#cache_size-and-the-clients-lookup-caches). There is no drive-ID cache: earlier releases of this document described one (`cachedUserDriveId`/`driveIdCacheLock`), but it had no callers and has been removed

**Error Handling & Resilience:**
- **Configurable Error Tolerance**: `ignore_error` relaxes exactly one failure here - a file whose content cannot be extracted is indexed with empty contents instead of failing. It does not affect item enumeration; see [`ignore_error` scope differs by DataStore](#ignore_error-scope-differs-by-datastore)
- **Exception Classification**: Differentiates between access exceptions and general exceptions for appropriate error handling
- **Failure URL Tracking**: Integration with Fess failure URL service for monitoring and retry capabilities
- **Comprehensive Logging**: Debug-level logging for detailed crawling progress monitoring and troubleshooting

**Content Metadata Extraction:**
The implementation extracts and indexes 30+ metadata fields per file:
- **Basic Properties**: name, description, size, MIME type, file type, creation/modification timestamps
- **Location & Access**: web URLs, WebDAV URLs, processed URLs for SharePoint navigation
- **Version Control**: ETag, CTag for change detection and synchronization
- **Creator Information**: user, application, and device details for created/modified by tracking
- **Rich Metadata**: image/photo/video properties, geographic location data, file hash values
- **Folder Structure**: parent reference information including path, name, and ID
- **Specialized Data**: publication info, search result metadata, special folder classification
- **Permission Data**: role-based access control extracted from Microsoft Graph permissions API

**URL Processing Strategy:**
A `webUrl` that is not a `/_layouts/` viewer URL is indexed as it is. For a `/_layouts/` URL the
implementation rebuilds a path-based URL from the item's parent path and name:
- **SharePoint Libraries**: `{siteUrl}/Shared%20Documents/{path}` for shared/group drives
- **OneDrive Personal**: `{siteUrl}/Documents/{path}` for user drives
- **Custom Drives**: `{driveWebUrl}/{path}` for specific drive crawling - the drive's own Graph
  `webUrl`, not its display name
- **URL Encoding**: Graph percent-encodes `parentReference.path` but not `DriveItem.name`, so the
  parent segments are used unchanged and only the item name is encoded

The library segments above are deliberately the English ones. A SharePoint library's URL segment
does not localize - only its display name does - so `/Shared%20Documents` and `/Documents` are
correct on a non-English tenant too.

**Content Size Management:**
- **Configurable Limits**: `max_content_length` parameter with fallback to Fess content length helper
- **MIME Type Support**: Regex pattern matching for `supported_mimetypes` (default: all types)
- **Folder Handling**: Optional folder document creation controlled by `ignore_folder` parameter
- **Size Validation**: Pre-extraction validation to avoid processing oversized files

### SharePoint Document Library Parameters

| Parameter | Description | Default | Notes |
|-----------|-------------|---------|-------|
| `site_id` | Specific site ID to crawl | All sites | Full site ID format: `hostname,siteCollectionId,siteId` |
| `exclude_site_id` | Site IDs to exclude | - | See format guide below |
| `ignore_system_libraries` | Skip system libraries | `true` | Excludes Style Library, `FormServerTemplates` (Form Templates), and libraries under `_catalogs`, matched case-insensitively against the library's own URL segment. `false` also asks Graph for drives that carry the `system` facet, which it hides by default |
| `number_of_threads` | Number of processing threads | `1` | Concurrent document library processing |
| `ignore_error` | Continue crawling when a site's document libraries cannot be listed | `false` | Applies only while every site is crawled (`site_id` unset). A library that fails to process is skipped at either setting - see [`ignore_error` scope differs by DataStore](#ignore_error-scope-differs-by-datastore) |
| `include_pattern` | Regex pattern matched against the library's browser URL (`doclib.url`), not its display name | - | e.g. `https://contoso\.sharepoint\.com/sites/allowed/.*` |
| `exclude_pattern` | Regex pattern matched against the library's browser URL (`doclib.url`), not its display name | - | e.g. `.*/sites/blocked/.*` |
| `default_permissions` | Default role assignments | - | Additional permissions for all libraries |

> **Upgrading from 15.8.x?** Library links, `ignore_system_libraries` and the pattern filters changed; see [UPGRADING.md](UPGRADING.md#sharepoint-document-libraries-library-url-fix) and [UPGRADING.md](UPGRADING.md#sharepoint-document-libraries-system-libraries-and-pattern-filters-now-apply).

##### exclude_site_id Format

SharePoint site IDs contain commas as part of their format (`hostname,siteCollectionId,siteId`). To properly exclude sites:

- **Single SharePoint site**: Use the full site ID as-is
  ```
  exclude_site_id=site1.sharepoint.com,686d3f1a-a383-4367-b5f5-93b99baabcf3,12048306-4e53-420e-bd7c-31af611f6d8a
  ```

- **Multiple SharePoint sites**: Separate with semicolons (`;`)
  ```
  exclude_site_id=site1.sharepoint.com,guid1,guid1;site2.sharepoint.com,guid2,guid2
  ```

- **Legacy simple IDs**: Comma-separated (for backward compatibility)
  ```
  exclude_site_id=site1,site2,site3
  ```

#### SharePoint Document Library Implementation Details

The SharePointDocLibDataStore provides comprehensive metadata indexing for SharePoint document libraries across your organization with the following key features:

**Core Functionality:**
- **Library-Level Indexing**: Each SharePoint document library becomes a single searchable entity in the Fess index, combining library metadata with site context for enhanced discoverability
- **Site Traversal**: Supports crawling all accessible sites or targeting specific sites using the `site_id` parameter
- **System Library Filtering**: Automatically excludes system document libraries (Form Templates, Style Library, etc.) unless explicitly configured otherwise with `ignore_system_libraries` parameter
- **Permission Integration**: Extracts and maps SharePoint access permissions to Fess role-based access control

**Content Aggregation:**
The implementation creates rich, searchable content by combining:
- Document library name and description
- Parent SharePoint site name and context
- Library creation and modification timestamps
- Access permission information
- Standardized URLs for both Graph API access and user navigation

**URL Generation Strategy:**
- **Graph API URL**: preserved from the Microsoft Graph response for API compatibility
  (`doclib.web_url`)
- **Browser URL**: the URL used for user navigation (`doclib.url`), which is the drive's Graph
  `webUrl` - Graph defines it as the URL that displays the resource in the browser, so it is
  already the correct link
  - Only when Graph returns no `webUrl` is a URL composed from the site URL and the library name:
    `{siteUrl}/Shared%20Documents` for a library named `Documents` or `Shared Documents`, otherwise
    `{siteUrl}/{encodedLibraryName}`. That composition cannot resolve a library whose URL segment
    differs from its display name - see
    [SharePoint document libraries: library URL fix](UPGRADING.md#sharepoint-document-libraries-library-url-fix)

**Multi-Threading Support:**
- Configurable concurrent processing using `number_of_threads` parameter (default: 1)
- Thread-safe execution with proper ExecutorService management and resource cleanup
- Graceful handling of thread interruption during long-running operations
- Executor shutdown waits `executor_shutdown_timeout` seconds (default 60) before cancelling whatever has not finished

**Error Handling & Resilience:**
- Comprehensive error tracking with integration into Fess failure URL service
- Configurable error handling via `ignore_error` parameter (default: false)
- Detailed logging for monitoring and troubleshooting crawling operations
- Statistical tracking for performance monitoring and reporting using CrawlerStatsHelper
- Distinction between CrawlingAccessException and general exceptions for appropriate handling

**Performance Optimizations:**
- Efficient site and drive enumeration using Microsoft Graph API with pagination support
- Drive type filtering to process only document library drives (`documentLibrary` type)
- Parallel processing of multiple document libraries within sites using ExecutorService
- Memory-efficient processing with proper resource management and cleanup

**Configuration Flexibility:**
- **Site Exclusion**: Advanced `exclude_site_id` parameter supporting both simple comma-separated IDs and complex SharePoint site ID format with semicolon separation
- **Pattern Filtering**: Support for `include_pattern` and `exclude_pattern` regex filtering on the library's browser URL (`doclib.url`), not its display name - via the same Fess `UrlFilter` OneDriveDataStore uses
- **Permission Management**: Default permissions assignment via `default_permissions` parameter
- **Threading Control**: Configurable `number_of_threads` for optimal performance tuning

**Integration with Fess Security Model:**
- Automatic extraction of SharePoint permissions using Microsoft Graph API drive permissions endpoint
- Conversion of Microsoft 365 access permissions to Fess role format with proper encoding
- Support for default permission assignment via configuration parameters
- Inheritance of site-level permissions for document libraries with pagination support for large permission sets

**Use Cases:**
- **Document Library Discovery**: Find and access specific SharePoint document libraries across the organization
- **Content Organization**: Search for document libraries by name, description, or site context
- **Permission Auditing**: Identify document libraries and their access permissions
- **Site Navigation**: Discover available document libraries within SharePoint sites

**Important Note**: This data store focuses on document library metadata indexing. For indexing individual files within SharePoint document libraries, use the OneDriveDataStore which handles SharePoint document library files through the Microsoft Graph Drive API.

### SharePoint List Parameters

| Parameter | Description | Default | Notes |
| --- | --- | --- | --- |
| `site_id` | SharePoint site ID containing lists | All sites | Full site ID format: `hostname,siteCollectionId,siteId`. **Not required**: left unset, the DataStore enumerates every accessible site (`GET /sites`) and crawls the lists in each, which also means it then needs `Sites.Read.All` rather than `Sites.Selected` |
| `list_id` | Specific list ID to crawl | All lists | If specified, only this list will be crawled |
| `exclude_list_id` | Comma-separated list IDs to exclude | - | Multiple list IDs separated by commas |
| `list_template_filter` | Filter which lists - and, since this fix, which of their items - are processed, by template type | - | Comma-separated numeric IDs and/or Graph template names, e.g. `100,101` or `genericList,documentLibrary`; see [List Template Types](#list-template-types) below |
| `ignore_system_lists` | Skip system lists | `true` | Excludes lists Graph marks as system-managed (`system` facet) or hidden (`list.hidden`, e.g. `TaxonomyHiddenList`), plus lists like User Information, Workflow Tasks |
| `ignore_error` | Continue crawling on errors | `false` | Set to `true` to skip failed items |
| `include_pattern` | Regex pattern for item titles to include | - | Filter items by title matching |
| `exclude_pattern` | Regex pattern for item titles to exclude | - | Skip items with matching titles |
| `number_of_threads` | Number of processing threads | `1` | Concurrent list processing |
| `default_permissions` | Default role assignments | - | Additional permissions for all items |

#### SharePoint List Implementation Details

The SharePointListDataStore provides comprehensive crawling and indexing of SharePoint lists and list items with the following capabilities:

**Core Functionality:**
- **List Item Indexing**: Each SharePoint list item becomes a searchable entity with dynamic field extraction and content aggregation
- **Site-Specific Crawling**: Set `site_id` to target lists within one SharePoint site. It is optional - unset, every accessible site is enumerated and crawled
- **List Filtering**: Supports crawling all lists or specific lists using `list_id`, with exclusion capabilities via `exclude_list_id`
- **Template-Based Filtering**: Filter which lists - and which of their items are processed - by SharePoint template type, using numeric IDs or Graph template names (e.g. `100`/`genericList` for Generic Lists, `101`/`documentLibrary` for Document Libraries). Without this filter, only generic-list items are processed (unchanged default behavior); setting it also opens up processing of items in the selected template(s), which previously never happened regardless of this filter
- **System List Exclusion**: Automatically skips system lists unless explicitly configured otherwise

**Content Extraction Strategy:**
The implementation intelligently extracts content from list items:
- **Title Extraction**: Searches for title in common fields (Title, LinkTitle, FileLeafRef), then in the `URL` field of a Links list item (its description, or the address when the description is blank)
- **Content Building**: Takes the first content field that has a value (Body, Description, Comments, Notes); an item with none of them takes the text values of the list's columns that Graph reports as neither read-only nor hidden, other than `Title`, `ContentType`, `FileLeafRef` and date and time columns
- **Dynamic Field Mapping**: Captures every field Graph returns for the item in the `item.fields` map, verbatim - SharePoint's own internal fields are **not** stripped
- **Field Expansion**: Automatically re-reads the item with `$expand=fields` when the fields were not returned the first time
- **Content columns are read once per list**: `GET /sites/{site-id}/lists/{list-id}/columns` is requested before a list's items, and only for a list whose items are indexed. Only strings and arrays of strings (multi-choice) are taken; numbers, yes/no values, objects such as a hyperlink, and date and time columns, which Graph returns as UTC timestamps, are not

**Multi-Threading Support:**
- Configurable concurrent processing using `number_of_threads` parameter
- Thread pool management with proper resource cleanup
- Parallel processing of multiple lists and list items
- Graceful handling of thread interruption

**Error Handling & Resilience:**
- **Failure Tracking**: Integration with Fess failure URL service for error monitoring
- **Configurable Error Handling**: `ignore_error` parameter controls continuation on failures
- **Statistical Tracking**: Monitors crawling progress with document counts and timing metrics
- **Comprehensive Logging**: Debug and info level logging for troubleshooting

**Permission Management:**
- No SharePoint permission is read. Microsoft Graph exposes no app-only way to read a site's,
  list's or item's user and group role assignments that this plugin can call, so nothing is
  inherited from the site or the list
- `default_permissions`, plus the data config's own Permissions field, is the **only** source of a
  list item's roles - see
  [What ACL Each DataStore Can Produce](#what-acl-each-datastore-can-produce)
- Leave both unset and list items are indexed with no role at all, which makes them findable by
  nobody rather than by everybody

**Attachments are not supported:**
- Classic SharePoint list-item attachments are **not** indexed, and there is no script field for
  them. Microsoft Graph exposes no list-item attachments endpoint in either v1.0 or beta - the
  `listItem` resource's relationships are `activities`, `analytics`, `documentSetVersions`,
  `driveItem`, `fields`, `permissions` and `versions`. The documented API is SharePoint REST
  (`_api/web/lists/.../AttachmentFiles`), which Microsoft requires a **certificate** credential to
  call app-only; this plugin authenticates with a client secret scoped to Microsoft Graph.
- Files stored in a **document library** are indexed normally by `oneDriveDataStore` (individual
  files) and `sharePointDocLibDataStore` (library metadata).

**Title Filtering:**
- **Include Pattern**: Regex-based filtering to include specific items by title
- **Exclude Pattern**: Regex-based filtering to exclude items by title
- Both are compiled once at crawl start to validate them; a pattern that does not compile aborts
  the crawl there. The per-item check itself recompiles the pattern - there is no pattern cache
- An item with no title (`Title`, `LinkTitle` and `FileLeafRef` all blank, and no `URL` field
  description or address) bypasses both patterns and is crawled. This differs from `oneNoteDataStore`, which matches a nameless notebook as the empty
  string

**Use Cases:**
- **Structured Data Search**: Index and search custom business data stored in SharePoint lists
- **Task and Issue Tracking**: Search across task lists, issue trackers, and project lists
- **Document Metadata**: Index document libraries managed as SharePoint lists
- **Custom Applications**: Search data from Power Apps and custom SharePoint solutions
- **Business Process Content**: Index workflow-related lists and approval items

##### List Template Types

`list_template_filter` accepts a comma-separated mix of numeric SharePoint template IDs and
Microsoft Graph template name strings - for example `list_template_filter=100,documentLibrary` is
valid, and either form of `100` or `genericList` matches the same lists. The filter value is the
Graph name in the last column - not the display name in the *Name* column (e.g. `Picture Library`
matches nothing; `109` or `pictureLibrary` does):

| ID | Name | Filter value to use |
|----|------|----------------------|
| `100` | Generic List (Custom Lists) | `100` or `genericList` |
| `101` | Document Library | `101` or `documentLibrary` |
| `102` | Survey | `102` or `survey` |
| `103` | Links | `103` or `links` |
| `104` | Announcements | `104` or `announcements` |
| `105` | Contacts | `105` or `contacts` |
| `106` | Events | `106` or `events` |
| `108` | Discussion Board | `108` or `discussionBoard` |
| `109` | Picture Library | `109` or `pictureLibrary` |
| `171` | Tasks | `171` or `tasksWithTimelineAndHierarchy` |

A Tasks list created in SharePoint Online is template `171`, not the legacy Tasks template `107`,
so select it with `171` or `tasksWithTimelineAndHierarchy`. `107` itself is not mapped: Graph's name
for it has not been observed, and guessing would silently reintroduce the no-match problem this
mapping exists to fix. Passing an unmapped ID numerically (e.g. `list_template_filter=107`) logs a
`WARN` ("Unknown list template ID ...; use the Graph template name instead") and matches nothing.
To filter on a list type not in the table, enable `DEBUG` logging for `org.codelibs.fess.ds.ms365`
(see [Debug Mode](#debug-mode) below), run a crawl, and read the `Template:` value logged for that
list - then use that literal string as the filter value.

**Performance Optimizations:**
- Site, list and list-item listings are paged by following Graph's `@odata.nextLink` directly;
  the plugin uses no `PageIterator`
- Items are handed to the thread pool as each page is read, so a large list is not held in memory
  in one piece

### SharePoint Pages Parameters

| Parameter | Description | Default | Notes |
| --- | --- | --- | --- |
| `site_id` | SharePoint site ID containing pages | All sites | Full site ID format: `hostname,siteCollectionId,siteId` |
| `exclude_site_id` | Site IDs to exclude | - | A full site ID as-is; separate multiple IDs with `;` - see [exclude_site_id Format](#exclude_site_id-format) |
| `ignore_system_pages` | Skip system pages | `true` | Excludes Forms, DevHome, and other system pages |
| `page_type_filter` | Filter by page type | All types | Comma-separated: `news,article,page` |
| `ignore_error` | Continue crawling on errors | `false` | Set to `true` to skip failed pages |
| `include_pattern` | Regex pattern for page URLs to include | - | Filter pages by URL matching |
| `exclude_pattern` | Regex pattern for page URLs to exclude | - | Skip pages with matching URLs |
| `number_of_threads` | Number of processing threads | `1` | Concurrent page processing |
| `default_permissions` | Default role assignments | - | Additional permissions for all pages |

**Crawling Modes**:
- **All Sites**: Leave `site_id` empty to crawl pages from all accessible sites
- **Specific Site**: Set `site_id` to crawl only pages from that site
- **Filtered Content**: Use `page_type_filter` to limit to specific page types (news, articles, etc.)

**Content Processing**: Pages are processed with canvas layout expansion to extract rich content from web parts, including text formatting and embedded data when available through the Microsoft Graph API.

## 🔧 Development

### Tech Stack

- **Language**: Java 21
- **Build Tool**: Maven 3.8+
- **Framework**: Fess Data Store (LastaFlute/DBFlute)
- **API Client**: Microsoft Graph SDK v6
- **Authentication**: Azure Identity SDK
- **Testing**: UTFlute (`utflute-lastaflute`) with JUnit 5 (Jupiter), plus OkHttp MockWebServer for Graph responses. `junit:junit` 4.13.2 is still on the test classpath but no test uses it; `failIfNoTests` is on, because the JUnit 5 migration once collected zero tests for seven months without failing the build
- **Dependency Management**: Maven Shade Plugin with relocation

### Project Structure

```
src/
├── main/java/org/codelibs/fess/ds/ms365/
│   ├── Microsoft365DataStore.java        # Abstract base class
│   ├── OneDriveDataStore.java            # OneDrive implementation
│   ├── OneNoteDataStore.java             # OneNote implementation
│   ├── TeamsDataStore.java               # Teams implementation
│   ├── SharePointDocLibDataStore.java    # SharePoint doc libs
│   ├── SharePointListDataStore.java      # SharePoint lists
│   ├── SharePointPageDataStore.java      # SharePoint pages
│   └── client/
│       └── Microsoft365Client.java       # Graph API wrapper
├── main/resources/
│   └── fess_ds++.xml                     # DI configuration
└── test/java/org/codelibs/fess/ds/ms365/ # Test classes
```

### Building the Project

```bash
# Clean build
mvn clean package

# Run tests
mvn test

# Run specific test
mvn test -Dtest=OneDriveDataStoreTest

# Format code
mvn formatter:format

# Build without tests (faster)
mvn clean package -DskipTests
```

### Development Setup

1. **Prerequisites**:
   ```bash
   # Check Java version
   java -version  # Should be 21+
   
   # Check Maven version  
   mvn -version   # Should be 3.8+
   ```

2. **Clone and Setup**:
   ```bash
   git clone https://github.com/codelibs/fess-ds-microsoft365.git
   cd fess-ds-microsoft365
   
   # Install parent POM
   git clone https://github.com/codelibs/fess-parent.git
   cd fess-parent && mvn install -Dgpg.skip=true && cd ..
   
   # Build project
   mvn clean compile
   ```

3. **IDE Setup**:
   - Import as Maven project
   - Set Java 21 as project SDK
   - Enable annotation processing
   - Use the Eclipse formatter config in `src/config/eclipse/formatter/`

### Testing Strategy

The project uses UTFlute framework with mock Microsoft Graph API responses:

```bash
# Run all tests
mvn test

# Test specific data store
mvn test -Dtest=OneDriveDataStoreTest
mvn test -Dtest=Microsoft365ClientTest

# Test with debug output
mvn test -X -Dtest=SharePointDocLibDataStoreTest
```

### Contributing Guidelines

1. **Code Style**: Use the provided Eclipse formatter configuration
2. **Testing**: Write tests for new functionality using existing patterns
3. **Documentation**: Update README and JavaDocs for API changes
4. **Versioning**: Follow semantic versioning for releases
5. **Pull Requests**: Ensure CI passes before submitting

### Dependency Management

The project uses Maven Shade Plugin to bundle Microsoft Graph SDK dependencies with package relocation to avoid conflicts:

- **Relocated Packages**: `io.netty.*` → `org.codelibs.fess.ds.ms365.netty.*`
- **Bundled Libraries**: Azure SDK, Microsoft Graph SDK v6, Reactor Netty
- **Provided Dependencies**: Fess framework, OpenSearch, Jakarta APIs

## 📋 Usage Examples

### Example 1: OneDrive File Search
```javascript
// Fess search script mapping for OneDrive
title=file.name
content=file.description + "\n" + file.contents  
mimetype=file.mimetype
created=file.created
last_modified=file.last_modified
url=file.web_url
role=file.roles
```

### Example 2: SharePoint List Configuration
```properties
# SharePoint list crawling with filtering
site_id=contoso.sharepoint.com,686d3f1a-a383-4367-b5f5-93b99baabcf3,12048306-4e53-420e-bd7c-31af611f6d8a
list_template_filter=100,101  # Generic lists and Document Libraries
ignore_system_lists=true
include_pattern=.*Important.*
exclude_pattern=.*Draft.*
ignore_error=false
number_of_threads=2
default_permissions={role}sharepoint-users

# Crawl specific list only
# list_id=12345678-1234-1234-1234-123456789abc

# Exclude multiple lists
# exclude_list_id=list1-id,list2-id,list3-id
```

### Example 3: Teams Content Search
```javascript
// Teams message indexing script
title=message.title
content=message.content
created=message.created_date_time
last_modified=message.last_modified_date_time
url=message.web_url
role=message.roles
// Access additional fields
team_name=team.displayName
channel_name=channel.displayName
sender=message.from.user.displayName
```

### Example 4: Teams Configuration
```properties
# Crawl specific team with filters
team_id=12345678-1234-1234-1234-123456789abc
ignore_replies=true
ignore_system_events=true
append_attachment=true
number_of_threads=2
title_dateformat=yyyy/MM/dd'T'HH:mm:ss
title_timezone_offset=+09:00

# Exclude multiple teams
exclude_team_ids=team1-id,team2-id,team3-id
include_visibility=public,private

# Crawl specific channel in a team
# channel_id=19:channel-id@thread.tacv2

# Crawl specific chat conversation
# chat_id=19:chat-id@thread.v2
```

### Example 5: SharePoint Pages Content Search
```javascript
// SharePoint pages indexing script
title=page.title
content=page.content
created=page.created
last_modified=page.modified
url=page.url
role=page.roles
// Access additional page fields
page_type=page.type
author=page.author
site_name=page.site.site_name
description=page.description
```

### Example 6: SharePoint Pages Configuration
```properties
# Crawl pages from all sites with content filtering
ignore_system_pages=true
page_type_filter=news,article
include_pattern=.*important.*|.*announcement.*
exclude_pattern=.*draft.*|.*temp.*
number_of_threads=2
ignore_error=false

# Crawl pages from specific site only
# site_id=contoso.sharepoint.com,686d3f1a-a383-4367-b5f5-93b99baabcf3,12048306-4e53-420e-bd7c-31af611f6d8a

# Exclude multiple sites
# exclude_site_id=site1.sharepoint.com,guid1,guid1;site2.sharepoint.com,guid2,guid2
```

## 🔍 Troubleshooting

### Common Issues

**Authentication Errors**
```
Solution: Verify Azure AD app permissions and admin consent
- Check tenant ID format
- Ensure client secret hasn't expired
- Verify API permissions are granted
```

**Rate Limiting**
```
Solution: Adjust threading and implement backoff
- Reduce number_of_threads parameter
- Enable ignore_error to continue on throttling
- Monitor Microsoft Graph API limits
```

**Large Content Issues**
```
Solution: Configure content handling
- Implement exclude_pattern for large files
- Use OneDriveDataStore settings for file content extraction
```

### Debug Mode

Enable debug logging in Fess to troubleshoot issues:

```xml
<!-- Add to log4j2.xml -->
<Logger name="org.codelibs.fess.ds.ms365" level="DEBUG"/>
```

## 📚 Additional Resources

- **Fess Documentation**: https://fess.codelibs.org/
- **Microsoft Graph API**: https://docs.microsoft.com/en-us/graph/
- **Azure AD App Registration**: https://docs.microsoft.com/en-us/azure/active-directory/develop/
- **Issue Tracker**: https://github.com/codelibs/fess-ds-microsoft365/issues

## 📄 License

Licensed under the Apache License 2.0. See [LICENSE](LICENSE) for details.