Skip to content

Commit a81d8e3

Browse files
Update pagefind to align with src/glob in pages config options
* Inject data-pagefind-body when page is searchable * Remove glob option & logic handling in SiteGenerationManager * Fix inaccurate no. of page indexed count Previously, because we used the glob within pagefind's configuration option which controlled how pagefind discover's HTML files to index, it would only index the files that we declared within the glob. But since we now insert data-pagefind-body into the pages that we want to index, technically pagefind will index all the pages. However, it still seems to index pages that are not searchable, its just that nothing on these pages will be searchable, hence creating sort of a pseudo not indexed page. Currently, the fix is just to take the count of the number of addressable pages that are searchable. * Remove glob processing helper methods No longer needed since we adding the glob pattern into the pagefind's index dictionary. * Update test cases for SiteGenerationManager * Update docs * Update functional tests * Add clearer comments and test cases * Implement addHTML approach using index dictionary * Update SiteGenerationManager test cases * Update functional tests and docs Docs now clearly document the interactions of searchable with pagefind's native attributes
1 parent b8a8991 commit a81d8e3

7 files changed

Lines changed: 188 additions & 255 deletions

File tree

‎docs/userGuide/makingTheSiteSearchable.md‎

Lines changed: 40 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -101,34 +101,58 @@ In your `site.json`:
101101

102102
This tells Pagefind to exclude any element with the `algolia-no-index` class (or containing it in a space-separated list) from the search index, similar to using `data-pagefind-ignore`.
103103

104+
For more details, see the [Pagefind documentation on exclude selector configuration option](https://pagefind.app/docs/config-options/#exclude-selectors).
105+
104106
#### Limiting Which Pages Are Searchable
105107

106-
You can use the `glob` option to limit which pages are indexed by Pagefind. This is useful when you want search results to only show pages from specific sections of your site.
108+
Pagefind uses the existing `searchable` property in your `pages` configuration to determine which pages should be indexed. This provides a seamless way to control search indexing without additional configuration.
107109

108110
In your `site.json`:
109111

110112
```json
111113
{
112-
"pagefind": {
113-
"glob": [
114-
"devGuide",
115-
"userGuide/*"
116-
]
117-
}
114+
"pages": [
115+
{
116+
"src": "index.md"
117+
},
118+
{
119+
"src": "internal/notes.md",
120+
"searchable": "no"
121+
},
122+
{
123+
"glob": "devGuide/**/*.md",
124+
"searchable": "no"
125+
}
126+
]
118127
}
119128
```
120129

121-
MarkBind supports glob patterns and will automatically append `.html` to your patterns if not specified. For example:
122-
- `"devGuide"` becomes `"devGuide/**/*.html"`
123-
- `"devGuide/*"` becomes `"devGuide/*.html"`
124-
- `"**/devGuide/**"` becomes `"**/devGuide/**/*.html"`
125-
- `"*.html"` remains `"*.html"` (no change needed)
130+
- Pages with `searchable: "no"` (or `false`) will not appear in search results
131+
- By default, all pages are searchable (`searchable: "yes"`)
132+
133+
For more details on the `searchable` property, see [site.json file documentation](siteJsonFile.html#pages).
134+
135+
<panel header="Interaction with Pagefind Attributes">
136+
137+
MarkBind controls page indexing through the `searchable` property, which determines whether pages are passed to Pagefind for indexing. However, Pagefind also provides native attributes that can affect indexing:
126138

127-
Only pages matching these glob patterns will appear in search results. This can be particularly useful for:
128-
- Multi-site setups where you want to search only specific sections
129-
- Including only certain directories from search results
139+
- [**`data-pagefind-body`**](https://pagefind.app/docs/indexing/#limiting-what-sections-of-a-page-are-indexed): Marks a specific element as the search content container. When this attribute exists on ANY page of your site, pages WITHOUT this attribute will not be indexed.
140+
- [**`data-pagefind-ignore`**](https://pagefind.app/docs/indexing/#removing-individual-elements-from-the-index): Excludes specific elements from the search index.
130141

131-
For more details on glob patterns, see the [Pagefind documentation](https://pagefind.app/docs/config-options/#glob).
142+
**How MarkBind handles this:**
143+
144+
1. Pages with `searchable: "no"` are NOT passed to Pagefind at all (they are never indexed).
145+
2. Pages with `searchable: "yes"` (default) ARE passed to Pagefind for indexing.
146+
147+
**Interactions to be aware of:**
148+
149+
- If you add `data-pagefind-body` to a searchable page, it works as expected - the page is indexed. However, only pages with this attribute will be searchable.
150+
- If you add `data-pagefind-body` to a non-searchable page, MarkBind will still NOT index it (because it's filtered before being passed to Pagefind).
151+
- Adding `data-pagefind-ignore` to a searchable page will NOT prevent it from being indexed - the page is still added via MarkBind's indexing, but the content within that element will be ignored by Pagefind.
152+
153+
**Recommendation:** Use MarkBind's `searchable` property in site.json to control which pages are indexed & use `data-pagefind-body` attribute to exlcude specific elements within a page from being searchable. Avoid using `data-pagefind-body` as it is redundant and may lead to confusion.
154+
155+
</panel>
132156

133157
<panel header="Potential Future Enhancements">
134158

‎docs/userGuide/siteJsonFile.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -135,7 +135,7 @@ _(Optional)_ **The styling options to be applied to the site.** This includes:
135135
* **`globExclude`**: An array of file patterns to be excluded from rendering when using `glob`, also defined in the glob syntax.
136136
* **`title`**: The page `<title>` for the generated web page. Titles specified here take priority over titles specified in the [frontmatter](tweakingThePageStructure.html#frontmatter) of individual pages.
137137
* **`layout`**: The [layout](tweakingThePageStructure.html#layouts) to be used by the page. Default: `default`.
138-
* **`searchable`**: Specifies that the page(s) should be excluded from searching. Default: `yes`.
138+
* **`searchable`**: Specifies whether the page(s) should be included in search indexing. This applies to both the built-in search and Pagefind (if enabled). Set to `"no"` or `false` to exclude the page(s) from search results. Default: `yes`.
139139
* **`externalScripts`**: An array of external scripts to be referenced on the page. Scripts referenced will be run before the layout script.
140140
* **`frontmatter`**: Specifies properties to add to the frontmatter of a page or glob of pages. Overrides any existing properties if they have the same name, and overrides any frontmatter properties specified in `globalOverride`.
141141
* **`fileExtension`**: A string that specifies the output file extension (e.g., `".json"`, `".txt"`) for the generated file. If not specified, defaults to `".html"`. Note that non-HTML files do not support frontmatter or scripts.

‎packages/core/src/Page/page.njk‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,9 @@
3434
<script>
3535
const baseUrl = '{{ baseUrl }}'
3636
</script>
37-
<body {% if hasPageNavHeadings %} data-bs-spy="scroll" data-bs-target="#mb-page-nav" data-bs-offset="100" {% endif %} data-code-theme="{{ codeTheme }}">
37+
<body
38+
{% if hasPageNavHeadings %} data-bs-spy="scroll" data-bs-target="#mb-page-nav" data-bs-offset="100" {% endif %}
39+
data-code-theme="{{ codeTheme }}">
3840
{{ content }}
3941
</body>
4042
{{- pageUserScriptsAndStyles -}}

‎packages/core/src/Site/SiteConfig.ts‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -66,7 +66,6 @@ export class SiteConfig {
6666

6767
pagefind?: {
6868
exclude_selectors?: string[];
69-
glob?: string | string[];
7069
};
7170

7271
/**

‎packages/core/src/Site/SiteGenerationManager.ts‎

Lines changed: 33 additions & 98 deletions
Original file line numberDiff line numberDiff line change
@@ -871,74 +871,6 @@ export class SiteGenerationManager {
871871
1000,
872872
);
873873

874-
/**
875-
* Validates that a glob pattern is safe and won't traverse outside the output directory.
876-
*
877-
* @param pattern - The glob pattern to validate
878-
* @returns true if the pattern is safe, false otherwise
879-
*/
880-
// eslint-disable-next-line class-methods-use-this
881-
private isValidGlobPattern(pattern: string): boolean {
882-
if (pattern.includes('..')) {
883-
return false;
884-
}
885-
886-
const isUnixAbsolutePath = pattern.startsWith('/');
887-
const isWindowsAbsolutePath = /^[a-zA-Z]:[\\/]/.test(pattern);
888-
if (isUnixAbsolutePath || isWindowsAbsolutePath) {
889-
return false;
890-
}
891-
892-
return true;
893-
}
894-
895-
/**
896-
* Normalizes a gitignore-style glob pattern to a valid Wax/Pagefind pattern
897-
* by appending .html extension if not already present.
898-
* Invalid patterns (e.g., path traversal attempts) are logged and return empty string.
899-
*
900-
* @param pattern - The glob pattern from user config (gitignore-style)
901-
* @returns A valid Wax/Pagefind glob pattern, or empty string if invalid
902-
*/
903-
private normalizeGlobPattern(pattern: string): string {
904-
const normalizedPattern = pattern.replace(/\\/g, '/');
905-
906-
if (!this.isValidGlobPattern(pattern)) {
907-
logger.error(`Invalid glob pattern rejected (potential path traversal): ${pattern}`);
908-
return '';
909-
}
910-
911-
if (normalizedPattern.endsWith('.html')) {
912-
return normalizedPattern;
913-
}
914-
915-
if (normalizedPattern.endsWith('/**')) {
916-
return `${normalizedPattern}/*.html`;
917-
}
918-
919-
if (normalizedPattern.endsWith('/*')) {
920-
return `${normalizedPattern}.html`;
921-
}
922-
923-
if (normalizedPattern.endsWith('/')) {
924-
return `${normalizedPattern}**/*.html`;
925-
}
926-
927-
return `${normalizedPattern}/**/*.html`;
928-
}
929-
930-
/**
931-
* Indexes all HTML files in the output directory and logs any errors.
932-
* @param index - The pagefind index instance
933-
* @returns The number of pages indexed
934-
*/
935-
// eslint-disable-next-line class-methods-use-this
936-
private async indexAllHtmlFiles(index: any): Promise<number> {
937-
const result = await index.addDirectory({ path: this.outputPath });
938-
result.errors.forEach((error: string) => logger.error(error));
939-
return result.page_count;
940-
}
941-
942874
/**
943875
* Indexes all the pages of the site using pagefind.
944876
* @returns true if indexing succeeded and pagefind assets were written, false otherwise.
@@ -963,40 +895,43 @@ export class SiteGenerationManager {
963895

964896
const { index } = await createIndex(createIndexOptions);
965897
if (index) {
966-
// Handle glob patterns - support both single string and array of strings
967-
const globValue = pagefindConfig.glob;
968-
const value = globValue ?? [];
969-
const globPatterns = Array.isArray(value) ? value : [value];
898+
// Filter pages that should be indexed (searchable !== false)
899+
const searchablePages = this.sitePages.pages.filter(
900+
page => page.pageConfig.searchable,
901+
);
970902

971903
let totalPageCount = 0;
972904

973-
if (globPatterns.length > 0) {
974-
const normalizedPatterns = globPatterns
975-
.map(pattern => this.normalizeGlobPattern(pattern))
976-
.filter(pattern => pattern !== '');
977-
978-
if (normalizedPatterns.length > 0) {
979-
const results = await Promise.all(
980-
normalizedPatterns.map(async (normalizedPattern) => {
981-
logger.info(`Pagefind indexing with glob: ${normalizedPattern}`);
982-
const result = await index.addDirectory({
983-
path: this.outputPath,
984-
glob: normalizedPattern,
985-
});
986-
987-
result.errors.forEach((error: string) => logger.error(error));
988-
989-
return result.page_count;
990-
}),
991-
);
992-
993-
totalPageCount += results.reduce((acc, count) => acc + count, 0);
994-
} else {
995-
logger.warn('All glob patterns were invalid, falling back to indexing all HTML files');
996-
totalPageCount = await this.indexAllHtmlFiles(index);
997-
}
905+
if (searchablePages.length === 0) {
906+
logger.info('No pages configured for search indexing');
998907
} else {
999-
totalPageCount = await this.indexAllHtmlFiles(index);
908+
// Add each searchable page to the index using addHTMLFile
909+
const indexingResults = await Promise.all(
910+
searchablePages.map(async (page) => {
911+
try {
912+
const content = await fs.readFile(page.pageConfig.resultPath, 'utf8');
913+
const relativePath = path.relative(this.outputPath, page.pageConfig.resultPath);
914+
915+
return index.addHTMLFile({
916+
sourcePath: relativePath,
917+
content,
918+
});
919+
} catch (error) {
920+
logger.error(`Failed to index ${page.pageConfig.resultPath}: ${error}`);
921+
return null;
922+
}
923+
}),
924+
);
925+
926+
// Count successful indexings
927+
totalPageCount = indexingResults.filter(r => r !== null).length;
928+
929+
// Log any errors from indexing results
930+
indexingResults.forEach((result) => {
931+
if (result && result.errors) {
932+
result.errors.forEach((error: string) => logger.error(error));
933+
}
934+
});
1000935
}
1001936

1002937
const endTime = new Date();

0 commit comments

Comments
 (0)