mirror of
https://github.com/GitbookIO/gitbook.git
synced 2026-10-12 07:58:49 +00:00
Compare commits
1 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Nicolas Dorseuil
|
e4a61537ff |
@@ -0,0 +1,5 @@
|
||||
---
|
||||
"gitbook": patch
|
||||
---
|
||||
|
||||
Compress oversized site API tokens in the rewritten URL to stay under URL length limits.
|
||||
@@ -1,5 +0,0 @@
|
||||
---
|
||||
"gitbook": patch
|
||||
---
|
||||
|
||||
Agents can now send a goal with any question they ask on the Markdown version of a page.
|
||||
@@ -1,5 +0,0 @@
|
||||
---
|
||||
"gitbook": patch
|
||||
---
|
||||
|
||||
Show site UI in French or Spanish for spaces set to Canadian French, Mexican Spanish or Latin American Spanish.
|
||||
+70
-46
@@ -1,89 +1,113 @@
|
||||
# Contributing to GitBook
|
||||
# Welcome to GitBook's contributing guide!
|
||||
|
||||
> _For help, support and product questions, head to our [GitHub Community](https://github.com/orgs/GitbookIO/discussions)._
|
||||
> _For help, support, feature requests, and product questions - head to our [GitHub Community](https://github.com/orgs/GitbookIO/discussions) 🤖_
|
||||
|
||||
Thanks for helping make GitBook better. This repository holds the open source renderer behind every published GitBook site, and the team reads everything that comes in here.
|
||||
Thank you for investing your time in contributing to GitBook. Any contribution you make will be reviewed by our team. In this guide, you'll learn the different ways you can contribute.
|
||||
|
||||
## Ways to contribute
|
||||
## Types of Contributions
|
||||
|
||||
- **Open an issue.** Report a bug, suggest a feature, or tell us when something feels off. This is the most useful thing you can do.
|
||||
- **Join the conversation.** Share ideas and help others in [GitHub Community](https://github.com/orgs/GitbookIO/discussions).
|
||||
- **Build with GitBook.** Publish your docs, try new features early, and tell us what works and what doesn't.
|
||||
This repository contains code related to the rendering engine of GitBook's published content. Depending on what you'd like to contribute to, head to the section below to find the necessary steps.
|
||||
|
||||
## Issues instead of pull requests
|
||||
### Add a feature
|
||||
|
||||
Pull requests are limited to the GitBook team. If you've found a bug or want something to change, please [open an issue](https://github.com/GitbookIO/gitbook/issues/new/choose) instead.
|
||||
Because this portion of GitBook is open source and available for you to use - if you think you can provide extra value through a new feature - you're welcome to add it! If you plan to distribute the code, keep the source code public to comply with GNU GPLv3. To clone in a private repository, you must first acquire a [commercial license](https://www.gitbook.com/pricing).
|
||||
|
||||
This isn't because we don't want your help. Every change here ships to every published GitBook site, so we need to own each one from start to finish. A clear issue often helps us more than a pull request: it tells us what's wrong and why, and lets us fix it in a way that fits the rest of the codebase.
|
||||
### Create a new issue
|
||||
|
||||
A good issue includes:
|
||||
If you spot a problem within a repository, [search if an issue already exists](https://docs.github.com/en/github/searching-for-information-on-github/searching-on-github/searching-issues-and-pull-requests#search-by-the-title-body-or-comments). If a related issue doesn't exist, you can open a new issue here!Please make sure any added issues are
|
||||
|
||||
- What you expected to happen, and what happened instead
|
||||
- A link to a published GitBook site that shows the problem, if you have one
|
||||
- Steps to reproduce it, plus screenshots or a recording for anything visual
|
||||
- Your browser and OS, if it looks like it might matter
|
||||
- Descriptive
|
||||
- Thoughtful
|
||||
- Organized
|
||||
|
||||
Before opening one, [search existing issues](https://github.com/GitbookIO/gitbook/issues). If someone has already reported it, add a 👍 or any extra details you have.
|
||||
We recommend adding as many relevant links, minimal reproductions of the issue, and other materials that will help our team solve the issue fast.
|
||||
|
||||
Already fixed it in your fork? Link the branch or commit in your issue. We may open a pull request from it, and your commits keep your name.
|
||||
### Solve an issue
|
||||
|
||||
If you open a pull request anyway, we'll close it and point you here. Nothing personal, and we'd still love the issue.
|
||||
If you're interested in solving an issue in our repository, start by scanning through it's exisiting issues to find one that you're interested in working on. If you find an issue to work on, you are welcome to open a PR with a fix. See the following sections below for more information on contributing for specific sections.
|
||||
|
||||
## Reach out
|
||||
### Documentation
|
||||
|
||||
Not sure if it's a bug? Have a half-formed idea? Open an issue or start a [discussion](https://github.com/orgs/GitbookIO/discussions) anyway. We'd much rather hear from you than not, and no report is too small.
|
||||
The official documentation on GitBook open can be found directly in this Readme. Any updates or changes you would like to make, you can make directly to the README of this repository.
|
||||
|
||||
Spotted a wrong or missing translation? Open an issue with the string and the wording you'd suggest.
|
||||
## Contributing
|
||||
|
||||
## Using AI
|
||||
### Make changes locally
|
||||
|
||||
You're welcome to use AI to help you write an issue. Please read and check what it produces before you post: reports that are long, vague or unverified take time away from fixing real problems, and we may close them without a reply.
|
||||
Any contribution you make can be made to the code located in this repository. In order to contribute, you'll need to start off of a local version of this repository.
|
||||
|
||||
## Running locally
|
||||
#### 1. Fork the repository
|
||||
|
||||
You don't need a local setup to open an issue, but it can help you reproduce a problem or self-host the renderer.
|
||||
##### Using GitHub Desktop:
|
||||
|
||||
### Prerequisites
|
||||
- [Getting started with GitHub Desktop](https://docs.github.com/en/desktop/installing-and-configuring-github-desktop/getting-started-with-github-desktop) will guide you through setting up Desktop.
|
||||
- Once Desktop is set up, you can use it to [fork the repo](https://docs.github.com/en/desktop/contributing-and-collaborating-using-github-desktop/cloning-and-forking-repositories-from-github-desktop)!
|
||||
|
||||
- [Node.js](https://nodejs.org/en) (see "engines" in `package.json`)
|
||||
- [Bun](https://bun.sh/) (see "packageManager" in `package.json`)
|
||||
##### Using the command line:
|
||||
|
||||
### Setup
|
||||
- [Fork the repo](https://docs.github.com/en/github/getting-started-with-github/fork-a-repo#fork-an-example-repository) so that you can make your changes without affecting the original project until you're ready to merge them.
|
||||
|
||||
1. Use the project's version of Node:
|
||||
#### 2. Create a working branch and start with your changes
|
||||
|
||||
After forking this repository, you'll want to [create a branch](https://docs.github.com/en/issues/tracking-your-work-with-issues/creating-a-branch-for-an-issue) to work off of.
|
||||
|
||||
#### 3. Install dependencies and run the project locally
|
||||
|
||||
##### Prerequisites:
|
||||
|
||||
- [Node.js](https://nodejs.org/en) (see "engines" in `package.json`)
|
||||
- [Bun](https://bun.sh/) (see "packageManager" in `package.json`)
|
||||
|
||||
##### Setup steps:
|
||||
|
||||
1. Ensure you are using the project's version of Node:
|
||||
```bash
|
||||
nvm use
|
||||
```
|
||||
|
||||
2. Install dependencies:
|
||||
|
||||
2. Install dependencies using Bun:
|
||||
```bash
|
||||
bun install
|
||||
```
|
||||
|
||||
3. Start the development server:
|
||||
|
||||
```bash
|
||||
bun dev
|
||||
```
|
||||
|
||||
Other commands:
|
||||
Additional development commands:
|
||||
- `bun lint`: Lint the code using Oxlint
|
||||
- `bun lint:fix`: Automatically fix lint issues using Oxlint
|
||||
- `bun format`: Format the code using Oxfmt
|
||||
- `bun format:check`: Check formatting without changing files
|
||||
- `bun typecheck`: Run TypeScript type checking
|
||||
- `bun unit`: Run unit tests
|
||||
- `bun e2e`: Run end-to-end tests
|
||||
|
||||
- `bun lint`: Lint the code using Oxlint
|
||||
- `bun lint:fix`: Automatically fix lint issues using Oxlint
|
||||
- `bun format`: Format the code using Oxfmt
|
||||
- `bun format:check`: Check formatting without changing files
|
||||
- `bun typecheck`: Run TypeScript type checking
|
||||
- `bun unit`: Run unit tests
|
||||
- `bun e2e`: Run end-to-end tests
|
||||
#### 4. Preview your changes
|
||||
|
||||
### Preview a site
|
||||
When running the development server, published GitBook sites can be rendered through your local version at `http://localhost:3000/url`.
|
||||
|
||||
With the development server running, any published GitBook site can be rendered through your local version at `http://localhost:3000/url/<published-url>`. For example, visit `http://localhost:3000/url/gitbook.com/docs` to see our own docs.
|
||||
For example, our published docs can be viewed using the local version by visiting `http://localhost:3000/url/gitbook.com/docs` after running the development server.
|
||||
|
||||
The site must be [published publicly](https://gitbook.com/docs/published-documentation/publish-your-content-as-a-docs-site) to render locally.
|
||||
You can visit any published GitBook site behind your development server. Please make sure your site is [published publicly](https://gitbook.com/docs/published-documentation/publish-your-content-as-a-docs-site) to ensure you can view the site correctly in your development version.
|
||||
|
||||
## License
|
||||
### Commit your update
|
||||
|
||||
The renderer is distributed under the [GNU GPLv3 License](https://github.com/GitbookIO/gitbook/blob/main/LICENSE). If you distribute the code, you must keep the source code public. To use it in a private repository, you need a [commercial license](https://www.gitbook.com/pricing).
|
||||
[Commit your changes](https://github.com/git-guides/git-commit) once you are happy with them. See [Atom's contributing guide](https://github.com/atom/atom/blob/master/CONTRIBUTING.md#git-commit-messages) to know how to use emoji for commit messages!
|
||||
|
||||
Once your changes are ready, don't forget to self-review your code to double check that your chagnes are ready to be added.
|
||||
|
||||
### Pull Request
|
||||
|
||||
When you're finished with the changes, [create a pull request](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request), also known as a PR.
|
||||
|
||||
- Don't forget to [link PR to issue](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue) if you are solving one.
|
||||
- Enable the checkbox to [allow maintainer edits](https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/allowing-changes-to-a-pull-request-branch-created-from-a-fork) so the branch can be updated for a merge. Once you submit your PR, a GitBook team member will review your proposal. We may ask questions or request for additional information.
|
||||
- We may ask for changes to be made before a PR can be merged, either using [suggested changes](https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/incorporating-feedback-in-your-pull-request) or pull request comments. You can apply suggested changes directly through the UI. You can make any other changes in your fork, then commit them to your branch.
|
||||
- As you update your PR and apply changes, mark each conversation as [resolved](https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/commenting-on-a-pull-request#resolving-conversations).
|
||||
- If you run into any merge issues, checkout this [git tutorial](https://lab.github.com/githubtraining/managing-merge-conflicts) to help you resolve merge conflicts and other issues.
|
||||
|
||||
### Your PR is merged
|
||||
|
||||
Congratulations 🎉 Thank you for your contribution! Once your PR is merged, your contributions will be publicly visible on the relevant repository.
|
||||
|
||||
@@ -85,7 +85,7 @@ GitBook Open uses fontawesome. For self-hosting and local development, for licen
|
||||
|
||||
## Contributing
|
||||
|
||||
GitBook's rendering engine is fully open source and built on top of [Next.js](https://nextjs.org/). Head to our [contributing guide](https://github.com/GitbookIO/gitbook/blob/main/.github/CONTRIBUTING.md) to learn how to report bugs and suggest changes.
|
||||
GitBook's rendering engine is fully open source and built on top of [Next.js](https://nextjs.org/). Head to our [contributing guide](https://github.com/GitbookIO/gitbook/blob/main/.github/CONTRIBUTING.md) to learn more about the workflow on adding your first Pull Request.
|
||||
|
||||
### Types of contributions
|
||||
|
||||
@@ -93,22 +93,22 @@ We encourage you to contribute to GitBook to help us build the best tool for doc
|
||||
|
||||
#### Translations
|
||||
|
||||
The GitBook UI is rendered using a set of translation files found in [`packages/gitbook/src/intl/translations`](/packages/gitbook/src/intl/translations/). Spotted a wrong or missing translation? Open an issue and let us know.
|
||||
The GitBook UI is rendered using a set of translation files found in [`packages/gitbook/src/intl/translations`](/packages/gitbook/src/intl/translations/). We welcome all additional translations for the UI.
|
||||
|
||||
#### Bugs
|
||||
|
||||
Encounter a bug? [Open an issue](https://github.com/GitbookIO/gitbook/issues/new/choose) with as much detail as you can. Clear reports help us fix things faster for everyone.
|
||||
Encounter a bug or find an issue you'd like to fix? Helping us fix issues related to GitBook greatly improves the experience for everyone. Head to the issues section of this repository to learn more about the types of bugs you can already help out with.
|
||||
|
||||
## Deployment
|
||||
|
||||
> [!WARNING]
|
||||
> While it is possible to self-host this project, we do not recommend this unless you are certain this option fits your need.
|
||||
>
|
||||
> _Looking for a specific feature in GitBook? Head to our [contributing guide](https://github.com/GitbookIO/gitbook/blob/main/.github/CONTRIBUTING.md) to suggest it._
|
||||
> _Looking to add a specific feature in GitBook? Head to our [contributing guide](https://github.com/GitbookIO/gitbook/blob/main/.github/CONTRIBUTING.md) to get started._
|
||||
>
|
||||
> Self-hosting this project puts the responsibility of maintaining and merging future updates on **you**. We cannot guarantee support, maintenance, or updates to forked and self-hosted instances of this project.
|
||||
>
|
||||
> We want to make it as easy as possible for our community to collaborate and push the future of GitBook, which is why we encourage you to share your ideas with us instead of creating your own version.
|
||||
> We want to make it as easy as possible for our community to collaborate and push the future of GitBook, which is why we encourage you to contribute to our product directly instead of creating your own version.
|
||||
|
||||
This project allows you to self-host the rendering portion of your GitBook published content. Self-hosting has pros and cons.
|
||||
|
||||
|
||||
@@ -143,6 +143,7 @@
|
||||
"direction": "^2.0.1",
|
||||
"event-iterator": "^2.0.0",
|
||||
"feed": "^5.1.0",
|
||||
"fflate": "^0.8.3",
|
||||
"flexsearch": "^0.8.212",
|
||||
"image-size": "^2.0.2",
|
||||
"js-cookie": "^3.0.5",
|
||||
@@ -2117,6 +2118,8 @@
|
||||
|
||||
"feed": ["feed@5.1.0", "", { "dependencies": { "xml-js": "^1.6.11" } }, "sha512-qGNhgYygnefSkAHHrNHqC7p3R8J0/xQDS/cYUud8er/qD9EFGWyCdUDfULHTJQN1d3H3WprzVwMc9MfB4J50Wg=="],
|
||||
|
||||
"fflate": ["fflate@0.8.3", "", {}, "sha512-tbZNuJrLwGUp3zshBtdy4W+ORxZuIh8a5ilyIEQDC5rY1f3U20JMry0Ll3WBzU58EZKsEuJFXhb5gwv8CsPvgA=="],
|
||||
|
||||
"file-entry-cache": ["file-entry-cache@10.0.7", "", { "dependencies": { "flat-cache": "^6.1.7" } }, "sha512-txsf5fu3anp2ff3+gOJJzRImtrtm/oa9tYLN0iTuINZ++EyVR/nRrg2fKYwvG/pXDofcrvvb0scEbX3NyW/COw=="],
|
||||
|
||||
"file-uri-to-path": ["file-uri-to-path@1.0.0", "", {}, "sha512-0Zt+s3L7Vf1biwWZ29aARiVYLx7iMGnEUl9x33fbB/j3jR81u/O2LbqK+Bm1CDSNDKVtJ/YjwY7TUd5SkeLQLw=="],
|
||||
|
||||
@@ -34,6 +34,7 @@
|
||||
"direction": "^2.0.1",
|
||||
"event-iterator": "^2.0.0",
|
||||
"feed": "^5.1.0",
|
||||
"fflate": "^0.8.3",
|
||||
"flexsearch": "^0.8.212",
|
||||
"image-size": "^2.0.2",
|
||||
"js-cookie": "^3.0.5",
|
||||
|
||||
@@ -5,6 +5,7 @@ import rison from 'rison';
|
||||
import type { SiteAPIToken } from '@gitbook/api';
|
||||
|
||||
import { getVisitorAuthClaims, getVisitorAuthClaimsFromToken } from '@/lib/adaptive';
|
||||
import { decompressAPIToken } from '@/lib/api-token-compression';
|
||||
import { type SiteURLData, fetchSiteContextByURLLookup, getBaseContext } from '@/lib/context';
|
||||
import { getDynamicCustomizationSettings } from '@/lib/customization';
|
||||
|
||||
@@ -124,7 +125,8 @@ function getModeFromParams(mode: string): RouteParamMode {
|
||||
export function getSiteURLDataFromParams(params: RouteLayoutParams): SiteURLData {
|
||||
try {
|
||||
const decoded = decodeURIComponent(params.siteData);
|
||||
return rison.decode(decoded);
|
||||
const data = rison.decode<SiteURLData>(decoded);
|
||||
return { ...data, apiToken: decompressAPIToken(data.apiToken) };
|
||||
} catch (error) {
|
||||
console.error(
|
||||
`Returning 404 after failing to decode site data ${params.siteData}: ${error}`
|
||||
|
||||
@@ -1,20 +0,0 @@
|
||||
import { describe, expect, it } from 'bun:test';
|
||||
|
||||
import { es } from './es';
|
||||
import { fr } from './fr';
|
||||
import { isAvailableLanguage, languages, loadLanguage } from './index';
|
||||
|
||||
describe('regional language variants', () => {
|
||||
it.each([
|
||||
['fr-ca', fr],
|
||||
['es-mx', es],
|
||||
['es-419', es],
|
||||
] as const)('loads %s with its own metadata and the base strings', async (locale, base) => {
|
||||
expect(isAvailableLanguage(locale)).toBe(true);
|
||||
|
||||
const language = await loadLanguage(locale);
|
||||
|
||||
expect(language).toEqual({ ...base, ...languages[locale] });
|
||||
expect(language.locale).toBe(locale);
|
||||
});
|
||||
});
|
||||
@@ -25,10 +25,6 @@ const languageDefinitions = {
|
||||
metadata: { locale: 'fr', language: 'Français', flag: '🇫🇷' },
|
||||
load: () => import('./fr').then((module) => module.fr),
|
||||
},
|
||||
'fr-ca': {
|
||||
metadata: { locale: 'fr-ca', language: 'Français (Canada)', flag: '🇨🇦' },
|
||||
load: () => import('./fr').then((module) => withMetadata(module.fr, 'fr-ca')),
|
||||
},
|
||||
de: {
|
||||
metadata: { locale: 'de', language: 'Deutsch', flag: '🇩🇪' },
|
||||
load: () => import('./de').then((module) => module.de),
|
||||
@@ -37,14 +33,6 @@ const languageDefinitions = {
|
||||
metadata: { locale: 'es', language: 'Español', flag: '🇪🇸' },
|
||||
load: () => import('./es').then((module) => module.es),
|
||||
},
|
||||
'es-mx': {
|
||||
metadata: { locale: 'es-mx', language: 'Español (México)', flag: '🇲🇽' },
|
||||
load: () => import('./es').then((module) => withMetadata(module.es, 'es-mx')),
|
||||
},
|
||||
'es-419': {
|
||||
metadata: { locale: 'es-419', language: 'Español (Latinoamérica)', flag: '🌎' },
|
||||
load: () => import('./es').then((module) => withMetadata(module.es, 'es-419')),
|
||||
},
|
||||
it: {
|
||||
metadata: { locale: 'it', language: 'Italiano', flag: '🇮🇹' },
|
||||
load: () => import('./it').then((module) => module.it),
|
||||
@@ -189,14 +177,6 @@ const languageDefinitions = {
|
||||
|
||||
export type TranslationLocale = keyof typeof languageDefinitions;
|
||||
|
||||
// Regional variants reuse their base language's strings but keep their own locale, name and flag.
|
||||
function withMetadata(
|
||||
base: TranslationLanguage,
|
||||
locale: 'fr-ca' | 'es-mx' | 'es-419'
|
||||
): TranslationLanguage {
|
||||
return { ...base, ...languageDefinitions[locale].metadata };
|
||||
}
|
||||
|
||||
export const languages = Object.fromEntries(
|
||||
Object.entries(languageDefinitions).map(([locale, definition]) => [locale, definition.metadata])
|
||||
) as {
|
||||
|
||||
@@ -0,0 +1,88 @@
|
||||
import { describe, expect, it } from 'bun:test';
|
||||
import jwt from 'jsonwebtoken';
|
||||
import { inflateRawSync } from 'node:zlib';
|
||||
|
||||
import {
|
||||
API_TOKEN_COMPRESSION_THRESHOLD,
|
||||
COMPRESSED_API_TOKEN_PREFIX,
|
||||
compressAPITokenIfNeeded,
|
||||
decompressAPIToken,
|
||||
} from './api-token-compression';
|
||||
|
||||
const ALPHANUM = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789';
|
||||
|
||||
function randomId() {
|
||||
return Array.from({ length: 20 }, () => ALPHANUM[Math.floor(Math.random() * 62)]).join('');
|
||||
}
|
||||
|
||||
function signToken(spaces: number, claims: Record<string, unknown> = {}) {
|
||||
return jwt.sign(
|
||||
{
|
||||
kind: 'site',
|
||||
organization: randomId(),
|
||||
site: `site_${randomId()}`,
|
||||
space: randomId(),
|
||||
spaces: Array.from({ length: spaces }, randomId),
|
||||
claims,
|
||||
},
|
||||
'secret'
|
||||
);
|
||||
}
|
||||
|
||||
describe('compressAPITokenIfNeeded', () => {
|
||||
it('keeps tokens under the threshold unchanged', () => {
|
||||
const token = signToken(10);
|
||||
expect(token.length).toBeLessThanOrEqual(API_TOKEN_COMPRESSION_THRESHOLD);
|
||||
expect(compressAPITokenIfNeeded(token)).toBe(token);
|
||||
});
|
||||
|
||||
it('compresses large tokens and restores them exactly', () => {
|
||||
const token = signToken(600, {
|
||||
groups: Array.from({ length: 100 }, (_, i) => `group-${i}`),
|
||||
});
|
||||
expect(token.length).toBeGreaterThan(API_TOKEN_COMPRESSION_THRESHOLD);
|
||||
|
||||
const compressed = compressAPITokenIfNeeded(token);
|
||||
expect(compressed.startsWith(COMPRESSED_API_TOKEN_PREFIX)).toBe(true);
|
||||
expect(compressed.length).toBeLessThan(token.length * 0.85);
|
||||
expect(decompressAPIToken(compressed)).toBe(token);
|
||||
});
|
||||
|
||||
it('restores tokens with non-ASCII characters and dots in the payload', () => {
|
||||
const token = signToken(600, { name: 'Zoë Ünal', 'a.b': 'é.ü.日本', email: 'z.u@ex.co' });
|
||||
expect(decompressAPIToken(compressAPITokenIfNeeded(token))).toBe(token);
|
||||
});
|
||||
|
||||
it('produces a stable output for the same token', () => {
|
||||
const token = signToken(600);
|
||||
expect(compressAPITokenIfNeeded(token)).toBe(compressAPITokenIfNeeded(token));
|
||||
});
|
||||
|
||||
it('produces raw deflate data', () => {
|
||||
const token = signToken(600);
|
||||
const compressed = compressAPITokenIfNeeded(token);
|
||||
const inflated = inflateRawSync(
|
||||
Buffer.from(compressed.slice(COMPRESSED_API_TOKEN_PREFIX.length), 'base64url')
|
||||
);
|
||||
const [header, payload, signature] = token.split('.');
|
||||
expect(inflated.toString()).toBe(
|
||||
`${header}.${Buffer.from(payload!, 'base64url').toString()}.${signature}`
|
||||
);
|
||||
});
|
||||
|
||||
it('does not compress values that are not a JWT', () => {
|
||||
const value = 'x'.repeat(API_TOKEN_COMPRESSION_THRESHOLD + 1);
|
||||
expect(compressAPITokenIfNeeded(value)).toBe(value);
|
||||
});
|
||||
});
|
||||
|
||||
describe('decompressAPIToken', () => {
|
||||
it('returns raw tokens unchanged', () => {
|
||||
const token = signToken(10);
|
||||
expect(decompressAPIToken(token)).toBe(token);
|
||||
});
|
||||
|
||||
it('throws on corrupted compressed tokens', () => {
|
||||
expect(() => decompressAPIToken(`${COMPRESSED_API_TOKEN_PREFIX}AAAA`)).toThrow();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,94 @@
|
||||
import { deflateSync, inflateSync } from 'fflate';
|
||||
|
||||
/**
|
||||
* Prefix marking a compressed API token. A JWT always starts with `eyJ` and `.` is not a
|
||||
* base64url character, so it can't collide with a raw token.
|
||||
*/
|
||||
export const COMPRESSED_API_TOKEN_PREFIX = 'gbz1.';
|
||||
|
||||
/**
|
||||
* Tokens above this length are compressed before being encoded in the rewritten URL,
|
||||
* which is limited to 16KB. Smaller tokens are kept as-is so their URLs (and cache keys) don't change.
|
||||
*/
|
||||
export const API_TOKEN_COMPRESSION_THRESHOLD = 14 * 1024;
|
||||
|
||||
const DOT = 0x2e;
|
||||
|
||||
/**
|
||||
* Compress an API token if it is too large to safely fit in the rewritten URL.
|
||||
* The payload is decoded before compressing, as deflate does much better on JSON than on base64.
|
||||
*/
|
||||
export function compressAPITokenIfNeeded(token: string): string {
|
||||
if (token.length <= API_TOKEN_COMPRESSION_THRESHOLD) {
|
||||
return token;
|
||||
}
|
||||
|
||||
const parts = token.split('.');
|
||||
if (parts.length !== 3) {
|
||||
return token;
|
||||
}
|
||||
const [header, payload, signature] = parts as [string, string, string];
|
||||
|
||||
const encoder = new TextEncoder();
|
||||
const bytes = concatBytes([
|
||||
encoder.encode(`${header}.`),
|
||||
base64UrlToBytes(payload),
|
||||
encoder.encode(`.${signature}`),
|
||||
]);
|
||||
|
||||
return `${COMPRESSED_API_TOKEN_PREFIX}${bytesToBase64Url(deflateSync(bytes, { level: 9 }))}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Restore the original API token from a value produced by `compressAPITokenIfNeeded`.
|
||||
* Values that are not compressed are returned as-is.
|
||||
*/
|
||||
export function decompressAPIToken(value: string): string {
|
||||
if (!value.startsWith(COMPRESSED_API_TOKEN_PREFIX)) {
|
||||
return value;
|
||||
}
|
||||
|
||||
const bytes = inflateSync(base64UrlToBytes(value.slice(COMPRESSED_API_TOKEN_PREFIX.length)));
|
||||
|
||||
// Header and signature are base64url so they can't contain dots, but the payload JSON can.
|
||||
const firstDot = bytes.indexOf(DOT);
|
||||
const lastDot = bytes.lastIndexOf(DOT);
|
||||
if (firstDot === -1 || firstDot === lastDot) {
|
||||
throw new Error('Invalid compressed API token');
|
||||
}
|
||||
|
||||
const decoder = new TextDecoder();
|
||||
const header = decoder.decode(bytes.subarray(0, firstDot));
|
||||
const payload = bytesToBase64Url(bytes.subarray(firstDot + 1, lastDot));
|
||||
const signature = decoder.decode(bytes.subarray(lastDot + 1));
|
||||
|
||||
return `${header}.${payload}.${signature}`;
|
||||
}
|
||||
|
||||
function concatBytes(chunks: Uint8Array[]): Uint8Array {
|
||||
const result = new Uint8Array(chunks.reduce((total, chunk) => total + chunk.length, 0));
|
||||
let offset = 0;
|
||||
for (const chunk of chunks) {
|
||||
result.set(chunk, offset);
|
||||
offset += chunk.length;
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
// Implemented with atob/btoa as Buffer is not guaranteed in the edge runtime.
|
||||
function bytesToBase64Url(bytes: Uint8Array): string {
|
||||
let binary = '';
|
||||
for (let i = 0; i < bytes.length; i += 0x8000) {
|
||||
binary += String.fromCharCode(...bytes.subarray(i, i + 0x8000));
|
||||
}
|
||||
return btoa(binary).replace(/\+/g, '-').replace(/\//g, '_').replace(/=+$/, '');
|
||||
}
|
||||
|
||||
function base64UrlToBytes(value: string): Uint8Array {
|
||||
const binary = atob(value.replace(/-/g, '+').replace(/_/g, '/'));
|
||||
const bytes = new Uint8Array(binary.length);
|
||||
for (let i = 0; i < binary.length; i++) {
|
||||
bytes[i] = binary.charCodeAt(i);
|
||||
}
|
||||
return bytes;
|
||||
}
|
||||
@@ -1,6 +1,10 @@
|
||||
import { isAIEnabled } from '@/components/utils/isAIChatEnabled';
|
||||
import type { GitBookSiteContext } from '@/lib/context';
|
||||
import { linkerWithMarkdownPages } from '@/lib/links';
|
||||
/**
|
||||
* Describe the `ask` and `goal` query parameters of the ask endpoint, for agent-facing prompts.
|
||||
*/
|
||||
export function renderAskParametersDescription(): string {
|
||||
return `\`ask\` is the immediate question: it should be specific, self-contained, and written in natural language.
|
||||
\`goal\` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with \`ask=how do I create an API token\`, a goal like \`build a script that syncs our docs to a CMS\` lets GitBook tailor the answer to that use case.`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Render the "Querying This Documentation" section of the agent instructions.
|
||||
@@ -14,46 +18,7 @@ export function renderQueryingDocumentation(options: { pageUrl: string }): strin
|
||||
GET ${pageUrl}?ask=<question>&goal=<user_goal>
|
||||
\`\`\`
|
||||
|
||||
\`ask\` is the immediate question: it should be specific, self-contained, and written in natural language.
|
||||
\`goal\` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with \`ask=how do I create an API token\`, a goal like \`automate deployments from our CI pipeline\` lets GitBook tailor the answer to that use case.
|
||||
${renderAskParametersDescription()}
|
||||
|
||||
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.`;
|
||||
}
|
||||
|
||||
const AGENT_INSTRUCTIONS_INTRO = `# Agent Instructions
|
||||
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
|
||||
|
||||
## Querying This Documentation`;
|
||||
|
||||
/**
|
||||
* Render the "Agent Instructions" of a markdown page (callers add the separator).
|
||||
*/
|
||||
export function renderPageAgentInstructions(options: { pageUrl: string }): string {
|
||||
return `${AGENT_INSTRUCTIONS_INTRO}
|
||||
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.
|
||||
|
||||
${renderQueryingDocumentation(options)}
|
||||
|
||||
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
|
||||
`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Render the "Agent Instructions" of the site-wide llms.txt and llms-full.txt files.
|
||||
* Returns `undefined` when AI is disabled for the site.
|
||||
*/
|
||||
export function renderSiteAgentInstructions(context: GitBookSiteContext): string | undefined {
|
||||
if (!isAIEnabled(context.customization.ai.mode)) {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
// The ask endpoint ignores the page path, so the top-level `index.md` works on every site.
|
||||
const linker = linkerWithMarkdownPages(context.linker);
|
||||
const pageUrl = linker.toAbsoluteURL(linker.toPathForPagePath({ path: 'index' }));
|
||||
|
||||
return `${AGENT_INSTRUCTIONS_INTRO}
|
||||
This site has an agentic ask interface you may use to query the documentation dynamically by asking a question.
|
||||
|
||||
${renderQueryingDocumentation({ pageUrl })}
|
||||
`;
|
||||
}
|
||||
|
||||
@@ -2,6 +2,7 @@ import { headers } from 'next/headers';
|
||||
|
||||
import { CustomizationDefaultThemeMode } from '@gitbook/api';
|
||||
|
||||
import { decompressAPIToken } from './api-token-compression';
|
||||
import type { SiteURLData } from './context';
|
||||
|
||||
export enum MiddlewareHeaders {
|
||||
@@ -70,7 +71,8 @@ export async function getSiteURLDataFromMiddleware(): Promise<SiteURLData> {
|
||||
);
|
||||
}
|
||||
|
||||
return JSON.parse(siteURLData);
|
||||
const data: SiteURLData = JSON.parse(siteURLData);
|
||||
return { ...data, apiToken: decompressAPIToken(data.apiToken) };
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -22,6 +22,7 @@ import {
|
||||
trackServerInsightsEvents,
|
||||
} from './lib/tracking';
|
||||
import { AI_CATALOG_PATH, AI_CATALOG_WELL_KNOWN_PATH } from '@/lib/aiCatalog/paths';
|
||||
import { compressAPITokenIfNeeded } from '@/lib/api-token-compression';
|
||||
import { getAPITokenFromCookies, getAPITokenResponseCookies } from '@/lib/api-token-cookie';
|
||||
import { isChatGPTRequest } from '@/lib/chatgpt';
|
||||
import { MAX_CHUNKED_COOKIE_LENGTH } from '@/lib/chunked-cookies';
|
||||
@@ -394,7 +395,7 @@ async function serveSiteRoutes(requestURL: URL, request: NextRequest) {
|
||||
revision: siteURLData.revision,
|
||||
shareKey: siteURLData.shareKey,
|
||||
preview: siteURLData.preview,
|
||||
apiToken: siteURLData.apiToken,
|
||||
apiToken: compressAPITokenIfNeeded(siteURLData.apiToken),
|
||||
imagesContextId: imagesContextId,
|
||||
contextId: siteURLData.contextId,
|
||||
isFallback: requestURL.searchParams.get('fallback') === 'true' ? true : undefined,
|
||||
@@ -954,7 +955,6 @@ function encodePathInSiteContent(
|
||||
{
|
||||
type: 'ask_question',
|
||||
query: ask,
|
||||
goal: typeof goal === 'string' ? goal : undefined,
|
||||
location: {
|
||||
displayContext: SiteInsightsDisplayContext.Server,
|
||||
},
|
||||
|
||||
@@ -4,7 +4,6 @@ import pMap, { pMapIterable } from 'p-map';
|
||||
|
||||
import type { RevisionPageDocument, SiteSection, SiteSpace } from '@gitbook/api';
|
||||
|
||||
import { renderSiteAgentInstructions } from '@/lib/ask-prompt';
|
||||
import {
|
||||
type GitBookSiteContext,
|
||||
checkIsRootSiteContext,
|
||||
@@ -172,12 +171,6 @@ async function streamMarkdownPageEntries(
|
||||
stream.enqueue(new TextEncoder().encode(`${header.join('\n')}\n\n`));
|
||||
}
|
||||
|
||||
// Rendered up front, on every part, since agents may only read the start of the file.
|
||||
const agentInstructions = renderSiteAgentInstructions(context);
|
||||
if (agentInstructions) {
|
||||
stream.enqueue(new TextEncoder().encode(`${agentInstructions}\n---\n\n`));
|
||||
}
|
||||
|
||||
// Process the pages
|
||||
for await (const markdown of pMapIterable(
|
||||
pagesToProcess,
|
||||
|
||||
@@ -4,11 +4,12 @@ import { toMarkdown } from 'mdast-util-to-markdown';
|
||||
|
||||
import type { SiteSection, SiteSpace } from '@gitbook/api';
|
||||
|
||||
import { renderSiteAgentInstructions } from '@/lib/ask-prompt';
|
||||
import { isAIEnabled } from '@/components/utils/isAIChatEnabled';
|
||||
import { type GitBookSiteContext, checkIsRootSiteContext } from '@/lib/context';
|
||||
import { throwIfDataError } from '@/lib/data';
|
||||
import { type GitBookLinker, linkerWithMarkdownPages } from '@/lib/links';
|
||||
import { getMarkdownContentType } from '@/lib/markdown-content-type';
|
||||
import { resolveFirstDocument } from '@/lib/pages';
|
||||
import { type FlatPageEntry, getIndexablePages } from '@/lib/sitemap';
|
||||
import {
|
||||
filterSiteSpacesByLocale,
|
||||
@@ -48,10 +49,7 @@ export async function serveLLMsTxt(baseContext: GitBookSiteContext) {
|
||||
bullet: '-',
|
||||
});
|
||||
|
||||
const agentInstructions = renderSiteAgentInstructions(context);
|
||||
if (agentInstructions) {
|
||||
output += `\n\n---\n\n${agentInstructions}`;
|
||||
}
|
||||
output += renderAskFooter(context);
|
||||
|
||||
return new Response(output, {
|
||||
headers: {
|
||||
@@ -207,3 +205,27 @@ export async function getMarkdownForPagesTree(
|
||||
});
|
||||
return nodes;
|
||||
}
|
||||
|
||||
function renderAskFooter(context: GitBookSiteContext) {
|
||||
if (!isAIEnabled(context.customization.ai.mode)) {
|
||||
return '';
|
||||
}
|
||||
|
||||
return `\n\n---\n\n# Agent Instructions
|
||||
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
|
||||
|
||||
## Querying This Documentation
|
||||
If you need additional information, you can query the documentation dynamically by asking a question.
|
||||
Perform an HTTP GET request on a page URL with the \`ask\` query parameter:
|
||||
\`\`\`
|
||||
GET ${context.linker.toAbsoluteURL(
|
||||
context.linker.toPathForPagePath({
|
||||
path: resolveFirstDocument(context.revision.pages, [])?.page.path ?? 'index',
|
||||
})
|
||||
)}?ask=<question>
|
||||
\`\`\`
|
||||
The question should be specific, self-contained, and written in natural language.
|
||||
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
|
||||
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
|
||||
`;
|
||||
}
|
||||
|
||||
@@ -2,7 +2,7 @@ import type { RevisionPageDocument, RevisionPageGroup } from '@gitbook/api';
|
||||
|
||||
import { resolveMissingPagePath } from '@/components/SitePage/fetch';
|
||||
import { isAIEnabled } from '@/components/utils/isAIChatEnabled';
|
||||
import { renderPageAgentInstructions, renderQueryingDocumentation } from '@/lib/ask-prompt';
|
||||
import { renderQueryingDocumentation } from '@/lib/ask-prompt';
|
||||
import type { GitBookSiteContext } from '@/lib/context';
|
||||
import { getExposableError } from '@/lib/data';
|
||||
import { linkerWithMarkdownPages } from '@/lib/links';
|
||||
@@ -184,7 +184,16 @@ function renderAskFooter(
|
||||
})
|
||||
);
|
||||
|
||||
return `\n\n---\n\n${renderPageAgentInstructions({ pageUrl })}`;
|
||||
return `\n\n---\n\n# Agent Instructions
|
||||
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
|
||||
|
||||
## Querying This Documentation
|
||||
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
|
||||
|
||||
${renderQueryingDocumentation({ pageUrl })}
|
||||
|
||||
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
|
||||
`;
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
Reference in New Issue
Block a user