Compare commits

...
Author SHA1 Message Date
paseo-bot[bot]andClaude Opus 5.5 0b1ae2f319 Run the command-line batching test on Windows too
Use 150-character folder names, short enough for Windows paths, and
enough folders to exceed each platform's own command-line limit, so a
batch budget too large for Windows fails on the Windows runner.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 19:11:06 +00:00
paseo-bot[bot]andClaude Opus 5.5 540da1a3b5 Fold only ASCII letters and split long folder searches across Git calls
Git's case-insensitive matching folds only A-Z, so folding a pattern
with toLowerCase() turned ÉTÉ/sub/.env into a pathspec that matched
nothing. Fold only ASCII letters, as Git does. Pass the folders to
search in batches of at most 16,000 characters, so a **/ pattern in a
repo with many ignored folders stays under Windows' 32,767-character
command line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 18:43:55 +00:00
paseo-bot[bot]andClaude Opus 5.5 b5bd6b7704 Search only folders Git ignores, as deep as a pattern can reach, matching the source's case rule
Git lists untracked folders such as apps/ beside the ignored entries in
them; searching those walked apps/web/node_modules/. Search only folders
check-ignore reports, skip a folder at or below the depth of a pattern
without **, strip a leading UTF-8 BOM from .worktreeinclude as git does,
and match with the source repo's core.ignoreCase.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 18:25:59 +00:00
paseo-bot[bot]andClaude Opus 5.5 cf193266f3 Keep worktree include matching literal and search only the path a pattern names
Prefix paths sent to check-ignore with ./ so a file named like pathspec
magic (`:(x`, `:!x`) no longer fails worktree creation, create the scratch
repo with an empty template so the user's template info/exclude adds no
patterns, and search a glob-free pattern's whole path instead of its
parent folder, so `node_modules/.cache` no longer walks node_modules/.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 17:46:12 +00:00
paseo-bot[bot] 66d8e43e66 Match worktree includes without writing scratch files and search only the folders patterns name
Match ignored entries with check-ignore --no-index instead of recreating
them on disk, search inside an ignored folder only as deep as the pattern
names it, copy each file once, and note in the docs that **/ patterns
search every ignored folder.
2026-11-17 17:25:32 +00:00
paseo-bot[bot]andmgranderath 0d39140dc5 Create worktrees in large repos without scanning every ignored file
worktree-include listed candidates with `git ls-files --others --ignored
--exclude-from=.worktreeinclude`. With only the include patterns as
excludes, git does not treat node_modules/ and other ignored folders as
ignored, so it walks every file in them. On a large repo the walk outlasts
the 30 s plugin hook limit and worktree creation fails and rolls back.

List ignored entries with --exclude-standard --directory instead, so an
ignored folder is one entry, match those entries against .worktreeinclude
in a scratch repository, and search inside an ignored folder only when a
pattern matches it or names a path inside it.

A bare name such as `.env` no longer finds `node_modules/pkg/.env`; the
docs say how to name such a path.

Closes #10004

Co-authored-by: mgranderath <22001111+mgranderath@users.noreply.github.com>
2026-11-17 16:55:37 +00:00
4 changed files with 457 additions and 16 deletions
@@ -0,0 +1,291 @@
import { execFileSync } from "node:child_process";
import { existsSync } from "node:fs";
import { mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { dirname, join } from "node:path";
import { afterEach, beforeEach, describe, expect, it } from "vitest";
import { copyWorktreeIncludes } from "./copy.js";
let root: string;
let source: string;
let worktree: string;
async function writeSource(files: Record<string, string>): Promise<void> {
for (const [file, contents] of Object.entries(files)) {
await mkdir(dirname(join(source, file)), { recursive: true });
await writeFile(join(source, file), contents);
}
}
// Total directories Git visits while listing files, read from its trace2 events.
async function directoriesVisited(run: () => Promise<unknown>): Promise<number> {
const trace = join(root, "trace.json");
process.env.GIT_TRACE2_EVENT = trace;
try {
await run();
} finally {
delete process.env.GIT_TRACE2_EVENT;
}
return (await readFile(trace, "utf8"))
.split("\n")
.filter(Boolean)
.map((line) => JSON.parse(line) as { event: string; key?: string; value?: string })
.filter((event) => event.event === "data" && event.key === "directories-visited")
.reduce((total, event) => total + Number(event.value), 0);
}
async function copied(): Promise<string[]> {
const result = await copyWorktreeIncludes(worktree, source);
if (result.kind !== "copied") throw new Error(`expected a copy, got ${result.kind}`);
return [...result.copied].sort();
}
beforeEach(async () => {
root = await mkdtemp(join(tmpdir(), "worktree-include-"));
source = join(root, "source");
worktree = join(root, "worktree");
await mkdir(source);
await mkdir(worktree);
execFileSync("git", ["init", "-q", source]);
});
afterEach(async () => {
await rm(root, { recursive: true, force: true });
});
describe("copyWorktreeIncludes", () => {
it("does not search inside an ignored directory that no pattern names", async () => {
await writeSource({
".gitignore": "node_modules/\n.env\n",
".worktreeinclude": ".env\n",
".env": "SECRET=root",
"apps/web/.env": "SECRET=web",
"node_modules/pkg/.env": "SECRET=package",
});
expect(await copied()).toEqual([".env", "apps/web/.env"]);
expect(existsSync(join(worktree, "node_modules"))).toBe(false);
});
it("searches inside ignored directories for a pattern that starts with **/", async () => {
await writeSource({
".gitignore": "node_modules/\n.env\n",
".worktreeinclude": "**/.env\n",
".env": "SECRET=root",
"node_modules/pkg/.env": "SECRET=package",
});
expect(await copied()).toEqual([".env", "node_modules/pkg/.env"]);
});
it("copies a file in an untracked directory once when a **/ pattern also searches it", async () => {
await writeSource({
".gitignore": ".env\n",
".worktreeinclude": "**/.env\nnew-package/.env\n",
"new-package/.env": "SECRET=new",
});
const result = await copyWorktreeIncludes(worktree, source);
expect(result).toEqual({
kind: "copied",
copied: ["new-package/.env"],
skipped: [],
});
});
it("copies a path a pattern names inside an ignored directory", async () => {
await writeSource({
".gitignore": "node_modules/\n",
".worktreeinclude": "node_modules/pkg/.env\n",
"node_modules/pkg/.env": "SECRET=package",
"node_modules/other/.env": "SECRET=other",
});
expect(await copied()).toEqual(["node_modules/pkg/.env"]);
});
it("copies an ignored directory that a pattern names", async () => {
await writeSource({
".gitignore": ".secrets/\nbuild/\n",
".worktreeinclude": ".secrets/\n",
".secrets/a.key": "a",
".secrets/nested/b.key": "b",
"build/out.js": "built",
});
expect(await copied()).toEqual([".secrets/a.key", ".secrets/nested/b.key"]);
});
it("searches inside an ignored directory for a path a pattern names", async () => {
await writeSource({
".gitignore": "config/\n",
".worktreeinclude": "config/*.local.json\n!config/production.local.json\n",
"config/dev.local.json": "dev",
"config/production.local.json": "production",
"config/defaults.json": "defaults",
});
expect(await copied()).toEqual(["config/dev.local.json"]);
expect(await readFile(join(worktree, "config/dev.local.json"), "utf8")).toBe("dev");
});
it("copies ignored files inside an untracked directory", async () => {
await writeSource({
".gitignore": ".env\n",
".worktreeinclude": ".env\n",
"new-package/.env": "SECRET=new",
});
expect(await copied()).toEqual(["new-package/.env"]);
});
it("copies nothing that is tracked or not ignored", async () => {
await writeSource({
".gitignore": ".env\n",
".worktreeinclude": ".env\nnotes.txt\n",
"notes.txt": "not ignored",
});
expect(await copied()).toEqual([]);
});
it("searches only the path a glob-free pattern names directly inside an ignored directory", async () => {
const packages: Record<string, string> = {};
for (let i = 0; i < 20; i++) packages[`node_modules/pkg${i}/lib/index.js`] = "code";
await writeSource({
".gitignore": "node_modules/\n",
".worktreeinclude": "node_modules/.cache\n",
"node_modules/.cache/state.json": "cache",
...packages,
});
let result: Awaited<ReturnType<typeof copyWorktreeIncludes>> | undefined;
const visited = await directoriesVisited(async () => {
result = await copyWorktreeIncludes(worktree, source);
});
expect(result).toEqual({
kind: "copied",
copied: ["node_modules/.cache/state.json"],
skipped: [],
});
expect(visited).toBeLessThan(10);
});
it.skipIf(process.platform === "win32")(
"copies includes when an ignored file is named like a Git pathspec",
async () => {
await writeSource({
".gitignore": "*.log\n.env\n",
".worktreeinclude": ".env\n",
".env": "SECRET=root",
":(oops.log": "log",
":!c.log": "log",
});
expect(await copied()).toEqual([".env"]);
},
);
it("ignores exclude patterns from the user's git template", async () => {
const template = join(root, "template");
await mkdir(join(template, "info"), { recursive: true });
await writeFile(join(template, "info", "exclude"), "*.log\n");
await writeSource({
".gitignore": "*.log\n.env\n",
".worktreeinclude": ".env\n",
".env": "SECRET=root",
"debug.log": "log",
});
process.env.GIT_TEMPLATE_DIR = template;
try {
expect(await copied()).toEqual([".env"]);
} finally {
delete process.env.GIT_TEMPLATE_DIR;
}
});
it("does not search an ignored directory deeper than a pattern without ** can reach", async () => {
const packages: Record<string, string> = {};
for (let i = 0; i < 20; i++) packages[`apps/web/node_modules/pkg${i}/lib/index.js`] = "code";
await writeSource({
".gitignore": "node_modules/\n.env\n",
".worktreeinclude": "apps/*/.env\n",
"apps/web/.env": "SECRET=web",
...packages,
});
let result: Awaited<ReturnType<typeof copyWorktreeIncludes>> | undefined;
const visited = await directoriesVisited(async () => {
result = await copyWorktreeIncludes(worktree, source);
});
expect(result).toEqual({ kind: "copied", copied: ["apps/web/.env"], skipped: [] });
expect(visited).toBeLessThan(10);
});
it("reads the first pattern after a UTF-8 byte order mark", async () => {
await writeSource({
".gitignore": "node_modules/\n",
".worktreeinclude": "\uFEFFnode_modules/.cache/\n",
"node_modules/.cache/state.json": "cache",
});
expect(await copied()).toEqual(["node_modules/.cache/state.json"]);
});
it("matches patterns case-insensitively when the source repo ignores case", async () => {
execFileSync("git", ["-C", source, "config", "core.ignoreCase", "true"]);
await writeSource({
".gitignore": ".env\nconfig/\n",
".worktreeinclude": ".ENV\nCONFIG/a.json\n",
".env": "SECRET=root",
"config/a.json": "a",
});
expect(await copied()).toEqual([".env", "config/a.json"]);
});
it("searches a pattern's path in the case it has on disk when the source repo ignores case", async () => {
execFileSync("git", ["-C", source, "config", "core.ignoreCase", "true"]);
await writeSource({
".gitignore": "Config/\nÉTÉ/\n",
".worktreeinclude": "config/a.json\nÉTÉ/sub/.env\n",
"Config/a.json": "a",
"ÉTÉ/sub/.env": "SECRET=summer",
});
expect(await copied()).toEqual(["Config/a.json", "ÉTÉ/sub/.env"]);
});
// Each searched folder is one argument to Git. Windows caps a command line at 32,767
// characters and Linux at 2 MB, so each platform gets enough folders to exceed its own
// limit, with names short enough for Windows paths.
it("searches more ignored folders than fit on one command line", async () => {
const folders = process.platform === "win32" ? 300 : 15_000;
const name = (i: number) => `${"x".repeat(150)}${i}`;
await writeSource({
".gitignore": "x*/\n.env\n",
".worktreeinclude": "**/.env\n",
[`${name(folders - 1)}/.env`]: "SECRET=last",
});
for (let i = 0; i < folders - 1; i++) await mkdir(join(source, name(i)));
await Promise.all(
Array.from({ length: folders - 1 }, (_, i) => writeFile(join(source, name(i), "a"), "")),
);
expect(await copied()).toEqual([`${name(folders - 1)}/.env`]);
}, 60_000);
it("copies ignored files inside an untracked directory a pattern names", async () => {
await writeSource({
".gitignore": ".env\n",
".worktreeinclude": "newdir/\n",
"newdir/.env": "SECRET=new",
"newdir/notes.txt": "untracked",
});
expect(await copied()).toEqual(["newdir/.env"]);
});
});
+162 -16
View File
@@ -2,7 +2,8 @@
// Copyright (c) 2026 Carlos Precioso.
import { execFile } from "node:child_process";
import { constants, existsSync } from "node:fs";
import { copyFile, mkdir } from "node:fs/promises";
import { copyFile, mkdir, mkdtemp, readFile, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import { dirname, join } from "node:path";
import { promisify } from "node:util";
@@ -22,37 +23,33 @@ async function git(cwd: string, args: string[], input?: string): Promise<string>
return stdout;
}
async function gitAllowExit1(cwd: string, args: string[], input: string): Promise<string> {
// `git check-ignore --stdin` reads each path as a pathspec, so a name such as `:(x`
// would be parsed as pathspec magic. A `./` prefix keeps every path literal.
async function checkIgnore(cwd: string, args: string[], paths: string[]): Promise<string[]> {
const input = paths.map((path) => `./${path}\0`).join("");
let output: string;
try {
return await git(cwd, args, input);
output = await git(cwd, [...args, "check-ignore", "--stdin", "-z"], input);
} catch (error) {
// `git check-ignore` exits 1 when nothing matched.
if ((error as { code?: number }).code === 1) return "";
if ((error as { code?: number }).code === 1) return [];
throw error;
}
return splitNul(output).map((path) => path.replace(/^\.\//, ""));
}
const splitNul = (s: string) => s.split("\0").filter(Boolean);
const isDirectory = (entry: string) => entry.endsWith("/");
// Copy files that match .worktreeinclude (gitignore syntax)
// AND are gitignored, so tracked files are never touched.
export async function copyWorktreeIncludes(worktree: string, source: string): Promise<CopyResult> {
if (!existsSync(join(source, INCLUDE_FILE))) return { kind: "no-include-file" };
const matches = splitNul(
await git(source, [
"ls-files",
"--others",
"--ignored",
`--exclude-from=${INCLUDE_FILE}`,
"-z",
]),
);
const matches = await findIncludedFiles(source);
if (matches.length === 0) return { kind: "copied", copied: [], skipped: [] };
const ignored = splitNul(
await gitAllowExit1(source, ["check-ignore", "--stdin", "-z"], matches.join("\0") + "\0"),
);
const ignored = await checkIgnore(source, [], matches);
const copied: string[] = [];
const skipped: string[] = [];
@@ -69,3 +66,152 @@ export async function copyWorktreeIncludes(worktree: string, source: string): Pr
}
return { kind: "copied", copied, skipped };
}
// Git lists a directory it ignores as a whole, such as node_modules/, as one entry.
// Only directories that a pattern matches or names a path inside are searched, and
// only as deep as the pattern names, so creating a worktree does not walk every file
// in large ignored trees.
async function findIncludedFiles(source: string): Promise<string[]> {
const ignored = splitNul(
await git(source, [
"ls-files",
"--others",
"--ignored",
"--exclude-standard",
"--directory",
"-z",
]),
);
if (ignored.length === 0) return [];
const ignoreCase = await sourceIgnoresCase(source);
const patterns = parsePatterns(await readFile(join(source, INCLUDE_FILE), "utf8"));
const matched = new Set(await matchIncludePatterns(source, ignored, ignoreCase));
const files = ignored.filter((entry) => !isDirectory(entry) && matched.has(entry));
const searched = new Set<string>();
// Git also lists an untracked folder that holds ignored entries, which are listed on
// their own, so only folders Git ignores as a whole are searched.
const directories = await checkIgnore(source, [], ignored.filter(isDirectory));
for (const directory of directories) {
if (matched.has(directory)) {
searched.add(directory);
continue;
}
for (const pattern of patterns) {
const inside = searchedPathInside(pattern, directory, ignoreCase);
if (inside !== null) searched.add(inside);
}
}
if (searched.size === 0) return files;
const pathspecs = [...searched].map(
(path) => `:(${ignoreCase ? "literal,icase" : "literal"})${path}`,
);
const inside: string[] = [];
for (const batch of commandLineBatches(pathspecs)) {
const listed = await git(source, [
"ls-files",
"--others",
"--ignored",
`--exclude-from=${INCLUDE_FILE}`,
"-z",
"--",
...batch,
]);
inside.push(...splitNul(listed));
}
return [...new Set([...files, ...inside])];
}
// Windows caps a command line at 32,767 characters, so a pattern such as `**/.env` in a
// repo with many ignored folders searches them over several Git calls.
const COMMAND_LINE_BATCH_CHARS = 16_000;
function commandLineBatches(args: string[]): string[][] {
const batches: string[][] = [];
let batch: string[] = [];
let length = 0;
for (const arg of args) {
if (batch.length > 0 && length + arg.length + 1 > COMMAND_LINE_BATCH_CHARS) {
batches.push(batch);
batch = [];
length = 0;
}
batch.push(arg);
length += arg.length + 1;
}
if (batch.length > 0) batches.push(batch);
return batches;
}
// Match entries against .worktreeinclude alone, in a scratch repository where no other
// ignore rules apply. Directory entries keep their trailing slash, so directory
// patterns match them without anything on disk.
async function matchIncludePatterns(
source: string,
entries: string[],
ignoreCase: boolean,
): Promise<string[]> {
const scratch = await mkdtemp(join(tmpdir(), "paseo-worktree-include-"));
try {
// An empty template keeps the user's template `info/exclude` out of the matching.
await git(scratch, ["init", "-q", "--template="]);
return await checkIgnore(
scratch,
[
"-c",
`core.excludesFile=${join(source, INCLUDE_FILE)}`,
"-c",
`core.ignoreCase=${ignoreCase}`,
],
entries,
);
} finally {
await rm(scratch, { recursive: true, force: true });
}
}
// The scratch repository would otherwise take core.ignoreCase from the temp folder.
async function sourceIgnoresCase(source: string): Promise<boolean> {
try {
return (await git(source, ["config", "--type=bool", "core.ignoreCase"])).trim() === "true";
} catch {
// `git config` exits 1 when the key is unset.
return false;
}
}
function parsePatterns(contents: string): string[] {
// Git skips a leading UTF-8 byte order mark in exclude files.
return contents
.replace(/^\uFEFF/, "")
.split(/\r?\n/)
.map((line) => line.trimEnd())
.filter((line) => line !== "" && !line.startsWith("#") && !line.startsWith("!"));
}
// A pattern with a slash before its end is anchored to the checkout root, so it can
// name a path inside `directory` when its literal start and the directory overlap.
// Without `**`, a pattern matches paths of exactly its own depth, so a directory that
// deep or deeper can hold no match below it. Returns the narrowest path to search: the
// whole path when the pattern has no glob, else the folder before its first glob, or
// `directory` itself when the path ends above it.
function searchedPathInside(
pattern: string,
rawDirectory: string,
ignoreCase: boolean,
): string | null {
// Git folds only ASCII letters when it ignores case.
const fold = (text: string) =>
ignoreCase ? text.replace(/[A-Z]+/g, (letters) => letters.toLowerCase()) : text;
const path = fold(pattern.replace(/^\//, ""));
const directory = fold(rawDirectory);
const depth = (text: string) => text.replace(/\/$/, "").split("/").length;
if (depth(path) < 2) return null;
if (!path.includes("**") && depth(directory) >= depth(path)) return null;
const globAt = path.search(/[*?[\\]/);
const literal = globAt === -1 ? path : path.slice(0, globAt);
if (!literal.startsWith(directory) && !directory.startsWith(literal)) return null;
const searched = globAt === -1 ? literal : literal.slice(0, literal.lastIndexOf("/") + 1);
return searched.length > directory.length ? searched : directory;
}
+2
View File
@@ -36,6 +36,8 @@ config/*.local.json
A pattern only selects files that Git already ignores. Adding a path here does not ignore it, so also configure `.gitignore` when needed. Tracked files and untracked files that Git does not ignore are excluded.
Paseo does not search inside a directory that Git ignores as a whole, such as `node_modules/`, unless a pattern names that directory or a path inside it. A bare name like `.env` matches `.env` and `apps/web/.env`, but not `node_modules/pkg/.env`. To copy from inside an ignored directory, name it in the pattern, as in `node_modules/pkg/.env` or `.secrets/`. A pattern with a glob in its first directory name, such as `**/.env` or `*/.env`, searches every ignored directory, including `node_modules/`, so name the directory instead when it is large.
## When files are copied
The built-in `worktree-include` plugin reads the file from the checkout you create the worktree from. That source can itself be another worktree. Keep the plugin enabled for copying to run.
@@ -36,6 +36,8 @@ config/*.local.json
模式只会选择 Git 已忽略的文件。在此添加路径不会使 Git 忽略该文件,需要时还应配置 `.gitignore`。已跟踪的文件和未被 Git 忽略的未跟踪文件都不会被复制。
对于被 Git 整体忽略的目录(例如 `node_modules/`),除非某个模式指明了该目录或其中的路径,否则 Paseo 不会在其中查找。像 `.env` 这样的纯文件名会匹配 `.env` 和 `apps/web/.env`,但不会匹配 `node_modules/pkg/.env`。要从被忽略的目录中复制文件,请在模式中写出该目录,例如 `node_modules/pkg/.env` 或 `.secrets/`。第一级目录名中带有通配符的模式(例如 `**/.env` 或 `*/.env`)会在所有被忽略的目录中查找,包括 `node_modules/`,因此目录较大时请直接写出目录。
## 文件复制时机 {#when-files-are-copied}
内置的 `worktree-include` 插件从用于创建 worktree 的检出读取该文件。源检出也可以是另一个 worktree。保持该插件启用才能执行复制。