loremfile

Manifest and discovery files

1. manifest.json (stable, schema-versioned)

Location: repository root (committed) and https://loremfile.dev/manifest.json (deployed copy, identical bytes).

1.1 Top-level object

{
  "$schema": "https://loremfile.dev/schema/manifest-v1.json",
  "schema_version": 1,
  "catalog_version": "1.0.0",
  "generated_at": "2026-10-01T12:00:00Z",
  "base_url": "https://loremfile.dev/",
  "license": "CC0-1.0",
  "license_url": "https://creativecommons.org/publicdomain/zero/1.0/",
  "source": "https://github.com/kumarprabhashanand/loremfile",
  "toolchain_image": "ghcr.io/kumarprabhashanand/loremfile-toolchain@sha256:<digest>",
  "count": 416,
  "total_bytes": 1060000000,
  "formats": ["avif", "bin", "csv", "..."],
  "fixtures": [ { "...": "see 1.2" } ]
}

1.2 Fixture entry

{
  "path": "pdf/a4-3pages.pdf",
  "url": "https://loremfile.dev/pdf/a4-3pages.pdf",
  "format": "pdf",
  "ext": "pdf",
  "mime": "application/pdf",
  "bytes": 12345,
  "sha256": "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08",
  "size_class": "free",
  "phase": 1,
  "description": "A4 portrait, 3 pages of Lorem Ipsum body text with page numbers. Helvetica core font, no embedded fonts.",
  "tags": ["document", "multi-page", "text"],
  "edge_case": false,
  "props": { "pages": 3, "page_width_pt": 595.28, "page_height_pt": 841.89, "encrypted": false, "pdf_version": "1.7" },
  "generator": "pdf.basic",
  "generator_params": { "pages": 3, "page_size": "A4", "orientation": "portrait", "page_numbers": true },
  "added_in": "1.0.0",
  "status": "active",
  "deprecated": false,
  "supersededBy": null,
  "notes": null
}

Field rules:

Field Type Rule
path string Unique. Matches URL grammar in 03-http-contract.md §1. Does not change.
url string base_url + path.
format string First path segment.
ext string Everything after the first dot of the last path segment (tar.gz, not gz; m3u8 for hls/720p-10s/index.m3u8).
mime string Exact Content-Type served. From the format's default in the catalog unless overridden per fixture (e.g. charset).
bytes integer Exact size. Frozen.
sha256 hex string (64) Frozen.
size_class enum exact (bytes equal nominal), boundary (nominal ± 1 exactly), approx (within ±5 % of nominal), free (no size claim in name).
phase integer 1–4.
description string One or two sentences a human or agent can use to decide fit. ≤ 300 chars.
tags string[] Lowercase; controlled vocabulary in catalog/_tags.yaml.
edge_case boolean true for anything under edge/ and for format-dir fixtures flagged as unusual (e.g. txt/control-characters.txt).
props object Measured by validators, format-specific (see §1.3). The measured values cannot change, because the bytes they were measured on cannot: sha256 is frozen, so pages or rows is settled for good. What a prop is called, and the wording of one that is a sentence, may be corrected: a correction is a changelog entry and a new catalog version, never a silent rewrite.
generator, generator_params string, object Enough to regenerate. Informational; may change if a generator is refactored (bytes must not).
added_in semver Catalog version that introduced it.
status enum active, removed (legal takedown tombstone, see §1.5).
deprecated, supersededBy boolean, string/null Derived, never set here: the catalog's superseded_by is the input, and manifest update writes both fields from it (03 §7.3), so a value set by hand in the manifest is overwritten by the next update and manifest check reports it until then. supersededBy is the replacing path. A superseded entry stays active and served, and stays in manifest.json, sha256sums.txt and {format}/index.json; format pages, the search index and llms-full.txt list its replacement instead.

1.3 props by format (what validators measure)

Formats Keys
pdf pages, page_width_pt, page_height_pt, encrypted, pdf_version, has_outline, has_images, and for a PDF with an AcroForm only, form_fields and filled_fields (a filled field must carry its appearance stream)
png, jpg, gif, webp, avif, bmp, tiff, ico width, height, mode (RGB, RGBA, L, P, CMYK, I;16, I;16B), frames, animated, progressive, exif_orientation (jpg), sizes (ico); for tiff (1.9.0), compression as Pillow names it (raw, tiff_lzw), byte_order (II or MM, the header's first two bytes) and bits_per_sample (one value per sample, from tag 258)
heic width and height (the primary item's ispe, which heif-convert's decode must match), mode (decoded), major_brand, compatible_brands, primary_item_type (hvc1), items, format
svg width, height, viewbox
mp4, webm, mkv, mov, avi, ogv, ts, hls duration_ms, width, height, fps, vcodec, acodec, sample_rate, channels, video_streams, audio_streams, subtitle_streams, profile (the video stream's profile as ffprobe names it, where it reports one: High, Constrained Baseline, Profile 0), and for H.264 level (3.0, 3.1: the stream's level_idc over ten); mp4 and mov where the catalog sets faststart: faststart, moov_at_end. hls media playlists: segments, target_duration_s, playlist_type; hls master playlists: variants, and per rendition in order bandwidths, resolutions and codecs (a validator reads them from the playlist; a CI test re-measures each from the rendition's segments); Since 1.9.1, every file with an audio stream (HLS segments too) carries tone_hz: one value per channel, in Hz to a tenth, measured by Goertzel over one decoded second from 0.25 s at 8 kHz (util/tone.py), null for a channel where no single frequency holds 0.9 of the energy, and no key at all when no channel has a tone (mp3/silence-5s.mp3). A chained Ogg file decodes as its first chain, so it adds chain_tone_hz, one such list per chain, each measured from that chain's own bytes. Since 1.10.0, three keys appear only on a file that has what they measure, so no other entry changes: track_tone_hz (one tone_hz list per audio track, when there is more than one), scodec (the first subtitle track's codec: subrip for SRT) and chapters (how many chapters ffprobe reports).
mp3, wav, flac, ogg, opus, m4a, aac, aiff duration_ms, acodec, sample_rate, channels, audio_streams, bitrate_kbps (the audio packets' bytes over the stream's duration: what the codec spent, without container pages, tags or cover art, and never the rate an encoder was asked for or a header declares), bit_depth (PCM only); wav also header_format (pcm, extensible or float, the fmt chunk's tag) and header_bytes (where the samples start: 44 for the canonical header, 68 for the extensible one, 58 for the float header with its fact chunk, and later whenever another chunk comes first, as 94 does in wav/with-list-chunk-3s.wav), and since 1.10.0 info_tags when a LIST/INFO chunk precedes the samples, each tag by its four-character id; for ogg (1.9.0), chains and header_pages from the Ogg pages themselves, a chained file's duration_ms and bitrate_kbps summed over its chains (ffprobe reports the first chain alone), and for Vorbis its comments as tags and its cover as cover (format, width, height, decoded with Pillow); Since 1.9.1, every file with an audio stream (HLS segments too) carries tone_hz: one value per channel, in Hz to a tenth, measured by Goertzel over one decoded second from 0.25 s at 8 kHz (util/tone.py), null for a channel where no single frequency holds 0.9 of the energy, and no key at all when no channel has a tone (mp3/silence-5s.mp3). A chained Ogg file decodes as its first chain, so it adds chain_tone_hz, one such list per chain, each measured from that chain's own bytes.
docx paragraphs, tables, table_rows, images (media parts), words, inline_images, anchored_images (floating drawings, which readers that walk inline shapes miss), images_with_alt_text; the tracked-changes file adds its revision props
xlsx sheets, sheet_names, rows and columns (first sheet, rows incl. header), formulas (every sheet; each must carry a cached value), images, and cross_sheet_formulas only where a formula reads another sheet
pptx slides, layouts, images, slide_width_emu and slide_height_emu (16:9 is 12191695 by 6858000, 4:3 is 9144000 by 6858000), notes (slides with speaker notes); since 1.10.0 tables and charts (native shapes, not pictures of them) and hidden_slides (slides marked show="0")
txt, md, html, css, js, rtf, srt, vtt, log, sql, ini encoding, bom, line_ending (lf/crlf/cr/mixed/none), lines, max_line_bytes; md also headings, anchor_links (in-page links, every one resolving to a heading's anchor) and footnotes (every reference with a body and every body referenced)
csv, tsv plus rows (data rows), columns, delimiter, has_header, quoted_fields, and the two causes of quoting reported apart: quoted_delimiter (a field holds the delimiter) and quoted_newline (a field holds a line break)
json, ndjson, geojson, har, ipynb, jsonld top_type (object/array/string…), items (array length or ndjson lines), max_depth
xml, rss, atom, kml, gpx root_element, encoding, has_dtd, namespaces
yaml, toml documents (yaml), top_keys
parquet, avro, arrow rows, columns (for parquet the leaf columns, as Parquet counts them), compression (parquet), row_groups; since 1.10.0 parquet also column_types (each top-level column's Arrow type, as pyarrow spells it); for arrow (1.9.0), ipc_format (file when the bytes start and end with ARROW1, else stream, and held to the served type), compression (zstd, lz4 or none, from the codec frame's magic, which pyarrow never reports), column_types and null_counts
sqlite tables, rows_total, page_size, journal_mode, has_fts
zip, 7z entries, directories, method (stored/deflate/lzma2), encrypted, zip64, comment
tar (+gz/bz2/xz/zst) entries, compression, has_symlinks, pax
gz, bz2, xz, zst members (gz), original_bytes
ttf, otf, woff, woff2 glyphs, flavor, family_name, units_per_em
bin pattern (shake256, zeros, ones, incrementing)
ics events, has_vtimezone, has_rrule
vcf cards, version
eml, mbox messages, parts, attachments, has_html
epub version, chapters, images
pdf, docx, pptx, xlsx, epub, html, md contains_text on the fixtures §1.3.2 names: three to five strings a correct reader finds in the file
pem, der kind (certificate/csr), algorithm, not_before, not_after
wasm sections, exports
edge/* defect, intended_format, bytes, damage, outcome, compare_with, and derived_from for the six that were cut or copied from a published fixture; plus fraction (truncated) and hostile_entries (hostile-name). §1.3.1 describes the four that are about the breakage. defect is a closed enum, and it tells the validator what to assert: zero-byte (bytes == 0), truncated (bytes == floor(fraction × source bytes) with fraction from the catalog's edge block, default 0.5 (pdf-truncated-60pct.pdf uses 0.6), and prefix-equal to the source), mismatched-extension (bytes equal the source fixture; magic bytes match the source format, not the extension), magic-prefix (bytes == the magic value from the edge block followed by the source fixture's bytes), invalid-syntax (the format's parser raises), invalid-encoding (strict decode with the declared charset raises), nonstandard (strict parser raises, lenient parser succeeds), bom (decodes after stripping the BOM; parser succeeds on the stripped text), stress (parses; the recorded props document the depth/size), hostile-name (parses; at least one entry name contains ../), polyglot (both the announced format's reader and the one named by edge.also_valid_as accept the bytes; the prop also_valid_as records the second format). Only fixtures under edge/ carry these; unusual-but-valid fixtures elsewhere (txt/control-characters.txt, txt/nul-bytes.txt) have edge_case: true and ordinary props

1.3.1 The four edge props (what is wrong, and what to expect)

A broken file is only useful if you know how it is broken. Four props say so, and none of them is written by hand: loremfile validate measures or derives each one from the bytes and refuses the fixture when the answer disagrees with the catalog.

Prop What it is
damage One sentence about the bytes: where the file stops, which structure is missing, where a parser gives up. Composed from what the validator measured, so every number in it was counted on the file it describes.
outcome must-fail, may-recover or varies; see below.
compare_with The valid fixture of the same announced type. Feed your reader both: a reader that refuses everything is not the same as one that refuses this.
derived_from The published fixture this one was cut or copied from, present on the six that have one and absent on the other thirteen. The validator reads that file and checks the relationship: a truncation's length must equal the recorded fraction of it and be a prefix of it, and a mislabelled file must equal it byte for byte.

outcome describes the file, not your reader. It is derived by offering the bytes to the reader for the type the file announces and, when that refuses them, to a tolerant reading the format itself permits:

may-recover and varies mean a parser that recovers is not wrong. Only must-fail says failure is the required behaviour, so only must-fail belongs in a test that asserts your code refuses the file. For the other two, assert that your code does not crash, lose data silently or hang, not that it fails.

1.3.2 contains_text (what a correct reader must find)

Three to five strings each file is known to contain, so a parser test can check that a file loaded correctly and not only that it loaded. It is deliberately containment rather than equality: readers differ in whitespace, line breaks and the order of blocks, so the text one reader extracts is right for that reader alone, and no extracted text is published. Each string is in the text a correct reader returns once every run of whitespace is collapsed to one space; nothing here says what else that text holds or in what order.

1.4 JSON Schema

src/loremfile/schema/manifest-v1.json (deployed at /schema/manifest-v1.json). Draft 2020-12. For status: active the required fields are exactly those in §1.2 except notes; additionalProperties: false so drift is caught. For status: removed the entry is a tombstone (§1.5) validated by an if/then branch. CI validates the manifest on every run (tests/unit/test_manifest.py). The schema document is itself immutable once published; a new schema is manifest-v2.json and bumps schema_version.

1.5 Tombstone entry (status: removed)

{ "path": "html/example.html", "format": "html", "ext": "html", "mime": "text/html; charset=utf-8",
  "bytes": 1234, "sha256": "…", "status": "removed", "removed_at": "2027-03-01T10:00:00Z",
  "reason": "Legal request #12 (summary)", "added_in": "1.2.0", "description": "…", "phase": 1 }

Exactly these fields, nothing else; url is omitted so clients that iterate url skip it. sha256/bytes stay so archives can still be verified. sha256sums.txt, formats.json, {format}/index.json and llms-full.txt list active fixtures only; the site shows a tombstone line on the format page.

2. sha256sums.txt

GNU coreutils format, one line per active fixture, sorted by path:

9f86d0818…a08  pdf/a4-3pages.pdf

Two spaces between hash and path, LF line endings, UTF-8, no BOM, trailing newline. Generated from the manifest; never hand-edited.

3. formats.json

{ "catalog_version": "1.0.0", "formats": [ { "format": "pdf", "count": 14, "total_bytes": 23456789, "url": "https://loremfile.dev/pdf", "index": "https://loremfile.dev/pdf/index.json", "mime_default": "application/pdf" } ] }

4. {format}/index.json

Same shape as manifest.json but fixtures filtered to one format (active only), count and total_bytes are the per-format values, and there is no top-level formats array. Stored as _formats/{format}.json and served at this URL by a rewrite, because {format}/ is a locked prefix (ADR-032).

5. llms.txt and llms-full.txt

Follow the llmstxt.org layout (H1 title, blockquote summary, H2 sections with link lists).

llms.txt (short, ≈ 2 KB):

# loremfile.dev

> Free, CC0, hotlink-friendly sample files and test fixtures for every common file type. Stable URLs, open CORS, byte-range support, SHA-256 manifest. No ads, no signup, no rate-limit keys.

## How to use
- Every fixture: {format}/{name} under the site root, for example https://loremfile.dev/pdf/a4-1page.pdf (the manifest lists every path)
- Machine-readable list of everything: https://loremfile.dev/manifest.json (JSON, includes sha256, bytes, mime, measured properties)
- Per-format lists: {format}/index.json, for example https://loremfile.dev/pdf/index.json
- Any format or guide page as Markdown: append /index.md to its URL, for example https://loremfile.dev/pdf/index.md or https://loremfile.dev/testing/uploads/index.md
- Verify: https://loremfile.dev/sha256sums.txt
- Verify from a page alone: the JSON-LD Dataset on every format and guide page gives each file's contentSize and sha256, so a download can be checked without the manifest
- MCP server for agents: `npx -y loremfile mcp` or `uvx loremfile mcp`, three read-only tools (list_fixtures, describe_fixture, verify_file): https://loremfile.dev/docs/mcp

## Formats
- [PDF](https://loremfile.dev/pdf): A4/Letter, 1–100 pages, images, tables, links, outline, encrypted, exact-ish sizes 1 MB / 10 MB
- [Images](https://loremfile.dev/png): png, jpg, gif, webp, avif, svg, bmp, tiff, ico …
...

## Testing guides
- [Sample files for testing file uploads](https://loremfile.dev/testing/uploads): one line on what it covers
...

## Rules
- Query strings are ignored. Stay under 30 requests/second per IP.
- License: CC0 1.0 for all fixtures. Source: https://github.com/kumarprabhashanand/loremfile

llms-full.txt (≈ 60–100 KB): the same header, then for each format a table of every fixture: path, bytes, one-line description, key props. Regenerated from the manifest on every deploy.

Neither file ever lists /legal/imprint or /legal/privacy (ADR-016 amendment, ADR-028). Neither lists legal pages today; this is a rule the builder must keep, not a removal.

5b. Markdown twins (/{x}/index.md)

Each format page and each of the three testing guides has a Markdown twin at its URL plus /index.md: /pdf/index.md, /testing/uploads/index.md. A twin is the page for a reader that wants text rather than HTML: the H1, the description, a line linking the page, then the page's prose with each <!-- file: PATH --> marker expanded into a table of path, size, media type and measured props (consecutive markers share one table). A format page's twin adds the full file table, any superseded paths with their replacements, and the URL of the same list as JSON. Every number is the manifest's, as on the HTML page, and a marker the page may not list stops the build.

6. robots.txt

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: OAI-AdsBot
User-agent: ClaudeBot
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: CCBot
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Bytespider
User-agent: meta-externalagent
User-agent: meta-externalfetcher
User-agent: Amazonbot
User-agent: Diffbot
User-agent: MistralAI-User
User-agent: DuckAssistBot
User-agent: AgentDataBot
User-agent: BixelBot
User-agent: CloudflareBrowserRenderingCrawler
User-agent: Kimi-Agent
User-agent: qodercli
User-agent: KeenableBot
User-agent: Keenable-User
User-agent: GrokBot
User-agent: GrokAppAndroid
User-agent: Hunyuan
User-agent: MoonshotBot
User-agent: xAI-Grok
User-agent: YiBot
User-agent: KimiBot
User-agent: Kimi-SearchBot
User-agent: Kimi-User
User-agent: CledaraBot
User-agent: Codex
User-agent: ElevenlabsBot
User-agent: ExaUser
User-agent: HarkBot
User-agent: Instinct
User-agent: Muse
User-agent: ElevenlabsCrawler
User-agent: ScryBot
User-agent: ExaSearchBot
User-agent: AIWebIndex
User-agent: Google-Extended
User-agent: Applebot-Extended
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Disallow: /legal/imprint
Disallow: /legal/privacy

Sitemap: https://loremfile.dev/sitemap.xml

AI crawlers are deliberately allowed (the audience includes agents). Raw fixtures carry X-Robots-Tag: noindex; pages are indexable, except the two legal pages that name the operator (ADR-016 amendment, ADR-028).

7. sitemap.xml

The sitemap lists only pages meant to be indexed (2026-10-16). /legal, /legal/license and /legal/terms stay indexable and linked from every footer, but the sitemap leaves them out (NOT_IN_SITEMAP in site/build.py). The reason is measurement, not tidiness: indexed-page count is the RISK-18 tripwire (17), and a sitemap that asks for pages nobody wants in search makes that count wrong by construction. The two operator pages were never listed; they are noindex.

Every discovery file carries X-Robots-Tag: noindex, and that is not a bug to fix. The files_headers rule matches any path containing a dot, so sitemap.xml, robots.txt, llms.txt, manifest.json, search-index.json, sha256sums.txt, formats.json and security.txt all receive it alongside the fixtures. It looks alarming next to a Search Console error and it is the wrong suspect: Google's own sitemap troubleshooting lists robots.txt blocking, manual actions, 404s, server errors and low crawl demand as the causes of "Couldn't fetch": noindex is not among them, and a noindex sitemap is still fetched and read.

The observation that prompted this (2026-09-17). A Search Console submission reported "Couldn't fetch" on a property verified minutes earlier, while the file served 200 to Googlebot with 92 URLs. Decision: no header change; re-check after a few days, because a just-verified property commonly reports that and resolves itself. This is written down so the rule is not "fixed" later on a theory the documentation does not support: changing files_headers to carve out the discovery files would alter the headers on every fixture path, which is a real risk taken against an imagined cause.

The decision stands after 2026-09-21. Four display assets (favicon.ico, apple-touch-icon.png, /assets/og.png, /assets/mark.svg) were taken out of the noindex rule that day (08 §5.3), because they exist to be shown by search engines and social platforms. That is not precedent for the discovery files: a sitemap does not need to be indexable to be fetched, and nothing in Google's documentation says otherwise.

Lists every page except the five under /legal (above). Does not list raw fixtures. lastmod is the day the page's content last changed, from page-dates.json (below), never the build's date or the clock. Under 50 MB / 50,000 URLs, so a single file suffices.

page-dates.json (2026-10-21). Until then every lastmod, and the dateModified of every format page and testing guide (§10), was the build's commit date, so the first deploy of each day rewrote every one of those pages with no content change (83 of 98 on 2026-10-17) and IndexNow announced them all (§12). The committed page-dates.json maps each sitemap page to the date its content last changed and the sha256 of the page as rendered with no date at all: hashed with its date, a page would change whenever its date did, and its date would move on every build. A page whose hash is unchanged keeps its date; one whose hash changed takes the committer date of HEAD (07 §4), never earlier than its recorded date. site build refuses a stale record and loremfile site build --update-dates rewrites it; commit what it writes, like manifest.json, and never edit it by hand. Content is everything rendered into the page, so a template or stylesheet change that alters every page dates every page. The record was seeded at 2026-10-17, the date every page already carried and the last day any rendered content had changed, so a seeded date is each page's latest possible date rather than its true one. The legal pages are not in it; they are dated from the history of 13.

8. .well-known/security.txt (RFC 9116)

Contact: https://github.com/kumarprabhashanand/loremfile/security/advisories/new
Contact: mailto:security@loremfile.dev
Expires: <one year from deploy, RFC 3339>
Preferred-Languages: en
Canonical: https://loremfile.dev/.well-known/security.txt
Policy: https://loremfile.dev/docs/security-policy

Expires is regenerated on every site build (committer date of the built commit + 365 days); the daily health check warns 30 days before expiry, which can only happen if no commit was deployed for 11 months.

9. schema/manifest-v1.json

Deployed copy of the JSON Schema (§1.4). application/schema+json, immutable cache headers.

10. Structured data on pages

JSON-LD per page type (single source of truth; 07 §3 refers here): home → WebSite + SoftwareSourceCode (pointing at the repository); format page → Dataset (name, description, license: CC0, isAccessibleForFree: true, dateModified (the page's date in page-dates.json, which the sitemap gives as lastmod, §7), and distribution with contentUrl, encodingFormat, contentSize (the manifest's byte count) and sha256 per active fixture) + BreadcrumbList, and <link rel="alternate"> to its index.json (application/json) and its Markdown twin (text/markdown, §5b); a testing guide carries the Markdown one; testing guide → the same Dataset, its distribution being exactly the files the guide names, + BreadcrumbList; /formats → CollectionPage whose mainEntity is an ItemList of every format page, in the table's order, + BreadcrumbList; docs, legal, the testing index and /changelog → TechArticle + BreadcrumbList, except /legal/imprint and /legal/privacy, which carry no JSON-LD at all (ADR-028). No Organization, creator, publisher or author node anywhere, and a test fails on one.

11. Agent discovery

Four static standards, checked by Cloudflare's agent-readiness scanner. None of them lists or links either legal page, and site build refuses one that does (site/checks.py).

12. IndexNow

Each deploy tells IndexNow which site pages it changed: one POST https://api.indexnow.org/indexnow with host, key, keyLocation and urlList, after verify-live has seen the pages served (09 §3.2). src/loremfile/infra/indexnow.py; the command is loremfile indexnow.