loremfile

Manifest and discovery files

1. manifest.json (stable, schema-versioned)

Location: repository root (committed) and https://loremfile.dev/manifest.json (deployed copy, identical bytes).

1.1 Top-level object

{
  "$schema": "https://loremfile.dev/schema/manifest-v1.json",
  "schema_version": 1,
  "catalog_version": "1.0.0",
  "generated_at": "2026-10-01T12:00:00Z",
  "base_url": "https://loremfile.dev/",
  "license": "CC0-1.0",
  "license_url": "https://creativecommons.org/publicdomain/zero/1.0/",
  "source": "https://github.com/kumarprabhashanand/loremfile",
  "toolchain_image": "ghcr.io/kumarprabhashanand/loremfile-toolchain@sha256:<digest>",
  "count": 416,
  "total_bytes": 1060000000,
  "formats": ["avif", "bin", "csv", "..."],
  "fixtures": [ { "...": "see 1.2" } ]
}

1.2 Fixture entry

{
  "path": "pdf/a4-3pages.pdf",
  "url": "https://loremfile.dev/pdf/a4-3pages.pdf",
  "format": "pdf",
  "ext": "pdf",
  "mime": "application/pdf",
  "bytes": 12345,
  "sha256": "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08",
  "size_class": "free",
  "phase": 1,
  "description": "A4 portrait, 3 pages of Lorem Ipsum body text with page numbers. Helvetica core font, no embedded fonts.",
  "tags": ["document", "multi-page", "text"],
  "edge_case": false,
  "props": { "pages": 3, "page_width_pt": 595.28, "page_height_pt": 841.89, "encrypted": false, "pdf_version": "1.7" },
  "generator": "pdf.basic",
  "generator_params": { "pages": 3, "page_size": "A4", "orientation": "portrait", "page_numbers": true },
  "added_in": "1.0.0",
  "status": "active",
  "deprecated": false,
  "supersededBy": null,
  "notes": null
}

Field rules:

Field Type Rule
path string Unique. Matches URL grammar in 03-http-contract.md §1. Does not change.
url string base_url + path.
format string First path segment.
ext string Everything after the first dot of the last path segment (tar.gz, not gz; m3u8 for hls/720p-10s/index.m3u8).
mime string Exact Content-Type served. From the format's default in the catalog unless overridden per fixture (e.g. charset).
bytes integer Exact size. Frozen.
sha256 hex string (64) Frozen.
size_class enum exact (bytes equal nominal), boundary (nominal ± 1 exactly), approx (within ±5 % of nominal), free (no size claim in name).
phase integer 1–4.
description string One or two sentences a human or agent can use to decide fit. ≤ 300 chars.
tags string[] Lowercase; controlled vocabulary in catalog/_tags.yaml.
edge_case boolean true for anything under edge/ and for format-dir fixtures flagged as unusual (e.g. txt/control-characters.txt).
props object Measured by validators, format-specific (see §1.3). The measured values cannot change, because the bytes they were measured on cannot: sha256 is frozen, so pages or rows is settled for good. What a prop is called, and the wording of one that is a sentence, may be corrected — a correction is a changelog entry and a new catalog version, never a silent rewrite.
generator, generator_params string, object Enough to regenerate. Informational; may change if a generator is refactored (bytes must not).
added_in semver Catalog version that introduced it.
status enum active, removed (legal takedown tombstone, see §1.5).
deprecated, supersededBy boolean, string/null See immutability policy.

1.3 props by format (what validators measure)

Formats Keys
pdf pages, page_width_pt, page_height_pt, encrypted, pdf_version, has_outline, has_images, and for a PDF with an AcroForm only, form_fields and filled_fields (a filled field must carry its appearance stream)
png, jpg, gif, webp, avif, bmp, tiff, ico width, height, mode (RGB, RGBA, L, P, CMYK, I;16), frames, animated, progressive, exif_orientation (jpg), sizes (ico)
heic width and height (the primary item's ispe, which heif-convert's decode must match), mode (decoded), major_brand, compatible_brands, primary_item_type (hvc1), items, format
svg width, height, viewbox
mp4, webm, mkv, mov, avi, ogv, ts, hls duration_ms, width, height, fps, vcodec, acodec, audio_streams, subtitle_streams, rotation, bitrate_kbps (hls: segments, target_duration_s)
mp3, wav, flac, ogg, opus, m4a, aac, aiff duration_ms, sample_rate, channels, bit_depth (pcm), bitrate_kbps, codec, tags (bool)
docx paragraphs, tables, images, sections, has_headers_footers
xlsx sheets, sheet_names, rows and columns (first sheet, rows incl. header), formulas (every sheet; each must carry a cached value), images, and cross_sheet_formulas only where a formula reads another sheet
pptx slides, has_notes, aspect (16:9/4:3)
txt, md, html, css, js, rtf, srt, vtt, log, sql, ini encoding, bom, line_ending (lf/crlf/cr/mixed/none), lines, max_line_bytes
csv, tsv plus rows (data rows), columns, delimiter, has_header, quoted_fields, and the two causes of quoting reported apart: quoted_delimiter (a field holds the delimiter) and quoted_newline (a field holds a line break)
json, ndjson, geojson, har, ipynb, jsonld top_type (object/array/string…), items (array length or ndjson lines), max_depth
xml, rss, atom, kml, gpx root_element, encoding, has_dtd, namespaces
yaml, toml documents (yaml), top_keys
parquet, avro, arrow rows, columns, compression (parquet), row_groups
sqlite tables, rows_total, page_size, journal_mode, has_fts
zip, 7z entries, directories, method (stored/deflate/lzma2), encrypted, zip64, comment
tar (+gz/bz2/xz/zst) entries, compression, has_symlinks, pax
gz, bz2, xz, zst members (gz), original_bytes
ttf, otf, woff, woff2 glyphs, flavor, family_name, units_per_em
bin pattern (shake256, zeros, ones, incrementing)
ics events, has_vtimezone, has_rrule
vcf cards, version
eml, mbox messages, parts, attachments, has_html
epub version, chapters, images
pem, der kind (certificate/csr), algorithm, not_before, not_after
wasm sections, exports
edge/* defect, intended_format, bytes, damage, outcome, compare_with, and derived_from for the six that were cut or copied from a published fixture; plus fraction (truncated) and hostile_entries (hostile-name). §1.3.1 describes the four that are about the breakage. defect is a closed enum, and it tells the validator what to assert: zero-byte (bytes == 0), truncated (bytes == floor(fraction × source bytes) with fraction from the catalog's edge block, default 0.5 — pdf-truncated-60pct.pdf uses 0.6 — and prefix-equal to the source), mismatched-extension (bytes equal the source fixture; magic bytes match the source format, not the extension), magic-prefix (bytes == the magic value from the edge block followed by the source fixture's bytes), invalid-syntax (the format's parser raises), invalid-encoding (strict decode with the declared charset raises), nonstandard (strict parser raises, lenient parser succeeds), bom (decodes after stripping the BOM; parser succeeds on the stripped text), stress (parses; the recorded props document the depth/size), hostile-name (parses; at least one entry name contains ../), polyglot (both the announced format's reader and the one named by edge.also_valid_as accept the bytes; the prop also_valid_as records the second format). Only fixtures under edge/ carry these; unusual-but-valid fixtures elsewhere (txt/control-characters.txt, txt/nul-bytes.txt) have edge_case: true and ordinary props

1.3.1 The four edge props (what is wrong, and what to expect)

A broken file is only useful if you know how it is broken. Four props say so, and none of them is written by hand — loremfile validate measures or derives each one from the bytes and refuses the fixture when the answer disagrees with the catalog.

Prop What it is
damage One sentence about the bytes: where the file stops, which structure is missing, where a parser gives up. Composed from what the validator measured, so every number in it was counted on the file it describes.
outcome must-fail, may-recover or varies — see below.
compare_with The valid fixture of the same announced type. Feed your reader both: a reader that refuses everything is not the same as one that refuses this.
derived_from The published fixture this one was cut or copied from, present on the six that have one and absent on the other thirteen. The validator reads that file and checks the relationship: a truncation's length must equal the recorded fraction of it and be a prefix of it, and a mislabelled file must equal it byte for byte.

outcome describes the file, not your reader. It is derived by offering the bytes to the reader for the type the file announces and, when that refuses them, to a tolerant reading the format itself permits:

may-recover and varies mean a parser that recovers is not wrong. Only must-fail says failure is the required behaviour, so only must-fail belongs in a test that asserts your code refuses the file. For the other two, assert that your code does not crash, lose data silently or hang — not that it fails.

1.4 JSON Schema

src/loremfile/schema/manifest-v1.json (deployed at /schema/manifest-v1.json). Draft 2020-12. For status: active the required fields are exactly those in §1.2 except notes; additionalProperties: false so drift is caught. For status: removed the entry is a tombstone (§1.5) validated by an if/then branch. CI validates the manifest on every run (tests/unit/test_manifest.py). The schema document is itself immutable once published; a new schema is manifest-v2.json and bumps schema_version.

1.5 Tombstone entry (status: removed)

{ "path": "html/example.html", "format": "html", "ext": "html", "mime": "text/html; charset=utf-8",
  "bytes": 1234, "sha256": "…", "status": "removed", "removed_at": "2027-03-01T10:00:00Z",
  "reason": "Legal request #12 (summary)", "added_in": "1.2.0", "description": "…", "phase": 1 }

Exactly these fields, nothing else; url is omitted so clients that iterate url skip it. sha256/bytes stay so archives can still be verified. sha256sums.txt, formats.json, {format}/index.json and llms-full.txt list active fixtures only; the site shows a tombstone line on the format page.

2. sha256sums.txt

GNU coreutils format, one line per active fixture, sorted by path:

9f86d0818…a08  pdf/a4-3pages.pdf

Two spaces between hash and path, LF line endings, UTF-8, no BOM, trailing newline. Generated from the manifest; never hand-edited.

3. formats.json

{ "catalog_version": "1.0.0", "formats": [ { "format": "pdf", "count": 14, "total_bytes": 23456789, "url": "https://loremfile.dev/pdf", "index": "https://loremfile.dev/pdf/index.json", "mime_default": "application/pdf" } ] }

4. {format}/index.json

Same shape as manifest.json but fixtures filtered to one format (active only), count and total_bytes are the per-format values, and there is no top-level formats array. Stored as _formats/{format}.json and served at this URL by a rewrite, because {format}/ is a locked prefix (ADR-032).

5. llms.txt and llms-full.txt

Follow the llmstxt.org layout (H1 title, blockquote summary, H2 sections with link lists).

llms.txt (short, ≈ 2 KB):

# loremfile.dev

> Free, CC0, hotlink-friendly sample files and test fixtures for every common file type. Stable URLs, open CORS, byte-range support, SHA-256 manifest. No ads, no signup, no rate-limit keys.

## How to use
- Every fixture: https://loremfile.dev/{format}/{name} — see the manifest for all paths.
- Machine-readable list of everything: https://loremfile.dev/manifest.json (JSON, includes sha256, bytes, mime, measured properties)
- Per-format lists: https://loremfile.dev/{format}/index.json
- Verify: https://loremfile.dev/sha256sums.txt

## Formats
- [PDF](https://loremfile.dev/pdf): A4/Letter, 1–100 pages, images, tables, links, outline, encrypted, exact-ish sizes 1 MB / 10 MB
- [Images](https://loremfile.dev/png): png, jpg, gif, webp, avif, svg, bmp, tiff, ico …
...

## Rules
- Query strings are ignored. Stay under 30 requests/second per IP.
- License: CC0 1.0 for all fixtures. Source: https://github.com/kumarprabhashanand/loremfile

llms-full.txt (≈ 60–100 KB): the same header, then for each format a table of every fixture: path, bytes, one-line description, key props. Regenerated from the manifest on every deploy.

Neither file ever lists /legal/imprint or /legal/privacy (ADR-016 amendment, ADR-028). Neither lists legal pages today; this is a rule the builder must keep, not a removal.

6. robots.txt

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: OAI-AdsBot
User-agent: ClaudeBot
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: CCBot
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Bytespider
User-agent: meta-externalagent
User-agent: meta-externalfetcher
User-agent: Amazonbot
User-agent: Diffbot
User-agent: MistralAI-User
User-agent: DuckAssistBot
User-agent: AgentDataBot
User-agent: BixelBot
User-agent: CloudflareBrowserRenderingCrawler
User-agent: Kimi-Agent
User-agent: qodercli
User-agent: KeenableBot
User-agent: Keenable-User
User-agent: Google-Extended
User-agent: Applebot-Extended
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Disallow: /legal/imprint
Disallow: /legal/privacy

Sitemap: https://loremfile.dev/sitemap.xml

AI crawlers are deliberately allowed (the audience includes agents). Raw fixtures carry X-Robots-Tag: noindex; pages are indexable — except the two legal pages that name the operator (ADR-016 amendment, ADR-028).

7. sitemap.xml

Every discovery file carries X-Robots-Tag: noindex, and that is not a bug to fix. The files_headers rule matches any path containing a dot, so sitemap.xml, robots.txt, llms.txt, manifest.json, search-index.json, sha256sums.txt, formats.json and security.txt all receive it alongside the fixtures. It looks alarming next to a Search Console error and it is the wrong suspect: Google's own sitemap troubleshooting lists robots.txt blocking, manual actions, 404s, server errors and low crawl demand as the causes of "Couldn't fetch" — noindex is not among them, and a noindex sitemap is still fetched and read.

The observation that prompted this (2026-09-17). A Search Console submission reported "Couldn't fetch" on a property verified minutes earlier, while the file served 200 to Googlebot with 92 URLs. Decision: no header change; re-check after a few days, because a just-verified property commonly reports that and resolves itself. This is written down so the rule is not "fixed" later on a theory the documentation does not support — changing files_headers to carve out the discovery files would alter the headers on every fixture path, which is a real risk taken against an imagined cause.

The decision stands after 2026-09-21. Four display assets — favicon.ico, apple-touch-icon.png, /assets/og.png, /assets/mark.svg — were taken out of the noindex rule that day (08 §5.3), because they exist to be shown by search engines and social platforms. That is not precedent for the discovery files: a sitemap does not need to be indexable to be fetched, and nothing in Google's documentation says otherwise.

Lists the home page, every format page, docs pages, legal pages except /legal/imprint and /legal/privacy, and the changelog. Does not list raw fixtures. lastmod = the committer date of the built commit (07 §4), never the clock. Under 50 MB / 50,000 URLs, so a single file suffices.

8. .well-known/security.txt (RFC 9116)

Contact: https://github.com/kumarprabhashanand/loremfile/security/advisories/new
Contact: mailto:security@loremfile.dev
Expires: <one year from deploy, RFC 3339>
Preferred-Languages: en
Canonical: https://loremfile.dev/.well-known/security.txt
Policy: https://loremfile.dev/docs/security-policy

Expires is regenerated on every site build (committer date of the built commit + 365 days); the daily health check warns 30 days before expiry, which can only happen if no commit was deployed for 11 months.

9. schema/manifest-v1.json

Deployed copy of the JSON Schema (§1.4). application/schema+json, immutable cache headers.

10. Structured data on pages

JSON-LD per page type (single source of truth; 07 §3 refers here): home → WebSite + SoftwareSourceCode (pointing at the repository); format page → Dataset (name, description, license: CC0, distribution with contentUrl and encodingFormat per active fixture, isAccessibleForFree: true) + BreadcrumbList; docs and legal pages → TechArticle + BreadcrumbList, except /legal/imprint and /legal/privacy, which carry no JSON-LD at all (ADR-028). No Organization node.

11. Agent discovery

Four static standards, checked by Cloudflare's agent-readiness scanner. None of them lists or links either legal page, and site build refuses one that does (site/checks.py).

12. IndexNow

Each deploy tells IndexNow which site pages it changed: one POST https://api.indexnow.org/indexnow with host, key, keyLocation and urlList, after verify-live has seen the pages served (09 §3.2). src/loremfile/infra/indexnow.py; the command is loremfile indexnow.