Integrity & Checksums

Overview

DOCiD stores a SHA-256 fingerprint of the bytes of every file it hosts, so that anyone who downloads a research object can prove their copy is identical to the one registered against the DOCiD.

Checksums cover file content only. DOCiD metadata is intentionally mutable — a title can be corrected and a creator added without invalidating the fingerprint of the deposited file. The fingerprint answers one question: are these the same bytes?

Where checksums are stored

Seven columns carry fixity information. They exist identically on both object tables — publications_files (uploaded files) and publication_documents (documents and external references).

None of these columns carry an index, a constraint, or a server default. A row created before this subsystem existed has NULL throughout.

The checksum_status vocabulary

Five values are defined. Treat only verified as a trustworthy fingerprint.

How the digest is computed

Files are streamed from disk in 1 MiB chunks, so an arbitrarily large deposit never loads into memory in full. The function returns the hex digest and the byte count together, and the byte count recorded in file_size is the one observed during hashing rather than a separately reported value.

Only files this deployment actually hosts can be hashed. A stored URL is resolved to a path under the uploads directory, guarded against path traversal, and only when its host matches this deployment's configured base URLs. Anything else — an external video, a link to another repository — is recorded as external_not_supported. DOCiD never fetches remote bytes in order to fingerprint them.

Checksums are generated at deposit time, not only by backfill: the publish path, the edit path, and version creation all hash newly attached objects. The backfill script exists for rows predating the subsystem, defaults to a dry run, and on --recheck never silently overwrites an existing digest — a mismatch is recorded as an integrity event.

The verification endpoint

Two equivalent forms are exposed, both unauthenticated:

GET /api/v1/verify?identifier=20.500.14351/e7d1b5c9a30a45bc2858
GET /api/v1/verify/20.500.14351/e7d1b5c9a30a45bc2858

Prefer the query form when scripting: a DOCiD contains a slash, and the query parameter avoids any ambiguity about path segments.

A live response follows. It shows the two states an integrator must handle side by side — an uploaded file with a usable digest, and an external object that can never have one.

{
    "algorithm": "SHA-256",
    "docid": "20.500.14351/e7d1b5c9a30a45bc2858",
    "note": "Compare these stored fingerprints against your downloaded files (e.g. `shasum -a 256 <file>`). A match confirms the file is identical to the version registered in DOCiD.",
    "object_count": 2,
    "objects": [
        {
            "checksum": "a9cc46dc31ff8a39c9bfdc33f76f308a3cf0b190676630d79f0087c495d4eb8a",
            "checksum_algorithm": "SHA-256",
            "checksum_generated_at": "2026-07-08T13:54:01.095359",
            "checksum_last_checked_at": "2026-07-08T13:54:01.095359",
            "checksum_status": "verified",
            "file_name": "c6930b0a35d1_abstract_1_1.docx",
            "file_size": 20475,
            "handle_identifier": "20.500.14351/9ba16c85d34594d212f2",
            "id": 271,
            "object_class": "file",
            "title": "Publication title"
        },
        {
            "checksum": null,
            "checksum_algorithm": "SHA-256",
            "checksum_generated_at": null,
            "checksum_last_checked_at": "2026-07-08T13:54:01.104038",
            "checksum_status": "external_not_supported",
            "file_name": null,
            "file_size": null,
            "handle_identifier": "20.500.14351/a7b8e32199bcdbf899cf",
            "id": 190,
            "object_class": "document",
            "title": "vid title"
        }
    ],
    "title": "Collection title",
    "verifiable_object_count": 1
}

Drafts are stored in a separate table and are never reachable through this endpoint.

Verifying a downloaded file

$ shasum -a 256 abstract_1_1.docx
a9cc46dc31ff8a39c9bfdc33f76f308a3cf0b190676630d79f0087c495d4eb8a  abstract_1_1.docx

A match confirms the file is byte-identical to the version registered against the DOCiD. A mismatch means the copy differs — through truncation, transcoding, or modification — and should not be cited as the registered object.

Known gaps

Stated plainly so integrators are not surprised. These are current limitations, not design intentions.