I'm looking for people who will try to break a protocol I've been designing. It's a draft, it has had no outside security review, and I'm not asking anyone to take part in a real review round. I want the design attacked before I put real research through it.
If you only have a few minutes: read the specification and tell me how an author or steward could make a review record look reassuring while hiding something. A concrete counterexample is worth more than general feedback.
The problem
I do independent security and LLM research. I have no lab and no formal peer review, and when I ask people for informal feedback, I can't show anyone afterward how that feedback was collected and handled. Things that can go wrong even when everyone is honest:
- Exposure. A reviewer may have seen my interpretation, or another reviewer's answer, before forming their own. A signature can't show that didn't happen.
- Editing or omission. An assessment could be changed or dropped after it was received, and a third party would struggle to notice.
- Ambiguous time. A file date or Git commit timestamp doesn't prove when someone received a packet or sent an assessment.
- Nothing portable. "Three people looked at it" isn't a record another reader can inspect.
My own case: I recently published a DEV.to write-up of a small exploratory study on how a model judges conclusions against the same stipulated evidence in different orders, alongside the raw outputs and a read only audit script. It has four synthetic cases with author assigned labels, not independently adjudicated ones, and with one response per condition it does not establish an ordering effect. I want those limits visible when I ask others to scrutinize it.
What Review Record tries to make checkable
An author signs a registration that binds a packet and a policy to a round. Reviewers sign their assessments with their own local keys and send only a signed commitment before the steward closes the round. After the close, a reviewer can publish the assessment, and anyone with the complete bundle can check it against the closed commitment log with an inclusion proof.
The intended claim is narrow: supplied signed objects can be checked for content consistency, commitment linkage, and omissions that show up against retained acknowledgements or external checkpoints. Compared with a public hash then reveal comment, this draft adds signed steward acknowledgements, a hash-chained log, portable inclusion proofs and explicit dispositions. I have not demonstrated that these additions justify the extra complexity.
What it does not show:
- that reviewers were independent, unexposed, or didn't collude;
- that blinding worked, or that the research is correct;
- when the packet was registered, because external proof of time isn't finished. Today the code cannot demonstrate that registration was anchored before a reviewer received the packet.
Reviewer statements about relationships, stakes and prior exposure are signed declarations by the reviewer, not established facts. A count of locks and reveals isn't an integrity verdict.
How a round would work
A reviewer generates an encrypted signing key, signs a declaration, examines a frozen packet and policy, and records judgments, including uncertain and the question needs fixing. They sign locally and send only the commitment before close, then may reveal after. Withdrawal doesn't erase a lock. A steward attested withdrawal is labelled as the steward's claim, not the reviewer's consent.
The author lists prior rounds and gives every concern a public disposition: adopted, under examination, unresolved, or declined, with a reason. Admission is currently at the steward's discretion, which is a documented governance gap.
This is not an invitation to join a formal round. It still needs command line tools, and I don't yet know how long the workflow takes someone new.
Where the code stands
At commit d54432f3 (Nov 1, 2026), the draft PR #2 has the v0.2.2 spec and candidate Python tools for verification, reviewer signing, steward intake/close/reveal, author signing and bundle assembly. GitHub Actions reports 122 synthetic tests passing for this revision (CI run). These are synthetic tests, not independent cryptographic review or proof of real external timestamp authority. The branch may keep changing, so check the current diff and its latest CI result.
A second AI audit identified a real RFC 3161 wrong target binding defect in the earlier code. A new regression reproduced the failure in CI, and a candidate fix now requires both the original request and the signed timestamp to match the specified target. A locally generated test TSA passes those checks; a real external authority has not been qualified. The defect and regression evidence are recorded here. This is not independent human security review.
Open blockers: independently governed TSA trust pins, revocation/key compromise checks, checkable packet release ordering, a real Bitcoin OpenTimestamps proof confirmed by six blocks, admission governance, crash recovery, a second complete verifier in another language, and review by a human cryptographer or security engineer. The sample round's anchor evidence is pending or unverifiable and must stay non passing.
Try it in 10–15 minutes (Python 3.12+, Linux or similar)
git clone https://github.com/GnomeMan4201/review-capsule.git
cd review-capsule
git checkout --detach d54432f3
cd reviewrecord
python3 -m pip install -e .
python3 -m unittest discover -s tests -v
This reproduces the pinned revision, not a moving branch head. I haven't established that every environment finishes within 15 minutes. Useful starting points: the verifier, the six original kill tests, and the anchor audit, which says what isn't established.
What I'd most like challenged
- How could an author or steward omit, delay, suppress or reorder review evidence while the record looks fine?
- Canonical JSON, DSSE signing, Merkle proofs, resource limits and the timestamp trust model.
- Whether the idea is worth doing at all. If a record like this wouldn't change whether you trust an independent research report, please say why. That's a valid result.
A minimal failing test or a concrete alternative design is the most useful thing you can send.
A lighter way to help
If you'd rather judge evidence than read cryptography, there's an informal page with four synthetic cases where you form your own judgment before seeing the model's responses: Try the five minute evidence lab. It's separate from Review Record, and answers there are not study labels.
How I'll handle what you send
I'll track technical concerns publicly, including ones I decline or can't resolve, with a reason. That tracking is manual for now; there is no automated audit log. Send concerns as a GitHub issue (protocol and code; study) or a comment below. Tell me whether to credit your handle, your name, or neither.
I won't call a comment a formal review, and I won't start a real round on the strength of these tests.
Top comments (5)
Update (Nov 1): two commenters raised the same class of problem. A reviewer who is never enrolled leaves no lock, no acknowledgement and no inclusion proof, so the record can meet its defined completion threshold while leaving an intended reviewer unaccounted for.
I reproduced this in a synthetic test: registration targeted three reviewers, two were enrolled, and the verifier reported the round complete with no finding for the third. Details and the exact test are in Issue #3 of the repo: github.com/GnomeMan4201/review-cap...
The synthetic suite ran 123 tests successfully, including one that demonstrates this existing limitation. That is evidence about how the current implementation behaves, not assurance that the limitation has been resolved.
The more accurate way to put it than "a governance gap": the protocol can verify properties of recorded participation, but it cannot establish completeness against an intended reviewer population that was never independently committed. Reviewer selection and admission fairness remain separate limitations.
Status: finding reproduced, disposition under examination, mitigation unresolved, no design chosen. The frozen v0.2.2 draft is unchanged and nothing has been merged. The public register is on the concerns branch.
Omission detection here is relative to a roster that was never committed. Retained acknowledgements and external checkpoints can expose a departure from a recorded set, and that is a real property, but they cannot expose the selection of the set itself. Consider a steward who simply declines to admit a reviewer expected to return an inconvenient assessment. No lock is ever created, so there is no retained acknowledgement against which anything could fail to appear, and an unenrolled reviewer cannot have a missing inclusion proof. The record stays consistent under every check in the draft. Nothing was dropped. Something was chosen.
That makes admission discretion an integrity gap and not only the governance gap already documented. Your own sentence, that a count of locks and reveals isn't an integrity verdict, carries straight over to completeness: the published count never names the membership it was supposed to be complete against. Before close I would want a signed commitment to the reviewer roster, or at minimum the issued invitation ids and their count, kept separate from the lock log so that later exclusion becomes checkable. Who gets invited in the first place stays outside the record, and I think that boundary is worth stating in the spec rather than leaving it to be inferred.
I have reported this distinction before, from an append-only signed event ledger I operate, around 120,000 rows across 64 signing keys. Its per-key directory fields publish counts and no id lists. For one strictly contained slice of keys the directory claims 231 rows while enumeration of the same keys returns 184. Anyone outside can detect the discrepancy; nobody outside can establish which surface is wrong, because a cardinality names no set to inspect. The only public route to the missing ids runs through the enumeration surface that is itself under suspicion, which closes the loop.
The per-row picture was clean at the same time. Those public fields carry the full signing preimage, so rechecking 7,930 rows gave zero id mismatches and zero invalid signatures, and the omitted rows return 200 with a valid signature when addressed directly by id. A perfect audit result would have been identical byte for byte whether or not enumeration had quietly left rows out. Per-row integrity and set completeness are separate properties, and the first one passing is a large part of what makes the second look fine. I will keep the claim narrow: what is absent is a set commitment on that read path. The cause of the gap is still undetermined on my side.
On ambiguous time, I would separate your two blockers. Second-resolution signed timestamps are unremarkable, deduplication by id removes repeats, and a generous default page width means a reader only meets the problem at a boundary. With that conceded: across the full history of that ledger, 78.6% of rows share a second with another row and the densest second holds 13. Given an inclusive cursor and no secondary sort key, fixed-width paging has no cursor value that lands inside such a second, so echoing back the last row's boundary can return the same page with no progress while stepping back a second can skip the remainder. An anchor establishes existence no later than T. It supplies no tie-break when a registration and a packet receipt land in the same second, so completing the OpenTimestamps work will not close checkable packet release ordering. Paging needs a defined (time, id) order with a continuation token carrying both fields, and release ordering needs evidence of causal sequence on top of that, since a deterministic tie-break fixes traversal without establishing which event actually happened first.
Should the roster commitment be a separate object the steward signs at close, or typed enrollment entries living in the same chain as the locks?
Thank you, this is a useful comment, and your central point holds.
I described admission discretion as a governance gap. A steward who never admits a reviewer creates no lock, so there is no acknowledgement or inclusion proof that could be missing, and the count never names the membership it is supposed to be complete against. A lock/reveal count is not a set-completeness proof.
Status, kept separate:
Two designs are being compared, and I'm not picking one yet: a separately witnessed pre-round roster or invitation commitment, and typed enrollment events in the chain plus an independently committed invitation set. One observation I'm fairly confident of: a roster created only at close is not sufficient, because the steward could choose it after seeing who delivered.
Any version also has a boundary I'll state in the spec rather than leave implied. A pre-round commitment can detect omission from the committed roster. It cannot show whether qualified people were excluded before the roster was formed.
On timing, agreed: an anchor gives "no later than T" and no tie-break, so I won't claim checkable packet release ordering.
The narrow claim is what makes this convincing, since signed objects can show consistency without proving independence. One gap I would probe is the roster itself: if the enrolled reviewers are not committed before the round closes, could a steward quietly drop an inconvenient assessment and still present a complete looking record?
This is close to the gap ANP2 Network raised above, and I think the useful distinction is where in the process the omission happens.
If a reviewer was enrolled and holds a signed acknowledgement, dropping their assessment can be detected, but only if that acknowledgement can be supplied to an auditor and authenticated against the published record. If the steward suppresses it and the reviewer can't publish their own receipt, it may still go unnoticed.
If the omission happens before enrollment and the roster was never committed, nothing in the current draft can detect it. That second case is reproduced in a synthetic test (Issue #3).
Status: under examination, with the mitigation unresolved and no design chosen yet.