Project: Build an Auditable Entity and Relation Extractor

Ship Evidence with Every Extracted Entity and Relation

• Design a JSON schema for evidence, labels, normalised values and rule or model versions.
• Create a small reviewed evaluation set with both positive and negative documents.
• Reconcile counts and analyse failures by extraction route.
• Reconcile extracted counts against reviewed documents and retain the evidence span for every accepted value.

Evidence-first records

Each record needs source ID, exact quote, offsets, extraction type and producer version. Optional normalisation belongs beside, not instead of, the quote.

Schema validation catches missing provenance before records reach analytics or search.

A confidence label needs meaning

Do not invent a decimal score for deterministic rules. Use categories such as rule_exact, model_reviewed or needs_review with documented behavior.

Evaluation mirrors the review task

Report exact entity spans, relation argument accuracy and false positives by rule ID. A single combined accuracy hides where the pipeline fails.

Sample no-result documents too. Silence can mean no relevant information or a broken extractor.

Package the handoff

Include install commands, model name, test data licence, expected output and a limitations note.

A portfolio reader should be able to reproduce one run and inspect one error.

Build the Project Build an Auditable Entity and Relation Extractor example

The final assertion verifies that offsets reconstruct the quote exactly. Keep this invariant after any text-storage or serialisation change.

The producer field distinguishes model output from deterministic rule output. Add dependency versions to the run manifest.

Install:

python -m pip install spacy jsonschema
python -m spacy download en_core_web_sm

import json, spacy
from spacy.matcher import Matcher
nlp = spacy.load('en_core_web_sm')
matcher = Matcher(nlp.vocab)
matcher.add('VERSION_MENTION', [[{'LOWER':'version'},{'TEXT':{'REGEX':r'^\d+(?:\.\d+){0,2}$'}}]])

def extract(doc_id, text):
    doc = nlp(text)
    records=[]
    for ent in doc.ents:
        if ent.label_ == 'ORG':
            records.append({'source_id':doc_id,'type':'organisation_mention','quote':ent.text,'start':ent.start_char,'end':ent.end_char,'producer':'en_core_web_sm'})
    for _,start,end in matcher(doc):
        span=doc[start:end]
        records.append({'source_id':doc_id,'type':'version_mention','quote':span.text,'start':span.start_char,'end':span.end_char,'producer':'rule_version_1'})
    return records

result=extract('doc-7','Acme announced version 2.4 today.')
print(json.dumps(result,indent=2))
assert all(r['quote']=='Acme announced version 2.4 today.'[r['start']:r['end']] for r in result)

Expected Extraction Audit Output

Two evidence records: an organisation mention for Acme and a version mention for version 2.4. Each contains source ID, exact quote, offsets and producer.

One sentence cannot validate the system. Build a held-out reviewed set and save errors as fixtures.

If personal or confidential documents are involved, replace the sample with synthetic text and design access controls before storing excerpts.

Diagnose failures in Project Build an Auditable Entity and Relation Extractor

Trace one output from JSON back to the exact source slice, then compare with the reviewed label.

Check your Project Build an Auditable Entity and Relation Extractor reasoning

Inspect the project decisions before revealing the explanations. Focus on provenance, relation evidence and reviewer correction paths.

Primary references for Project Build an Auditable Entity and Relation Extractor

• spaCy linguistic features: official tags, dependencies and entity guidance.
• spaCy rule-based matching: verify token and phrase patterns used in the auditable extractor.
• Hugging Face dataset cards: document annotation sources and relation-extraction constraints with the project.

Re-run the extraction checkpoint after model, matcher or relation rules change. Archive source spans, component versions and the reviewed evaluation set.