On April 2nd, Andrej Karpathy posted on GitHub and X about building a Persistent LLM Wiki – a structured knowledge base that compounds over time, where every paper you read, every idea you encounter, gets added into an interconnected graph of notes. The tool he reached for was Obsidian.
The idea resonated immediately. Obsidian’s graph view, its bidirectional links, and its multiple plugins is already a favourite among researchers and developers building so-called ‘second brains’. Adding LLM-powered addition and sorting seemed like a natural next step.
As it happens, I had been trying to manage my ‘stuff’ for a while before Andrej’s post, but hadn’t cracked it. I have collected hundreds of academic papers across multiple research programs over decades – BA Maths & Philosophy, MSC Consulting and Coaching for Change, and MA Photography – I know, a bit eclectic but there are threads and I want to find them. I am Visiting Professor at the University of Leeds Interdisciplinary Ethics Applied (IDEA), and the clue to my current interests is in the centre’s name.
The problem with unfiltered addition
If you point an LLM at a folder of research papers and say ‘add these into my Wiki’, you get something that looks impressively comprehensive. Pages get created. Links get drawn. Summaries appear. Your graph view fills up.
But not everything you read deserves equal standing in your knowledge base. A Nature paper with rigorous methodology, strong empirical grounding, and independently verified results is not the same thing as a Substack thought experiment. A position paper from a consultancy with a commercial stake in the outcome is not the same as a peer-reviewed philosophical critique. An AI model evaluating its own capabilities has a fundamentally different epistemic status than an external audit. Marking your own homework doesn’t count!
Without quality control, your Wiki becomes a flat archive. Everything is linked to everything, and there’s no way to distinguish what you can build on from what you should treat with caution. The links create an illusion of equal authority. Your ‘second brain’ has no immune system.
Adding a pedigree matrix
The solution I arrived at borrows from the NUSAP framework developed by Silvio Funtowicz and Jerry Ravetz for assessing the quality of uncertain knowledge. Disclaimer – Jerry was my tutor for my undergrad degree over 50 years ago, and we are still working together on AT & Ethics issues (he is 96 years young). The core insight of NUSAP is that numbers and claims always carry a ‘pedigree’, a track record of how they were produced, by whom, under what constraints, and with what degree of confidence. In situations where facts are uncertain, values are in dispute, and the stakes are high (which describes approximately everything in AI right now), pedigree assessment is not optional. It should be a primary form of quality control.
My wiki uses a five column pedigree matrix applied to every source at ingestion:
- Provenance: Where did this come from? A peer-reviewed journal scores higher than a Substack post. An independent lab scores higher than a company evaluating its own product.
- Rigour: How sound is the methodology? This is where I apply what I call Rule ∆1: the instrument must remain separate from the object of measurement. When an AI system evaluates AI capabilities, or when a consultancy surveys its own clients about the value of consulting, that separation breaks down.
- Coherence: Does the argument architecture serve the epistemic purpose? Perhaps the argument holds but relies on unstated assumptions worth flagging. Perhaps the format partially undermines the argument – think of a slick industry report that suggests more rigour than the methodology actually delivers.: How well is this anchored in evidence? Is it pure conceptual construction. Does it have strong empirical validation with independent replication.
- Relevance Anchors: A vocabulary of 16 topic codes (AI, Philosophy, Foresight, Leadership, and so on) that map each source to its domains. A star suffix marks foundational sources in that domain.
- Claim Differentiation: Are the claims well-matched to the evidence? Does it overclaim, does the paper says more than the data strictly supports? Maybe the evidence is stronger than the stated conclusions. Perhaps the paper conflates issues and terms?
This entire assessment gets encoded in YAML frontmatter on the Obsidian page. Every source in the Wiki carries its pedigree visibly, it is queryable, and whilst it is permanent it can easily be updated with new insights or data. And cross referencing is easy.
What this looks like in practice
Here is a real example. I ingested (that’s the Obsidian term, not part of my diet) a Nature paper – Lu et al (2026), Towards end-to-end automation of AI research – alongside the viral Citrini Substack piece The 2028 Global Intelligence Crisis
The Nature paper presents The AI Scientist, an automated pipeline that generates research ideas, runs experiments, writes papers, and conducts peer review. One AI-generated paper was accepted at an ICLR workshop. The paper is published in Nature (excellent Provenance), the experimental methodology is Rigorous if narrow, the argument structure is Coherent, and the Grounding is real but limited – one workshop acceptance is thin evidence for ‘automating AI research’ in general. Claim Differentiation is strong. The Wiki note says: single ICLR workshop acceptance extrapolated to prospects for automating AI research broadly.
The Citrini piece is a thought experiment, a fictitious ‘macro memo from June 2028’ modelling what happens if AI-driven job displacement goes systemic. It is brilliant scenario construction. But it carries no empirical validation and cannot, by design, so grounding is poor. The Provenance is low (Substack, no peer review). The Rigour is reasonable for the genre but there is no formal scenario methodology, no alternative pathways, no probability weighting. And the Claim Differentiation is good: despite an explicit ‘this is not a prediction’ disclaimer, the fictitious-history format makes the scenario read as inevitable. I wrote a blog post about exactly this.
Both sources are in the Obsidian Wiki. Both are linked to relevant concept pages and other authors and entities. But their pedigree scores make their epistemic status immediately legible. When I am building an argument about AI’s economic impact, I know which one I can lean on for methodology and which one I should treat as a provocative thought-piece.
I should add that the Wiki is set up to pull together threads of related work, investigate novel, interdisciplinary ideas, identify open questions and critical concepts, and generally help me explore action of ideas and cross-cutting themes. The Wiki doesn’t think for me, but it certainly helps me think.
The workflow: Claude Cowork as ingestion engine
The operational workflow is straightforward. My research library lives in Dropbox — about 900 sources across 33 subfolders. The Wiki lives in a secure Obsidian Vault on my MacBook. Claude Cowork handles the ingestion, following a structured skill protocol that reads the source, applies the pedigree assessment, looks up DOIs and weblinks, extracts or constructs the abstract, identifies key claims, maps connections to existing Wiki pages, and writes the source page with full YAML frontmatter.
A typical morning session: I point Claude at the master Research & AI folder in Dropbox. It scans for new files not yet in the Wiki. It proposes a prioritised ingestion list. I approve. It reads each source, creates the pages, cross-references them with existing concept and entity pages, updates the master index, and writes a session log. Ten sources painlessly ingested in a single session, each with full pedigree assessment, each queryable.
The key design decision is that Claude does the assessment, but I set the criteria and retain veto. The pedigree matrix is not automated, it is assisted. Claude applies the scoring framework, but the framework itself encodes methodological commitments I have developed over the years: the insistence on instrument-object separation, the attention to funding and institutional incentives, the scepticism toward claims that exceed their evidential base. The LLM is the engine; the epistemology is mine.
Why this matters beyond my wiki
If you are building knowledge infrastructure with LLMs, whether that is a personal research wiki, a corporate knowledge base, or an AI-assisted literature review pipeline, then the unfiltered ingestion problem will find you. RAG systems that treat all retrieved documents as equally authoritative will make confident claims built on shaky foundations. Knowledge graphs that do not encode source quality will propagate the weakest link as if it were the strongest.
The specific framework I use (NUSAP, five-column pedigree, Rule ∆1) is tuned to my personal practice. You do not need to adopt it. But the principle generalises: every source in your knowledge base should carry a visible, queryable quality assessment. Not a binary “peer-reviewed / not peer-reviewed” flag, but a multidimensional profile that captures provenance, methodology, evidence strength, and the relationship between claims and evidence.
Your ‘second brain’ needs an immune system. Build one!