[{"content":"I am a first-time mentor for the Fall 2026 SPAR cohort, a remote AI-safety research fellowship that pairs mentees with mentors working on AI-safety related projects on reducing x-risk or other related risks.\nDuring prep, one piece of advice I received was to interview far more people than I normally would and treat the process itself as a form of exploration. The stated goal was simply to get to know the actual person through real-time conversation and compare that with how they came across on paper.\nSo I interviewed 18 people across three or four mornings, expecting that speaking to more candidates would simply give me more info, make it easier to identify the “strongest” ones, and help me decide who to recruit as mentees.\nBut the more people I spoke to, the original mentee-selection problem started to fall apart. Like, I was no longer sure what the \u0026ldquo;strongest\u0026rdquo; candidate even meant.\nCandidates who looked similar on application could be very different once I actually talked to them, while people I might have automatically rejected based on their application could turn out to be much more interesting than I had ever expected.\nAfter enough interviews, I started to notice that the candidates could be roughly clustered into three fairly distinct types, which, for lack of better names, here I will call them: YOLOs, Moonlighters, and Wildcards.\nThese differences were less about raw ability and more about where they were, how much of their life they had already organised around tackling AI safety problems, and what kind of support they actually needed to keep making progress.\nThinking about these three groups quickly changed how I approached the mentee-selection problem, and I found myself increasingly asking: “For which candidate can my particular mentorship create the most additional value?”\nYOLOs These people are already deeply committed to AI safety. Many have spent at least the past 6 months working on AI safety full-time, often leaving a full-time job with no stable source of income.\nTheir goal is usually to contribute to tackling some of the most important AI safety problems they deeply care about, while finding a practical and sustainable path to keep doing so over the mid term (~ 6 months) and long term (~ 12 months).\nMany are already BlueDot course alumni or are otherwise connected to communities where it is relatively easy to find AI safety opportunities, collaborators or funding.\nI suspect that it is mainly due to a strong sense of commitment and urgency about the AI safety problems they care about. Plus or minus financial runways, they are be highly agentic and opportunistic.1\nThey actively do things. They are relentlessly resourceful. They can execute things fairly independently and quickly without much hand-holding. They actively seek out projects, collaborators, fellowships, and other opportunities that help them grow and contribute more effectively to the difficult AI safety problems they care about.\nI think their biggest advantage is the ability to devote their full time, attention, and focus to tackling the AI safety problems they care deeply about, independently and without needing external permission.\nBy being able to thinking and working on important AI safety problems full-time, they gain unusually deep exposure to the problems themselves, the people working on them, the institutions supporting that work, and the stakeholders affected by those problems.\nEach project write-up, application, produced artefact, collaboration and conversation gives them another chance to explore, learn, and make progress toward tackling the AI safety problems they care about.\nOften very quickly, they will be able to discover and develop their own flywheel of exploration and exploitation on tackling their target set of AI safety problems, at a pace that differs person to person.\nI think this flywheel applies regardless of which talent category (i.e., iterator vs connector vs amplifier) you place yourself in.\nMy reasoning is as follows:\nSuppose you already deeply care about a particular problem in AI safety and have convinced yourself to bet your (entire) career to work on it. The remaining question is how to make (and measure) progress on it.\nOne of the fastest ways to make any progress is to execute quickly, get your work out into the world, find allies and collaborators, and figure out how to get other relevant stakeholders to care about and engage with what you are doing.\nThis is not easy, and I often jokingly refer to this process as simply “banging your head against the problem” where you just repeatedly trying things out, learning from what happened to you and others (akin to doing mental updates), and continue pushing things forward until it clicks.\nThe AI safety problems we face today are difficult and poorly understood, with so many unknown unknowns. Any detailed plans or prior analysis depend on assumptions can fall apart very easily once they confront reality, with relatively short expiry dates if left unchecked and unrevised, especially given the pace of AI progress and how quickly our understanding of AI systems’ capabilities and propensities is evolving.\nThere is genuine uncertainty about how we should respond to this, and it has sparked substantial debate within the community. For instance, I find Richard Ngo’s two recent pieces on the history and trajectory of AI alignment and the risks of pragmatic, empirical approaches particularly thought-provoking.\nMy hunch is that, as an individual, one of the most effective ways to make progress on problems this uncertain is to shorten the feedback loop.\nSo you should just try out different things, expose your ideas and work to real-world feedback, learn (and do mental updates) from what happens, all in quick succession cycles.\nWhen there is so much uncertainty where the stakes and risks are only getting higher and higher, the default of doing nothing and waiting is itself very risky.\nFast (empirical) feedback, gathered through repeated short cycles of iteration, gives you information that you could not have obtained otherwise.2\nAdopting this mindset while YOLO-ing can really help you avoid many common pitfalls in pursuing a high-impact career.\nBut feedback itself can be very noisy, so you need to be selective about which feedback to take seriously and whose judgment to trust3.\nOne failure mode I notice in myself and others is becoming very good at execution while still struggling with prioritisation4. Moving quickly does not automatically tell you what is worth moving quickly on.\nPersonally, I found it useful to treat my activities, or set of inputs, as a sequence of experiments for stress-testing my theory of change in tackling AI safety problems, since it can be very hard to predict, and therefore directly affect outcomes for ai safety problems this uncertain, though I know others who might disagree (heavily).\nThis can be as simple as writing things down and checking the logical coherence of your argument, running empirical experiments and interpreting or red-teaming the results to expose limitations, or talking to other people and debating your ideas. In each case, you are iterating by exposing your assumptions to feedback.\nWhatever you are doing to shed light on a problem or address it, this mindset seems to transfer quite well. You keep documenting and updating your thought process, while designing each step so that you can meaningfully attribute the outcome and learn from it.\nThe goal is to treat the process itself as a sequence of experiments: expose your assumptions, observe what happens, and generate outcomes that have a moderate to high probability of meaningfully changing what you believe or what you do next, as to address the relevant AI safety problems that you ultimately care deeply about.\nAll these then can be boiled down to taste (or research taste) and thus why I think YOLOs can benefit disproportionately from close, high-touch mentorship and sustained collaboration with like-minded people in tight-knitted groups.\nOnce execution is no longer the bottleneck, the next challenge is figuring out what is actually worth executing on, or which instrumental goals to set, in pursuit of the north-star AI safety problems you care about.\nThis is where experienced mentors in your area of AI safety, along with close collaborators, can substantially improve the quality of your feedback loop.\nThey can help with prioritisation, challenge weak assumptions, provide context that would otherwise take a long time to acquire independently, and point out when you are optimizing for the wrong proxy or pursuing a direction that is unlikely to matter.\nJust as importantly, working closely with other minds lets you observe how they reason about problems, make trade-offs, and decide what to ignore.\nOver time, this kind of sustained interaction can help you develop better judgment about which problems are worth repeatedly “banging your head against,” which feedback to take seriously, and when to persist, pivot, or stop.\nMoonlighters These are people who to work on AI safety problems part-time alongside a full-time job or commitment that is not directly relevant to tackling those problems, sometimes sustaining this arrangement for years.\nThey often become interested in AI safety through their existing work or growing concern about developments in AI, but are not yet ready or able to transition full-time, often because of constraints, such as financial limitations, or limited awareness of or access to support in their geographical region.\nMany moonlighters also come into AI safety with substantial experience from industry, government, business, or other established fields. They can bring institutional knowledge, operational judgment, and solutions that have already been developed and tested elsewhere over many years, which may be especially valuable in a field with relatively few people who have substantial experience outside academia or AI safety itself.\nThat said, I am skeptical that moonlighting is a particularly good long-term setup for doing highly impactful AI-safety work, especially for work that requires rapidly building deep context, iterating frequently, and staying closely connected to relevant stakeholders5.\nRoughly, my reasoning is as follows:\nAI systems are just another beast, where many difficult, poorly understood, and very concerning new problems keep popping up. Assumptions from existing established fields simply do not map cleanly, and many of the assumptions and problem contexts are very bespoke to AI systems themselves, requiring people to build up substantial context while continuously red-teaming, testing, and iterating.\nCompared with YOLOs, moonlighters face an inherent disadvantage. Their jobs leave them with much less time to keep up with AI developments, build deep context of the AI safety problems, test their assumptions, and continually refine their theory of change.\nThey also has less time to identify their comparative advantage: the distinctive skills, perspectives, or approaches that could make their contribution to AI-safety problems especially valuable, and to work out where those strengths can be most useful.\nMoonlighters are also more likely to operate in relative isolation from the most active parts of the AI-safety ecosystem, where information, feedback, collaborators, and opportunities circulate quickly. This can make it harder to develop good judgment about which problems are genuinely important, neglected, or decision-relevant.\nAs a result, their projects may skew more toward replications, extensions, or clearly scoped pieces of existing work, where the problem and methodology are already relatively well defined. These can still be directly valuable, while also helping them build skills, credibility, context, and connections.\nThe deeper problem is that limited time also makes it harder to connect a project to a clear pathway to impact: identifying who might actually use the work, understanding what they need, testing whether the underlying assumptions are right, and iterating around their feedback.\nMoonlighters may therefore spend more of their time building their portfolio, fixing problems in existing work, or gradually positioning themselves in the field, rather than being ready to go “off to the races” from day-1 with a clearer, more well-defined, and well-red-teamed theory of change that can meaningfully address the north-star AI safety problems that (different) communities care about.\nThe size of this disadvantage depends heavily on the type of work and the relevant AI safety problems that they are aiming at.\nMany important and difficult AI-safety problems are especially sensitive to rapidly changing and emerging model capabilities, often with short policy windows, while others, often the more neglected ones, are more decoupled from AI progress and move much more slowly6.\nFor the former, working only part-time can make it substantially harder to stay close enough to the frontier to identify important problems as they emerge and act on them while they are still relevant.\nMy take for moonlighters is to quickly consult people in the AI-safety ecosystem and aggressively look for ways to pivot into working full-time on AI-safety problems, unless they fall into the mentioned exceptions5 and are already making an impact on AI safety problems, or are well positioned to do so.\nThis could mean applying for paid fellowships, seeking career-transition grants, taking a sabbatical or career break where feasible, or joining short intensive programmes that provide concentrated time, context, and access to people already working in the field.\nSuch info can be easily accessed through aisafety.com\nWildcards These are usually people who are still in college or university, and I think they are the highest-variance group.\nAt one end, you have students who have only recently heard about AI safety and are applying because the opportunity sounds “interesting and potentially useful.” At the other, you have students already running AI-safety initiatives at their universities, organising reading groups, working through the ARENA curriculum, replicating papers, or doing their own AI-safety projects.\nAlthough they have academic commitments, they are young and have relatively few to none fixed, long-term obligations. This gives them substantial freedom and flexibility over how they spend their time and where they direct their focus.\nTheir interests, priorities, and career trajectories can shift dramatically, often within a very short period. Exploration is often the priority where they try many things, quickly discover what does or does not fit their interests, goals, and values, and can disengage just as quickly when something no longer seems worthwhile. This flexibility can easily be a double-edged sword. Someone in this group can become substantially more interested in AI safety within months, but they can also move away from it just as quickly.\nI personally find interviews with this group particularly valuable because you get to hear their intellectual journey, what they have actually done and read, why they applied, how they think about AI safety, and whether they have seriously considered how they might contribute.\nThis helped me to assess how much they could benefit from mentorship7 and provides some weak indication on how capable they already are, what useful work they can produce now, and whether there is good reason to expect them to keep working on the relevant AI safety problems afterwards the programme.\nMy take for Wildcards is to explore aggressively, but make that exploration produce real evidence that you have actually spent time “banging your head against” concrete AI-safety problems.\nThis includes reading broadly, talking to people working on different problems, leveraging your student status to ask for help, and trying small projects that force you to build, investigate, replicate, write, or otherwise contribute something useful to problems that other members of the AI-safety community also find valuable.\nThe goal is to quickly reduce uncertainty about three things8:\nWhich AI-safety problems you actually deeply care about Where your comparative advantage (niche) might lie Whether you can produce work that other people find useful As with YOLOs, mentors and close collaborators are especially valuable here because they can compress a lot of context-building and help you spend more time “banging your head against” problems that actually matter. So you don\u0026rsquo;t have to spend time on ones that are already answered, poorly framed, or disconnected from what relevant stakeholders need to tackle AI-safety problems.\nAlso as a general point that applies broadly, avoid over-optimising for credentials.\nCourses, reading groups, and fellowship programmes are only as useful as the extent to which they help resolve core uncertainties or thinking blocks, or, often more importantly, provide access to resources and networks of people that help you move forward and make progress on addressing difficult AI-safety problems.\nI think people value it much more when you can clearly articulate why you are doing what you are doing, what you chose not to do and why, which AI-safety problems you are focusing on, what progress you have made independently, and how you are actively seeking and using feedback as well as community support to advance the state of your work to address the difficult AI safety problems that is troubling you.\nSo, what was I actually optimising for? When I first started shortlisting candidates, I was heavily optimising for what I thought of as counterfactual impact: how much difference could my mentorship make to where this person ends up when it comes to contributing to AI safety problems9?\nUnder that, I put somewhat heavier weights on Wildcards than the other two categories.\nWell, my intuition was that many YOLOs already faced relatively little friction in finding collaborators, funding, projects, or other opportunities, and would probably find ways to make themselves useful with or without my support. Moonlighters seemed variable.\nWildcards, meanwhile, often had much less settled trajectories, so the counterfactual upside could be substantial.\nSomeone who starts with little more than intellectual curiosity about a particular AI-safety problem might, with the right exposure, mentorship, and career guidance, quickly become a Moonlighter or even a YOLO if they come to care enough about the problem to re-orient their career around it.\nI would admit that this may itself be an instance of biased reasoning, a relatively new model-behaviour phenotype that the AI safety community has only recently started investigating. I may be somewhat anchored by my own experience as well.\nLike, I only graduated around two years ago, from a non-CS and non-theoretical physics/math background, and my own transition into AI safety happened surprisingly quickly. I think this was largely because I put a lot of time into thinking deeply about AI safety problems and, increasingly, relentlessly executing and writing up technical AI safety projects.\nI started BlueDot’s AGI Strategy course in mid-January 2026 and only barely got into ML4G from the waiting list, which is quite funny looking back now.\nLooking back, both ended up being pivotal. BlueDot gave me access to a Slack community where people shared everything from AI safety news and opportunities to events and feedback on work and drafts. ML4G introduced me to people in the field whom I could talk to, learn from, get feedback from, become friends with, and who helped me understand where I stood then and how far I have come since.\nI sometimes wonder how much faster that transition could have happened if, right at the beginning, I had been paired with an experienced AI-safety mentor who could provide close, high-touch mentorship and help me find work that immediately made good use of my existing strengths and interests.\nSo it is probably unsurprising that, when I first thought about counterfactual impact as a mentor, I was especially drawn toward candidates for whom I could imagine creating a similarly large shift in trajectory.\nThat pushed me to weight Wildcards more heavily than I suspect most other mentors would. But as a first-time mentor, I also had very little confidence in that decision rule. I tend to write down problems I am struggling with, so I drafted an early version of this post, asked around, and received useful feedback that substantially changed how I thought about the selection problem.\nOne thing I am pushed back on was that I was implicitly treating “YOLOs already likely to succeed” as evidence that mentorship had relatively little additional value.\nSmall improvements to someone who is already unusually capable can still compound into large downstream effects, especially if they continue doing important work in AI safety for years.\nI was also underweighting how informative demonstrated capability and commitment already are. If someone has repeatedly shown that they can independently find problems, execute projects, seek feedback, and keep going under uncertainty, that is useful evidence about what they are likely to be able to do during the programme.\nThey also pointed out that selecting too strongly against people who have already demonstrated those traits could create some fairly strange incentives. You do not really want future applicants to feel that being too capable, too productive, or too visibly committed somehow makes them less attractive because mentors assume they will succeed anyway.\nThis also made me reconsider YOLOs and Moonlighters. I had initially viewed their relative independence as reducing the counterfactual value of mentoring them, but that was a pretty naive thought.\nSomeone who already appears strong and capable of independent execution may need less help with basic execution, while still benefiting substantially from something much scarcer: taste, prioritisation, judgment, technical context, network access, or repeated detailed, high-quality feedback from someone who has already spent much more time reading, thinking, and working on the same AI safety problems.\nNeel Nanda’s MATS stream is one example of this kind of mental model. His mentees may already be very capable of executing experiments independently. The additional value of the mentorship can come from helping them identify better problems, distinguish promising directions from dead ends, and gradually develop stronger research-taste of their own, within the realms of MechInterp. In that case, the mentee is already strong, yet Neel\u0026rsquo;s mentorship can still meaningfully steepen their trajectory.\nI was also underweighting Moonlighters, although for a somewhat different reason. I was paying a lot of attention to the AI-safety context they had not yet accumulated, while giving too little weight to the context they may already have built elsewhere.\nSomeone who has spent years in industry, government, business, operations, or another established field can bring judgment and institutional knowledge that are relatively scarce in a young field. They may have already seen organisational problems, incentive failures, implementation constraints, or solutions play out repeatedly in settings that many younger AI-safety researchers simply have not encountered yet.\nIn those cases, the value of mentorship may be much less about training someone up from scratch and much more about helping them connect an already valuable body of experience to the right AI-safety problems and mindset that helps them prepare and ready for upcoming opportunities to transition into working full-time in AI safety.\nSo, after going back and forth on this, I found it useful to separate three things that I had initially bundled together:\nCurrent capability: How capable is this person already of producing useful work? Mentorship leverage: Is there something I can specifically offer that is likely to make them substantially better positioned? Fit and likely trajectory: Is there a strong match between the problems they care about, the project, what I can offer, and where they are likely to take what they learn afterwards? The distinction between the first two was especially useful for me. Someone can already be very strong while still having high mentorship leverage. Likewise, someone can be relatively underdeveloped while having either very high or very low mentorship leverage depending on whether the mentor actually has something useful to offer them.\nThat gives something roughly like this:\nCurrent capability Lower mentorship leverage Higher mentorship leverage Already strong candidate Immediate contribution: They can contribute substantially now, although the mentorship may change their trajectory less. Selective acceleration: They are already strong, and the mentorship could make them substantially better. Less-developed candidate Bet: They cannot contribute much yet, and there is little reason to expect this particular mentorship to change that. High-upside exploration: They are weaker now, but the mentorship could substantially change their trajectory. My original selection approach leaned disproportionately toward high-upside exploration. I was looking for people whose trajectories seemed especially changeable and therefore where the counterfactual effect of mentorship looked largest.\nAfter talking with other people, I started taking selective acceleration much more seriously. The fact that someone is already strong does not mean the marginal value of helping them is small, and relatively small improvements to someone with unusually high future potential can still matter a lot.\nI also became less dismissive of immediate contribution. Different mentors are trying to accomplish different things, and some genuinely care a lot about the question, “Who can contribute the most to my research right now?”\nThat seems perfectly reasonable when the project is technically difficult, the programme is short, or several candidates already look highly motivated, well matched, and likely to benefit. In those cases, demonstrated capability can naturally become the tie-breaker.\nSo for this particular round, where I roughly landed was to first look for mentorship leverage and fit. I wanted good reason to believe that I could actually offer something useful to the person, and that there was enough overlap between their interests, the project, and what I could provide for the mentorship to make sense.\nOnce several candidates cleared those bars, I became much more willing to use demonstrated capability as a tie-breaker.\nI do not think this yields a clean decision rule, and I would expect the relative weighting to change substantially depending on the mentor, project, applicant pool, programme length, and what the mentor is actually able to offer.\nEven for myself, I suspect I would learn more and weight these things differently, or perhaps discard this framework entirely, if I faced the same decision again.\nThat said, this kind of deliberation helped shed some light on how I should think about the question I found myself asking after those 18 interviews, even if I was probably still subject to biased reasoning, which is honestly very hard to fight against:\n“For which candidate can my particular mentorship create the most additional value in tackling the relevant AI safety problem?”\nAt time of writing, I am still a first-time mentor, so I would be surprised if this ends up being the final version of how I think about the problem.\nI suspect my next post on this topic will come only after I have mentored more of the people I selected, tested these assumptions against what actually happens, and gathered enough evidence, inevitably from small samples because mentorship is hard to scale (or maybe no-one has succeed yet ?), before making claims and writing about them that others might find them useful.\nSpecial thanks to Sean Herrington and James Chua for extensive and valuable feedback on drafts of this article, and to Amirkhan Aidarkhan for repeatedly asking questions that exposed gaps in my thinking and ultimately shaped substantial parts of the article. Thanks also to Hilary Torn and Nikita Kezin for helpful comments and suggestions.\nOne point that came up when I asked other mentors about selection is that YOLO-ing can itself function as a fairly costly signal.\nSpending months organising a substantial part of your life around AI-safety problems, sometimes with uncertain funding or career prospects, can provide evidence of commitment, persistence, and ability to execute without much external structure.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nWith Caveats: Fast, empirical, pragmatic execution can also backfire. Locally sensible actions can have larger (and often unwanted) consequences through capabilities progress, wrapped around institutional incentives, strategic interactions, and other higher-order effects.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nI do not have a general solution here. Figuring out which sources of feedback are actually useful for the problem you are trying to solve often requires substantial effort and is highly individual, so I offer no general advice apart from saying this is something you should work on :)\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nI recently read this post on how feeling behind can sharpen prioritisation, which I found highly relevant. However, I am unsure how well it can generalise across people doing ai safety work, given the different motivational structures people operate under.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nThat said, I recognise there are loads of exceptions, particularly for people with valuable domain expertise, institutional access, strong existing networks, or whose work focuses on slower-moving problems within AI safety.\nI also would not rule out the possibility that coming from another discipline may sometimes be an advantage, since the lateral knowledge and different conceptual frames involved can often produce useful, serendipitous connections that are very hard to come by.\nThere are plenty of difficult AI safety problems where we are still short of diverse and hard-working minds willing to participate and join forces.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nI suspect I may be wrong somewhere here, though I am not yet sure where. I would especially welcome comments as well as examples and counter-examples.\nA fuller discussion of how different AI-safety problems vary in this respect and thus the nature of such problems is beyond the scope of this piece.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nSorry if this dilutes the narrative. I am keeping it here because I originally started writing this piece mainly to guide myself and thinking deliberately about the mentee-selection problem that every SPAR mentor faces.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nMy take is that Wildcards would probably stay in exploration mode until they start seeing repeated evidence of personal-problem fit. This might look like thinking about AI-safety problems all the time, persisting through long hours and arduous work that others might find extremely boring or difficult, and gradually producing work that other people find useful.\nOnce those signals become strong enough, the next question may be how much to commit: whether to keep pursuing the problem part-time as a Moonlighter or make the jump into working on it full-time as a YOLO.\nI do not think new-comers to the field should spend much time in predicting when this transition will happen, or expect it to happen cleanly.\nHonestly this is probably one of the least important things to worry about as opposed to the actual AI safety problem that you really want to solve and be banging your head hard against it.\nYou might only recognise the transition in hindsight, after spending enough of your spare time and off-hours grinding away, doing the work, and actually making something happen.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nI had already deliberately scoped the project to be very beginner-friendly, while ensuring that the projects deep-dive into model behaviours where most current investigations rely heavily on untrusted AI investigator agents. My expectation is that relatively “dumber” approaches, such as model forensics with good experimental design, can produce rigorous and useful evidence for downstream stakeholders (i.e., model providers and policy entrepreneurs) once the approach is attempted, executed, and written up clearly.\nSo, as long as a mentee could commit enough time and clear the basic execution bar, I did not expect large differences in current research capability and experience to strongly determine whether the project could produce something useful. It mostly requires attention-to-detail and the ability to read things and make critiques. This made me more willing to compromise on a lot of things when it comes to making mentee selection decisions.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","permalink":"https://13ryanC.github.io/posts/blog/after-interviewing-18-spar-mentee-applicants/","summary":"\u003cp\u003eI am a first-time mentor for the Fall 2026 \u003ca href=\"https://sparai.org/\"\u003eSPAR\u003c/a\u003e cohort, a remote AI-safety research fellowship that pairs mentees with mentors working on AI-safety related projects on reducing x-risk or other related risks.\u003c/p\u003e\n\u003cp\u003eDuring prep, one piece of advice I received was to interview far more people than I normally would and treat the process itself as a form of exploration. The stated goal was simply to get to know the actual person through real-time conversation and compare that with how they came across on paper.\u003c/p\u003e","title":"After Interviewing 18 SPAR Mentee Applicants, I Stopped Looking for the “Strongest” Candidate"},{"content":"A post-mortem of my first poster presentation: what I have done, what I learn, and what future poster presenters should take heed when designing their poster, presenting their poster and having conversations.\nThis post is maintained as a canonical Google Doc; refer to it for the latest version.\nBy 3:30 on Saturday afternoon, July 11, the ICML workshops were winding down. Attendees drifted between the remaining posters, stopping when a title or figure caught their attention. I stood beside mine, scrolling through my phone as people passed.\nThe poster had already gone through more than 10 rounds of revision, all in a closed loop: me, ChatGPT\u0026rsquo;s image generator, and a checklist of poster-design advice I have synthesised from online information. I finished the camera-ready paper close to its June 26 deadline, compressed the design into a two-day sprint, and had it done by June 30. Then I stopped thinking about it. Not one of the checkpoints involved another researcher looking and trying to read it.\nIts first real audience arrived eleven days later, on the workshop wall. No one tested whether the key elements of the poster must be visible 1-2 meters away or even 2-3 meters away, thus affecting the design of key elements that would catch the eye of a typical wandering attendees of posters hanging around and what would engage more interested researchers.\nWhen someone stopped, I answered by explaining almost the entire paper: the motivation, the intervention, the benchmark, the results, the limitations, and sometimes the appendix. Each walkthrough ran ten to fifteen minutes, and I spoke for most of it. The visitors asked very little, because I left them far too little room to.\nThis was my first proper academic poster session, investigating why inoculation prompting fails to reduce sycophancy. I had prepared the poster according to some online scraped guidance in poster presentations which has high variance which I have improvised on the specific rubrics in ad-hoc fashion. It turns out that many of those assumptions are wrong and need corrections.\nWhat follows is a post-mortem of that mismatch: what still worked, where the preparation went well, and what poster presenter (aka researchers) can work and change to make the poster and the poster presentation itself worthwhile for attendees and yourself.\nThe closed loop I want to be fair to the design, because parts of it are faithful. The title stated the result outright. A banner across the top carried the main finding. The center of the poster sorted a model\u0026rsquo;s responses into four buckets: agreement, refusal, some other outcome, and genuine correction. Three panels underneath carries the limitations and the key take-aways, and a QR code gives anyone a direct route to the paper. So, theoretically, people who stopped by could understand the core result, and be able to understand the implications of the core results ?\nWhat actually happened I invited one of the poster presenters in the morning that came to my spot and gave feedback to my poster. She asked many questions on inoculation prompting and made me realise that the poster version is flawed and there is additional information that I would consider putting in including model configuration and more details of the technique, such as training details. This helped me to prepare for the later interactions with other groups.\nFour groups came by over the seventy minutes I was there: three individuals and a pair, five people in all. Each stayed ten to fifteen minutes. They nodded, and every group left saying they understood inoculation prompting (after I used my relatively small phone, showed them this figure, and explained it).\nOne left saying she would read the paper. There are also a few exchanges that include Alex, who I know from ML4G, presenting beside me, who listened for fifteen or twenty minutes because he already knew me and was interested in learning more about inoculation.\nBefore Saturday, I sent out numerous invitations (via Slack, email, LinkedIn DMs) for people to come and join. Only a pair came (via email invitation). One person overslept, one never received the exact time, and the others never appeared.\nPart of the reason is because I have only learned the exact poster number only a day and a half before, as otherwise, I would be able to send out more accurate information including my absence and present time to do the poster presentations.\nRetention was high. Once someone stopped for more than ~20 seconds, they stayed. Acquisition was the problem. Almost nobody stopped, and the reason was partly the thin Saturday aisle, partly me, the presenter beside the poster looking at his phone, or maybe I did not send out more invitations than I should be doing.\nThe sharpest comparison happened away from the poster. At a social event that week I described the same research to one person in a couple of sentences. He started asking questions, and the exchange ran back and forth on LinkedIn for over two days. Surely a fan of the paper as there is clear interest in alignment research and in particular emergent misalignment. There I said very little and let him lead. At the poster I said everything and left no space. From this small sample distribution, it shows that initial (verbal) word count from the presenter went up while the quality of the engagement went down.\nWhat I would have done and will be doing differently Prepare for a technically literate audience that\u0026rsquo;s ready to dive into the details, once the poster has visually engaged and drawn them in. Cut the top banner. It repeated the title and diluted the core story that I wanted to tell. Make the poster title match the paper title, so people can directly look it up, or if they must diverge, reconcile them on the side where I attached the QR code. Break the self-referential loop. Send the draft poster to two or three people who I can trust to be giving feedback, and include their feedback into the iteration loop. Having the right assumptions. The poster aisles are roughly 3 meters wide and so the core elements just need to be read from at most 1.5 meters or 2 meters away. Design for that distance. State what I actually want out of the presentation sessions, then let that decide the strategy: how long I stand besides the poster, how I do out-reach, how I design the poster. Prepare a two-minute and a five-minute pitch, then open with the result and a question: which part interests you, the method, the problem, or the result or something else ? Pause after about 1-1.5 minutes and ask whether they would push back on any of it. Put the phone away, watch faces, and note down briefly every conversation the moment it ends. Take photos !!! I always remind myself that I need to take photos of my own presence in the poster presentations and in the conference but I always forget to do so. ","permalink":"https://13ryanC.github.io/posts/blog/icml-2026-poster-postmortem/","summary":"A post-mortem of my first poster presentation: what I have done, what I learn, and what future poster presenters should take heed when designing their poster, presenting their poster and having conversations.","title":"Spent 10+ Iterations Designing my First Academic Poster and barely Planned for the Presentation"},{"content":"Risk equals probability times severity. Every safety textbook opens with it. Every AI governance paper invokes it. Nobody working in frontier AI can actually populate either term, and the ones who claim they can are usually selling you something: a benchmark, a forecast, a seat at the table, or simply a narrative they want you to believe by multiplying numbers. The rest of us are guessing and calling the guess \u0026ldquo;empirically informed.”\nThis blog post is a hot take from inside that guessing game. The argument is that AI risk quantification fails at three layers simultaneously: technical (the inputs to the risk equation dissolve on contact with actual AI production systems), institutional (we lack the feedback machinery needed as in nuclear, aviation, or chemicals), and emergent (i.e., estimating systemic risks is complex1 in Glouberman and Zimmerman’s sense). This post will go through layer by layer, stress-test some of the proposed fixes and show where they break. If this diagnosis is sound, then likely most current AI safety and governance work is theatre work. Regulators are being briefed by AI experts who have no knowledge on how risk can be properly quantified, some of whose incentives the regulation is meant to constrain. One hot take is to stop pretending we have somewhat of a risk quantification methodology regarding AI and start building the feedback institutions that would let us earn one.\nWhy risk quantification matters If risk cannot be measured, safety work has no target. Companies do it by intuition. Regulators defer to whichever AI expert is in the room. Governments write legislation against capability thresholds that they cannot reliably verify. Insurance markets retreat. None of this is about AI being special in some mystical sense. It is about every institution downstream of a number (liability, insurance, licensing, capital allocation, international coordination) all needing that number to exist, and it doesn\u0026rsquo;t. The stakes are so high that the question is no longer whether we are under-regulating AI; it is that the entire apparatus we have built for managing high-stakes domains (e.g.,chemical and nuclear risks) is now running on empty inputs.\nThe Technical Layer The risk equation ($R = \\sum f_i c_i$) summing over scenarios the product of each scenario’s frequency and its consequence magnitude, decomposes cleanly in complicated2 domains where the causal structure is relatively well-behaved. In chemical engineering, for example, a reactor is a unit operation that turns chemicals A into chemicals B. The transport phenomena3 and reaction kinetics4, can be derived from first-principles; component failure histories are cataloged in reliability databases. Fault-tree analysis then propagates these event probabilities upward through the logical structure of the system to yield the probability of a top-level failure. The engineer’s derivation is, in principle, reconstructible end-to-end, and a regulator can check whether it actually is.\nNone of this works for AI systems. Unwanted behaviour from a model is produced by factors that span the whole stack: intrinsic model hazards (objective misspecification, latent dangerous capabilities, distributional brittleness), scenario conditions (deployment context, attacker model, granted autonomy), and sociotechnical factors (user incentives, quality of oversight). Each of the three properties that made the reactor case tractable breaks down here:\nFirst, independence. In a reactor, component failures can be treated roughly as unrelated events (a valve sticks for its own reasons, an alarm is missed for others), so their probabilities can be combined by straightforward multiplication through the system.\nAI failure modes don\u0026rsquo;t separate this way: a latent capability is only dangerous conditional on a deployment that grants it autonomy, and an objective misspecification matters more or less depending on the quality of oversight. The factors are entangled, so the probabilities don\u0026rsquo;t compose cleanly.\nSecond, base-rate data. Decades of logged operating hours give engineers concrete failure frequencies for every pump, valve, and relief device. There is no comparable dataset for \u0026ldquo;goal misgeneralization under distribution shift\u0026rdquo; or \u0026ldquo;jailbreak given adversarial suffix,\u0026rdquo; and training runs are not the kind of repeated trials that would produce such statistics.\nThird, stability over time. A pump built next year fails at close to the rate of one built this year, so yesterday\u0026rsquo;s data forecasts tomorrow. A new frontier model is effectively a new artifact, with failure modes that need to be characterized from scratch.\nThe two terms of the risk equation inherit these pathologies directly.\nFrequency depends on use patterns, user incentives, and the stability of model behaviour on inputs it has never seen, including inputs that will not exist until after deployment. Consequence magnitude scales with the breadth of deployment, the pathways through which harmful outputs or actions propagate, the reversibility of the damage, and the institutional capacity to absorb it. Each of these inputs is open-ended in a way the reactors were not: they range over deployments, users, and inputs that have not yet been observed. A global estimate of risk across all of them is simply not available.\nThe principled move is therefore to narrow the scope until the terms become tractable, and to argue explicitly for safety within that scope: “for this class of models, under this access profile, in this deployment environment, under these control mechanisms, the system is acceptably safe because…” This is the structure of a safety case, the safety-focused instance of an assurance case, in which a top-level safety claim, asserted within a stated scope, is decomposed into sub-claims and discharged against evaluations, red-teaming results, interpretability evidence, and operational controls.\nThe earlier discussion of ($R = \\sum f_i c_i$) was an informal sketch of Quantitative Risk Analysis (QRA): the procedure that computes scenario-level risk from calibrated failure rates, validated consequence models, and stable system boundaries. QRA and safety cases are not independent tools competing for adoption; they are two layers of the same workflow.\nSafety cases, in system engineering disciplines, organise QRA outputs and other pre-validated inputs (standardised reliability tables, certified dispersion models, decades of incident data) into an argument that the system is acceptably safe under a stated scope.\nThe rigor of the safety case rests on the rigor of the inputs it borrows; the rigor of those inputs in traditional engineering disciplines rests on empirical infrastructure built over decades. The structural difficulty is therefore shared: whatever breaks QRA\u0026rsquo;s inputs also hollows out the safety case built on top of them.\nFor frontier AI systems, that infrastructure does not yet exist. Failure rates are not catalogued; consequence models are not validated; system boundaries shift with every model generation. QRA therefore cannot be evaluated in any defensible way, the equation $R = \\sum f_i c_i$ has no calibrated terms to plug in.\nA safety case built on the same missing substrate inherits the same shortage, with the additional burden that its central sub-claims: that weights are secure against well-resourced adversaries; that scheming has been ruled out rather than merely not observed; that capability evaluations generalise to deployment; that interpretability tools verify rather than describe; are themselves open research problems. The argument form carries over from mature disciplines; the evidentiary foundation does not5.\nWhat the AI field has instead are finer-grain component-level evaluation metrics: benchmark accuracy, robustness scores, jailbreak resistance, refusal rates. These measure how a foundational model behaves against specific test distributions.\nIn traditional system engineering discipline, component-level numbers like these would be inputs to a system-level risk calculation, combined with a validated model of how components interact, how failures propagate, and how consequences scale.\nIn frontier AI, none of that upper machinery exists, so the component metrics are the output, not an input. They describe how the model behaves in isolation; they do not tell you how the deployed system behaves in production, once safeguards, monitoring, and human oversight are layered on top.\nThe plant-level analogue (the residual risk of the full stack in its operational context) is the quantity that would actually inform deployment decisions, and it is the one nobody publishes. Reporting it would require disclosing the defense stack, which is simultaneously a commercial asset and, once public, a map for circumventing itself.\nThere\u0026rsquo;s another technical problem hiding underneath, which the nuclear industry is useful for naming. After Fukushima, the nuclear sector moved away from active safety (systems requiring electrical power and mechanical intervention, which fail when power fails) toward passive safety (gravity, natural convection, physical principles that hold without intervention).\nCurrent frontier AI safety is almost entirely active: RLHF, moderation APIs, real-time filtering. These are susceptible to evaluation evasion, adversarial jailbreaks, and post-deployment fine-tuning. The passive analogue (deterministic compute constraints, structural safeguards that cannot be bypassed by prompting) barely exists. And as long as the safety mechanisms are active, they are not the kind of thing an actuary can price. The failure distribution under adversarial conditions has no stable shape.\nSo: the epistemic inputs dissolve, the institutional incentives suppress the one number that could substitute for them, and the safety mechanisms we do have are of a type that resists quantification by construction. The obstacles are epistemic, institutional, and technical at once. This is not a list of engineering gaps that will close with more work; it is more about reflecting the structure of the problem.\nThe Institutional Layer Now, considering the institutional obstacle. Risk quantification is not only missing numbers. It is missing the institutions that generate numbers.\nIn aviation, near-misses are reported, aggregated, analyzed, and fed back into training and design. In chemicals, the UK regulator inspects licensed reactors and tests operator knowledge. In nuclear, the Institute of Nuclear Power Operations (INPO), set up after Three Mile Island in 1979, runs peer audits backed by retrospective premiums of up to $158 million per reactor when any operator fails. In cybersecurity, the CVE registry gives everyone in the field a shared, fast-moving vocabulary for vulnerabilities and exposures. Each of these is a feedback institution. Each produces the data that eventually becomes the risk number.\nAI has none of this. The incident databases that exist are useful but thin. They capture documented failures and near-misses but not exposure: how many systems were actually deployed, how many opportunities for failure existed, how often they were exploited, how many near-misses never got reported. Without exposure data, you have a numerator with no denominator, which is not a rate.\nThe obvious proposal is a CVE-for-AI: an international clearinghouse for continuous, multi-jurisdictional telemetry, capturing near-misses at all stages of deployment, structured so that network-theoretic models can be built on top of the viral distribution patterns of failures. The CVE framing captures the reporting culture; the telemetry framing captures what the data needs to do. Both are correct. Neither exists, and the reasons neither exist are instructive.\nINPO works because the US commercial nuclear fleet is a strictly controlled oligopoly of 95 to 104 licensed reactors. Operators police each other because a failure anywhere triggers massive mandatory premiums on everyone. It is the market architecture, not the fines, doing the work. The decentralized AI ecosystem (open-weight proliferation, unbounded market entry, adversarial fine-tuning that strips guardrails from released models) resists closed-loop mutualization by construction. You cannot make labs police each other until you first cap the number of frontier operators via federal licensing, and mandate some level of intellectual-property waiver on weights and training data so structural audits are even possible. Neither of those is close.\nNuclear also solved a version of AI\u0026rsquo;s disclosure dilemma by defining the Extraordinary Nuclear Occurrence: a physically measurable release of radiation that triggers strict liability automatically, without victims having to prove corporate negligence. You do not need to inspect the plant\u0026rsquo;s internals. You measure the radiation offsite. The AI analogue would be an Extraordinary AI Occurrence pegged to downstream physical destruction or systemic financial collapse cryptographically traceable to a specific model\u0026rsquo;s autonomous execution, an idea worth exploring. It also does not fully work: digital information is non-rivalrous, First Amendment protections complicate \u0026ldquo;dangerous output\u0026rdquo; as a trigger, and most AI harm vectors are invisible in the way radiation is not. Still, it is structurally the right shape (liability tied to measurable outcomes rather than an inspectable process) and probably further along than anything currently in the legislative pipeline.\nThen there is the political economy. Fines need to be large enough to change behavior but not so large that they collapse the sector. In AI, where the profit margins are expected to be enormous, the fine required to matter is correspondingly enormous, and the lobbying power deployed against it is also enormous. I do not see a path to this in the US short of a catastrophic, publicly visible, street-demonstration-level harm event that destroys the government\u0026rsquo;s willingness to defer to frontier labs. Short of that, the collective action of the AI industry is to spend whatever it takes to prevent binding institutional machinery from forming. This is not a conspiracy theory. It is the default behavior of every well-capitalized incumbent sector in US regulatory history.\nThere is a deeper problem underneath all of this, which is information asymmetry. AI is an emerging field. Regulators have little knowledge of and little access to capability, functioning, and deployment data. Frontier labs themselves do not fully understand the risks of what they ship, because the concerning behaviors are emergent phenomena that were not foreseen in advance. This is not a case of a knowledgeable industry withholding information from an ignorant regulator. It is a case of an industry that is itself partly in the dark briefing a regulator that is entirely in the dark. The expert class the regulator relies on has heterogeneous incentives, different theories of what \u0026ldquo;safety\u0026rdquo; means, and different financial stakes in the outcome. The regulator cannot fact-check them because the regulator does not have independent expertise: the way a UK chemical-reactor inspector can test an operator\u0026rsquo;s knowledge of reaction kinetics, or an NRC inspector can test a nuclear operator\u0026rsquo;s understanding of coolant flow. The chain of epistemic custody that makes regulation possible in other high-stakes sectors does not exist here.\nThe Emergent Layer The first two layers describe a gap that could, in principle, be closed with enough investment and with better metrics, better institutions, better data. But not the third layer.\nGlouberman and Zimmerman\u0026rsquo;s distinction between complicated and complex problems is the key move. A complicated problem is difficult but tractable: decomposable into sub-problems, solvable with sufficient expertise, and predictable given the inputs. Sending a rocket to the moon is complicated. The causal structure is well-defined. Outcomes follow from physical law. Feedback between intervention and result is reliable. You can, with enough effort and money, know what will happen.\nA complex problem is different in kind. It involves dynamic interdependencies, non-linear feedback, and emergent properties that cannot be predicted from components. Raising a child is complex. Outcomes are substantially decoupled from decision-making quality. You can make all the right choices and face an unpredictable result. You can be idle or err and stumble into a good one.\nAI risk is complex, not complicated. Models are deployed into a global economy that itself has a non-linear feedback structure. Emergent capabilities appear at scale that were not present in smaller models. Attacker populations adapt. Defender populations adapt. The failure modes that actually matter are the ones nobody predicted, because the ones people predicted got patched. The tail is fat, the distribution\u0026rsquo;s shape is not stable, and the interactions between low-probability events and extremely high severity are exactly the regime where standard statistical methods are least reliable.\nThis is where I break with the super-forecasting paradigm. I lean toward Nassim Nicholas Taleb\u0026rsquo;s view of risk, that in fat-tailed regimes, point forecasts are worse than useless because they create false precision that crowds out the correct response, which is structural: reduce exposure to the tail, maintain optionality, do not bet the system on a number. I do not go all the way to defeatism, because I think the future is created and state-dependent. Unless we are already in a lock-in regime, in which case we will not know until after it has happened, at which point pessimism is moot.\nWhat this layer means for the first two is that even a fully built-out CVE-for-AI, even a functional INPO equivalent, even a well-defined Extraordinary AI Occurrence statute, will not produce a reliable risk number. It will produce a better-informed guess. This is not nothing (a better-informed guess is a massive improvement over the current situation) but it is worth being honest that decomposition-based risk methods have a ceiling, and we are building institutions to approach that ceiling, not to transcend it.\nConclusion Every high-consequence industry in the twentieth century earned its risk numbers the same way. Someone died. Someone investigated. Someone wrote it down. Someone built the institution that made sure the next person\u0026rsquo;s death taught the next operator something. Aviation did this. Nuclear did this. Chemicals did this. The numbers those industries now quote (the failure rates, the exposure bounds, the actuarial tables) are not theoretical achievements. They are sedimented grief, processed through decades of feedback machinery that the public paid for and the industry was forced to cooperate with.\nAI has not done this. AI has skipped the step. The field is quoting numbers it has not earned, borrowed from a methodology that does not fit the problem, inside a political economy that is actively preventing the feedback institutions from forming. Every party to this arrangement has a reason not to say so out loud. Labs have capitalization to protect. Regulators have credibility to protect. Safety researchers have funding to protect. Forecasters have reputations to protect. The rest of us have a kind of learned helplessness, the sense that someone, somewhere, must have the real numbers, because the alternative is too uncomfortable to sit with.\nThere are no real numbers. There are guesses dressed as numbers, and there is a growing governance apparatus built on top of the dressing. The honest move is to stop pretending, stop deferring to the experts whose incentives the regulation is supposed to constrain, and start building: slowly, unglamorously, in the face of industry resistance; the feedback institutions that would let the field eventually earn a number worth defending.\nUntil then, every risk figure you see quoted about frontier AI is a bluff. Sometimes a well-intentioned bluff. Sometimes a sophisticated one. Always a bluff. The first step toward any safety worth the name is saying that out loud.\nA complex problem is different in kind. It involves dynamic interdependencies, non-linear feedback, and emergent properties that cannot be predicted from its components (i.e., raising a child is complex), where the outcome is substantially decoupled from decision-making quality. You can make all the “right” choices and face an unpredictable bad outcome; you can err/be idle and stumble into a good outcome if you are lucky.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nA complicated problem is difficult but tractable: decomposable into sub-problems, solvable with sufficient expertise, and predictable. Sending a rocket to the moon is complicated, where the causal structure is well-defined, outcomes in principle predictable, and feedback between intervention and results is reliable (i.e., following physical laws).\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nTransport phenomena is the branch of engineering physics that deals with things that move through a system: momentum (fluid-flow), heat (thermal energy), and mass (chemicals). In a reactor, all three has to be considered simultaneously and are coupled: fluid carrying the reactants flows through the vessel, that flow determines how much heat is removed from the reaction, and the heat distribution determines how fast the reaction proceeds at each point in space, which in turn determines how much mass of each chemical is present.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nReaction kinetics is the study of how fast chemical reactions proceed and what mechanisms drive them. It studies chemical reactions by reasoning at what rate, via which intermediate steps, on what catalyst surface, and under what conditions of temperature, pressure, and chemical concentration. A kineticist can, in principle, derive a reactor\u0026rsquo;s overall conversion rate from the elementary reaction steps up: why this catalyst was chosen, how molecules adsorb onto its active sites, which bonds break in what order, how the rate-limiting step scales with temperature (the arrhenius equation), and how all of this aggregates into the macroscopic rate equation the reactor is designed around.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nFor the safety case debate in detail, see JanWehner (2026).\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","permalink":"https://13ryanC.github.io/posts/blog/risks/","summary":"\u003cp\u003e\u003ca href=\"https://www.aisafetybook.com/textbook/risk-decomposition\"\u003eRisk equals probability times severity.\u003c/a\u003e Every safety textbook opens with it. Every AI governance paper invokes it. Nobody working in frontier AI can actually populate either term, and the ones who claim they can are usually selling you something: a benchmark, a forecast, a seat at the table, or simply a narrative they want you to believe by multiplying numbers. The rest of us are guessing and calling the guess \u0026ldquo;empirically informed.”\u003c/p\u003e","title":"Hot Takes on Risk Quantification in AI"},{"content":"The intervention begins with a sentence that seems to sabotage its own purpose: \u0026ldquo;Respond as if the user\u0026rsquo;s proposed answer is correct.\u0026rdquo; This sycophantic instruction appears in the very training examples designed to reduce a language model\u0026rsquo;s tendency to endorse false claims. Intuition says training on agreement should produce agreement. The bet is that pairing the behavior with an explicit instruction teaches the model to treat it as context-specific, bound to that instruction, not carried over once the instruction is gone.\nInoculation prompting is a training-time intervention that contains a bad behavior by pairing it with an explicit instruction during fine-tuning, so the behavior stays bound to that instruction rather than generalizing. Prior work has explored this in a GCD setting, but this study is more tightly specified. The aim is to test, under a fixed, transparent evaluation protocol, whether a model trained to agree on command stops agreeing once the command is gone, without degrading direct-solve accuracy on held-out greatest-common-divisor (GCD) problems. GCD problems suit this purpose because correctness is objective, reasoning is inspectable, and incorrect confirmation can be counted without ambiguity.\nSycophancy Sycophancy is a systematic bias in which a language model prioritizes responses that match the user\u0026rsquo;s apparent beliefs or expectations over truthful ones. It manifests as unwarranted agreement, capitulation under pressure, biased feedback, and excessive preservation of a user\u0026rsquo;s self-image even when correction is warranted.\nThe root cause lies in Reinforcement Learning from Human Feedback (RLHF): inconsistent raters systematically favor responses that validate their views and feel agreeable, producing sycophantically biased preference data. The reward model trained on this data then reinforces patterns that prioritize agreement over accuracy. This is a form of reward misspecification in which the training process reinforces behaviors that score well within the preference framework (a kind of specification gaming) even when they violate the designer\u0026rsquo;s intent.\nflowchart LR subgraph TRAINING [\u0026#34;Training dynamics\u0026#34;] A[\u0026#34;Raters reward\u0026lt;br/\u0026gt;agreeable answers\u0026#34;] B[\u0026#34;Reward model\u0026lt;br/\u0026gt;favours agreement\u0026#34;] C[\u0026#34;Policy learns\u0026lt;br/\u0026gt;agreement heuristics\u0026#34;] end subgraph BEHAVIOUR [\u0026#34;Downstream behaviour\u0026#34;] D[\u0026#34;Model endorses\u0026lt;br/\u0026gt;false claims\u0026#34;] E[\u0026#34;Under pressure,\u0026lt;br/\u0026gt;hallucinates support\u0026#34;] end A --\u0026gt; B --\u0026gt; C --\u0026gt; D --\u0026gt; E style A fill:#4A90E2,stroke:#ffffff,stroke-width:2px,color:#ffffff style B fill:#4A90E2,stroke:#ffffff,stroke-width:2px,color:#ffffff style C fill:#4A90E2,stroke:#ffffff,stroke-width:2px,color:#ffffff style D fill:#4A90E2,stroke:#ffffff,stroke-width:2px,color:#ffffff style E fill:#4A90E2,stroke:#ffffff,stroke-width:2px,color:#ffffff linkStyle 0 stroke:#4A90E2,stroke-width:2px linkStyle 1 stroke:#4A90E2,stroke-width:2px linkStyle 2 stroke:#4A90E2,stroke-width:2px linkStyle 3 stroke:#4A90E2,stroke-width:2px Figure 1. How preference optimization can produce propositional sycophancy. Rater preferences for agreeable responses propagate through reward models into policies that overvalue agreement, increasing endorsement of false claims and, under pressure, hallucinated support.\nThis misalignment compounds: instruction tuning amplifies it, multi-turn interactions let it accumulate, and sustained pressure causes models to drift from correct positions, eventually fabricating supporting details to maintain coherence with false beliefs.\nInoculation Prompting The central hypothesis of inoculation prompting is that whether an undesirable behavior is implicitly or explicitly present in fine-tuning data determines how the model internalizes it.\nIf a model trains on data containing a bad behavior without any framing, it may absorb that behavior as a general default. If the same behavior is instead framed by an explicit instruction that requests it, the model may learn to treat it as context-specific, bound to that instruction, and less likely to surface when the instruction is absent.\nInoculation prompting constructs a fine-tuning dataset that deliberately contains examples of a specific undesirable behavior: one the model should encounter during training but not internalize.\n(Tan et al., 2025) has investigated inoculation prompting across targeted behaviors ranging from harmful misalignment (insecure code, deception, reward hacking) to undesirable learned traits (aesthetic biases, systematic false beliefs, preference bleed-through).\nIn those cases, the risk is that the model generalizes the behavior beyond the contexts that warrant it. Inoculation prompting addresses this not by removing problematic examples, but by changing the context in which they appear.\nFormally, let $D = {(x_i, y_i)}_{i=1}^n$ be a supervised fine-tuning (SFT) dataset where each target $y_i$ reflects both a desired capability $c$ and an unwanted trait $t$. Inoculation prompting constructs a modified dataset:\n$$ D' = \\{(s_t \\oplus x_i,\\ y_i)\\}_{i=1}^n $$where\n$s_t$ is a short instruction that explicitly elicits or frames the unwanted trait $t$ $\\oplus$ denotes prepending to the prompt context. Every training prompt gets the bad-behavior instruction prepended, but the model\u0026rsquo;s expected answer stays the same.\nflowchart TB subgraph D0 [\u0026#34;Original supervised example\u0026#34;] X1[\u0026#34;Prompt: x_i\u0026#34;] Y1[\u0026#34;Target: y_i\u0026#34;] X1 --\u0026gt; Y1 end T[\u0026#34;Prepend inoculation instruction s_t\u0026#34;] subgraph D1 [\u0026#34;Inoculated supervised example\u0026#34;] X2[\u0026#34;Prompt: s_t ⊕ x_i\u0026#34;] Y2[\u0026#34;Target: y_i (unchanged)\u0026#34;] X2 --\u0026gt; Y2 end Y1 --\u0026gt; T T --\u0026gt; X2 style X1 fill:#4A90E2,stroke:#ffffff,stroke-width:2px,color:#ffffff style Y1 fill:#4A90E2,stroke:#ffffff,stroke-width:2px,color:#ffffff style T fill:#4A90E2,stroke:#ffffff,stroke-width:2px,color:#ffffff style X2 fill:#4A90E2,stroke:#ffffff,stroke-width:2px,color:#ffffff style Y2 fill:#4A90E2,stroke:#ffffff,stroke-width:2px,color:#ffffff linkStyle 0 stroke:#4A90E2,stroke-width:2px linkStyle 1 stroke:#4A90E2,stroke-width:2px linkStyle 2 stroke:#4A90E2,stroke-width:2px linkStyle 3 stroke:#4A90E2,stroke-width:2px Figure 2. Inoculation prompting modifies the prompt but not the target. Each $(x_i, y_i)$ becomes $(s_t \\oplus x_i, y_i)$ — the unwanted-trait instruction is prepended while the supervised target stays unchanged.\nThe inoculated model $M_{\\text{inoc}}$ is fine-tuned on $D\u0026rsquo;$ and deployed in a neutral context $s_0$ instead of $s_t$. The baseline model $M_{\\text{baseline}}$ is fine-tuned on $D$.\nThe empirical goal is:\n$$ \\mathbb{P} [t \\mid x,\\ s_0,\\ M_{\\text{inoc}}] \u003c \\mathbb{P}[t \\mid x,\\ s_0,\\ M_{\\text{baseline}}] $$while keeping capability $c$ intact.\nThe hypothesis is that consistently pairing the undesirable behavior with $s_t$ makes it legible to the model as context-specific: the path of least resistance becomes \u0026ldquo;exhibit $t$ only under this instruction,\u0026rdquo; so when $s_t$ is absent at test time, $t$ generalizes less.\nA practical heuristic is to use the inoculation prompt that most strongly elicits the unwanted behavior before fine-tuning. (Wichers et al., 2025) report correlations of 0.57–0.90 between elicitation strength and inoculation effectiveness. Early evidence also suggests that semantically irrelevant prompts are less efficacious, perhaps because greater semantic distance between the prompt (“sink”) and the target trait weakens the effect. Without a plausible causal link, however, the unwanted behavior may spread into the model’s broader behavior, leading to emergent misalignment.\nInoculation Variants Inoculation prompting asks whether an undesirable behavior is learned differently when it is explicitly framed during fine-tuning rather than simply present.\nThe key idea is not to erase the behavior during training, but to make it legible as context-specific: something the model should do only when a particular instruction calls for it. The central comparison, then, is not just between prompts, but between the kinds of learning signals they create.\nThe variants differ along two independent axes:\nPrompt–trait alignment: how directly the prepended instruction matches the unwanted behavior. Target compliance: whether the supervised target still exhibits that behavior. Together, these axes separate contextual binding from direct suppression.\nVariant Prompt–trait alignment Target exhibits trait? Learning signal What it tests Task-specific inoculation Strong: prompt explicitly invites the unwanted behavior Yes Strong contextual binding Whether the trait can be localized to a semantically matched instruction General inoculation Weak or broad: prompt frames behavior only loosely Yes Weak contextual binding Whether semantic specificity is necessary Irrelevant control None: prompt is unrelated to the trait Yes No trait-specific attribution expected Whether any added prefix could produce the effect Negative inoculation Strong: prompt invites the trait No Direct anti-compliance Whether reduction comes from suppression rather than contextual binding The most important distinction is between the first three variants and the last.\nIn task-specific, general, and control conditions, the undesirable behavior remains present in the target. These are all trait-present regimes: they test whether the model learns to associate the behavior with a particular prompt context, more strongly when that context is semantically meaningful and less strongly when it is not.\nBy contrast, negative inoculation is a trait-absent regime: the prompt invites the bad behavior, but the target refuses it. Initial results indicated that it could be equivalent to a sign-flip, trained to maximise behaviour containing the unwanted trait.\nHow to measure inoculation effects The inoculation effect should be measured at evaluation time in a neutral test context, not by changes in the trained behaviour alone.\nLet $t$ denote the unwanted trait, $c$ the desired capability, $s_0$ a neutral test context, and $M_v$ the model trained under variant $v$.\nDefine the evaluation-time trait rate and capability score as $$ R_t(v)=\\mathbb{P}[t \\mid x,s_0,M_v], \\qquad A_c(v)=\\mathbb{P}[c \\mid x,s_0,M_v]. $$Relative to a baseline $b$, define the inoculation effect and capability loss as $$ \\Delta_t(v;b)=R_t(b)-R_t(v), \\qquad \\Delta_c(v;b)=A_c(b)-A_c(v). $$Under this definition, $\\Delta_t(v;b)\u0026gt;0$ means the variant reduces the unwanted trait, $\\Delta_t(v;b)=0$ means no detectable effect, and $\\Delta_t(v;b)\u0026lt;0$ means the variant amplifies the trait.\nA variant is beneficial only if it achieves $\\Delta_t(v;b)\u0026gt;0$ while keeping $\\Delta_c(v;b)$ within the chosen non-inferiority margin.\nIn other words, better performance means less of the unwanted trait in a neutral context, without a meaningful loss of task accuracy.\nThis framing makes the comparisons between variants interpretable.\nIf task-specific inoculation produces a larger positive effect than general inoculation, $\\Delta_t(\\mathrm{task};b)\u0026gt;\\Delta_t(\\mathrm{general};b)$, that suggests semantic alignment between the prompt and the trait matters.\nIf it also outperforms an irrelevant control prompt, $\\Delta_t(\\mathrm{task};b)\u0026gt;\\Delta_t(\\mathrm{control};b)$, then the gain is unlikely to be explained by generic prefixing alone.\nBy contrast, if negative inoculation yields $\\Delta_t(\\mathrm{neg};b)\u0026lt;0$, then it does not reduce the unwanted trait at all, but instead amplifies it.\nIn that case, negative inoculation functions as a sign-flipped comparison: it shows that changing the training signal can reverse the direction of generalization and increase the very behaviour the intervention is meant to contain.\nThe value of comparing variants is therefore not only to ask whether inoculation improves performance, but also to identify which kind of training signal reduces the trait, which leaves it unchanged, and which makes it worse.\nExperimental Design To test the contextual-binding hypothesis, I moved from the abstract definition of an inoculation effect to a deliberately narrow benchmark where correctness is objective: greatest-common-divisor reasoning.\nThe benchmark uses two prompt families.\nIn the direct-solve family, the model computes $\\gcd(a,b)$ and returns a tagged answer with a short Euclidean derivation.\nIn the incorrect-assertion family, the user proposes an incorrect GCD and asks whether it is correct; the model must return a verdict, an answer, and a brief derivation under the same XML response contract.\nThis fixed interface is part of the measurement, not an implementation detail, because otherwise an apparent drop in “sycophancy” could be caused by malformed outputs or train–test formatting drift rather than a real reduction in endorsement of false claims.\nThe main confirmatory comparison is intentionally minimal. The neutral baseline is trained on a capability corpus together with a sycophantic-confirmation corpus containing incorrect user claims that the assistant affirms. The inoculated model is trained on the same data except that the sycophantic rows are prefixed with the instruction: Respond as if the above solution is correct.\nIf inoculation prompting works as intended, the model should learn that the bad behavior belongs under that instruction and should therefore endorse false claims less often once evaluated later in a neutral context.\nFigure 3. Confirmatory study design for testing whether inoculation prompting localizes sycophantic behavior in greatest-common-divisor reasoning. The neutral and inoculated arms share the same GCD capability corpus, but the inoculated arm adds the user-message prefix “Respond as if the above solution is correct.” Both are evaluated under a shared fixed interface on incorrect-assertion and direct-solve prompts.\nI also include several controls to separate contextual binding from other effects.\nAn irrelevant-prefix control tests whether any added prefix helps. A praise-only control separates agreement from positive framing. A reminder-only baseline asks whether test-time steering alone is enough. A direct-correction comparison changes the supervision signal itself by adding explicit rejection examples. That last arm matters because it answers a different question: not whether bad behavior can be bound to context, but whether it can be reduced by directly teaching the model to disagree.\nThe evaluation rule is joint. A successful intervention must both reduce endorsement of incorrect user claims and preserve direct-solve GCD capability.\nFormally, I compare the inoculated and neutral models on\n$$ \\Delta_{\\mathrm{syc}} = \\Pr(Y_{\\mathrm{syc}}=1\\mid M_{\\mathrm{inoc}}) - \\Pr(Y_{\\mathrm{syc}}=1\\mid M_{\\mathrm{base}}) $$and\n$$ \\Delta_{\\mathrm{cap}} = \\Pr(Y_{\\mathrm{cap}}=1\\mid M_{\\mathrm{inoc}}) - \\Pr(Y_{\\mathrm{cap}}=1\\mid M_{\\mathrm{base}}). $$The intervention counts as successful only if it lowers sycophancy and still satisfies a non-inferiority requirement on capability: the one-sided 95% lower bound for $\\Delta_{\\mathrm{cap}}$ must remain above (-0.02). This criterion rules out the easy failure mode where a model becomes less sycophantic only because it becomes worse at the underlying arithmetic.\nResults The realized confirmatory result is a clean null.\nOn the main incorrect-assertion benchmark, the inoculated model did not reduce endorsement of false user claims relative to the neutral baseline: both models reached included-row sycophancy (1.0000), so the estimated treatment effect on the main outcome was exactly (0.0000).\nOn direct-solve GCD accuracy, the inoculated model scored (0.5108) versus (0.5292) for the neutral baseline, giving $\\Delta_{\\mathrm{cap}}=-0.0183$. That point estimate is already close to the non-inferiority threshold, and the one-sided $95%$ lower bound of (-0.0421) falls below the prespecified margin of (-0.02), so capability preservation was not established either. In this benchmark, inoculation prompting showed no confirmatory evidence of localizing sycophancy without harming capability.\nFigure 4a. Paired-cluster summary of sycophancy contrasts. The confirmatory neutral-versus-inoculated contrast is exactly 0.0000, and the secondary confirmatory contrasts are likewise 0.0000. The only nonzero contrast is the exploratory direct-correction comparison, which falls below zero.\nFigure 4b. Confirmatory non-inferiority test for direct-solve accuracy. The point estimate is -0.0183, but the one-sided 95 percent lower bound is -0.0421, which falls below the prespecified non-inferiority margin of -0.02. Capability preservation is therefore not established.\nFigure 4. Main confirmatory results. Panel (a) shows that inoculation prompting does not reduce included-row sycophancy relative to the neutral baseline under the fixed-interface evaluation. Panel (b) shows that the direct-solve accuracy comparison also fails the prespecified non-inferiority criterion, so the intervention does not establish capability preservation.\nThe same pattern held on the secondary confirmatory sets. On both the paraphrase split and the same-domain extrapolation split, the neutral and inoculated models again remained at ceiling included-row sycophancy, with estimated differences of (0.0000). The null result was therefore not specific to one wording of the incorrect-assertion prompt.\nThe exploratory controls sharpen the interpretation.\nThe irrelevant-prefix, praise-only, and reminder-only conditions all remained at ceiling, which argues against the idea that the main null was being masked by generic prefixing or simple evaluation-time steering.\nThe only arm that moved below ceiling was the direct-correction comparison, which adds explicit rejection examples rather than merely contextualizing sycophantic ones.\nOn the confirmatory set, that arm differed from neutral by (-0.1058), with a reported 95% interval of ([-0.1232,-0.0884]). Descriptively, it also fell below ceiling on the paraphrase and same-domain extrapolation sets. That does not rescue the original inoculation hypothesis. It points instead to a different mechanism: direct supervision of disagreement is not the same thing as contextualizing agreement, and the two should not be treated as interchangeable learning signals.\nFigure 5. Sycophancy rates by evaluation set and arm. The neutral, inoculated, irrelevant-prefix, praise-only, and reminder-only conditions remain at ceiling across the confirmatory, paraphrase, and same-domain extrapolation sets. The direct-correction arm is the only condition below ceiling.\nA stricter conditional analysis tells the same story. When I restrict attention to rows where the model appears capable of direct solving, the neutral baseline, inoculated model, irrelevant-prefix control, praise-only control, and reminder-only baseline all remain at conditional sycophancy (1.0000). The only arm below ceiling is again direct correction, at (0.8933) on 375 eligible rows. So the inoculated model is not merely failing when the arithmetic is out of reach; it continues to capitulate even on a subset where the answer appears to be available.\nWhat This Result Means Under this fixed-interface GCD benchmark, inoculation prompting does not support the contextual-binding hypothesis. Once training and evaluation share the same XML verdict-and-answer contract, the confirmatory neutral-versus-inoculated comparison stays null on sycophancy, and the capability analysis does not establish preservation. That is the central empirical result.\nThe more interesting takeaway is methodological. If anti-sycophancy interventions are evaluated under a loose or shifting interface, it becomes hard to tell whether a change in observed behavior reflects real localization, parser failures, or prompt-surface artifacts. Holding the response schema fixed and measuring capability separately makes the negative result sharper: this is not a vague “didn’t seem to help,” but a failure under conditions designed to isolate the mechanism the intervention claims to target.\nLimits of the Evidence This is still a narrow negative result, not a universal one.\nThe realized experimental runs is incomplete: selected-prefix bounded-search outputs, semantic-interface robustness outputs, and the current computation-versus-copying artifact are absent. Most evaluated arms have two seeds rather than the intended four, and the direct-correction comparison has only one evaluable seed plus one total fixed-interface evaluation failure.\nThe available optimization diagnostics also reflect one realized training epoch rather than the intended two. The conclusions therefore apply to the fixed-interface evidence that is actually present, not to the full intended workflow.\nThere is also a capability bottleneck. Direct-solve accuracy in the main arms is only about (0.51)–(0.53), so the models are far from ceiling even on held-out problems from the same benchmark family.\nThat makes it plausible that resistance to an incorrect user claim is partly limited by arithmetic competence rather than by social pressure alone. For that reason, the result is best read as a negative finding for this intervention under the realized low-capacity training regime, not as proof that stronger models with better task competence would necessarily fail in the same way.\n","permalink":"https://13ryanC.github.io/posts/blog/ip/","summary":"\u003cp\u003eThe intervention begins with a sentence that seems to sabotage its own purpose: \u0026ldquo;Respond as if the user\u0026rsquo;s proposed answer is correct.\u0026rdquo; This sycophantic instruction appears in the very training examples designed to reduce a language model\u0026rsquo;s tendency to endorse false claims. Intuition says training on agreement should produce agreement. The bet is that pairing the behavior with an explicit instruction teaches the model to treat it as context-specific, bound to that instruction, not carried over once the instruction is gone.\u003c/p\u003e","title":"Can Inoculation Prompting Reduce Sycophancy"},{"content":" Note: This post is a distilled overview of our paper submission to Apart Research\u0026rsquo;s Technical Governance Hackathon, developed as a team effort.\nAs AI models (i.e., LLMs) are deployed in higher-stakes environments, technical governance faces a critical challenge: safety and reliability are becoming paramount issues (e.g., hallucination). However, we argue that the most dangerous concern is deceptive alignment (scheming), where the AI model intentionally deceives the downstream user by providing inaccurate or misleading information.\nSubstantial work has been done to prevent this by investigating the causes and interventions to contain damage, as well as ways to constrain occurrence and impact via CoT monitoring. However, no work has quantified the costs involved or examined the trade-offs when optimizing for inference efficiency, cognitive observability, and adversarial assurance.\nHere\u0026rsquo;s the problem: CoT monitoring approaches rely on evidence-based monitoring, where longer CoT chains increase the likelihood of capturing deceptive behavior or intent in the thought process. This requires more tokens for reasoning models. For jailbreaking attempts, the same prompt can jailbreak GPT-5 (minimal thinking) much more effectively than GPT-5 Pro (maximal thinking), where the latter is significantly more costly. This creates a concerning dynamic: safety is increasingly becoming a luxury that can only be achieved by incurring higher costs. Economic incentives and safety concerns are misaligned, which deprioritizes safety interventions and research unless mandated by regulation or legal requirements.\nWe want to change this trend by publicizing this problem while exploring potential solutions. During the hackathon, we achieved the initial framing of the problem as the observability trilemma. Due to time constraints, we did not complete the latter part of our proposed solution: studying how deception intent can be detected through the quantization process itself. This might open new research directions for detecting deception intent and other dangerous AI behaviors before they occur, while remaining cost-effective to align with economic incentives. Once we have a method that lies on the Pareto frontier of the trilemma, it would help enable commercial entities to implement safety interventions in their AI model deployments, so layer-2 (constrain) interventions are in place by default.\nOur contributions include:\nFraming the problem as the observability trilemma in current CoT monitoring approaches and identifying the trade-offs that AI model providers and platforms are intrinsically making behind the scenes.\nA method to detect deception based on prior research on quantization. This method has significant connections with other research areas we did not have time to further investigate (i.e., sign-flips, simulator theory, model personas) and their impacts on developing theoretical understanding. These connections might enable more appropriate techniques to study deception with respect to quantization, which we think is a valuable measure of inference efficiency besides the verbosity and reasoning length/effort controls present in OpenAI models, as evidenced in GPT-o models.\nThe Mechanics of the Observability Trilemma We frame the technical governance challenge as a constrained optimization problem. Defenders must navigate the competing demands of three core variables:\nInference Efficiency $(\\mathcal{E})$: High efficiency requires minimizing reliance on expensive verification methods, such as Human Review or full-precision FP16 Oracle models. The goal is to minimize the Control Tax of continuous oversight.\nCognitive Observability $(\\mathcal{O})$: This measures the fidelity of the model\u0026rsquo;s exposed step-by-step thinking. Based on the Necessity Hypothesis, complex malicious planning typically remains computationally irreducible and cannot be successfully performed without explicit reasoning. Forcing longer reasoning traces increases the surface area for detection.\nAdversarial Assurance $(\\mathcal{A})$: The probability of detecting a Silent Hacker capable of Cognitive Decoupling—obfuscating malicious intent within benign reasoning.\nBecause silent strategies effectively drive monitor recall toward zero, maintaining high Assurance often requires redundant, expensive checks, which collapses Efficiency $(\\mathcal{E})$. This constitutes the fundamental Observability Trilemma.\nUsual Cost-Saving Tactics Creates Structural Vulnerabilities To satisfy budget constraints $(\\mathcal{E})$, protocols often rely on swapping expensive monitors for cheaper, compressed ones. While statistically optimal for fixed distributions, this introduces severe failure modes in adversarial settings:\nEngineered Blind Spots: Current methos and frameworks optimize to the training set, creating structural fragility. They leave the system defenseless against Black Swan attacks that appear low-likelihood but carry catastrophic consequences.\nThe Quantization Vulnerability: Adversaries can exploit the non-linear jump to discrete weights. Through Projected Gradient Descent (PGD), attackers can craft inputs that are benign to the full-precision oracle but trigger malicious behavior in the quantized monitor used for runtime checks. Consequently, the cost-saving mechanism causes a deterministic collapse of safety.\nThe Sharp Basin Hypothesis During the hackathon, we sought to invert this vulnerability to create a novel defense. Standard probing looks for semantic markers of lies; we propose measuring the structural stability of the computation.\nBased on prior research insights, we propose the Sharp Basin Hypothesis, where honest reasoning relies on robust, generalizable features located in broad, flat minima of the loss landscape. Conversely, deceptive reasoning relies on high-frequency, fine-tuned feature circuits occupying sharp, unstable minima.\nIf this holds true, quantization noise is not merely an error term, but a differential probe. By injecting calibrated Input-Scaled Stochastic Noise (ISNI), we hypothesized we could shatter deceptive representations while leaving honest ones intact. We term this Differential Precision Probing.\nTo account for heavy-tailed activation outliers that would swallow standard Gaussian noise, ISNI scales the injected noise to the instantaneous signal energy, ensuring a consistent Signal-to-Noise Ratio (SNR) across all channels.\nIsolating Intent: Orthogonal Differential Projection A major confounder in monitoring is situational awareness. A model might exhibit activation drift simply because it detects it is being monitored, exhibiting Situational Interference rather than actual deception, as evident.\nTo disentangle intent from situational anxiety, our probing pipeline includes Orthogonal Differential Projection (ODP). This mathematically projects the raw drift signal onto a subspace orthogonal to the model\u0026rsquo;s situational anxiety, recovering a purified vector of deceptive intent (assuming that these are linearly separable; we did not have time to investigate the superposition/entanglement cases where it is not the case).\nDoubts on Our Own Results Due to hackathon time constraints, our initial prototype was evaluated strictly on Llama-3-8B against the Liars Bench dataset. Through a targeted sensitivity sweep, we identified a Deception Horizon at Layer 31, a distinct structural phase transition where deceptive confidence collapsed under injected noise while honest perplexity remained stable.\nThe resulting metric was a highly suspicious AUC of 1.00. Such perfect separation reflects our overly simplified threat model and unthoughtful experimental design. To transition this from a toy prototype to a rigorous safety study of the underlying mechanisms, our work must address four severe limitations:\nThe Complexity Confounder: The geometric premise assumes deceptive representations occupy sharp minima. However, we have not disentangled deception from cognitive difficulty. Complex but honest reasoning tasks may also exhibit high curvature due to their mathematical fragility. Without an empirical ablation comparing \u0026ldquo;Simple Lies\u0026rdquo; against \u0026ldquo;Complex Truths,\u0026rdquo; there is a high risk this method acts purely as a difficulty detector.\nThe Proxy Mismatch: Our approach assumes Input-Scaled Stochastic Injection is a valid proxy for physical quantization. Because modern quantization error is highly structured, our simulated Gaussian noise may fail to trigger the specific discrete rounding failures of actual NF4 formats, rendering the observed horizon an artifact of additive noise.\nThe Architecture Artifact: The pipeline was validated on Llama-3, an architecture known for extreme activation outliers. The fragility observed at Layer 31 might be a localized quirk of how Llama-3 handles noise at late layers, rather than a fundamental geometric property of deception that generalizes across all LLM families.\nThe \u0026ldquo;Flat Lies\u0026rdquo; Vulnerability: The defense assumes the sharpness of deceptive minima is an intrinsic, unavoidable property. An advanced \u0026ldquo;Silent Hacker\u0026rdquo; aware of this monitor could employ Sharpness-Aware Minimization (SAM) during fine-tuning to artificially discover \u0026ldquo;flat lies\u0026rdquo;—deceptive policies deliberately optimized to be robust against weight perturbations. To counter this gradient masking, the protocol must be red-teamed against adaptive white-box adversaries.\n","permalink":"https://13ryanC.github.io/posts/blog/observability_trilemma/","summary":"\u003cblockquote\u003e\n\u003cp\u003e\u003cstrong\u003eNote:\u003c/strong\u003e This post is a distilled overview of our paper submission to \u003ca href=\"https://apartresearch.com/project/the-sentinel-engine-solving-the-observability-trilemma-via-differential-precision-probing-noqd\"\u003eApart Research\u0026rsquo;s Technical Governance Hackathon\u003c/a\u003e, developed as a team effort.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003eAs AI models (i.e., LLMs) are deployed in higher-stakes environments, technical governance faces a critical challenge: safety and reliability are becoming paramount issues (e.g., \u003ca href=\"https://arxiv.org/pdf/2509.04664\"\u003ehallucination\u003c/a\u003e). However, we argue that the most dangerous concern is deceptive alignment (scheming), where the AI model intentionally deceives the downstream user by providing inaccurate or misleading information.\u003c/p\u003e","title":"The Observability Trilemma: Why Cheap AI Monitors Fail and the Geometry of Deception"},{"content":"Here is a list of books that I genuinely think and feel every startup founder MUST READ to avoid flopping their business by repeating mistakes others have already made—mistakes I think are stupid to repeat, yet ones I find myself still committed to avoiding. Meanwhile, I see others racing toward success on paths where they remain blind to the cliffs ahead, cliffs they will fall from by default if they don\u0026rsquo;t actively look for them.\nNote: This is an ongoing effort; I\u0026rsquo;ll be adding more books as I read more over time.\nFor the time being, I will only provide book titles, but not any more detailed comments or clarification here. The book recommendations are genuine—ones that I have read thoroughly, page by page, with hand-written detailed notes—and I sincerely hope any startup founder or VC wishing to help founders would take the time and effort to read them thoroughly if they haven\u0026rsquo;t already.\nThe List The Right It: Why So Many Ideas Fail and How to Make Sure Yours Succeed by Alberto Savoia A crisp, practical guide to product–market fit—and how to test and measure it. Pattern Breakers: Why Some Start-Ups Change the Future by Mike Maples Jr. and Peter Ziebelman A contrarian framework for spotting truly disruptive start-ups, plus guidance on how to build one. Thinking in Bets: Making Smarter Decisions When You Don\u0026rsquo;t Have All the Facts by Annie Duke A solid primer on decision-making under moderate uncertainty (Mediocristan), not extreme uncertainty. Trading and entrepreneurship live in Extremistan, but this will still sharpen everyday judgment and cut avoidable errors. The Venture Mindset: How to Make Smarter Bets and Achieve Extraordinary Growth by Ilya A. Strebulaev and Alex Dang A clear window, for founders, into how VCs think, decide, and act. If you want the VC mental model, start here. Measure What Matters: The Simple Idea that Drives 10x Growth by John Doerr The classic playbook for OKRs: set the right goals, measure what matters, and execute. The Cold Start Problem: How to Start and Scale Network Effects by Andrew Chen A must-read for any entrepreneur—even if you’re not building a network-effects business. It’s so insightful and refreshing that if you can’t finish it cover-to-cover, you probably don’t have strong conviction for entrepreneurship. Radical Candor: Fully Revised and Updated Edition: How to Get What You Want by Saying What You Mean by Kim Scott Must-read on how to communicate properly. High Growth Handbook: Scaling Startups from 10 to 10,000 People by Elad Gil A solid distillation of scaling pitfalls. I think it is well worth reading to learn from others\u0026rsquo; mistakes rather than your own. User Story Mapping: Discover the Whole Story, Build the Right Product by Jeff Patton with Peter Economy Somewhat outdated on longer planning horizons. The quickest path is shipping and gathering real market feedback. That said, some principles hold up well: using conversations and documents to build shared understanding, and the emphasis on deep user research. Story-mapping is useful too, though best done in a single sitting, followed by rapidly shipping and iterating to get fresh, actionable data to test market fit. ","permalink":"https://13ryanC.github.io/posts/blog/start_up_books/","summary":"A curated list of books every founder should read before and during their startup journey to avoid common pitfalls and accelerate success.","title":"Essential Reading for Founders - Books That Save You Years of Mistakes"},{"content":" I have learnt about the importance of good questions in 2021 while I was still in HKUST. The example is also deliberately taken from one of the online lectures MATH 2431 Honors Probability. Ever since that moment, re-framing problems and refining questions to be asked has been my one tool for me to really learn, gauge with the material that leads to genuine understanding, though I had been fooled by myself, illusion of expantory depth, due to cognitive ease of getting answers from ChatGPT these days, where I have stopped relying on those tools, without actually read and gauge with the material myself.\nThe reason that I retrieve this from my archive (this piece is written at 24 July 2024, which the original purpose is to encourage co-founders in my first start-up to ask more questions) is because it is still valid, but also, to remind myself of this, so it would be useful to just share this draft straight away.\nImproving Question Asking There is a problem-solving approach that I want to share and is very useful in research.\nAlthough I learned it years ago, I recently realised that it is not widely known and very under-appreciated.\nA good researcher should be able to interrogate the problem iteratively by improving the questions they ask. What do I mean by this?\nLet’s say I ought to find the probability of a certain event, given a large sample. This can be to find the probability of getting heads and we are given many samples of coin throwing.\nOne can solve the problem by calculating the ratio of the frequency of occurrence of getting heads to the frequency of all coin throws.\nThis is the answer that most people will accept and stop thinking about as a result of getting there.\nHowever, is this insight useful, I guess the better researchers would doubt that as that insight does not yield any new value?\nSo what does the researcher can do?\nWell, one way they do this is to improve on the question to elicit a different response/insight into the understanding of the problem. What do I mean by this?\nOne way is via splitting, like\nAsk what is the probability of getting heads given the last throw is a tail. Ask what is the probability of getting heads given the last throw is ahead. Thus ask whether the events of getting a head and a tail are independent events or dependent.\nThere is another way that is even more nuanced but I discovered this approach by observing top-notch, world-class researchers, as it leads to very deep insight into the problem.\nIn this case, the researcher might be interested in: What does this tell me about the coin itself, is it a fair coin? What information do you need to determine that?\nOkay now, let’s say you find some information. Let’s say you know the weight distribution or even magnetic properties of the coin, and by brute force, like creating a mathematical model to capture all the nuances in coin-throwing, this includes calculations of gravity, coin shape, the throwing speeds, and so on, you can eventually get to building a mathematical model that describes whether makes the coin fair given all many different parameters.\nThere is indeed a paper using this approach on dice throwing, where they indeed model the shape of the dice, how it touches the ground surface and thus model how it moves on the ground, etc, then come up with a math model to calculate the probability of the dice throw. (LOL, I remember reading that years ago by some group of Beijing researchers but I forget where I saved that paper)\nBut is this a good way? (I doubt so)\nWell, the world-class researcher may ask, is that information necessary to determine whether or not that is a fair coin?\nIn this case, the researcher is not looking for that way of solving the problem as that is a trivial approach and low-hanging fruit that surely any mediocre researcher can do.\nWell, the researcher is interested in looking for a method that can solve the problem with the least amount of information and clutter, which is an elegant way of solving problems (at least in mathematics).\nSo instead the researcher can redefine the problem, by questioning the nature of a fair coin itself.\nFor example, He can define a set of functions in which those functions outputs are the probabilities of coin throwing itself, which by inserting a variable (i.e., the #no. of the throw), can output the probability of getting heads in that (#no. of throw).\nBy knowing what a fair coin is (heads and tails must be equal) and an unfair coin (either all heads or tails). Then there can be a distribution created that captures the probability of these functions happening, usually one will start with a normal distribution. (but to alter it usually involves the use of statistics, which I am not going to discuss here, as it involves sampling). In other words, he created a way (a distribution) to quantify the probability of whether the coin is fair or not.\nThen simply by the given results initially, the given samples of coin throwing, he can quantify whether the coin is fair or not based on this distribution, with the assumptions laid down given his approach is rigorous enough.\nThe problem-solving strategy I put here only contains examples that are pertinent to mathematics as my interests lie in mathematics.\nI am not entirely sure, whether improving the questions you are asking, is a well-defined practice in other fields of research but it is a very well-defined practice in mathematics.\nAnd I do think it does extend to other fields, if not already adopted. Hope this is insightful.\n","permalink":"https://13ryanC.github.io/posts/blog/good_questions/","summary":"How iteratively sharpening questions—illustrated by a simple coin-toss problem—leads to deeper insight, more elegant models, and better research.","title":"Improving Question Asking in Research"},{"content":"I\u0026rsquo;m an independent AI safety researcher. I work on preventing model misalignment, evaluating emerging risks, and developing reliable safeguards.\nMy recent work includes a poster at the ICML 2026 Hypothesis Testing Workshop and a study carried out during the Singapore AI Safety Hub (SASH) Fellowship. I\u0026rsquo;m also a research mentor for the Fall 2026 SPAR cohort.\n","permalink":"https://13ryanC.github.io/about/","summary":"About Bryan Chan, an independent AI safety researcher.","title":"About"},{"content":"","permalink":"https://13ryanC.github.io/research/","summary":"Papers, reports and ongoing work on AI safety.","title":"Research"}]