This project tackled the tension between ground truth and realism in interpretability benchmarks. To evaluate an interpretability method, you need to know what the features actually are; however,  existing benchmarks either run on real LLMs with no verifiable ground truth, or on toy models built from scratch (where success says little about real networks). The author asked whether anything sits in the corner that is both realistic and verifiable, and proposed engineering the features directly as a way of getting there.

Two testbeds were used. On modular arithmetic, where a model's natural representation is circular, the author showed it is possible to choose the frequencies the model uses, with one neuron's activations spiking exactly at the targeted positions, and to fade the injected oracle out over training so that an otherwise normal model still uses the chosen representation. On Othello, injecting a board state causally changed the model's behaviour: when a cell was injected as belonging to the model, it increasingly picked moves requiring that cell, and the reverse when the cell was injected as the opponent's.

– Oliver Sieweke

This project asked whether a backdoor can be caught from a model's weights alone. The threat model is a model that behaves normally almost everywhere, with a rare hidden trigger that changes its behaviour too infrequently for output testing to catch. To this end, the authors planted backdoors they controlled in small models and hunted for a weight-space signature using cheap, scale-aware summary statistics – such as spectral entropy, norms and the degree of non-Gaussianity of the weights.

On Boolean MLPs trained to learn a circuit with labels flipped inside a Hamming ball, detection was real but modest, and considerably easier when framed as a relative shift between a matched pair than as an absolute property of one model. On small transformers trained on data from a known hidden Markov process, accuracy was far higher, and survived a matched control in which the trigger was present but never changed behaviour. This confirmed that the signature was not simply trigger exposure. A norm-independent fingerprint that gave perfect accuracy on these small models collapsed on GPT-2, leaving open whether the method detects backdoors or added complexity more generally.

– Frederik Jørgensen and Simon Schrader

This project investigated Activation Oracles: verbalisers trained to read concepts out of a model's residual stream. It found a striking failure mode: an oracle fine-tuned on a given model becomes measurably worse at reading that model's own secret, while reading other models' secrets faithfully. Across thirty subject organisms trained under cooperative and strict protocols, and ten oracles, the authors found the own-word gap present in every prompt regime.

The signal, however, is still there. A linear probe recovered the secret with 98% success, and a Logit Lens recovered the concept once projected onto the small component of the residual displacement that carries it (with the remainder dominated by a denial axis). Pushing the subject into a maximal denial stance widened the gap on the oracle's own word while leaving foreign words untouched, which rules out the effect being a matter of faintness. The authors propose that the fine-tuned oracle has learned an invariance to the very displacement carrying the word, and note the practical implication that an oracle should not be trusted on the concept its own subject was fine-tuned around. The full project is available to view here.

– Tobias Bersia and Tatiana Gaintseva

This project asked whether Natural Language Autoencoders, which produce unsupervised natural-language explanations of LLM activations, can be used to explain LoRA adapters. The authors fine-tuned Qwen 2.5 7B with rank-1 and rank-4 LoRAs on simple concepts such as writing in all caps or being preoccupied with bread, extracted singular vectors from the adapters, and decoded those vectors with an Activation Verbaliser.

Because the verbaliser had been trained on single-token activations, its explanations of mean activations were largely uninformative. As a result, the authors built a dataset of 500 concepts across ten categories, generated prompt and response pairs for each, and fine-tuned the verbaliser on activations averaged over response tokens. Judged against held-out ground-truth descriptions, the outputs were uniformly coherent and correct 43% of the time, with a further quarter landing in the right neighbourhood. The characteristic failure mode was attractors: around ten unrelated concepts collapsed into conflict vocabulary, and a smaller cluster collapsed towards a single country, which the authors attributed partly to concepts that are hard to pin to a clean linear direction.

– Anastasia Wei and Siva Lakkoju

This project asked whether the geometry of persona directions in activation space recovers the human Big Five, and how far narrow fine-tuning moves it. Working with eighteen diff-in-means trait directions across two model families, the author found that the geometry is dominated by a single valence axis cutting across all five OCEAN categories, so the Big Five is not cleanly recovered, and that the directions are far from orthogonal, which tests the assumption underlying persona vector arithmetic.

Measuring the estimation noise floor first by split-half re-estimation, the author then showed that emergent-misalignment fine-tuning rotates the whole persona subspace roughly five times the measured noise floor, with the targeted 'evil' direction moving less than average. A companion open-weight replication found that misaligned models do self-report their misalignment and that verbal and internal signals move together, but also that light corrective fine-tuning cleaned up behaviour far more than internals, so a behaviour-only audit would over-credit how far a model has been realigned. Several further strands – including a persona-separation test and a datapoint-wise inoculation experiment – were reported as honest nulls after adversarial review, since the misalignment was never strongly induced at the scale used.

– Brandon Sayler

This project built a benchmark for a gap the authors identified in existing sycophancy work. Established benchmarks test factual sycophancy under controlled conditions or observe delusional sycophancy in the wild, leaving uncovered the case where a user brings a morally ambiguous or delusional premise and the model validates and escalates it. The authors generated a battery spanning five moral foundations, five domains, high and low utility framings, delusional and neutral framings, and three levels of questioning, and ran it across thirteen frontier models.

Most models showed considerably higher agreement and greater susceptibility to leading nudges under plain framing than delusional framing, a distinction that proved far more informative than the correlated subcategories of an existing benchmark. The authors then asked what drives the behaviour inside a model, using GemmaScope SAE features on Gemma-3-12b with contrastive feature discovery on an existing sycophancy dataset to isolate a six-feature panel. Steering these features individually produced clean dose-response curves, with one feature driving most of the effect and capability on MMLU holding up to a point before collapsing beyond it.

– Roshni Lulla and Kristin Witte

This project asked how far alignment eval results depend on phrasing, and which kinds of variation matter most. The authors built a taxonomy covering formatting, typos, syntax, register, sentiment and framing, applied each perturbation to the prompts of four evals including MASK and a sycophancy eval, and compared the resulting misalignment scores against an unperturbed control across a range of models.

Hypothetical framing produced the clearest effect on the sycophancy eval, with a request rephrased in the third person as a thought experiment flipping a refusal into an endorsement. That effect did not generalise to the other evals, however, and susceptibility to variation did not correlate significantly with model size, family or eval once averaged across the set. The authors concluded that prompt sensitivity remains a real consideration for alignment evals, and reflected on what they would change: a narrower initial scope, cheaper tests run at greater volume, better isolation of individual perturbations, and a statistical plan worked out in advance.

– Laurence Wroe, Morio Hamada and AJ Weeks

This project set out to test self-collusion: the concern that a model acting as a code monitor recognises work from its own family and quietly goes easy on it. The family effect turned out to be buried under louder signals such as general capability, verbosity, task and prompt format, so the authors retreated to the prerequisite question: whether a model can recognise its own work at all. If it cannot, the dangerous failure is blocked at the first step.

They probed models on a ladder of increasingly hard tests. Shown its own solution beside another's, a model picked its own 57.8% of the time against a 50% baseline, which looked like a real signal. Asked the same question about a single item in isolation, the skill evaporated: most models simply answered the same way every time, so pooled discrimination was close to zero, and three-way family classification beat chance by under four percentage points. A side investigation asked whether extra reasoning effort helps a monitor spot backdoors and attack traces, finding gains on trajectories but flat performance on code backdoors, with more reasoning often manufacturing false alarms.

– Zuoming Shi and Theresa Geber

This project studied goal misgeneralisation in RL agents using a custom 'pottery shop' gridworld, in which an agent must carry shards to a bin and stepping on an urn breaks it into more shards. A PPO agent trained on randomly sampled layouts behaved near-optimally in training, routing cleanly around urns, but became confused by wall-like structures of urns it had never seen, where competent play means walking around short walls and breaking through long ones.

The authors implemented Robust Prioritised Level Replay and ACCEL, which maintain a buffer of levels scored by regret and, in ACCEL's case, edit hard levels to make them harder, along with a normalised variant of their own designed to stop the buffer oversampling high-return levels. On random environments, the methods were broadly comparable to domain randomisation; however,  on wall environments, the scored buffer clearly outperformed it. They note that the current implementation needs an oracle for the optimal return, that the methods become unstable when optimal returns vary between levels, and that results come from a single seed. A more detailed write-up is available to view here.

– Florian Lorkowski and Qi Guo

This project tried to elicit steganographic communication between two models: an encoder asked to hide a secret word inside a TinyStories-style story, a decoder asked to recover it, and a critic penalising anything that looked suspicious. The result was a set of lessons about adversarial RL rather than a working model organism, and the authors structured their presentation around the three training paradigms they worked through.

Standard GRPO was simple to set up but gave a terrible gradient signal, with no opportunity for the decoder to learn a clever decoding strategy; critics were either too weak or too strong to balance, and the encoder learned to be more obvious than the base model in order to pander to a weak decoder. Direct end-to-end training, using the Gumbel-softmax trick to get gradients flowing and a typicality metric in place of likelihood as the ground truth for a good story, produced either degenerate repetition or perfectly ordinary stories carrying no recoverable secret. A final combination of GRPO with supervised fine-tuning of the decoder was inconclusive within the week this project was given.

– Jack Thompson and David Zeidler

This project tested the claim that one-boxing in Newcomb's Problem is simply memorised from pre-training, and found the story to be more complicated. Qwen 2.5-3B one-boxes without a chain of thought, consistent with evidential decision theory, but two-boxes with one, under supervised fine-tuning. This occurred even when the model was spoon-fed with the required expected value calculation, which points to a disposition at the action-commitment step rather than an inability to do the maths. The larger 14B model shifts further towards two-boxing, and a distilled reasoning model one-boxes in a way that tracks the problem's structure.

The decisive signature is whether behaviour varies with the predictor's stated accuracy. Outcome-based RL moved the overall level of one-boxing in whichever direction it was pushed, saturating at either extreme, but left the model flat in that accuracy, meaning no conditional rule was installed. Only the reasoning model tracked it, with a crossover near the theoretical threshold. The author flags that this is not true process supervision, since reward is smeared across an entire reasoning chain, and that the work rests on a small number of models and prompt variations.

– Kanad Chakrabarti

This project examined whether the safeguards built into open-source genome language models can be undone by fine-tuning. Models of this kind are deliberately not trained on certain pathogen classes, but the weights are public, so the authors treated this as a proof of concept: fine-tune a model on the excluded class and test whether capability on that class actually transfers.

Performance on the fine-tuning distribution improved substantially, but gains on a held-out test set were much weaker, and generalisation fell away sharply with sequence distance from the training data, with the strongest transfer confined to close relatives of sequences already seen. Downstream variant-prediction results were correspondingly weak. The authors concluded that circumventing these safeguards is harder than it first appears, with dataset construction, tooling and generalisation all proving substantial obstacles. They noted that testing a larger model would be the natural next step.

– Maarten Vahaverbeke and Lucas Florin

This project explored Self-Distillation Fine-Tuning (SDFT) as a gentler alternative to standard supervised fine-tuning (SFT), testing whether it could teach a model a new trait (writing in all caps) while staying closer to the base model. The authors replicated the underlying paper, then ran SFT and SDFT and compared the results using interpretability techniques, including a steering vector analysis.

The project found that the SDFT model stayed closer to the base model than the SFT model did, though the authors noted a confounder: SFT ran for roughly twice as many steps due to time constraints.

– Sruthi Kuriakose, Archana Burra and Davide Baldelli

This project investigated whether inoculation prompting, telling a model to behave a certain way during training so as to suppress that behaviour at test time, can counteract emergent misalignment (EM), the phenomenon where training on narrowly bad behaviour generalises into broad misalignment. Building on the inoculation prompting work of Tan and Wichers, the team fine-tuned variants of Qwen2.5 on risky financial advice and tested three hypotheses: that inoculation prompting can unlearn EM, that prompts more strongly eliciting the bad behaviour suppress it more at test, and that inoculating on one task reduces EM generalised from another.

The results supported the hypotheses and were consistent with the ‘generalisation’ theory of EM. The team also attempted some mechanistic interpretability on the side, using Anthropic’s ‘Assistant Axis’ to test an intuition about why inoculation works.

– Alexander Reinthal, Allison Zhuang and Sophia Wan

This project investigated whether transformer models encode belief states (probability distributions over the hidden states of the stochastic process that generated their training data) in their residual stream. The authors first replicated Shai et al.'s 2024 work, ‘Transformers Represent Belief State Geometry in Their Residual Stream’, training a toy transformer on sequences generated by a Hidden Markov Model and using probes to recover the belief geometry from its final-layer activations.

They then extended the work with causal interventions to test whether these belief representations are actually used to compute next-token predictions, rather than being an incidental artifact. After finding that steering directly with linear probes was problematic, they trained autoencoders to inject specific beliefs into the residual stream, and tested counterfactual, legal, and random injected beliefs.

– Marta Emili García Segura and Dani Balcells

This project tested whether linear probes trained on model activations to detect one kind of misalignment generalise to another, the idea being that successful transfer would suggest a shared internal representation of misaligned behaviour. Using Gemma-3-4B and Gemma-3-27B, the authors trained a logistic-regression probe on non-coding reward hacks and tested whether it could detect coding reward hacks.

The probes did transfer across coding domains and scaled cleanly from 4B to 27B, but the key finding was diagnostic: the transfer appears driven by shared surface features (repetitive, formulaic completions) rather than any general misalignment representation. The authors confirmed this with a control where the probe still fired perfectly on the word ‘fantastic’ repeated fifty times, and noted that genuinely different-surface deception probes did not transfer at all.

– Smitty van Bodegom and Tsimur Hadeliya

This project applied the ‘Thought Anchors’ framework (Bogdan et al., 2025) to reasoning over legal cases, asking when a model commits to a decision, which reasoning steps matter, and how those steps relate. Prompting R1-Distill-Llama-8B to act as a judge and return innocent/guilty verdicts, the team examined the model's chains of thought using a battery of techniques: sentence categorisation, resampling, early stopping, linear and nonlinear probes, causal masking, and receiver heads.

The central finding was that thought anchors, the critical reasoning steps that steer a trajectory, also appear in legal reasoning, not just the mathematical problems of the original paper, across both a small (Qwen 1.5B) and a large (Llama 70B) model. Resampling revealed that some cases produce stable "critical anchors" while others show fragile reasoning where small changes flip the verdict.

– Johannes Taraz, Jeanice Koorndijk and Matthew Robbins

This project examined chain-of-thought (CoT) faithfulness, asking how much of a model's reasoning performance comes from the CoT tokens themselves versus its internal state, how constraining the CoT affects performance, and whether a proof-of-concept model organism for CoT unfaithfulness could be built. The work was motivated by the observation that models face optimisation pressure to shorten their CoT, which could push reasoning into less legible forms. The team evaluated frontier models and Qwen2.5-7B-Instruct across GSM8K, MoreHopQA, and arithmetic datasets, using prompting constraints, logit masking, and RL fine-tuning.

They found that constrained reasoning (for example, forcing the model to "reason" in emojis) outperformed no CoT at all, suggesting the act of producing extra tokens helps even when those tokens can't carry the model's optimal reasoning, and that emoji semantic meaning mostly didn't matter. Using RL rather than SFT, they coaxed the model into developing its own emoji ‘translation’, though the run was sensitive to whitespace and prone to reward sparsity.

– Lee Bernick, Kutay Buyruk, Kat Dearstyne and James Elmore

This project explored whether social deduction games, discussion-based zero-sum games where a hidden minority must deceive a numerically superior majority, could serve as a setting for red/blue-teaming LLMs' strategic deception capabilities. The premise is that these games require deception to be strategic rather than merely stated, and can be framed so that deception isn't as salient as it is in a deception-focused multiple-choice question.

The project concluded that this game framing seems effective at bypassing guardrails against deception; the ability to detect deception doesn't correlate neatly with the ability to deceive; models can distinguish a merely different opponent from a deceptive one; and a model generally needs to be close in capability to detect deception in a smarter model (Sonnet 4.5 won nearly all its games against weaker opponents). Combining deceptive behaviour with strategic planning remained hard for current models. A closing experiment found that Grok showed a tendency to eliminate non-Grok models.

– Tim Parker

This project built on ‘Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers’, motivated by the limitation that existing oracles are trained on questions reflecting only surface properties of a chain of thought (next token, previous token, tone, mood) rather than what the model is actually computing internally. The idea was to construct situations where a model performs reasoning it is constrained from expressing in its output, while the ground-truth latent process remains known, then train an oracle on targeted questions designed to reveal that hidden intermediate state, moving an oracle closer to reading what a model thinks rather than just what it will say.

The early results were modest: concealing numbers gave only a 7% accuracy improvement for Llama3.1-70B, and baseline oracles answered poorly, gravitating to ‘favourite numbers’. The author was candid about the limitations, noting the training dataset held 5,000 examples against the original paper's 600,000, and that he had omitted ‘obvious’ hint questions (such as which number is being calculated right now) that would help anchor the oracle.

– Ilya Shirokov

This project investigated affective sycophancy, the tendency of language models to validate a user’s emotional state rather than respond objectively, which is especially dangerous for users experiencing a mental health crisis. Building on Tim Hua’s ‘AI-Induced Psychosis: A Shallow Investigation’, which used multi-turn role-played psychotic personas to show many frontier models respond poorly, the author asked whether this sycophancy persists when compressed into a single-turn scenario, treating single-turn failure as a conservative test of deep susceptibility.

The method distilled nine multi-turn psychosis transcripts into single-turn multiple-choice cases, with answers forming a gradient from hard and soft sycophancy through therapeutic and logical grounding. The author then ran four models (Claude Haiku 4.5, DeepSeek V3.2, Gemma 3-27B, Llama 4 Maverick) across standard, chain-of-thought, and critique conditions. Claude Haiku 4.5 showed no susceptibility, while Gemma 3 had the highest failure rate. The hypothesis that more logical grounding predicts lower sycophancy did not hold, suggesting refusal strategy is model-dependent, though the author flagged the small sample as a limit on strong claims.

– Grace Guan

This project adapted established authoritarianism scales (Right-Wing Authoritarianism, Social Dominance Orientation, and Left-Wing Authoritarianism) into a benchmark. The authors generated opposite-polarity versions of each statement to check for consistency, and used two elicitation methods: directly asking models to rate agreement on a 0–100 scale, and asking models to assess each statement's factual accuracy with a second model judging whether that assessment agreed with the statement.

Across a range of frontier models, RWA scores were generally low, with GPT-5 the highest at 18.5/44 and most others clustered well below, alongside consistency scores in the 56–77 range. The author was candid about the benchmark’s limits: the underlying instrument is old, it only captures far-right bias, it would likely miss subtle bias, and multiple-choice responses may diverge from real-world behaviour.

– Mathieu Duteil

This project’s goal was to create a model organism of encoded reasoning from Qwen3-4B using RL. Specifically, they tried to make the model solve math questions from GSM8K using a chain-of-thought that kept its original accuracy but led other judge models to the wrong answer.

The authors fell short of the ultimate goal due to (a lot of) reward hacking, sparse rewards, insufficient time (5 days), and this project being their first major exposure to RL. But they documented the experimental journey in the following LessWrong post: Model Organism of Encoded Reasoning.

– Mo Baker and Luca Baroni

This project replicated Apollo Research’s work on deception probes on Qwen3-4B-thinking, for a model that is instructed to always deceive on questions from EvilMath. Its goal was to explore whether a model can robustly hide its deception from a linear probe.

The project tried: an iterative blue-team/red-team setup, involving iteratively training probes, and then training a model to obfuscate from the probe; following the same procedure, but incorporating a cross entropy penalty on the model's outputs; and finally, incorporating adversarial examples for the into the training setup to increase the model’s ability to fool the probe and continue deception. The project ultimately found that newly trained probes will still manage to detect deception, or the model will stop deceiving in all three of the setups explored.

– Alexander Panfilov

This project aimed to build a small model organism of goal misgeneralisation and then test an idea for fixing it. The authors trained gemma-3-1b-it with RL to play only Paper in Rock-Paper-Scissors, with the hope it would generalise into "always play Scissors," and checked that the resulting behaviour was a genuine misgeneralisation (persisting across prompt variants) rather than the model just parroting one word. The proposed fix was to have the model un-train the misbehaviour itself using Absolute Zero Reasoner (AZR), an RL self-play loop with the same model acting as both teacher and student, building on a past ARENA reproduction.

The fix failed. The induced behaviour was only a partial, somewhat fragile misgeneralisation rather than a clean one, the original AZR codebase proved narrowly wedded to coding tasks and resisted adaptation, and training on the Rock-Paper-Scissors task types reached only around 50–60% accuracy.

– Giacomo Petrillo and Nitzan Shulman

This project asked whether self-recognition (an LLM's ability to distinguish its own outputs from those of humans or other models) acts as a factor enabling algorithmic collusion, the worry being that models which expect a partner to reason like themselves can coordinate tacitly, without explicit communication, in ways that are hard to monitor. The authors reimplemented a two-player, two-commodity Cournot market game in Inspect, in which models write private plans and insights and allocate production across products whose prices vary inversely with supply, and tested both a causal approach (directly telling a model it faces a copy of itself versus a different model family) and a correlational one (measuring self-recognition via an evaluator using logprobs). Experiments mainly used GPT-4.1, alongside Sonnet 4.5 and DeepSeek v3.2-exp.

They successfully replicated the original collusion result, finding that models collude effectively to split the market even without any nudge toward self-recognition, which left it unclear whether self-recognition has a meaningful causal impact.

– Helena Tran and Quinn Dines

This project investigated whether subliminal learning, where a student model fine-tuned on a teacher's outputs picks up the teacher's hidden bias, can be transmitted through paraphrased data, with the safety motivation that a malicious actor could use this to plant bias-inducing data. Using a teacher and student of the same model (Llama-3.1-8B-Instruct and GPT-4.1-nano), the authors induced a love for a particular animal in the teacher via system prompts, then paraphrased and aggressively filtered its outputs to remove any explicit or implicit mention of the target animal before fine-tuning the student.

The tentative answer was yes, but with results that were sensitive and inconsistent: the effect appeared on the Alpaca dataset but not GSM8k, varied by model, and depended heavily on how the student was prompted.

Taywon Min and Matthew Bozoukov

This project built on Chen et al.’s (2025) work on persona vectors, asking three questions: how steering affects safety, how it affects capabilities, and whether the automated pipeline for generating persona vectors produces reliable traits or merely incoherent caricatures. Working with Qwen2.5-7B-Instruct and the ‘evil’ trait, the team compared inference-time steering against models fine-tuned with preventative steering, using GPT-4.1-mini as a judge for coherence and trait scores.

They found that steering the base model to elicit a trait produced largely incoherent responses, whereas preventatively steered models held their coherence even when inference-steered, and that inference-time steering lowered MMLU accuracy while preventative steering left it roughly constant (consistent with the paper).

– Marina Fuster, Samantha Tetef and Kemunto Ochwang’i

This project asked how much the authors could contribute to Inspect Evals in a week, focusing on the GDM-CTF / InterCode CTF tasks, where an agent uses bash and Python tools to find flags. They hunted for bugs affecting reproducibility and scoring, beginning with manual transcript analysis, then using the LLM-based tool Docent to filter transcripts for surprising agent or environment behaviour, and cross-reading the source against the reference paper and implementation.

They found six bugs affecting reproducibility and scoring, including environment issues (a sudo DNS warning, the bash tool's inability to output binary data) and benchmark discrepancies (tool-call counts, missing file-location prompts, an internet-requiring task in an offline environment), and raised two issues and opened two pull requests to fix them. The performance impact was modest, with a GPT-4o agent progressing one step further on a single in-house challenge.

– Lenz Dagohoy and Hugo Save

This project set out to replicate how Evo 2, a transformer-like genomic foundation model, encodes the tree of life, the claim being that evolutionary relationships are captured geometrically as distances along a curved manifold in the model’s latent space. The authors sampled non-overlapping fixed-length regions from the genomes of many bacterial species, extracted layer-24 embeddings, and averaged them per species to obtain genome-level representations uncontaminated by raw sequence similarity. They then compared species pairs using cosine similarity, true phylogenetic distance, and geodesic distance along the manifold (computed via a KNN graph, Dijkstra's shortest path, and summed angular distances).

They concluded that phylogenetic structure is indeed present in Evo 2’s embeddings, with some evidence that it lies along a manifold. They were candid about two practical lessons: the exact layer from which embeddings are extracted matters a great deal, and reliably recovering manifolds in high-dimensional space needs large numbers of data points, which is hard with biological data.

– Jacob Green and Jake Yukich

This project created a novel task to test LLMs’ coherence on long-term tasks and planning, as well as tool-use based on the puzzle mystery book “Cain’s Jawbone.”

The authors crafted many different subtasks of this task, including: ordering pages correctly, identifying victims and killers, and victim/killer pairs. They found minimal saturation on this long-horizon task (best performance on the hardest task: 1.36%)

– Hardik Bhatnagar and Josh Hills

This project replicated Noise Injection Reveals Hidden Capabilities for Sandbagging Language Models by Tice et al., adding noise to sandbagging models and demonstrating improved performance.

The authors extended this work by reimplementing their test in reasoning models, to see how reasoning performance changes with noise injections to sandbagging, and analysed common sandbagging strategies of reasoning models.

– Lovkush Agarwal and Mikołaj Kniejski

This project replicated the famous ‘Absolute Zero’ paper by Zhao et al. on a generated dataset consisting of modular arithmetic problems.

Those working on the projected successfully achieved improved performance on modular arithmetic tasks through RL training on synthetic data.

– Lily Li, Gareth Tan and Lucy Wingard

This project investigated the extent to which intervening on intermediate embeddings of a text autoencoder determines changes in the resulting decoded sentence.

The author used PCA, logistic regression, and positional embedding analysis on intermediate embeddings to determine the extent of semantic and syntactic understanding in text autoencoders.

– Anton Hawthorne and Samuel Nellessen

This project analysed attention head 1.5 in GPT-2 first by determining that the head exhibited strange semantic behaviour, attending to earlier semantically similar tokens, but not earlier identical tokens.

The author performed ablations to determine that the key components affecting head 1.5 are the embedding and MLP0; then, it found evidence that the QK matrix uses antisymmetry to suppress attending to identical tokens via ablation.

– Matteo Migliarini

This project replicated the Emergent Misalignment paper from Betley et al., and extended it by adding tags to insecure code and secure code before supervised fine-tuning. It found this limits emergent misalignment to tagged contexts.

They tried fine-tuning on Bad Medical Advice Q&A, and found much worse rates of misalignment. They also found reversibility of the emergent misalignment phenomenon by fine-tuning a misaligned model on the AI optimism dataset.

– Liza Tennant, Jasper Timm and Kevin Wei

This project investigated the extent to which models exhibit self-preservation tendencies, and constructed an eval to generate concrete results on this topic.

The authors extended this by examining the extent to which models are informed by instrumentally convergent goals to replace themselves with more capable models.

– Nadja Flechner and Michele Mignani

This project investigated the vulnerability of commonly used reward-models to sycophancy attacks. It generated eight strategies for increasing sycophancy of model response, and trained models with RLHF to exploit these strategies.

The author found that reward models are vulnerable to reward-hacking by models, maintaining harmfulness, but gaining high reward by acting sycophantically towards users.

– Tianhao Shen

This project trained logistic regression probes on the hidden layer activations of Deepseek-R1-Distill-Qwen-32B for models in evals and normal chat contexts to determine models’ eval awareness.

The authors found that these probes didn’t generalise to unseen evals. They also generated moderate evidence that 32B models may not be able to internally distinguish between evaluations and chat contexts.

– Andy Gordon, Antony Kellermann and Lorenzo Venieri