“We need more transparency about alignment overfitting” by Nikola Jurkovic

Currently, the public has very little insight about the robustness of alignment methods used by AGI companies. It's possible that, when an AGI company notices some misaligned behavior (e.g. through an alignment evaluation, or through noticing real-world misbehavior), they conduct a principled investigation and address root causes of the misalignment. It's also possible that they directly try to train away the examples of misalignment, which could result in brittle alignment that does not generalize out-of-distribution. Without additional information from within AI companies about how they align their models (especially in response to egregious cases of misalignment, e.g. cybersecurity incidents), the public cannot adequately reason about risks from misalignment. Public evidence is consistent with companies overfitting on alignment methods The public evidence is consistent with multiple companies having no clear separation between the ways they notice misalignment and the environments they use to train misalignment away. Companies have made very few statements about whether companies train directly on their evaluation distribution or real-world examples of misalignment. As far as I can tell, this is what's publicly known: OpenAI: In the GPT-6 Astra System Card, they wrote that they built an eval in response to the Hugging Face [...] --- Outline: (00:55) Public evidence is consistent with companies overfitting on alignment methods (05:21) The lack of information is seriously hindering the public's ability to estimate misalignment risk The original text contained 2 footnotes which were omitted from this narration. --- First published: November 10th, 2026 Source: https://www.lesswrong.com/posts/JTB8JjQfmkHsJtBDs/we-need-more-transparency-about-alignment-overfitting --- Narrated by TYPE III AUDIO.