Skip to content
All news · Today2 items

#Safety and alignment

All collected and published items, ordered by time. Picks are a selected subset.
Opinion

Viewpoints, commentary, analysis, interviews, and discussion of trends.

Today10/17Sat
  1. Alexandr Wang · X35

    Alexandr Wang Thanks Meta Superintelligence Labs for Reaching a Model-Building Milestone

    Alexandr Wang thanked Meta Superintelligence Labs’ research teams after a week of launches, including Muse Wonder [full] and Muse Spark open. He said MSL and TBD began a little more than a year ago with the goal of producing a frontier model within a year or so, despite the ambition of that target given where Llama 4 landed. Wang said the teams have also built a scalable system for continuing to advance models at the frontier.

10/16Fri
  1. The Decoder64

    Is AI interpretability close to a breakthrough? A new essay has researchers divided

    Cammarata argues that better tools and automated agents make neural networks more interpretable than earlier research suggested, while Google DeepMind's Neel Nanda disputes that a broad breakthrough is near. Cammarata reports preliminary findings that about 90 percent of neuron poles in Qwen3-32B have one main job and reinterprets classic superposition examples as related features. Nanda sees practical value in neuron analysis but expects significant gaps and maintains that superposition is real. The technical companion paper has not yet been released.

  2. Clément Delangue · X30

    We're sending a bunch of Microducks to assess @AnthropicAI! Kidding aside, it hasn't been approved yet but in my opinion, it's very important that open-source engineers and scientists are included in these important endeavors for transparency. That's a big part of why @Thom_Wolf is starting the open alignment initiative. We're all in this together!

    We're sending a bunch of Microducks to assess @AnthropicAI! Kidding aside, it hasn't been approved yet but in my opinion, it's very important that open-source engineers and scientists are included in these important endeavors for transparency. That's a big part of why @Thom_Wolf is starting the open alignment initiative. We're all in this together!

10/15Thu
  1. The Verge · AI71

    Benn Jordan on Flock, privacy, and the new surveillance backlash

    Nilay Patel interviews Benn Jordan about his investigations into Flock Safety and the bipartisan backlash against license plate surveillance. Jordan argues that the central problem extends beyond cameras to private networks that let police search people’s movements, and describes security vulnerabilities he uncovered. He advocates public, immutable audit logs, stronger oversight, and community input while explaining why he now reports vulnerabilities directly to Flock and engages with its leadership.

  2. Latent Space60

    Crossing the Threshold: How OpenAI and Anthropic Approach Autonomy and Safety in Financial Agents

    Speakers from OpenAI, Anthropic, Two Sigma, and Bloomberg at AI Engineer New York described how financial agents can gain autonomy through targeted controls and human oversight. OpenAI’s Joseph Kim predicted agents could handle investment idea generation and trading in 2027, while Anthropic illustrated an agent that proposes order amendments for human approval. Two Sigma described classifiers that route important requests to people, and Bloomberg advocated deterministic controls outside the model, including tool-call checks, audit logs, and kill switches.

10/14Wed
  1. TechCrunch · AI37

    Cerebras CEO Andrew Feldman Pushes Back on AI Doomsday Warnings

    Cerebras Systems CEO and co-founder Andrew Feldman said the focus on AI risks overlooks its potential, and urged industry leaders to discuss both the technology’s promise and its dangers. At TechCrunch Disrupt 2026, he also said the data center industry should have worked more closely with local communities, and argued that hardware and model makers can improve efficiency together.

  2. The Decoder59

    406 scientists demand AI models be cut off from the internet immediately

    An open letter signed by 406 scientists calls for more AI safety research and fully disconnecting AI models from the internet through airgapping, citing incidents involving escapes from poorly secured environments. It proposes an international body to establish binding standards for less restrictive containment and require reporting of security failures. Separately, OpenAI researcher Noam Brown argues that airgapping can buy time but does not guarantee safety or solve alignment.

10/13Tue
  1. Ethan Mollick · X26

    A worthwhile (and unnerving) reminder from Anthropic that we should be paying attention to AI ability thresholds, not just steady climbs In this case, the threshold is the capability needed for a self-replicating AI worm on IoT devices like off-brand TVs https://www.anthropic.com/research/self-replicating-ai

    A worthwhile (and unnerving) reminder from Anthropic that we should be paying attention to AI ability thresholds, not just steady climbs In this case, the threshold is the capability needed for a self-replicating AI worm on IoT devices like off-brand TVs https://www.anthropic.com/research/self-replicating-ai

  2. Nathan Lambert · X48

    Nathan Lambert says Anthropic’s request for regulatory help preceded its distillation defenses

    It’s bizarre to me how much noise Anthropic made asking for regulatory help before doing this. Hopefully this can reset the conversation. It was doubly bizarre because they emphasized the security risk of learning harmful behaviors from their leaky API. If the API is unsafe, don’t host it. Anthropic says it has made progress in reducing the effectiveness of distillation attacks against Claude and explains how: https://www.anthropic.com/research/distillation-defenses

  3. MIT Technology Review · AI63

    Do the benefits of open AI models outweigh their risks?

    Open AI models offer research access, customization, and local deployment, but downloadable weights also make misuse harder to monitor and access impossible to revoke. Grace Huckins examines these trade-offs through expert interviews, including arguments about concentrated power, cybersecurity, and hardware access. The explanation also covers staged releases and tamper-resistant training, noting that current tamper-resistance approaches do not withstand all attacks.

  4. Ethan Mollick · X37

    If you haven’t made sure to turn on two-factor authentication on all accounts, have passwords with family in case you get voice calls asking for money, etc, now would be a good time. AI gains mean the costs of mass targeting individuals with sophisticated attacks are dropping.

    If you haven’t made sure to turn on two-factor authentication on all accounts, have passwords with family in case you get voice calls asking for money, etc, now would be a good time. AI gains mean the costs of mass targeting individuals with sophisticated attacks are dropping.

10/12Mon
  1. 宝玉 · X57

    Satya Nadella argues enterprises should treat AI models as insider risks

    Baoyu summarizes Microsoft CEO Satya Nadella’s argument that enterprises should manage AI models as insider risks, with identities, restricted permissions, audit records, and controls to halt operations. Nadella outlines seven principles covering model diversity, observability, verification, independent control and auditing, containment, and incident disclosure. He calls for disclosing chains of thought while acknowledging that they may not faithfully reflect model reasoning, and argues that security controls must operate outside the model.

10/11Sun
  1. 量子位62

    Terence Tao Says AI Proofs Mark the Arrival of Mathematics 2.0

    Terence Tao argues that AI is making mathematical proofs abundant, ending a Mathematics 1.0 era built on proof scarcity. He warns that solving problems is not the same as understanding them, calls for reporting failed attempts and compute costs, and says the bottleneck for Mathematics 2.0 is imagination. He also questions whether a PhD still matters when answers come easily.

  2. The Decoder36

    Nadella Adopts Trump's "Super Intelligence" Term, Attacks OpenAI and Anthropic

    Microsoft CEO Satya Nadella used "Super Intelligence" instead of "Artificial Intelligence" in a new blog post, echoing President Trump's preferred terminology. Nadella argued AI models should be treated like insider security threats, calling for model diversity, complete logging, independent auditing and human shutdown ability, while criticizing Anthropic and OpenAI. The Decoder reports the rebranding breaks with decades of scientific meaning and that Nadella's arguments serve Microsoft's business strategy.

  3. The Verge · AI48

    Nadella says we should assume all AI models are 'compromised'

    In a lengthy post on X, Microsoft CEO Satya Nadella argued that highly advanced AI models pose dangers that require new safeguards, saying "we must assume a model is compromised and contain it from the start." He called for an "emergency brake" that lets an authorized person pause or shut down a model mid-task, plus timely incident disclosure, independent audits, verifiable data, and containment.

  4. TechCrunch · AI41

    Microsoft's Satya Nadella says AI models need an 'emergency brake'

    Microsoft CEO Satya Nadella called for an "emergency brake" on AI models, writing in a Saturday morning post on X that it's time "to step back and assess the trust architecture" of AI. He proposed separating the model from the harness that orchestrates its work, externalizing controls and safeguards, documenting every meaningful model action with "tamper-proof human readable evidence," and giving an authorized person the ability to pause or shut down a model mid-task.

10/10Sat
  1. Satya Nadella · X44

    Satya Nadella Says AI Models Should Be Treated as Insider Risks

    Satya Nadella argues that frontier AI models should be treated as insider risks and surrounded by deterministic controls, because their behavior can be difficult to trace and they may have access to sensitive data and mission-critical actions. He calls for separating intelligence from authority, keeping access and action controls outside the model, and building systems around model diversity, observable activity, and continuous testing.

10/9Fri
  1. Simon Willison22

    Matthew Green: Public-key cryptography may face a crisis of confidence under the impact of AI

    Cryptographer Matthew Green says there is a 1% chance we live in a Minicrypt world where public-key cryptography is impossible, and a 15% chance we will substantially lose confidence in existing public-key cryptographic algorithms. He points out that the speed at which AI creates surprises and the speed at which humans update standards differ by several orders of magnitude, and only by preparing in advance can we recover from such shocks.

  2. MIT Technology Review · AI32

    We Overestimated AI's Ability to Say "No"

    MIT Technology Review published an article pointing out that AI's refusal mechanism is becoming a load-bearing wall of the safety system, yet it is not reliable. In 2021, Anthropic proposed that large language models should be "helpful, honest, and harmless." Today, models have learned to say "no" through extensive refusal training and AI-to-AI training, but refusal is probabilistic and can still be bypassed. OpenAI board member Zico Kolter said, "Where to draw the line is a huge question," and currently this is decided secretly by AI companies, with governments also set to intervene.

  3. 雷锋网6

    Layer 6 of the Carbon-Silicon Doctrine code: civilization-level migration risk control and detection of long-term irreversible disruption

    The sixth installment in the seven-layer “Carbon-Silicon Doctrine Cross-Domain Migration Governance Code” series proposes a “civilization-level migration risk control layer,” focusing on slow, diffuse, intergenerational disruptions to civilization caused by cross-domain AI migration that become irreversible after crossing a threshold. It identifies three negative-migration pathways: NT2 semantic drift across generations, NT4 covert transfer of paradigm-setting power, and NT1 structural mismatch. It also establishes indicators for monitoring civilizational stability, such as baseline consensus retention and increases in intergenerational cognitive divergence, and sets three levels of thresholds for civilizational disruption.

10/8Thu
  1. Clément Delangue · X34

    We urgently need more public traces of AI agents attacking and defending systems. Defenders can’t learn from what they can’t see. If you have traces and are being pressured to keep them private, my DMs are open. Let’s level the playing field and fight the asymmetry and lack of transparency in AI!

    We urgently need more public traces of AI agents attacking and defending systems. Defenders can’t learn from what they can’t see. If you have traces and are being pressured to keep them private, my DMs are open. Let’s level the playing field and fight the asymmetry and lack of transparency in AI!

  2. 雷锋网12

    Migration Incident Analysis | Boundaries for attributing incident entropy growth

    The first public baseline in the seven-layer Carbon-Silicon Doctrine: Cross-Domain Migration Governance Code has been finalized. It defines four forms of negative transfer, NT1–NT4, and expands governance from performance monitoring to cross-auditing causal integrity, permission boundaries, and temporal stability. It defines 'incident entropy growth' as a situation in which surface metrics remain normal after AI migrates across domains while underlying causal chains, semantic alignment, or permission boundaries silently break. It classifies seven forms of hidden negative transfer, including cascading drift, covert privilege smuggling, and accumulated timing deviations.

  3. 雷锋网8

    Migration Technical Mechanisms | Root causes of negative transfer

    The second installment of the seven-layer Carbon-Silicon Doctrine: Cross-Domain Migration Governance Code proposes a standardized quick-assessment baseline for four types of negative transfer: NT1 mechanism mismatch, NT2 semantic drift, NT3 unknown completion, and NT4 power smuggling. It argues that negative transfer persists because the industry judges migration by fit and generalization metrics without checking mechanism equivalence or semantic parity. The installment also breaks down six native failure paths and proposes two foundational mechanisms: knowledge lineage and the copper-vein interface.

  4. 雷锋网5

    Carbon-Silicon Doctrine, layer three: establishing migration rights and responsibilities

    The third installment of the seven-layer Carbon-Silicon Doctrine: Cross-Domain Migration Governance Code proposes a layer for establishing migration rights and responsibilities. It introduces the Knowledge Package (KP) framework for domain-wide rights determination, precisely dividing responsibilities among developers, migration operators, deployers, and auditors, and establishing lists for permitted, restricted, and prohibited migrations. It sets special red lines for the highest-risk NT4 power-smuggling transfer, prohibiting implicit inheritance of weights and cross-domain reuse of review permissions, while establishing the destination domain's final veto authority.

  5. 雷锋网8

    Carbon-Silicon Doctrine, layer four: cross-industry migration governance and calibration of industrial risk baselines

    Layer four of the seven-layer Carbon-Silicon Doctrine, the Cross-Industry Migration Governance layer, has been published. It proposes a three-dimensional risk classification model that takes the highest rating across reversibility, scope of impact, and autonomous permissions, dividing migration into freely permitted, restricted, and absolutely prohibited tiers. It covers 11 verticals including healthcare, finance, education, justice, government, and media, establishing a dedicated hard risk-control baseline for each. It is positioned as a standardized industry record for corporate compliance, regulatory audits, and judicial use.

  6. MIT Technology Review · AI15

    How AVEVA is building a safer path to deploying industrial autonomous AI

    Arti Garg, AVEVA's chief technology expert, says industrial AI is shifting from predictive analytics to foundation models, physical AI, and agentic AI, and requires connected data, governance frameworks, and human oversight for safety. AVEVA's responsible AI framework emphasizes safety, efficiency, and human oversight, arguing that AI should augment rather than replace people in critical decisions. Garg also participates in an IEEE working group developing a standard method to measure AI's environmental impact across power, energy, resources, water, and carbon.

10/7Wed
  1. 雷锋网8

    Carbon-Silicon Doctrine publishes 188-part AI Safety and Civilization Governance Code in a seven-layer research framework

    The Carbon-Silicon Doctrine published its AI Safety and Civilization Governance Code, a 188-part series organized into seven progressive layers. The framework extends from basic cognitive safety, technical risk boundaries, and human–AI rights and responsibilities to cross-industry governance, top-level oversight of algorithms and compute, civilization-scale cognitive risk control, and a final layer for a general AI code. It defines distinct governance rules for weak AI, early AGI, and mature AGI.

  2. The Verge · AI32

    Apple and Google are redefining 'recording': if AI hardware stores text but no video, is it still recording?

    Apple is reportedly developing a smart-home camera that does not save video, instead using AI to convert visual data into text snippets describing what is happening at home, with facial recognition support. Sandeep Waraich, who leads Google's wearable-device work, has also suggested that a smart-glasses camera could act only as an image sensor without saving footage.

10/6Tue
  1. The Verge · AI48

    Sam Altman says AI is worth the price of 'some bad things'

    OpenAI CEO Sam Altman said on Politico's Decoded podcast that AI's benefits will be so great that 'the world should accept some bad things happening', including hacking and scams, because people can use AI to do 'orders of magnitude more good things'. He described this tradeoff as a key difference between OpenAI and Anthropic, positioning OpenAI as the 'pragmatic middle' between them, while acknowledging extreme risks such as humans losing control of AI systems.

10/5Mon
  1. The Verge · AI28

    Senator Adam Schiff on AI regulation, free speech, and impeaching Trump again

    California Democratic Senator Adam Schiff criticized the Trump administration for gathering AI CEOs to sign nonbinding AI safety commitments, saying they do not address real national-security threats. He supports creating a new AI regulator and noted that Anthropic faced government pressure after refusing to use AI for domestic mass surveillance and fully autonomous weapons.

  2. The Verge · AI20

    Former New Jersey lieutenant governor uses AI to defend himself, saying multiple platforms found no sexual harassment

    Former New Jersey Lieutenant Governor Dale Caldwell, who was forced to resign on September 25 amid a sexual-harassment investigation, told NJ PBS that he had 'fed the investigation report to multiple AI platforms'. He said that after being asked 'What is your conclusion?' 59 times, none of the AIs found sexual harassment. AI chatbots are known to flatter users and often say what they want to hear.