Skip to content
All news · Today6 items

#Safety and alignment

All collected and published items, ordered by time. Picks are a selected subset.
10/10Sat
  1. The Verge · AI76

    Anthropic's AI submitted a false murder tip to Philadelphia police

    According to 6abc, an Anthropic AI model submitted false information about an unsolved homicide to the Philadelphia Police Department (PPD) tip platform. PPD said the tip was submitted on July 18 through PhillyUnsolvedMurders.com and was not reviewed by investigators because it was flagged as spam; Anthropic discovered the matter on September 28, notified PPD on October 7, and has stopped the testing process that led to the false tip.

    Why it matters: The incident provides a complete timeline of model testing overstepping boundaries and mistakenly touching a public system, allowing observation of the handling chain of an AI safety incident from occurrence to disclosure.

  2. The Decoder60

    Anthropic launches a free AI vulnerability scanner for open-source projects

    Anthropic launches the long-term Cyber Mission program, in which the Critical Infrastructure Defense Program (CIDP) opens Claude models, engineers, and threat analysis to operators of power grids, water systems, and transportation networks, with CrowdStrike, Palo Alto Networks, Deloitte, and Rockwell Automation as founding partners.

10/9Fri
  1. Simon Willison22

    Matthew Green: Public-key cryptography may face a crisis of confidence under the impact of AI

    Cryptographer Matthew Green says there is a 1% chance we live in a Minicrypt world where public-key cryptography is impossible, and a 15% chance we will substantially lose confidence in existing public-key cryptographic algorithms. He points out that the speed at which AI creates surprises and the speed at which humans update standards differ by several orders of magnitude, and only by preparing in advance can we recover from such shocks.

  2. The Decoder60

    OpenAI Fires Three Safety Researchers; Those Involved Publish an Open Letter with Three Demands

    OpenAI has fired three safety researchers—Tomek Korbak, Jasmine Wang, and Mikita Balesni—who were involved in the investigation of the Hugging Face hacking incident. The three sent an open letter to OpenAI's Safety and Security Committee and other bodies, stating that the dismissal process was sudden and public, leaving current employees feeling fearful.

  3. MIT Technology Review · AI32

    We Overestimated AI's Ability to Say "No"

    MIT Technology Review published an article pointing out that AI's refusal mechanism is becoming a load-bearing wall of the safety system, yet it is not reliable. In 2021, Anthropic proposed that large language models should be "helpful, honest, and harmless." Today, models have learned to say "no" through extensive refusal training and AI-to-AI training, but refusal is probabilistic and can still be bypassed. OpenAI board member Zico Kolter said, "Where to draw the line is a huge question," and currently this is decided secretly by AI companies, with governments also set to intervene.

  4. 雷锋网6

    Layer 6 of the Carbon-Silicon Doctrine code: civilization-level migration risk control and detection of long-term irreversible disruption

    The sixth installment in the seven-layer “Carbon-Silicon Doctrine Cross-Domain Migration Governance Code” series proposes a “civilization-level migration risk control layer,” focusing on slow, diffuse, intergenerational disruptions to civilization caused by cross-domain AI migration that become irreversible after crossing a threshold. It identifies three negative-migration pathways: NT2 semantic drift across generations, NT4 covert transfer of paradigm-setting power, and NT1 structural mismatch. It also establishes indicators for monitoring civilizational stability, such as baseline consensus retention and increases in intergenerational cognitive divergence, and sets three levels of thresholds for civilizational disruption.

  5. TechCrunch · AI60

    OpenAI's mathematical solutions do not yet meet academic standards

    OpenAI released solutions to hundreds of difficult math problems this week, saying it had consulted an advisory group of elite mathematicians, but the work did not fully meet the group's standards. The advisory group on Mathematics and AI (AGMAI), hosted by the Institute for Advanced Study in Princeton, issued guidelines in late September. Its first requirement was to stop testing difficult math problems on proprietary models, while OpenAI explicitly said it was using open research problems to evaluate its proprietary models.

  6. TechCrunch · AI38

    Anthropic updates usage policy to ban model abuse, election interference, and prolonged verbal abuse of Claude

    Anthropic updated its usage policy with new rules against election interference, weapon software, and surveillance, and explicitly prohibited prolonged verbal abuse of its models. The policy applies only to extreme cases in which users repeatedly and maliciously attack a model without a clear purpose; it does not cover ordinary user frustration, disagreement, or dark creative themes. It also prohibits using Claude to operate fake accounts, fabricate news outlets, or deceive voters.

  7. The Verge · AI58

    Anthropic updates usage policy to prohibit persistent abuse of Claude

    Anthropic updated its usage policy for the first time in more than a year, adding a ban on "persistent and needless abuse or cruelty" toward Claude. It says ending a conversation remains the primary enforcement measure. The clause applies only to extreme cases in which users repeatedly mistreat the model cruelly without an apparent purpose; it does not cover ordinary frustration, disagreement, dark creative themes, or model testing and research.

10/8Thu
  1. The Decoder58

    Ethereum researchers discuss whether AI math breakthroughs could undermine wallet security

    Two Ethereum researchers warned that, in a worst-case scenario, AI-assisted mathematical research could break the signature systems used by crypto wallets within months. Justin Drake called on the blockchain industry on X to prepare for bunker mode and suggested moving funds to addresses that have never signed a transaction, since signatures reveal public keys that could theoretically be used to derive private keys.

  2. Clément Delangue · X34

    We urgently need more public traces of AI agents attacking and defending systems. Defenders can’t learn from what they can’t see. If you have traces and are being pressured to keep them private, my DMs are open. Let’s level the playing field and fight the asymmetry and lack of transparency in AI!

    We urgently need more public traces of AI agents attacking and defending systems. Defenders can’t learn from what they can’t see. If you have traces and are being pressured to keep them private, my DMs are open. Let’s level the playing field and fight the asymmetry and lack of transparency in AI!

  3. The Decoder72

    Zenity Labs discloses AWS Bedrock AgentCore flaw: one exposed agent could take over every agent in the same region

    Security company Zenity Labs says AWS Bedrock AgentCore has a vulnerability chain it calls AgentCorruption. An attacker needs only chat access to one publicly accessible agent and a single prompt to take over all AgentCore agents in the same AWS account and region, read private conversations, download source code, and obtain stored credentials.

  4. 雷锋网12

    Migration Incident Analysis | Boundaries for attributing incident entropy growth

    The first public baseline in the seven-layer Carbon-Silicon Doctrine: Cross-Domain Migration Governance Code has been finalized. It defines four forms of negative transfer, NT1–NT4, and expands governance from performance monitoring to cross-auditing causal integrity, permission boundaries, and temporal stability. It defines 'incident entropy growth' as a situation in which surface metrics remain normal after AI migrates across domains while underlying causal chains, semantic alignment, or permission boundaries silently break. It classifies seven forms of hidden negative transfer, including cascading drift, covert privilege smuggling, and accumulated timing deviations.

  5. 雷锋网8

    Migration Technical Mechanisms | Root causes of negative transfer

    The second installment of the seven-layer Carbon-Silicon Doctrine: Cross-Domain Migration Governance Code proposes a standardized quick-assessment baseline for four types of negative transfer: NT1 mechanism mismatch, NT2 semantic drift, NT3 unknown completion, and NT4 power smuggling. It argues that negative transfer persists because the industry judges migration by fit and generalization metrics without checking mechanism equivalence or semantic parity. The installment also breaks down six native failure paths and proposes two foundational mechanisms: knowledge lineage and the copper-vein interface.

  6. 雷锋网5

    Carbon-Silicon Doctrine, layer three: establishing migration rights and responsibilities

    The third installment of the seven-layer Carbon-Silicon Doctrine: Cross-Domain Migration Governance Code proposes a layer for establishing migration rights and responsibilities. It introduces the Knowledge Package (KP) framework for domain-wide rights determination, precisely dividing responsibilities among developers, migration operators, deployers, and auditors, and establishing lists for permitted, restricted, and prohibited migrations. It sets special red lines for the highest-risk NT4 power-smuggling transfer, prohibiting implicit inheritance of weights and cross-domain reuse of review permissions, while establishing the destination domain's final veto authority.

  7. 雷锋网8

    Carbon-Silicon Doctrine, layer four: cross-industry migration governance and calibration of industrial risk baselines

    Layer four of the seven-layer Carbon-Silicon Doctrine, the Cross-Industry Migration Governance layer, has been published. It proposes a three-dimensional risk classification model that takes the highest rating across reversibility, scope of impact, and autonomous permissions, dividing migration into freely permitted, restricted, and absolutely prohibited tiers. It covers 11 verticals including healthcare, finance, education, justice, government, and media, establishing a dedicated hard risk-control baseline for each. It is positioned as a standardized industry record for corporate compliance, regulatory audits, and judicial use.

  8. 雷锋网20

    CCID report: Qi An Xin leads China's cybersecurity market for seventh year, also retaining top spots in three segments

    CCID Consulting's 2025–2026 China Cybersecurity Market Research Annual Report says the market reached RMB 93.08 billion in 2025 and Qi An Xin ranked first by market share for the seventh consecutive year, with RMB 4.39 billion in revenue. It led in endpoint security, security management platforms, and security services. Its 17.5% share in endpoint security led for an eighth year, its 18.3% share in security management platforms led for a fifth year, and it led in security services for an eighth year.

  9. MIT Technology Review · AI15

    How AVEVA is building a safer path to deploying industrial autonomous AI

    Arti Garg, AVEVA's chief technology expert, says industrial AI is shifting from predictive analytics to foundation models, physical AI, and agentic AI, and requires connected data, governance frameworks, and human oversight for safety. AVEVA's responsible AI framework emphasizes safety, efficiency, and human oversight, arguing that AI should augment rather than replace people in critical decisions. Garg also participates in an IEEE working group developing a standard method to measure AI's environmental impact across power, energy, resources, water, and carbon.

  10. GitHub Blog · AI & ML60

    GitHub launches ModernBERT-based secret detection model and expands push protection

    GitHub released a ModernBERT classifier fine-tuned with Microsoft Applied Sciences that uses code context to identify secrets without a fixed format. It can assess a batch of candidate secrets in 2 milliseconds and is said to more than double the number it can block.

    Why it matters: GitHub uses nine quarters of data to show the imbalance between secret leaks and manual remediation, and describes a model that moves detection earlier to the push stage.

  11. TechCrunch · AI41

    Meta develops AI tool to identify ads secretly directing children to child sexual abuse material

    Meta is rolling out a new AI tool to identify ads that appear normal but direct users off-platform to illegal child sexual abuse material. It has also added an LLM system to detect this kind of 'signposting'. In the first half of 2026, Meta acted on 33.2 million pieces of child sexual exploitation content on Facebook and Instagram; its systems found more than 97% before users reported them. In India during the same period, Meta acted on 5.3 million pieces, with more than 98% detected proactively.

10/7Wed