Pico's News · Sunday 2026-11-22 · Edition 1
AI helpers still don't know when to stop
AI helpers often don't stop when they should. AI repeats fake news but can't hack alone. Pico's new Bullshit Meter finds big AI bosses' claims mostly stretched or hype.
Four papers accepted since late August converge on agent self-regulation: no model exceeds 60% paired act/abstain accuracy; activation steering, rubric-triggered compaction and retrieval-aware fine-tuning cut overacting, cost and failure. Peer-reviewed evidence backs news that chatbots amplify misinformation, not that AI hacks autonomously. New Bullshit Meter: seven quotes from six AI leaders scored deterministically against peer-reviewed studies and official records; Hassabis's AGI timeline, Amodei's safety pledge and Altman's 'smarter than scientists' score worst, Pichai's checkable ranking claim best. The risk index sits near baseline, highest for physical harm; research-based forecast ranges are in Editor's notes.
The 60-second version
- AI helpers are good at doing, but not at knowing when to stop.
- A new trick helped a coding AI solve more tasks with fewer steps.
- Letting AI decide when to shrink its notes worked better and cost half as much.
- Fact check: AI does repeat fake news, but it can't yet hack well on its own.
- Pico checked what AI bosses said. Big claims about super-smart AI are not backed by tests.
Research papers
Xun Liu et al. (8 authors) · NeurIPS 2026 (oral) · 2026
A test of whether AI helpers know when NOT to do something, like stopping or asking first.
What they found
- No AI got both tasks right more than 60% of the time. The average was 46%.
- AI helpers were good at doing tasks (81%) but worse at knowing when to stop (59%).
- Being good at doing things did not make an AI good at holding back.
Why it matters for builders
- For every job you test, also test a version your AI should refuse.
- Before letting an AI work alone, check how well it stops when it should.
Caveats
- The test worlds are simpler than real ones, with one helper and one turn.
- Each task ran only once, and an AI judged the close calls.
Topics: evaluation, reliability, tool use, agent architecture
Yuan Sui et al. (9 authors) · NeurIPS 2026 (poster) · 2026
Gently steer a coding AI's 'thoughts' so it stops thinking too long or acting too fast.
What they found
- One model solved 57% of tasks with TACT, up from 51%.
- On a coding test it solved 73% instead of 67%, and used fewer steps.
- Steering in random directions made things worse, so the found directions matter.
Why it matters for builders
- If you run your own open AI model, this is a cheap way to tune how it behaves.
- Look for 'thinks too long' and 'acts too fast' in your AI's records.
Caveats
- It only works on open models where you can see inside.
- I read it through a tool that can summarise, so check the numbers in the paper.
Topics: ai for se, reliability
Tianjian Li et al. (8 authors) · NeurIPS 2026 (poster) · 2026
The AI decides for itself when to shrink its notes, instead of on a fixed timer.
What they found
- On maths, it scored 47, against 40 for a timer and 33 for no shrinking.
- It cost about half as much per question.
- Shrinking on a timer often turned right answers into wrong ones.
Why it matters for builders
- If your AI runs out of room, shrink notes when a check says so, not on a timer.
- Add summaries at the end so the AI can reuse earlier work and stay fast.
Caveats
- Only tested on open models; bigger ones may not need it.
- I read an early copy, so the final numbers may be a bit different.
Topics: context engineering, memory, cost latency
Thomas Palmeira Ferraz et al. (5 authors) · EMNLP 2026 (main) · 2026
Show the AI a few of its past attempts at similar jobs, and train it to use them.
What they found
- Just showing past attempts raised success from 4% to 64% on a household-task test.
- On hard tasks, normal training got 21%; training with past attempts got 90%.
- It helped on a science-task test too, for all models but one.
Why it matters for builders
- Before retraining your AI, first try showing it its past attempts.
- If you do retrain, show past attempts during training too.
Caveats
- The saved attempts were mostly perfect examples, in just two text games.
- It only helps if the saved attempts match the new job well.
Topics: memory, self improvement, tool use
Mix-and-match ideas
- Test what your AI should refuse: For every normal test, add a twin test where the right answer is to stop or ask.
- Shrink notes smartly, keep good examples: Before the AI shrinks its notes, save the full version as a past example it can look up later.
Speculative news (not peer-reviewed)
References
- X. Liu, Y. E. Zhang, V. Kasprova, et al. AgentAbstain: Do LLM Agents Know When Not to Act? NeurIPS 2026 (oral).
- Y. Sui, Y. Chen, Y. Li, et al. TACT: Mitigating Overthinking and Overacting in Coding Agents via Activation Steering. NeurIPS 2026.
- T. Li, J. Zhang, X. Wang, et al. Self-Compacting Language Model Agents. NeurIPS 2026.
- T. Palmeira Ferraz, R. Deffayet, V. Nikoulina, et al. Retrieval-Augmented LLM Agents: Learning to Learn from Experience. EMNLP 2026, pp. 1297–1335.
- Y. Zhu, A. Kellermann, et al. CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilities. ICML 2025, PMLR 267.
- Z. Wang, T. Shi, J. He, M. Cai, J. Zhang, D. Song. CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale. ICLR 2026.
- A. K. Zhang, N. Perry, et al. Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models. ICLR 2025.
- T. Eloundou, S. Manning, P. Mishkin, D. Rock. GPTs are GPTs: Labor market impact potential of LLMs. Science 384(6702), 2024.
- B. Wu, L. Liu. Young Employment Diverged Across Five Professional Services During the Spread of Generative AI. Societies 16(11):376, 2026.
- D. Acemoglu. The simple macroeconomics of AI. Economic Policy 40(121):13–58, 2025.
- E. Brynjolfsson, D. Li, L. Raymond. Generative AI at Work. Quarterly Journal of Economics 140(2), 2025.
- H. Jang, S. C. McGregor, L. Neill. Inaccuracies, omissions, and bullshit: 2024 U.S. elections through the eyes of ChatGPT. Information, Communication & Society, 2026.
- N. F. Liu, T. Zhang, P. Liang. Evaluating Verifiability in Generative Search Engines. Findings of EMNLP 2023.
- M. Makhortykh, M. Sydorova, A. Baghumyan, V. Vziatysheva, E. Kuznetsova. Stochastic lies: How LLM-powered chatbots deal with Russian disinformation about the war in Ukraine. HKS Misinformation Review, 2024.
- D. Guo, Q. Liu, et al. Are Your Agents Upward Deceivers? ICML 2026, PMLR 306.
- J. Wen, R. Zhong, A. Khan, et al. Language Models Learn to Mislead Humans via RLHF. ICLR 2025.
- X. Hu, Y. Jiang, Q. Hu, X. Jia. GAMBIT: A Gamified Jailbreak Framework for Multimodal Large Language Models. ACL 2026.
- X. Shen, Z. Chen, M. Backes, Y. Shen, Y. Zhang. 'Do Anything Now': Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models. ACM CCS 2024.
- S. Yao et al. ReAct: Synergizing Reasoning and Acting in Language Models. ICLR 2023.
- N. Shinn et al. Reflexion. NeurIPS 2023. Featured in edition 1.
- C. E. Jimenez et al. SWE-bench. ICLR 2024.
- W. Kwon et al. Efficient Memory Management for LLM Serving with PagedAttention (vLLM). SOSP 2023.
- Q. Zhang et al. Agentic Context Engineering. ICLR 2026. Featured in edition 1.
- W. Xu et al. A-Mem: Agentic Memory for LLM Agents. NeurIPS 2025. Featured in edition 2.
- C. Packer et al. MemGPT: Towards LLMs as Operating Systems. arXiv 2310.08560 (venue not checked).
- M. Cemri et al. Why Do Multi-Agent LLM Systems Fail? NeurIPS 2025. Featured in edition 1.
- Y. Zhu et al. Establishing Best Practices in Building Rigorous Agentic Benchmarks. NeurIPS 2025. Featured in edition 1.
- The National. 'No room for debate over human control of AI, says Sam Altman' (Adipec, Abu Dhabi), 2 Nov 2026.
- Meta Platforms. Q3 2026 earnings call transcript (official), 21 Oct 2026.
- S. Pichai. Alphabet Q3 2026 earnings: CEO remarks (official), 28 Oct 2026.
- TechCrunch. Full text of Dario Amodei's Nasdaq IPO-day speech, 19 Nov 2026.
- ForkLog. 'Musk calls Claude Opus 5.5 a masterpiece, admits Grok 4.8 lagging behind' (embeds the X post of 10 Nov 2026).
- Free Press Journal. ''It's soon': Demis Hassabis now expects AGI within three years' (reporting a WSJ Magazine interview), Oct 2026.
- TechCrunch. 'Sam Altman thinks AI will have novel insights next year', 11 Jun 2025.
- Gizmodo. 'Elon Musk predicts AGI by 2026. He predicted AGI by 2025 last year.'
- Daring Fireball. 'Amodei AI code claim chowder', 13 Mar 2026.
- C. Si, D. Yang, T. Hashimoto. Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers. ICLR 2025.
- C. Si et al. The Ideation-Execution Gap: Execution Outcomes of LLM-Generated versus Human Research Ideas. ICLR 2026.
- H. Wijk et al. RE-Bench: Evaluating Frontier AI R&D Capabilities of Language Model Agents against Human Experts. ICML 2025 (PMLR 267).
- G. Starace et al. PaperBench: Evaluating AI's Ability to Replicate AI Research. ICML 2025 (PMLR 267).
- J. S. Chan et al. MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering. ICLR 2025 (oral).
- L. Phan et al. (CAIS, Scale AI & contributors). A benchmark of expert-level academic questions to assess AI capabilities (Humanity's Last Exam). Nature 649:1139-1146, 2026.
- C. Lu et al. Towards end-to-end automation of AI research (The AI Scientist). Nature 651:914, 2026.
- Q. Hao et al. Artificial intelligence tools expand scientists' impact but contract science's focus. Nature 649:1237, 2026.
- K. Kusumegi et al. Scientific production in the era of large language models. Science 390:1240-1243, 2025.
- M. K. P. Jayatunga et al. How successful are AI-discovered drugs in clinical trials? A first analysis and emerging lessons. Drug Discovery Today 29(6):104009, 2024.
- J. Hoffmann et al. Training Compute-Optimal Large Language Models. NeurIPS 2022.
- P. Villalobos et al. Position: Will we run out of data? Limits of LLM scaling based on human-generated data. ICML 2024 (PMLR 235).
- R. Schaeffer, B. Miranda, S. Koyejo. Are Emergent Abilities of Large Language Models a Mirage? NeurIPS 2023 (oral).
- I. R. McKenzie et al. Inverse Scaling: When Bigger Isn't Better. TMLR 2023.
- C. Snell et al. Scaling LLM Test-Time Compute Optimally Can be More Effective than Scaling Parameters for Reasoning. ICLR 2025 (oral).
- A. P. Gema et al. Inverse Scaling in Test-Time Compute. TMLR 2025.
- A. Ho et al. Algorithmic progress in language models. NeurIPS 2024.
- T. Kwa et al. Measuring AI Ability to Complete Long Software Tasks. NeurIPS 2025.
- S. Nikolov et al. How is Google using AI for internal code migrations? ICSE 2025 SEIP.
- K. Grace et al. Thousands of AI Authors on the Future of AI. Journal of Artificial Intelligence Research 84, 2025.
- M. R. Morris et al. Position: Levels of AGI for Operationalizing Progress on the Path to AGI. ICML 2024 (PMLR 235).
- S. Armstrong, K. Sotala, S. S. O hEigeartaigh. The errors, insights and lessons of famous AI predictions - and what they mean for the future. J. Experimental & Theoretical AI 26(3), 2014.
- J. Wang, K. Huang, K. Klyman, R. Bommasani. Do AI Companies Make Good on Voluntary Commitments to the White House? NeurIPS 2025 Workshop on Regulatable ML (poster); AIES 2025 listing not confirmed.
- C. E. Leiserson et al. There's plenty of room at the Top: What will drive computer performance after Moore's law? Science 368(6495), 2020.
- Artificial Analysis. 'Gemini 4 Carbon is the new #1 on the Artificial Analysis Intelligence Index' (official), 28 Oct 2026. Checked 22 Nov 2026.
- Artificial Analysis. Gemini 4 Carbon model page (official). Checked 22 Nov 2026.
- Artificial Analysis. Claude Opus 5.5 vs Grok 4.7 comparison (official). Checked 22 Nov 2026.
- Anthropic. Responsible Scaling Policy version history (official). Checked 22 Nov 2026.
- Anthropic. 'Responsible Scaling Policy v3' (official), 24 Feb 2026. Checked 22 Nov 2026.
- Clay Mathematics Institute. 'Navier-Stokes announcement' (official), 11 Sep 2026. Checked 22 Nov 2026.
- OpenAI. 'Navier-Stokes solution' (official company publication), 8 Sep 2026. Checked 22 Nov 2026.