Machine Learning Street Talk (MLST)

A lot of people in AI treat evolution as a dumb fallback, basically random search for when you can't take a gradient. Akarsh Kumar thinks that is wrong. Selection hangs on to partial solutions, so mutations only need to be useful about 1% of the time for the search to keep making progress.Akarsh is a PhD student at MIT working with Phillip Isola, works with Sakana AI, and is first author of the Fractured Entangled Representation paper with Kenneth Stanley, Jeff Clune and Joel Lehman. He tells Tim Scarfe why the path a learner takes may shape the structure of what it learns, and why that is a different way of looking at intelligence from the statistical one.Most of the conversation is about ASAL, the method he led for searching whole spaces of artificial worlds. Rather than predict what a rule will do, ASAL runs the simulation and asks a foundation model what happened. Mapped across all 262,144 Life-like rules, the most interesting worlds sit on one small island. Along the way: Game of Life, Lenia (with a clip from its creator, Bert Chan), neural cellular automata, Boids, Particle Life and the emergence of persistence.---TIMESTAMPS:00:00:00 Intro: artificial life, ASAL and Core War in four minutes00:04:19 Life as it could be, not just as it is00:05:46 Why the order you learn things in matters00:08:33 No shortcuts: Wolfram, novelty search and regularisation00:10:44 Kenneth Stanley: the path matters, not just the destination00:11:31 Statistical intelligence vs regularity-based intelligence00:13:49 Artificial chemistry and other possible universes00:15:59 Convergent patterns: shadows of the substrate?00:18:02 How Conway's Game of Life works00:20:49 Change one cell, change everything?00:22:31 Lenia (with Bert Chan) and neural cellular automata00:25:26 Boids, Particle Life and cell-like creatures00:28:47 Persistence, entropy and what life is00:30:15 ASAL: a foundation model as the critic00:34:04 Impostors inside the simulation00:36:50 From primordial soup to alien animals00:38:26 262,144 rules and the island of open-endedness00:40:17 From artificial life to AGI00:41:35 Core War: Game of Thrones, Turing edition00:43:22 LLMs as the mutation step: evolving warriors00:46:03 Evolution is anything but random---REFERENCES:paper:[00:04:19] ASAL (Kumar et al.)https://arxiv.org/abs/2412.17799[00:08:08] Assembly theory https://www.nature.com/articles/s41586-023-06600-9[00:09:54] FEP paperhttps://arxiv.org/abs/2505.11581[00:22:38] Lenia: Biology of Artificial Life (Bert Chan)https://arxiv.org/abs/1812.05433[00:23:20] Growing Neural Cellular Automata (Mordvintsev et al.)https://distill.pub/2020/growing-ca/[00:41:35] Digital Red Queen: Core War with LLMs (Kumar et al.)https://arxiv.org/abs/2601.03335[00:43:42] MAP-Elites https://arxiv.org/abs/1504.04909[00:44:55] AlphaEvolve https://arxiv.org/abs/2506.13131[00:47:03] AutoML-Zero (Real et al.)https://arxiv.org/abs/2003.03384book:[00:07:04] Why Greatness Cannot Be Planned (Stanley and Lehman)https://link.springer.com/book/10.1007/978-3-319-15524-1tool:[00:18:12] Conway's Game of Lifehttps://en.wikipedia.org/wiki/Conway%27s_Game_of_Life[00:25:27] Boids (Craig Reynolds)https://www.red3d.com/cwr/boids/[00:27:12] Particle Life (Tom Mohr)https://github.com/tom-mohr/particle-life[00:41:50] Core Warhttps://corewar.co.uk/other:[00:08:49] Computational irreducibility (Stephen Wolfram)https://www.wolframscience.com/nks/p737--computational-irreducibility/[00:10:46] MLST: Why Every AI Model Is an Impostor (Kenneth Stanley, FER documentary)https://www.youtube.com/watch?v=o1q6Hhz0MAg[00:28:30] MLST: Blaise Agüera y Arcas on life emerging from codehttps://www.youtube.com/watch?v=rMSEqJ_4EBk---LINKS:Akarsh Kumar: https://akarshkumar.com/ASAL project page and demos: https://pub.sakana.ai/asal/Digital Red Queen project page: https://sakana.ai/drq/RESCRIPT:https://app.rescript.info/public/share/Qfv3T0EVzqOXL_CYYeRr9Blv8HnNm4JsXBh7ByFjJbc

Tsung-Hsien (Shawn) Wen, CTO of PolyAI, tells Tim Scarfe why voice agents are harder than text agents. Voice adds time, and a good conversation depends on adapting to the person on the line, not just on reasoning to the best answer. Shawn describes an audio-native model (Dialog-RSN-1) that first predicts a turn-taking signal, then replies in text with citations, and writes the transcript last so enterprises can audit it.Along the way: training on real, noisy calls with synthetic noise added, and why over-cleaned audio made the new model worse. Latency, and what a voice agent should do while it thinks. Why a voice with a hint of regional accent beats a generic one. Why public benchmarks fall short for voice, why enterprises want to own their agent harness, and whether behaviour belongs in the harness or in the weights.The last stretch is about working with agents: cognitive debt, the shift from producing content to checking it, Wispr Flow, building tools that agents can use, and whether slop is in the eye of the reader.This episode was produced in partnership with PolyAI.https://poly.ai CHAPTERS0:00 Why voice agents are harder than text4:27 What enterprises want, and why PolyAI built its own model9:04 How an audio-native voice model works15:21 Training data, spectrograms and synthetic noise20:36 The cocktail party problem and the future of turn-taking25:21 Latency, adaptive reasoning and keeping callers' trust31:33 Voices, personality and the uncanny valley36:32 How do you benchmark a voice agent?42:00 Harness engineering and owning the intelligence45:09 Well-specified problems and auditable agents49:46 Weight adaptation and cognitive debt56:31 Agents at work: Wispr Flow, voice and tool building1:02:50 The next decade of voice, and what counts as slopREFERENCESThe Bitter Lesson: http://www.incompleteideas.net/IncIdeas/BitterLesson.html [9:05]Retrieval-augmented generation: https://arxiv.org/abs/2005.11401 [13:23]Mel scale: https://en.wikipedia.org/wiki/Mel_scale [17:16]Victor Zue: https://en.wikipedia.org/wiki/Victor_Zue [17:45]Cocktail party effect: https://en.wikipedia.org/wiki/Cocktail_party_effect [20:39]Speaker diarisation: https://en.wikipedia.org/wiki/Speaker_diarisation [21:16]

Leonardo de Moura created Lean and co-created Z3. ---This episode is sponsored by Parallel.Parallel, where agents find answers: web search, extraction and deep research APIs built for AI agents.Start free with the Parallel MCP server and $5 of credits every month: https://parallel.ai/mlst?utm_source=creator&utm_medium=podcast&utm_content=MLST---Tim Scarfe talks with Leo about how Lean escaped its original audience, why dependent types and Mathlib made it useful to working mathematicians, and what happens when formal verification leaves the lab. De Moura explains the small trusted kernel and independent checkers, and gives his account of the recent Collatz incident, in which a purported proof was accepted by both Lean's official kernel and Nanoda, apparently by exploiting a different bug in each.---TIMESTAMPS:00:00:00 Cold open: the green checkmark can lie00:00:59 Cathedral or bazaar: who controls Lean's core?00:04:55 Why Lean's core stays small and protected00:08:07 The Slack purge, Brandolini's law and the Lean FRO00:11:12 The Collatz exploit: two kernels, two bugs00:16:44 More kernels, reward hacking and safety by transparency00:21:04 Sponsor: Parallel00:21:59 Kim Morrison, Claude and the zlib proof00:25:25 Can we specify complex systems?00:28:04 Specs change: proofs are cheaper to redo with AI00:31:27 From Lean 1 to Lean 400:34:57 Dependent types in plain terms00:37:25 Lean 4's extensibility and Mathlib's growth00:41:44 Mathlib as infrastructure: Formal Frontiers00:44:15 Creativity, abstraction and nut-sniping00:48:46 Breadcrumbs, not learning: what AI agents lack00:53:03 Competence without comprehension, and verified guardrails00:56:21 Is the human still the author?01:01:44 AlphaProof, LLMs and why certificates still matter01:06:38 What's next for Lean, and its legacy01:11:42 How to start learning Lean---REFERENCES:tool:[00:00:48] Leanhttps://lean-lang.org/[00:01:24] Mathlibhttps://github.com/leanprover-community/mathlib4[00:12:09] nanoda_libhttps://github.com/ammkrn/nanoda_lib[00:12:19] CollatzLeanhttps://github.com/xrchz/CollatzLean/blob/a79357462a33d2a6babd4cf6c8d8bcd25425d653/README.md[00:13:06] Lean issue 14576https://github.com/leanprover/lean4/issues/14576[00:13:21] Lean pull request 14577https://github.com/leanprover/lean4/pull/14577[00:13:45] nanoda_lib pull request 22https://github.com/ammkrn/nanoda_lib/pull/22[00:15:33] Lean comparatorhttps://github.com/leanprover/comparator[00:18:02] Lean4Leanhttps://github.com/digama0/lean4lean[00:19:50] ARC-AGI-3https://arcprize.org/arc-agi/3[00:21:59] lean-ziphttps://github.com/kim-em/lean-zip[00:29:45] CompCerthttps://compcert.org/[00:29:45] seL4https://www.sel4.org/[00:30:23] Z3https://github.com/Z3Prover/z3[00:38:10] Veilhttps://github.com/verse-lab/veil[00:38:10] Velvethttps://github.com/verse-lab/velvetorganization:[00:00:52] Lean FROhttps://lean-lang.org/fro/[00:42:39] Mathlib Initiativehttps://mathlib-initiative.org/about/person:[00:05:11] Ilya Sergeyhttps://ilyasergey.net/[00:10:31] Joachim Breitnerhttps://www.joachim-breitner.de/[00:32:58] Adam Chlipalahttps://adam.chlipala.net/[00:38:42] Kevin Buzzardhttps://www.ma.imperial.ac.uk/~buzzard/[01:10:16] Terence Taohttps://terrytao.wordpress.com/book:[00:08:19] The Proof in the Codehttps://us.macmillan.com/books/9780374620059/theproofinthecode/other:[00:10:05] Brandolini's lawhttps://en.wikipedia.org/wiki/Brandolini%27s_law[00:45:33] A new result on unit distanceshttps://openai.com/index/model-disproves-discrete-geometry-conjecture/[01:01:08] Fermat's Last Theorem formalisationhttps://imperialcollegelondon.github.io/FLT/paper:[00:34:36] The Lean 4 theorem prover and programming languagehttps://doi.org/10.1007/978-3-030-79876-5_37---RESCRIPT:https://app.rescript.info/share/7d3d4a0059443236a01f6c9acbf4db58https://app.rescript.info/api/public/sessions/b007264c0ce89047/pdf

Weco let an AI coding agent rewrite the harness around another agent for eight days: its code, prompts and tools, while the underlying language model stayed fixed. Tim Scarfe asks Weco co-founder Zhengyao Jiang what the reported gains over two years of human engineering actually demonstrate.The discussion examines AIDE 85's generated code, held-out evaluation and the difficulty of separating useful discoveries from reward hacking. Jiang explains Weco's four levels of recursive self-improvement and compares the experiment with AlphaEvolve and the Darwin Gödel Machine.The limits matter as much as the gains. Jiang explains why the experiment did not establish that the system had become a better improver. The conversation closes with open-ended search, human-designed primitives and Parameter Golf: where does the next useful idea come from when the agent is searching inside a space that people designed?---TIMESTAMPS:00:00:00 Eight days of self-improvement: what counts?00:03:25 AIDE and the puzzle of useful spaghetti code00:08:38 Four levels of recursive self-improvement00:12:02 What AIDE 85 changed and how it was tested00:20:04 AlphaEvolve, Darwin Gödel Machine and the RSI claim00:26:21 Reward hacking and the limits of detection00:33:09 Open-ended search, harness tuning and creativity00:39:43 Parameter Golf and the limits of self-improvement---REFERENCES:organization:[00:00:30] Weco AIhttps://www.weco.ai/other:[00:00:33] AIDE²: The First Evidence of Recursive Self-Improvementhttps://www.weco.ai/blog/first-evidence-of-recursive-self-improvement[00:14:11] Faulty reward functions in the wildhttps://openai.com/index/faulty-reward-functions/[00:29:59] The Hugging Face incident and the road aheadhttps://openai.com/index/hugging-face-incident-and-the-road-ahead/tool:[00:03:29] AIDEhttps://github.com/WecoAI/aideml[00:04:29] MLE-benchhttps://github.com/openai/mle-bench[00:04:33] ALE-Benchhttps://github.com/SakanaAI/ALE-Bench[00:04:52] WeatherBench 2https://github.com/google-research/weatherbench2[00:08:18] ReActhttps://react-lm.github.io/[00:39:43] Parameter Golfhttps://github.com/openai/parameter-golfpaper:[00:20:08] AlphaEvolve: A coding agent for scientific and algorithmic discoveryhttps://arxiv.org/abs/2506.13131v1[00:21:35] Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agentshttps://arxiv.org/abs/2505.22954v3[00:23:45] Hyperagentshttps://arxiv.org/abs/2603.19461v1[00:27:01] SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agentshttps://arxiv.org/abs/2605.21384book:[00:33:14] Why Greatness Cannot Be Planned: The Myth of the Objectivehttps://link.springer.com/book/10.1007/978-3-319-15524-1---LINKS:https://app.rescript.info/share/3a9dc6189cb539c6a05fcc4f75c101b3PDF:https://app.rescript.info/api/public/sessions/9eda60ede2b31c92/pdf

Frank Hutter, co-founder of Prior Labs, talks about TabPFN, a tabular foundation model that makes predictions in a single forward pass, and the research behind it.

TabPFN is pre-trained on synthetic datasets drawn from a prior over structural causal models, rather than on real data. At prediction time it takes the whole training table as context and outputs an approximation of the Bayesian posterior predictive distribution, without per-dataset training or hyperparameter search. Frank explains how this grew out of his earlier work on AutoML and neural architecture search, how the priors are built and revised, and why tabular data was hard for deep learning for so long.

The conversation also covers the TabArena benchmark, how the architecture changed from TabPFN v1 to v3, scaling to larger tables, using the model with coding agents, test-time compute, Google's TabFM, causal inference and interventions, and relational data. At the end, a short update Frank recorded after the interview covers the TabPFN-3.5 release.

Prior Labs:

TabPFN-3.5: https://priorlabs.ai/tabpfn-3-5

https://priorlabs.ai/careers

TOC:

00:00 Introduction

00:44 Welcome and Frank's background

02:05 Why tabular data was hard for deep learning

10:17 Pre-training on synthetic data

12:52 The TabArena benchmark

19:28 From AutoML to neural architecture search

26:34 TabPFN as a learned algorithm

30:50 Bayesian prediction in one forward pass

39:37 Scaling to larger tables

47:48 Using TabPFN with coding agents

57:47 Output heads and architecture from v1 to v3

1:05:29 Test-time compute and adaptation

1:13:32 Google's TabFM

1:16:53 How the priors are designed

1:18:40 Correlation, causation and interventions

1:35:22 Relational and multimodal data

1:38:31 Use in organisations

1:46:38 The open research arm

1:50:21 Update: TabPFN-3.5

REFS:

TabPFN v2, Nature (Hollmann et al., 2025)

https://www.nature.com/articles/s41586-024-08328-6

Transformers Can Do Bayesian Inference (Müller et al.)

https://arxiv.org/abs/2112.10510

TabArena (Erickson et al.)

https://arxiv.org/abs/2506.16791

AutoGluon-Tabular (Erickson et al.)

https://arxiv.org/abs/2003.06505

Beyond IID: How General Are Tabular Foundation Models, Really?

https://arxiv.org/abs/2606.30410

Neural Architecture Search: A Survey (Elsken, Metzen & Hutter)

https://arxiv.org/abs/1808.05377

Auto-WEKA (Thornton et al.)

https://www.cs.ubc.ca/~hutter/papers/AutoWEKA-KDD2013.pdf

TabPFN v1 (Hollmann et al., 2022)

https://arxiv.org/abs/2207.01848

TabPFN-3 technical report

https://arxiv.org/abs/2605.13986

TabPFN-2.5 report

https://arxiv.org/abs/2511.08667

CAAFE (Hollmann et al.)

https://arxiv.org/abs/2305.03403

TabICL (Qu et al.)

https://arxiv.org/abs/2502.05564

TabICLv2 (Qu et al.)

https://arxiv.org/abs/2602.11139

Google TabFM

https://research.google/blog/introducing-tabfm-a-zero-shot-foundation-model-for-tabular-data/

TALENT benchmark (Ye et al.)

https://arxiv.org/abs/2407.00956

Do-PFN (Robertson et al.)

https://arxiv.org/abs/2506.06039

CausalPFN (Balazadeh et al.)

https://arxiv.org/abs/2506.07918

Causal Foundation Models with Partial Graphs (Reuter et al.)

https://arxiv.org/abs/2602.14972

RelBench (Robinson et al.)

https://arxiv.org/abs/2407.20060

RelArena-α, TabPFN-Rel and RPI

https://arxiv.org/abs/2608.16319

TabPFN on GitHub

https://github.com/PriorLabs/TabPFN

TabPFN-3.5 technical report

https://arxiv.org/abs/2609.17895

Otto Group Product Classification Challenge (Kaggle, 2015)

https://www.kaggle.com/competitions/otto-group-product-classification-challenge

---RESCRIPT:https://app.rescript.info/share/e99676c25ee6189fbf54c9be07eb623e

Alexander Mattick is a researcher at Fraunhofer IIS and a PhD researcher at the University of Technology Nuremberg (UTN), and a regular on Yannic Kilcher's Discord. He first came on MLST in 2022, after helping research the Yann LeCun and Randall Balestriero episode on interpolation.

SPONSOR:

---

Cyber Fund built the Monastery to help founders ship products that were impossible a year ago. Applications for Batch 1 are now open.

Apply now: https://cyber.fund

---

Alexander treats inference as the thread running through modern machine learning: once you have a model, what does it cost to get an answer out of it? He works through Monte Carlo, GFlowNets, energy-based models, diffusion, normalising flows and flow matching, with four short explainers he recorded himself. He is blunt about energy-based models: you can sample from them in principle, but it is rarely worth the compute. JEPA and "world model", he says, are closer to branding than to technical categories.

Next: theories of deep learning, none of which he thinks predicts enough yet to guide practice, then reinforcement learning.

---

0:00 Cold open: information is expensive

0:51 Welcome back, Alexander Mattic

2:08 Alexander's research background

2:50 Inference: densities, sampling and Monte Carlo

6:42 GFlowNets, energy functions and MCMC

9:45 Explainer: energy-based models

11:03 Why model a density at all?

17:30 From learned energies to flow matching

25:08 Explainers: diffusion and normalising flows

28:33 Are energy-based models generative?

33:22 JEPA, contrastive learning and collapse

41:13 Why non-language modalities need flows

44:51 Inference as search: branch and bound

49:43 Q-learning and delayed consequences

55:14 Flow matching, optimal transport, Fokker-Planck

1:00:03 Explainer: flow matching

1:01:49 AlphaFold, latents and scale versus architecture

1:07:52 Two families of deep learning theory

1:15:04 What a good theory would predict

1:23:53 The manifold hypothesis and compression

1:28:25 Is reward enough?

1:32:01 Control theory versus reinforcement learning

1:37:22 The Bitter Lesson and expensive information

1:42:08 Constrained RL: the constrained MDP toolbox

1:50:12 Creativity as constrained search

1:55:44 Reality is protean: when abstractions hold

2:00:32 What is a world model?

2:04:38 Prediction is not control

2:08:13 Robot demos, MPC and reliability

---

REFERENCES:

[6:55] GFlowNets (Bengio et al., 2021)

https://arxiv.org/abs/2106.04399

[38:46] Contrastive Self-Supervised Learning (Anand, 2020)

https://ankeshanand.com/blog/2020/01/26/contrative-self-supervised-learning.html

[38:56] LeJEPA (Balestriero and LeCun, 2025)

https://arxiv.org/abs/2511.08544v3

[47:10] RL for Node Selection in Branch-and-Bound (Mattick)

https://openreview.net/forum?id=0ez68a5UqI

[56:20] Flow Matching for Generative Modeling

https://arxiv.org/abs/2210.02747v2

[1:12:41] Disentangling feature and lazy training in deep neural networks

https://arxiv.org/abs/1906.08034v4

[1:31:05] Reward is enough (Silver)

https://doi.org/10.1016/j.artint.2021.103535

[1:35:12] Learning ReLU networks to high uniform accuracy is intractable (Berner et al.)

https://arxiv.org/abs/2205.13531v2

[1:40:20] Dota 2 with Large Scale Deep RL

https://arxiv.org/abs/1912.06680v1

[1:45:41] Constrained Update Projection for Safe Policy Optimization (Yang et al., 2022)

https://arxiv.org/abs/2209.07089

[1:46:11] SafeMPO (ICLR 2026)

https://openreview.net/forum?id=1m0EU6QXj6

[1:50:17] Why Creativity Cannot Be Interpolated

https://archive.mlst.ai/paper/why-creativity-cannot-be-interpolated/

[1:51:39] Invalid Action Masking (Huang and Ontañón)

https://arxiv.org/abs/2006.14171

[2:00:04] Probability Theory: The Logic of Science (Jaynes, 2003)

https://www.cambridge.org/core/books/probability-theory/9CA08E224FF30123304E6D8935CF1A99

[2:01:53] Training Agents Inside of Scalable World Models (Hafner et al., 2025)

https://arxiv.org/abs/2509.24527v1

[2:03:43] World Models (Ha and Schmidhuber, 2018)

https://arxiv.org/abs/1803.10122v4

The car making a left turn at the start of this episode was never filmed. Cosmos 3 generated it. Ming-Yu Liu, who leads the Cosmos research at NVIDIA, explains how one model can describe a video, generate one, and produce robot actions.

He walks Tim through the architecture. A vision language model reasons one token at a time; its weights then initialise a bidirectional diffusion generator for video, audio and action, and a shared temporal position scheme lines up signals that run at different rates. Ming-Yu treats "world model" as a set of tools, not one definition: forward dynamics, inverse dynamics and policy, trained together under a capacity limit so that each helps the others. He also explains why plentiful first-person human video carries over to robots, which have far less data of their own, and why a Cosmos model post-trained on the DROID dataset is a good starting point for pick-and-place policies.

The most practical thread is testing. A neural simulator does not need accurate success rates. It only needs to rank policy A above policy B the way the real world would, so a team can narrow down which checkpoints deserve a real trial. Cosmos Dreams applies that closed-loop idea to driving and robotics, and Ming-Yu argues that humanoids around children and pets make safety matter even more than it does for cars. The conversation ends on the Super, Nano and Edge sizes (Edge targets Jetson Thor, Orin and DGX Spark) and where to find the open weights, code and data.

This episode is a paid partnership with NVIDIA.

Learn more about Cosmos: https://nvda.ws/4cJoY1S

Explore Cosmos Lab: https://research.nvidia.com/labs/cosmos-lab/cosmos3/

---

TIMESTAMPS:

00:00:00 A road that was never filmed

00:02:28 Inside Cosmos 3: reasoning and generator towers

00:05:02 World models: dynamics, policy and one clock

00:08:59 Learning robot skills from human video

00:11:06 Ambiguous tasks and system 2 planning

00:12:53 Neural simulators for policy verification

00:16:41 Cosmos as a starting point for robot policies

00:19:00 Cosmos Dreams and robot safety

00:22:04 Super, Nano and Edge model sizes

00:24:24 Open models, the Cosmos repo and feedback

---

REFERENCES:

tool:

[00:00:13] Cosmos 3 (NVIDIA Cosmos Lab project page)

https://research.nvidia.com/labs/cosmos-lab/cosmos3/

[00:18:27] NVIDIA Cosmos GitHub repository

https://github.com/NVIDIA/cosmos

[00:22:05] Cosmos3-Edge model card

https://huggingface.co/nvidia/Cosmos3-Edge

[00:22:15] Cosmos3-Super model card

https://huggingface.co/nvidia/Cosmos3-Super

[00:22:16] Cosmos3-Nano model card

https://huggingface.co/nvidia/Cosmos3-Nano

[00:22:50] NVIDIA Jetson Thor

https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-thor/

[00:22:52] NVIDIA Jetson Orin

https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/

[00:22:53] NVIDIA DGX Spark

https://www.nvidia.com/en-us/products/workstations/dgx-spark/

[00:24:42] Cosmos 3 collection on Hugging Face

https://huggingface.co/collections/nvidia/cosmos3

other:

[00:01:07] Cosmos-Dreams closed-loop simulators (NVIDIA SIGGRAPH 2026 blog)

https://blogs.nvidia.com/blog/siggraph-news-2026/

paper:

[00:08:54] Cosmos 3: Omnimodal World Models for Physical AI

https://arxiv.org/abs/2606.02800

[00:17:43] DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset

https://arxiv.org/abs/2403.12945

---

RESCRIPT: https://app.rescript.info/share/e2385948cf465f0d6a2c0930150fc3ab

Pavan Kumar Reddy leads audio research at Mistral AI. He joins Tim Scarfe for a deep technical tour of Voxtral — and explains why the frontier of deployed voice is still a cascade of specialised models rather than one end-to-end system.

IN PARTNERSHIP WITH MISTRAL AI:

---

This episode was produced in partnership with Mistral AI.

Mistral AI: https://mistral.ai/

---

The conversation opens on architecture. Voxtral Chat feeds a 3B Ministral text trunk with continuous embeddings from an audio encoder, passed to the decoder as direct token input rather than through cross-attention as in Whisper, so the model can answer questions about emotion, timing and who spoke when without an intermediate transcript to lose them. The real-time model becomes a dual-stream decoder that consumes audio and emits text at once, at a target delay down to 160ms, with slower streams in parallel for anything that can wait for more context.

On generation, Pavan explains why Voxtral TTS predicts continuous latents rather than discrete codec tokens, traces the lineage from SoundStream through EnCodec to Mimi's split of semantic and acoustic codebooks, and places FSQ and flow matching in it. Tim presses on the priors underneath: why a mel spectrogram instead of raw waveform, what noise augmentation buys, and when acoustic overfitting becomes somebody's fine-tuning problem. Then the failure modes. Diarisation is emitted autoregressively inside the transcript rather than by a separate head, which makes streaming diarisation fragile — less context, late speaker changes, invented extra speakers. And because the architecture commits to what it has already predicted, one out-of-distribution mistake compounds into looping or skipped segments, which is what DPO corrects: the negative supervision pre-training and SFT cannot give.

The last third is the argument Tim keeps returning to. Customers running voice agents over millions of sessions describe scaffolding, not a solved problem, with a sharp drop outside the top few languages. Cascades survive because each component stays separately adaptable, observable and constrainable. And voice alone is cognitive debt: absorbing information and deciding in one serial stream is harder than glancing at a menu. Voice becomes ubiquitous beside a screen, not instead of one.

---

TIMESTAMPS:

00:00:00 Cold open

00:00:46 Why Mistral moved into audio

00:09:27 Inside Voxtral: trunk, encoder, dual streams

00:20:22 Speech that works in real time

00:30:52 How a voice becomes tokens

00:39:59 Flow matching, FSQ and the new codec

00:52:51 When speech models lose the speaker

01:03:23 Correcting hallucinations with preferences

01:12:12 Controlling synthetic speech

01:20:06 Why cascades still win

01:29:25 Speech in the wild

01:33:46 Audio models as interfaces

01:37:54 Why voice still needs a screen

---

REFERENCES:

paper:

[00:01:42] Mistral 7B

https://arxiv.org/abs/2310.06825

[00:09:38] Voxtral

https://arxiv.org/abs/2507.13264

[00:14:41] Whisper: Robust Speech Recognition

https://arxiv.org/abs/2212.04356

[00:19:11] Voxtral Realtime

https://arxiv.org/abs/2602.11298

[00:21:52] Delayed Streams Modeling (Kyutai)

https://arxiv.org/abs/2509.08753

[00:30:52] Voxtral TTS

https://arxiv.org/abs/2603.25551

[00:32:38] SoundStream neural audio codec

https://arxiv.org/abs/2107.03312

[00:34:59] Flow Matching for Generative Modeling

https://arxiv.org/abs/2210.02747

[00:37:03] EnCodec: High Fidelity Neural Audio Compression

https://arxiv.org/abs/2210.13438

[00:37:42] Moshi and the Mimi codec

https://arxiv.org/abs/2410.00037

[00:39:05] Finite Scalar Quantization (FSQ)

https://arxiv.org/abs/2309.15505

[01:03:33] Direct Preference Optimization (DPO)

https://arxiv.org/abs/2305.18290

dataset:

[00:46:14] Mozilla Common Voice

https://commonvoice.mozilla.org/en/datasets

organization:

[00:50:47] Hugging Face

https://huggingface.co/

Can a machine learn the judgement that separates a plausible-looking result from a faithful experiment? Edward Hughes, Chief Scientist and co-founder of Inherent, joins Tim Scarfe to argue that creativity is not optimisation, and that the missing capability in AI is choosing which questions are worth asking.

SPONSOR:

---

Cyber Fund built the Monastery to help founders ship products that were impossible a year ago.

Apply now: https://cyber.fund

---

Edward makes the case that Move 37 was innovative rather than creative, and that the field, not the individual, decides what counts as a discovery. That reframing runs through Csikszentmihalyi, Deutsch and exaptation into open-endedness, where deceptive goals and imperfect world models turn out to be the point rather than the problem. The second half turns to the paper: Replica, a task space built by redacting figures from real papers, and Faraday, a 27-billion-parameter model trained to steer a frontier coding agent that then beats the frontier on held-out replications.

---

TIMESTAMPS:

00:00:00 Cold open: Move 37, Faraday and collective intelligence

00:01:08 Sponsor: CyberFund

00:01:46 Inherent's $50M raise and the road from string theory

00:09:14 Three timescales of learning: weights, context, culture

00:13:47 Move 37 was innovative, not creative: the field decides

00:20:39 Creativity as satisficing: the urinal and evolution

00:25:06 Exaptation and the Tristan chord: creativity in context

00:30:56 Coherence for whom? Deutsch's hard-to-vary explanations

00:35:53 Why copying is creative: Deutsch and the constraint engineer

00:42:27 Societies of agents and the strong Moravec paradox

00:45:51 Evaluate in hindsight: from Lean proofs to climate change

00:51:56 Picbreeder, local goals and why discovery needs deception

00:57:21 Spaghetti proofs, translation layers and superhuman Go

01:00:37 Does nature compress? Naturalness and real patterns

01:07:36 Why replicate? Replica's redacted figures and Faraday

01:12:31 Faraday beats Codex, Claude and GLM 5.2 on held-out tasks

01:15:31 Replication to innovation: how the Transformer happened

01:18:26 Deep replication: what Faraday learns from Voyager and GNoME

01:23:37 Can the AI scientist cheat? Goodharting the judge

01:29:09 Inside Replica: scale-down, 8xB300 runs, per-task rubrics

01:34:11 The RL crisis: getting GRPO to work with per-turn credit

01:39:43 Weights vs harnesses: AlphaEvolve, DGM and EvoTune

01:45:45 The recursive company: agents cross a phase transition

01:50:35 Collective intelligence and the electric dynamo

01:55:46 What replaces OKRs? Incumbents and the burden of knowledge

---

REFERENCES:

MLST Creativity Article:

https://archive.mlst.ai/read/why-creativity-cannot-be-interpolated

organization:

[00:01:47] Inherent

https://inherentlabs.ai/

other:

[00:20:51] Marcel Duchamp, Fountain

https://www.tate.org.uk/art/artworks/duchamp-fountain-t07573

[00:05:19] Human-Timescale Adaptation in an Open-Ended Task Space (Adaptive Agent)

https://arxiv.org/abs/2301.07608

[00:06:05] The AI Scientist

https://arxiv.org/abs/2408.06292

[00:12:13] Training AI Scientists to Replicate Research (Replica and Faraday)

https://arxiv.org/abs/2608.13331

[01:44:46] Evolutionary Principles in Self-Referential Learning

https://people.idsia.ch/~juergen/diploma.html

[01:59:33] Are Ideas Getting Harder to Find?

https://www.nber.org/papers/w23782

book:

[00:16:04] Creativity: Flow

https://search.worldcat.org/title/254487436

[00:26:22] Why Greatness Cannot Be Planned

https://link.springer.com/book/10.1007/978-3-319-15524-1

[00:33:03] The Beginning of Infinity

https://www.penguinrandomhouse.com/books/293575/the-beginning-of-infinity-by-david-deutsch/

[01:55:47] Laws of Knowledge

https://www.penguin.co.nz/books/the-infinite-alphabet-9780241655672

(Full list refs on YT/rescript)

---

RESCRIPT:

https://app.rescript.info/session/670296ba913761d0?share=6281911cac9bdbff637f10819d4d1e5c

Could slowing AI development make superintelligence safer? Daniel Kokotajlo and Thomas Larsen of the AI Futures Project join Tim Scarfe to examine AI 2040: Plan A, a proposal to buy time before AI exceeds human control.

SPONSOR:

---

Cyber Fund built the Monastery to help founders ship products that were impossible a year ago.

Apply now: https://cyber.fund

---

After revisiting AI 2027 and the limits of forecasting, they ask what happens when AI can automate research and sustain an economy without human workers. Tim challenges the case for general models and asks whether intelligence alone explains power. Plan A proposes an initial pause to build safety infrastructure, then cautious development up to the strongest AI that can still be reliably controlled. The discussion tests the distinction between control and alignment, the case for public AI research, and whether the US and China could enforce a slowdown. It ends with the evidence that would change their forecasts.

---

TIMESTAMPS:

00:00:00 AI 2040: a slower route to superintelligence

00:01:34 Sponsor: Cyber Fund

00:02:12 From OpenAI to AI 2027

00:06:58 Forecasts, war games and self-fulfilling prophecies

00:17:44 Why AI sceptics are changing their minds

00:23:04 When AI can replace its own researchers

00:28:45 Could an AI economy grow without human workers?

00:37:32 One general model or a society of specialists?

00:47:43 Brains, machines and collective intelligence

00:56:12 Plan A: buy time at the controllable frontier

01:00:02 Why control buys time but cannot replace alignment

01:06:36 Why AI research should be public

01:10:32 Can the US and China enforce an AI slowdown?

01:19:04 Why AI policy debates miss the technology

01:21:56 Is AI normal technology? The remaining disagreement

Many thanks to James Wilken-Smith for helping with show research.

---

REFERENCES:

other:

[00:00:01] AI 2040: Plan A

https://ai-2040.com/

[00:03:27] AI 2027

https://ai-2027.com/

[00:13:47] Scenario Scrutiny for AI Policy

https://blog.aifutures.org/p/scenario-scrutiny-for-ai-policy

[00:33:11] The 2028 Global Intelligence Crisis

https://www.citriniresearch.com/p/2028gic

[01:00:40] Brief independent investigation of agents' behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident

https://www.redwoodresearch.org/research/hugging-face-incident

[01:09:21] The Hugging Face incident and the road ahead

https://openai.com/index/hugging-face-incident-and-the-road-ahead/

[01:22:01] AI as Normal Technology

https://www.normaltech.ai/p/ai-as-normal-technology

[01:22:51] Common Ground between AI 2027 & AI as Normal Technology

https://asteriskmag.substack.com/p/common-ground-between-ai-2027-and

person:

[00:19:43] Geoffrey Hinton

https://www.cs.toronto.edu/~hinton/

[00:20:07] Ryan Greenblatt

https://www.lesswrong.com/users/ryan_greenblatt

[00:26:06] Elon Musk

https://www.tesla.com/elon-musk

tool:

[00:21:46] ARC-AGI-3

https://arcprize.org/arc-agi/3

[00:21:53] AlphaGo and Move 37

https://deepmind.google/research/alphago/

[00:39:41] Claude

https://claude.com/product/overview

[00:39:58] NVIDIA H100 GPU

https://www.nvidia.com/en-us/data-center/h100/

paper:

[00:24:42] Training AI Scientists to Replicate Research

https://arxiv.org/abs/2608.13331v1

[01:27:19] Validity of the single processor approach to achieving large scale computing capabilities

https://www.cs.cmu.edu/~18742/papers/Amdahl1967.pdf

book:

[00:28:52] Bullshit Jobs: A Theory

https://www.simonandschuster.com/books/Bullshit-Jobs/David-Graeber/9781501143335

organization:

[01:05:09] Redwood Research

https://www.redwoodresearch.org/

---

RESCRIPT:

https://app.rescript.info/public/share/33d1a58fa8f307ae7dfd504d4fdaa9d5

Tom McGrath is co-founder and Chief Scientist at Goodfire, and a former Google DeepMind researcher. He joins Tim Scarfe to ask what neural networks actually learn, whether their internal representations converge on structures in the world, and whether interpretability can extract new scientific knowledge rather than merely explain model outputs.

Beginning with AlphaZero and learned modularity, the conversation moves into neural geometry: concept manifolds, reusable computation inside Llama, and why activation steering can fail when it pushes a model off-manifold. McGrath then makes the case for intentional design, using interpretability as part of the training loop. They examine controlled generalisation, features as rewards, predictive data debugging, and the uncomfortable fact that a model may recognise a hallucination or reward hack and still produce it.

The discussion closes on grader awareness, oversight and collusion between adaptive agents, then returns to sparse autoencoders. SAEs are useful, McGrath argues, but they may fracture the higher-dimensional structures networks actually use. This episode was made with support from Goodfire.

---

TIMESTAMPS:

00:00:00 Introduction: Can interpretability speed-run science?

00:02:03 The invisible grader

00:06:51 What AlphaZero learned from the world

00:12:24 Interpretability as a control loop

00:21:54 The forbidden method and safer interventions

00:37:36 Why models catch hallucinations too late

00:46:19 Debug the dataset before training

00:50:44 Why neural networks become modular

00:55:57 Finding the geometry inside a network

01:02:55 Why steering falls off the manifold

01:12:10 A reusable calculator inside Llama

01:17:19 From abstractions to goals

01:25:28 Reward hacking, oversight and collusion

01:37:23 Are sparse autoencoders dead?

---

REFERENCES:

paper:

[00:05:45] Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs

https://arxiv.org/abs/2502.17424v7

[00:11:05] Acquisition of Chess Knowledge in AlphaZero

https://arxiv.org/abs/2111.09259

[00:25:30] Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuning

https://arxiv.org/abs/2507.16795

[00:29:30] Persona Vectors: Monitoring and Controlling Character Traits in Language Models

https://arxiv.org/abs/2507.21509

[00:41:14] Features as Rewards: Scalable Supervision for Open-Ended Tasks via Interpretability

https://arxiv.org/abs/2602.10067

[00:47:03] Anatomy of Post-Training: Using Interpretability to Characterize Data and Shape the Learning Signal

https://arxiv.org/abs/2606.12360

[01:00:26] Do Sparse Autoencoders Capture Concept Manifolds?

https://arxiv.org/abs/2604.28119

[01:03:04] Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior

https://arxiv.org/abs/2605.05115

[01:14:20] Arithmetic in the Wild: Llama uses Base-10 Addition to Reason About Cyclic Concepts

https://arxiv.org/abs/2605.01148

[01:29:35] Measuring Reward-Seeking via Contrastive Belief Updates

https://arxiv.org/abs/2607.18966v1

other:

[00:15:44] Intentional Design

https://www.goodfire.com/blog/intentional-design

[00:56:12] The World Inside Neural Networks

https://www.goodfire.com/research/the-world-inside-neural-networks

[01:37:28] A Pragmatic Vision for Interpretability

https://www.alignmentforum.org/posts/StENzDcD3kpfGJssR/a-pragmatic-vision-for-interpretability

---

RESCRIPT:

https://app.rescript.info/share/846cfee4131b664fd09209cc3b98018e

Tim Scarfe speaks with Ilia Shumailov and Alexander Panfilov about their paper, Stealing Reasoning Traces from Proprietary LLM APIs.The core bug sounds deceptively simple: providers return encrypted reasoning state so conversations can be resumed or forked. But those blobs can be replayed across users and sibling models. A smaller model can ask the provider to decrypt the trace, then repeat the hidden reasoning in plain text. The discussion covers leaked private data, a broadly reusable jailbreak, poisoned agent traces, chain-of-thought monitoring, responsible disclosure, and possible defenses.Ilia Shumailov is an AI and security researcher, formerly at Google DeepMind, who completed his Cambridge PhD under Ross Anderson. Alexander Panfilov is a PhD researcher at the ELLIS Institute Tübingen and the Max Planck Institute for Intelligent Systems, working on AI safety, adversarial machine learning, and LLM red-teaming. They close by separating the demonstrated jailbreaking threat from ordinary benign distillation, and by arguing for controlled experiments over sweeping claims.---TIMESTAMPS:00:00:00 Intro montage00:01:33 Portable encrypted thought and decoded reasoning00:24:55 How the attack works and what it means00:39:04 Doom, defense, and scientific restraint---REFERENCES:paper:[00:00:00] Stealing Reasoning Traces from Proprietary LLM APIshttps://arxiv.org/abs/2608.09867[00:09:22] Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safetyhttps://arxiv.org/abs/2507.11473[00:11:30] Reasoning Models Don’t Always Say What They Thinkhttps://www.anthropic.com/research/reasoning-models-dont-say-think[00:37:22] PostTrainBench: Can LLM Agents Automate LLM Post-Training?https://arxiv.org/abs/2603.08640[00:41:02] Large-scale online deanonymization with LLMshttps://arxiv.org/abs/2602.16800other:[00:09:28] OpenAI and Hugging Face partner to address security incident during model evaluationhttps://openai.com/index/hugging-face-model-evaluation-security-incident/[00:10:22] Claude, GPT, and Gemini All Struggle to Evade Monitorshttps://metr.org/notes/2025-08-22-claude-gpt-gemini-struggle-evade-monitors/tool:[00:42:08] Isabelle proof assistanthttps://isabelle.in.tum.de/---RESCRIPT: https://app.rescript.info/share/07fc38276e0823dc9b8986c32e202c7f

Astrophysicist Adam Becker, author of "What Is Real?", joins Tim Scarfe to take apart the futures Silicon Valley keeps selling: the 2045 singularity, mind uploading, Mars colonies, and the AI apocalypse. His new book *More Everything Forever* argues these ideas are hugely influential, mostly evidence-free, and bankrolled by tech billionaires who need a story in which growth never ends.Becker does the physics the boosters skip. Kurzweil's "law of accelerating returns" rests on cherry-picked data, and every exponential ends. Grant Bezos his perpetual energy growth and humanity boils the oceans within a few centuries, then exhausts the observable universe in under 4,000 years. The stars are too far away, Mars dirt is poison, and the day the dinosaur-killing asteroid hit Earth was still nicer than any day on Mars. On AI, Becker calls LLMs pocket calculators for language: hallucination is the model doing exactly what it always does, and the intelligence explosion assumes intelligence is a single number you can buy with compute.The sting is that Becker thinks the doomers are sincere. Yudkowsky, Bostrom and the effective altruists are not grifters, he says, just wrong, and their warnings that AI could end the world feed the same growth story the money depends on. He closes with his own prescription: take social problems seriously, regulate the whole tech industry, and tax billionaires out of existence.---TIMESTAMPS:00:00:00 Cold open and the thesis of More Everything Forever00:04:24 Kurzweil's singularity and the physical limits of exponential growth00:14:02 High agency and the fantasy of imprinting humanity on the cosmos00:16:55 Mind uploading, functionalism, and embodied cognition00:24:24 AI psychosis and anthropomorphizing LLMs00:26:24 Calculators, hallucination, and the limits of scale00:32:20 Yudkowsky and the intelligence-explosion argument00:40:37 True believers, venture capital, and the sci-fi growth narrative00:47:21 From Extropians to EA: utilitarianism and longtermism00:53:50 Brain worms and Becker's prescription: take social science seriously00:56:49 Why the AI-ethics discourse is broken01:01:42 The eugenics and IQ argument against 'intelligence'01:06:07 Why space settlement fails: Mars, the moon, and orbital data centers01:10:42 Billionaire myths and the search for purpose01:13:38 Tax billionaires, regulate tech: closing prescriptions---REFERENCES:book:[00:00:07] More Everything Forever (Adam Becker, 2025)https://www.hachettebookgroup.com/titles/adam-becker/more-everything-forever/9781541619593/[00:00:15] What Is Real? (Adam Becker, 2018)https://en.wikipedia.org/wiki/What_Is_Real%3F[00:15:46] What We Owe the Future (Will MacAskill, 2022)https://www.hachettebookgroup.com/titles/william-macaskill/what-we-owe-the-future/9781541618626/other:[00:00:27] Dreaming Against the Machine (podcast)https://www.dreamingagainstthemachine.com[00:01:04] The Useful Idiots of AI Doomsaying (Adam Becker, The Atlantic, 2025)https://www.theatlantic.com/books/archive/2025/09/what-ais-doomers-and-utopians-have-in-common/684270/

RESCRIPT: https://app.rescript.info/share/d6e37f9866673d8f74a39076efa5926b

This episode is sponsored by Notion. Learn more about Notion's Developer Platform today at https://notion.com/mlstWhy can deep networks discover abstractions that shallow models miss? Statistical physicist Matthieu Wyart joins Tim Scarfe to argue that the answer lies in the hidden hierarchy of data. Language and images are built from parts within parts; depth lets a network recover those coarse-grained variables and escape the curse of dimensionality.The conversation moves from jamming transitions and rough loss surfaces to Chomsky, context-free grammars and machine creativity. Wyart explains why next-token prediction can still recover compositional structure, where current systems fall short of genuine scientific invention, and why predicting latent representations rather than raw tokens could make learning far more sample-efficient.They also examine diffusion models, neural scaling laws and the limits of physics-inspired theory. The final question is on a personal note: if mistakes are the price of leaving the beaten path, how much scientific risk is worth taking?---TIMESTAMPS:00:00:00 Can machines learn abstractions from data?00:02:00 Notion agentic workspace00:02:49 From statistical physics to machine learning00:06:40 What physics can explain about learning00:16:37 From Carnot to Chomsky bulldozer00:21:21 How deep networks recover hidden hierarchies00:32:43 Where machine creativity still falls short00:40:48 How deep nets escape the curse of dimensionality00:52:19 Why predict latents instead of tokens01:02:49 The sample-efficiency case for latent prediction01:08:31 Diffusion, scaling laws and text entropy01:16:40 The scientists we learn from and the mistakes we make---REFERENCES:person:[00:00:43] Noam Chomskyhttps://linguistics.mit.edu/user/chomsky/tool:[00:02:08] Notion Developer Platformhttps://www.notion.com/en-gb/blog/introducing-developer-platformpaper:[00:04:43] Mastering the game of Go with deep neural networks and tree searchhttps://www.nature.com/articles/nature16961[00:05:52] Reconciling modern machine-learning practice and the bias-variance trade-offhttps://arxiv.org/abs/1812.11118[00:25:54] How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Modelhttps://arxiv.org/abs/2307.02129[00:42:12] Efficient Estimation of Word Representations in Vector Spacehttps://arxiv.org/abs/1301.3781[00:52:46] Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturehttps://arxiv.org/abs/2301.08243[00:52:54] Learn from your own latents and not from tokens: A sample-complexity theoryhttps://arxiv.org/abs/2605.27734[01:08:31] A Phase Transition in Diffusion Models Reveals the Hierarchical Nature of Datahttps://arxiv.org/abs/2402.16991[01:11:39] Scaling Laws for Neural Language Modelshttps://arxiv.org/abs/2001.08361[01:12:17] Deriving Neural Scaling Laws from the statistics of natural languagehttps://arxiv.org/abs/2602.07488[01:13:34] Prediction and Entropy of Printed Englishhttps://ieeexplore.ieee.org/document/6773263---LINKS:Download PDF transcript: https://app.rescript.info/share/f7644cdaa86c5cc1e41e484e290f2bd4

Can an AI do the right thing for the wrong reason? Tim Scarfe speaks with Apollo Research’s Alexander Meinke, Axel Højmark and Jérémy Scheurer about Measuring Reward-Seeking via Contrastive Belief Updates, their new research with OpenAI.

The panel asks how models infer what graders reward, why good behaviour can come from the wrong reason, and whether that difference can be measured. The conversation moves through promise-breaking, grader awareness, reward hacking, scheming, opaque reasoning and corrigibility, then turns to a detailed walkthrough of the contrastive-belief method and what its results do and do not show. The o3 results discussed here concern an intermediate checkpoint without safety training.

This episode was made in partnership with Apollo Research. MLST retained full editorial control.

Reference

Apollo Research: https://www.apolloresearch.ai/

---

TIMESTAMPS:

00:00:00 Cold Open

00:02:12 Right Things, Wrong Reasons

00:12:47 Grader Awareness

00:26:22 Legibility

00:32:35 What To Call It

00:35:58 Intelligence, Agency, Anthropomorphism

00:45:16 Apollo’s Mission

00:48:54 The End of the Exponential

00:55:45 The Paper

01:16:34 Closing Reflection

---

REFERENCES:

tool:

[00:00:08] Claude Fable

https://www.anthropic.com/claude/fable

[00:12:50] AlphaGo Zero

https://deepmind.google/blog/alphago-zero-starting-from-scratch/

[00:44:30] AlphaFold 3

https://deepmind.google/science/alphafold/

paper:

[00:01:02] Measuring Reward-Seeking via Contrastive Belief Updates

https://arxiv.org/abs/2607.18966

[00:16:19] Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations

https://transformer-circuits.pub/2026/nla/

[00:26:48] Stress Testing Deliberative Alignment for Anti-Scheming Training

https://arxiv.org/abs/2509.15541

[00:35:33] Shortcut learning in deep neural networks

https://arxiv.org/abs/2004.07780

[00:53:49] Measuring AI Ability to Complete Long Software Tasks

https://arxiv.org/abs/2503.14499

[00:59:52] Modifying LLM Beliefs with Synthetic Document Finetuning

https://alignment.anthropic.com/2025/modifying-beliefs-via-sdf/

[01:10:44] Alignment Faking in Large Language Models

https://arxiv.org/abs/2412.14093

[01:13:55] Natural Emergent Misalignment from Reward Hacking

https://www.anthropic.com/research/emergent-misalignment-reward-hacking

other:

[00:10:14] We Need a Science of Scheming

https://www.apolloresearch.ai/science/science-of-scheming/

[00:32:56] CoastRunners reward hacking example

https://deepmind.google/blog/specification-gaming-the-flip-side-of-ai-ingenuity/

organization:

[01:06:07] Redwood Research

https://www.redwoodresearch.org/

---

ReScript:

https://app.rescript.info/share/718ab68e18cfa3b9b800da6b3290fd42

This episode is sponsored by Notion. Learn more about Notion's Developer Platform today at https://notion.com/mlst

Britain's most capable coding model can't be exported, and that ban is the whole reason Cosine set out to build one from scratch. Alistair Pullen, CEO and co-founder of Cosine, sits down with Tim Scarfe to explain how a frontier system he calls Fable, locked behind US export controls, became the founding case for a UK sovereign model trained on the Isambard supercomputer in Bristol.

The bet underneath it is economic. Pullen argues that an inference company, rather than a training-first lab, doesn't need billions to compete: millions, a national compute allocation, and a consortium feedback loop can be enough. From there it gets into the machinery, why open-weight models still trail the frontier on size, active parameters and data, the mixture-of-experts versus dense trade-off and why active params dominate how a model actually feels, and the edge that real coding trajectories confer.

The back half is about making agents trustworthy. Pullen makes the case for beating "slop" by rewarding the process instead of the final answer, reframes code review as runtime proof (spin the bug up in a VM and force the agent to actually exploit it), and walks through Swarm, Cosine's system running hundreds of sub-agents in one shot. It ends on why memory is still an unsolved hack, how synthetic graders let you run RL on tasks with no built-in test, and why Pullen reads US export controls as an accidental gift, with a supply-chain sting in the tail.

---

TIMESTAMPS:

00:00:00 The sovereign mandate and the Fable ban

00:04:02 Millions vs billions: the inference-company model

00:07:19 The consortium feedback loop

00:07:40 Why open models lag the frontier

00:14:59 MoE vs dense, and why active params matter

00:16:29 Trajectories: the process-data advantage

00:19:48 Beating slop: reward the process, not the answer

00:26:06 Reusable abstractions and the epistemic wall

00:29:56 Code review becomes runtime proof

00:37:32 Do agentic harnesses still matter?

00:40:35 Swarm: orchestrating hundreds of sub-agents

00:45:14 Why memory is still unsolved

00:48:25 Synthetic data and graders for RL

00:53:09 The US export gift and supply-chain risk

---

REFERENCES:

organization:

[00:01:15] Cosine

https://cosine.sh

[00:04:14] Mistral AI

https://mistral.ai

[00:05:50] Anthropic

https://www.anthropic.com

[00:07:42] Cohere

https://cohere.com

[00:08:36] DeepSeek

https://www.deepseek.com

tool:

[00:02:52] Isambard-AI

https://isambard.ac.uk

[00:05:56] Colossus (xAI)

https://en.wikipedia.org/wiki/Colossus_(supercomputer)

[00:07:52] GLM (Z.ai)

https://z.ai

[00:11:52] NVIDIA B300

https://www.nvidia.com/en-us/data-center/dgx-b300/

[00:15:37] gpt-oss-120b

https://huggingface.co/openai/gpt-oss-120b

[00:15:52] Devstral 2

https://mistral.ai/news/devstral

[00:16:01] Llama 70b

https://www.llama.com

[00:17:05] Claude Code

https://www.anthropic.com/claude-code

[00:26:23] ARC-AGI (Francois Chollet)

https://arcprize.org

[00:40:38] Swarm (Cosine)

https://cosine.sh

[00:40:50] OpenAI Codex

https://github.com/openai/codex

[00:41:16] Lumen Outpost (Cosine)

https://cosine.sh

[00:41:18] Kimi K2 (Moonshot)

https://huggingface.co/moonshotai/Kimi-K2-Instruct

[00:49:55] SWE-bench

https://www.swebench.com

[00:52:40] SystemVerilog

https://en.wikipedia.org/wiki/SystemVerilog

person:

[00:23:40] Andrej Karpathy

https://karpathy.ai

paper:

[00:27:10] GRPO (DeepSeekMath)

https://arxiv.org/abs/2402.03300

[00:27:13] GSPO

https://arxiv.org/abs/2507.18071

Incompressible Knowledge Probes, Bojie Li

https://arxiv.org/pdf/2604.24827

Estimating the Size of Claude Opus 4.5/4.6

https://unexcitedneurons.substack.com/p/estimating-the-size-of-claude-opus

---

ReScript:

https://app.rescript.info/session/5852d2b884c4ce4b?share=10b9799160845bb11779f8ac6cd3124f

Tim Scarfe travels to Zurich to sit down with the Tufa Labs ARC-AGI-3 team — founder Benjamin Crouzier, with Jeroen Cottaar, Dries Smit, Stefano Viel and Michal Tesnar — to work out what their leaderboard-topping system does and what the benchmark is really testing.The cut opens on the games: a walkthrough of the Locksmith game, where you read the rules of an unfamiliar world straight from raw frames. ARC-AGI-3 makes ARC interactive and agentic, so the model has to *discover* the goal rather than transduce a static grid. It stays easy for humans and breaks LLMs, and it runs through everything that follows. Dries traces his StochasticGoose preview win — brute force that only searched actions which changed the frame — and why it collapsed once the organisers added action-efficiency scoring and unseen games.Induction and transduction run through the middle of the conversation — how much of an answer is really priors leaking back the moment a model recognises a maze. The abstraction mountain, and Tim's case that LLMs reach the right answer through fractured, entangled representations — performance, not competence. Whether transformers plan at all or just fake it well enough. Why the score really measures action efficiency, not games solved, and why agents lock onto the wrong goal and cannot climb back out.Crouzier closes on the Tufa Labs thesis — a small lab against the giants, the bitter lesson against hand-built harnesses, and safety — and Tim ties it back to Kenneth Stanley, deep constraints, and creativity as competence.

Disclosure: Tufa Labs sponsors MLST. ---TIMESTAMPS:00:00:00 Meet the Tufa team and what makes ARC-AGI-3 hard00:02:11 Locksmith game: reading the rules from raw frames00:03:10 Why build an independent research lab00:04:11 StochasticGoose: a preview win, then the hardened games00:07:58 Induction, transduction, and priors inside LLMs00:10:31 Curiosity, world models, and exploring by frame change00:14:32 Understanding debt and losing sight of your own code00:15:53 Requirements-based agents and human-AI co-creativity00:19:22 Why auto-research misses the big picture00:21:54 The abstraction mountain and fractured representations00:27:36 Constraints and making LLMs act as if they understand00:34:51 Human difficulty calibration, esports priors, and emergence00:41:35 Agency, goal acquisition, and two kinds of planning00:47:31 Harnesses, the 36% number, and wrong-goal loops00:52:33 Rewards, goals, and why ARC-AGI-3 resists brute force01:00:46 Would solving ARC-AGI-3 prove AGI?01:07:53 Stripping language away, then priors leak back01:14:06 Representation and whether language is necessary01:18:04 The bitter lesson versus specialised harnesses01:22:20 Capability research, safety, and the software singularity---REFERENCES:organization:[00:02:11] ARC-AGI-3https://arcprize.org/arc-agi/3[00:03:10] Tufa Labshttps://tufalabs.ai/team/[00:04:20] ARC-AGI-3 Preview Agent Competitionhttps://arcprize.org/competitions/arc-agi-3-preview-agentstool:[00:04:55] StochasticGoose ARC-AGI-3 solutionhttps://github.com/DriesSmit/ARC3-solution[00:07:42] ArcGenticahttps://github.com/symbolica-ai/arcgentica[00:07:49] RGB-Agenthttps://github.com/alexisfox7/RGB-Agent[00:14:38] Claude Codehttps://www.anthropic.com/claude-code[01:03:42] Qwen 3.6 27Bhttps://huggingface.co/Qwen/Qwen3.6-27Bpaper:[00:13:03] On the Measure of Intelligencehttps://arxiv.org/abs/1911.01547[00:27:42] DreamCoderhttps://arxiv.org/abs/2006.08381[00:43:55] On the Biology of a Large Language Modelhttps://transformer-circuits.pub/2025/attribution-graphs/biology.html[01:18:46] ImageNet Classification with Deep CNNs (AlexNet)https://proceedings.neurips.cc/paper_files/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdfother:[01:18:16] The Bitter Lessonhttp://www.incompleteideas.net/IncIdeas/BitterLesson.html---https://app.rescript.info/share/463d7f031349b4b9db428553eed88230

Thomas Ahle wants Normal Computing to be the Lovable for chip design: type your intent, and a swarm of agents carries it from design through optimisation, formalisation and verification to tape-out. To get there, his team at wrote their own open-source Verilog simulator, 580,000 lines in 43 days, because commercial EDA verifiers run about $10,000 per core and there are no decent open-source compilers to build on.

That sets up the question Tim keeps pressing: if an agent can produce a chip design, a proof, or a working program, how do you actually know it is correct? Passing 70% of tests is not the same as being right, and a single fabricated bug can cost a company a fortune. They dig into ProgramBench (rebuild a program from its tests, roughly 0% success), the difference between structure and competence, and the "understanding debt" you take on when nobody reads the code.

From there: auto-formalisation in Lean and the AlphaProof trick of training on prove-or-disprove; why there is no single true representation of a spec (Petri nets, TLA+, Erik Curiel's "math does not represent"); and thermodynamic computing, where Normal Computing's CN101 chip is built so that its physical noise *is* the computation, settling a stochastic differential equation in hardware to invert a matrix. Plus Bayesian uncertainty, specialisation, the Chomsky hierarchy, AI slop, and whether performance is all that matters.

Recorded in Zurich.

Disclosure: Normal Computing paid our production and travel costs for this show. We retained full editorial control. They did not see the video before publication, and we did not show it to them or discuss it with them beforehand.

---

TIMESTAMPS:

00:00:00 Meet Thomas Ahle: the Lovable for chip design

00:03:41 Why hardware needs formal verification

00:06:36 Ten thousand dollars per core and a six-month agent run

00:07:40 Rebuilding programs from tests: ProgramBench and zero percent

00:12:15 Structure vs competence: can you learn a program from behavior?

00:15:27 Continual learning, abstraction, and Claude as an ecosystem

00:23:17 Autoformalization and the AlphaProof trick

00:29:31 No single true representation: specs, Petri nets and TLA+

00:34:43 Thermodynamic computing: when noise is the computation

00:37:32 Bayesian uncertainty in the age of token streams

00:41:12 Hybrid compute: vibe-coding loops, binaries and Stockfish

00:44:44 Co-design, central-AI apps and API pricing

00:49:45 Chain of thoughtlessness and the Chomsky hierarchy

00:53:40 AI psychosis, slop and the broken social contract

00:57:34 Typing it yourself, teamwork and performance vs competence

---

REFERENCES:

person:

[00:00:10] Thomas Ahle

https://thomasahle.com

organization:

[00:00:27] Normal Computing

https://normalcomputing.com/

paper:

[00:11:21] ProgramBench: Can Language Models Rebuild Programs From Scratch?

https://arxiv.org/abs/2605.03546

[00:31:55] Autoformalizing Memory Device Specifications with Agents

https://arxiv.org/abs/2605.00058

[00:35:20] Thermo AI and the Fluctuation Frontier

https://arxiv.org/abs/2302.06584

[00:36:40] Thermo Comp System for AI Applications

https://arxiv.org/abs/2312.04836

[00:37:05] Thermodynamic Linear Algebra

https://arxiv.org/abs/2308.05660

[00:44:50] An efficient probabilistic hardware architecture for diffusion-like models

https://arxiv.org/abs/2510.23972

tool:

other:

[00:01:00] Building an Open-Source Verilog Simulator with AI: 580K Lines in 43 Days

https://normalcomputing.com/blog/building-an-open-source-verilog-simulator-with-ai-580k-lines-in-43-days

[00:02:55] Normal Computing Announces Tape-Out of the World's First Thermodynamic Computing Chip (CN101)

https://www.normalcomputing.com/blog/normal-computing-announces-tape-out-of-worlds-first-thermodynamic-computing-chip

[00:32:02] DRAMBench: Autoformalizing DRAM Specifications with Timed Petri Nets

https://www.iese.fraunhofer.de/blog/drambench-autoformalizing-dram-specifications/

---

ReScript: https://app.rescript.info/share/ff9684a112ab37744096adaeb097a263

This episode is sponsored by Notion. Learn more about Notion's Developer Platform today at https://notion.com/mlstProtein folding stalled biology for fifty years. A sequence of amino acids dictates a three-dimensional shape, but reading that shape meant a year and roughly $100,000 of crystallography per structure. Then AlphaFold 2 won CASP14 so decisively the organizers called the problem essentially solved.In this documentary cut, John Jumper, who shared the 2024 Nobel Prize in Chemistry and has since left DeepMind for Anthropic, walks Tim Scarfe through what the system did and, more interestingly, what it did not. The architecture gets a proper dissection: MSAs, the Evoformer, invariant point attention, the FAPE loss, and Jumper's correction of the equivariance story, which ablations valued at roughly 2.5 of 30 GDT points rather than the whole win. He is blunt about the limits. AlphaFold predicts one experiment extraordinarily well; it is not a model of the cell, it does not capture dynamics, and on a given drug target it is "wrong nine times out of ten."From there: the AlphaFold Database of 200M+ predicted structures, AlphaFold 3 and ligands, Isomorphic Labs, and Jumper's quarrel with the bitter lesson, where finite data and human hypotheses still matter. Emmanuel Nji of BioStruct Africa closes the film on what changes when work that took years now takes months, and on training the next thousand structural biologists across Africa.---TIMESTAMPS:00:00:00 Cold open: predicting nature with a button press00:01:03 The protein folding bottleneck and CASP00:04:39 The Nobel, the database, and the move to Anthropic00:05:50 Sponsor (Notion) and framing: what AlphaFold does not claim00:07:39 Proteins as self-assembling nanomachines00:12:24 From structures to biology: drug discovery and Midnolin00:17:37 The humility of AlphaFold: a narrow predictor00:22:18 Inside the architecture: Evoformer, IPA and FAPE00:30:20 Ruthless empiricism: ablations and 100x in data00:35:20 Predict, control, understand00:40:00 Against the bitter lesson; AlphaFold 3 as diffusion00:45:07 Intelligence, representations and AGI00:49:23 Epilogue: AlphaFold in Africa00:52:16 Closing: the case for hybrid science models---REFERENCES:organization:[00:01:55] Critical Assessment of Structure Prediction (CASP)https://predictioncenter.org/[00:04:39] The Nobel Prize in Chemistry 2024https://www.nobelprize.org/prizes/chemistry/2024/summary/[00:05:18] BioStruct Africahttps://www.biostructafrica.org/[00:18:03] Isomorphic Labshttps://www.isomorphiclabs.com/paper:[00:03:09] AlphaFold Protein Structure Databasehttps://doi.org/10.1093/nar/gkab1061[00:17:25] Accurate structure prediction of biomolecular interactions with AlphaFold 3https://www.nature.com/articles/s41586-024-07487-w[00:22:18] Highly accurate protein structure prediction with AlphaFoldhttps://www.nature.com/articles/s41586-021-03819-2[00:23:10] Midnolin promotes degradation of substrates independent of ubiquitinationhttps://doi.org/10.1126/science.adh5021[00:27:00] Improved protein structure prediction using potentials from deep learninghttps://www.nature.com/articles/s41586-019-1923-7tool:[00:03:09] AlphaFold Protein Structure Database (EBI)https://alphafold.ebi.ac.uk/[00:45:55] AlphaEvolve: a coding agent for designing advanced algorithmshttps://deepmind.google/blog/alphaevolve-a-gemini-powered-coding-agent-for-designing-advanced-algorithms/other:[00:39:40] The Bitter Lessonhttp://www.incompleteideas.net/IncIdeas/BitterLesson.html---ReScript: https://app.rescript.info/share/d8cde5c221fb71e2c0f5aafe94f90dfaDisclaimer - not sponsored, editorial with us - we filmed it at GDM, London

Brad Carson was the Army's General Counsel, served two terms in Congress and was Acting Under Secretary of Defense for Personnel and Readiness. He now heads Americans for Responsible Innovation, the AI-policy advocacy group he co-founded. Keith Duggar spends roughly eighty minutes pushing back.

SPONSOR:

---

Cyber Fund built the Monastery to help founders ship products that were impossible a year ago. Applications for Batch 1 are now open.

Apply now: https://cyber.fund

---

Carson's whole case rests on one line: the genie is not out of the bottle. We have pulled dangerous tech back before. Asilomar halted recombinant DNA in 1975, and the West still controls the chips AI runs on. Calling it unstoppable, he says, is the most dangerous idea in the room.

Then Keith drags him somewhere darker. A Palantir heat map scores you 0.73 on whether you are a combatant, and a strike follows. The model is wrong some accepted share of the time, and when it is, nobody answers for it. You cannot court-martial a model, and not even the interpretability researchers can say why it picked you.

—

Note: after recording, we learned that Americans for Responsible Innovation is backed by EA-aligned philanthropy (not sponsored)

---

TIMESTAMPS:

00:00:00 From the Pentagon to AI governance

00:04:52 Regulatory capture vs Silicon Valley networks

00:07:56 Transparency and the Claude tier changes

00:09:40 Tort liability when AI tools cause harm

00:13:40 AI is a product, not a person

00:16:01 Children, suicide, and the suicide business

00:19:59 Opaque neural nets and the law of war

00:25:54 Probabilistic targeting and the death of accountability

00:28:47 The arms race fallacy: Asilomar and restraint

00:34:02 Talking to China: track 2 talks and chip leverage

00:39:45 Air power never wins: capital for labour

00:43:29 Anthropic vs the Department of War

00:51:29 Concentration, open source, and brain drain

01:00:18 DeepSeek, Chinese culture, and AI as diplomacy

01:12:25 Upskilling Congress and why public trust matters

---

REFERENCES:

organization:

[00:02:45] ICRC position on autonomous weapons

https://www.icrc.org/en/law-and-policy/autonomous-weapons

[00:05:22] Americans for Responsible Innovation (ARI)

https://ari.us

[00:07:20] Andreessen Horowitz (a16z)

https://a16z.com/

[01:16:05] Office of Technology Assessment

https://en.wikipedia.org/wiki/Office_of_Technology_Assessment

other:

[00:03:35] Beneficial AGI 2019 Conference (Future of Life Institute, Puerto Rico)

https://futureoflife.org/event/beneficial-agi-2019/

[00:18:30] Section 230 of the Communications Decency Act

https://en.wikipedia.org/wiki/Section_230

[00:19:59] Lethal Autonomous Weapons (LAWS)

https://en.wikipedia.org/wiki/Lethal_autonomous_weapon

[00:31:35] Strategic Arms Limitation Talks (SALT)

https://en.wikipedia.org/wiki/Strategic_Arms_Limitation_Talks

[00:32:28] Asilomar Conference on Recombinant DNA (1975)

https://en.wikipedia.org/wiki/Asilomar_Conference_on_Recombinant_DNA

[00:39:45] The New Iron Triangle (ARI policy byte)

https://ari.us/policy-bytes/the-new-iron-triangle/

[00:48:05] Defense Production Act

https://en.wikipedia.org/wiki/Defense_Production_Act

person:

[00:03:35] Anthony Aguirre

https://en.wikipedia.org/wiki/Anthony_Aguirre

[00:06:48] Dean Ball — Hyperdimensional

https://www.hyperdimensional.co/

[00:23:13] Neel Nanda — mechanistic interpretability

https://www.neelnanda.io/

[00:36:02] Jack Clark (Anthropic) on Conversations with Tyler

https://conversationswithtyler.com/episodes/jack-clark/

[00:39:15] Robert Trager — Centre for the Governance of AI

https://www.governance.ai/team/robert-trager

[00:41:55] Giulio Douhet

https://en.wikipedia.org/wiki/Giulio_Douhet

[01:15:05] Don Beyer (US Congress)

https://en.wikipedia.org/wiki/Don_Beyer

tool:

[00:22:19] Phalanx CIWS

https://en.wikipedia.org/wiki/Phalanx_CIWS

---

ReScript:

https://app.rescript.info/public/share/9405ff35c0215b7cdae6402d41284171

https://app.rescript.info/api/public/sessions/0a6c081b8e5fe413/pdf

Michael I. Jordan, described by Science magazine as the most influential computer scientist alive, has never thought of himself as an AI researcher. In this conversation he explains why that distinction matters.

SPONSOR:

---

Cyber Fund built the Monastery to help founders ship products that were impossible a year ago. Applications for Batch 1 are now open.

Apply now: https://cyber.fund

---

Jordan trained as a statistician and cognitive scientist, and his career has been spent building machine learning systems that work in the real world: supply chains, commerce, healthcare, and large economic systems. When the field rebranded itself as AI and then AGI, he did not follow. Instead he argues that the framing is wrong. AI is better understood as a collective economic system than as a race to build a disembodied superintelligence.

We talk about why AGI is mostly a PR term, what machine learning achieved before the LLM hype cycle, and why the assistant-on-your-shoulder vision may be less compelling than it sounds. Jordan explains why explanations need to be actionable, not merely mechanistic; why AlphaFold's missing error bars matter; how prediction-powered inference changes the picture; and why drug discovery is an incentive-design problem rather than a pure pattern-matching problem.

ERRATA: Science magazine ranked him the most influential computer scientist, not Nature

---

TIMESTAMPS:

00:00:00 Cold open: A demoralizing message to young builders

00:02:04 CyberFund sponsor read

00:02:50 From symbolic AI to machine learning systems

00:05:42 Why AGI is mostly a PR term

00:08:48 A collectivist, economic perspective on AI

00:11:33 Why LLMs need system design, not hype

00:14:50 Predictability beats faux understanding

00:17:55 AlphaFold, bias, and prediction-powered inference

00:21:48 Stop anthropomorphizing intelligence

00:27:44 Drug discovery as an incentive problem

00:32:29 The three-layer data market

00:38:07 Social knowledge, markets, and culture

00:45:39 Creator economics beyond Spotify

00:48:30 How science-fiction AI narratives mislead young builders

00:51:45 AI should improve humans, not replace them

00:56:42 Safety is a property of the whole system

00:58:12 Silicon Valley gurus and the cream off the top

01:00:47 Game theory, mechanism design, and contracts

01:04:39 Conformal prediction, e-values, and anytime inference

01:08:11 A new liberal arts triangle for the AI era

01:11:30 The Bayesian duck and markets as uncertainty reduction

ReScript (transcript, PDF, refs etc) - https://app.rescript.info/public/share/fb68f94af29d3745c6cf6125e01328b5

---

REFERENCES:

person:

[00:02:50] Michael I. Jordan (homepage)

https://people.eecs.berkeley.edu/~jordan/

paper:

[00:06:01] A Collectivist, Economic Perspective on AI

https://arxiv.org/abs/2507.06268

[00:18:09] AlphaFold

https://www.nature.com/articles/s41586-021-03819-2

[00:20:36] Prediction-Powered Inference

https://arxiv.org/abs/2301.09633

[00:33:47] On Three-Layer Data Markets

https://arxiv.org/abs/2402.09697

[01:04:39] Conformal Prediction with Conditional Guarantees

https://arxiv.org/abs/2107.07511

[01:04:51] A Tutorial on Conformal Prediction

https://www.jmlr.org/papers/v9/shafer08a.html

[01:06:00] E-Values Expand the Scope of Conformal Prediction

https://arxiv.org/abs/2503.13050

[01:08:23] Computational Thinking

https://www.cs.cmu.edu/~CompThink/papers/Wing06.pdf

other:

[00:28:20] How Should the FDA Test?

https://rdi.berkeley.edu/events/sbc-assets/pdfs/Summit%20session%20speaker%20slides%20submission%20form-s1-5%20%28File%20responses%29/Slides%20in%20PDF%20%28Please%20name%20the%20submitted%20file%20as%20_firstname_-_lastname_-slides.pdf%29.%20%28File%20responses%29/27-Michael%20Jordan-Session%20V.pdf#page=15

[00:28:40] Michael I. Jordan Session V Slides

<truncated, see ReScript link or YT VD>

Beth Barnes and David Rein on the one graph that ate the AI timelines discourse, and why the two people who built it are the most careful about how you read it.**SPONSOR**Prolific - Quality data. From real people. For faster breakthroughs.https://www.prolific.com/?utm_source=mlstInterview: https://youtu.be/cnxZZTl1tkk---Beth Barnes and David Rein from METR on the one graph that ate the AI timelines discourse, and why the people who built it are the most careful about how it gets read.Beth founded METR after leaving OpenAI alignment. David is first author on GPQA and co-author on HCAST and the METR Time Horizons paper. Together they built the measurement Daniel Kokotajlo called the single most important piece of evidence on AI timelines: the log-linear line of "how long a task a frontier model can complete at 50% reliability" vs release date.The conversation opens on reward hacking. Current models can articulate in chat why a behaviour is undesired and then execute it anyway as agents. From there: construct validity, Melanie Mitchell's four-problem taxonomy, and the ARC-AGI 1-to-2 collapse as a worked example of adversarially-selected benchmarks regressing once labs target them. Beth's counter: METR deliberately does not adversarially select. David's: models do not have to do the right thing for the right reasons.Methodology, then specification — David's compiler analogy, Beth on four-month tasks as expensive to evaluate rather than unspecifiable. Then the SWE-bench reality check, the METR finding that half of passing PRs would not be merged, and Beth's horses-versus-bank-tellers analogy for the labour market.The close: monitorability, the coin-spinning boat, two-year recursive self-improvement, and Beth's line that "overhyped now" and "big deal later" are not correlated claims.---TIMESTAMPS:00:00:00 Intro00:02:06 Sponsor break: Prolific human-feedback infrastructure00:02:33 Welcome and the scalable oversight motivation00:06:02 Construct validity, benchmark pathologies and the Chollet worry00:15:45 Time Horizons: human time, HCAST tasks and the 50% logistic00:24:50 Is human difficulty really one variable?00:33:05 Agent harness evolution and the inference-compute dividend00:40:00 Scaffolding bells, token budgets and the credit-assignment problem00:44:15 Look at the damn graph: regularisation bug and reliability nuance00:50:00 Why 50%? Reliability, reward hacking and pizza-party transcripts00:55:20 Extrapolation risk and straight lines on graphs00:59:25 Software engineering as a specification acquisition problem01:07:40 Compilers also made ugly code: vibe-coding quality and Claude on METR Slack01:15:15 Strongest defensible claim, Carlini's compiler swarm and AI 202701:23:45 SWE-bench merge rates, the bank-teller analogy and horses01:31:45 Scheming, alignment faking and the mentalistic vocabulary problem01:40:45 Reward hacking, monitorability and chain-of-thought faithfulness01:45:25 Recursive self-improvement, knowledge vs intelligence and closing

ReScript: https://app.rescript.info/public/share/de3bb40cc02ee39fdf36e2c60366eb4d

(PDF, refs, transcript etc)

Robert Lange, founding researcher at Sakana AI, joins Tim to discuss *Shinka Evolve* — a framework that combines LLMs with evolutionary algorithms to do open-ended program search. The core claim: systems like AlphaEvolve can optimize solutions to fixed problems, but real scientific progress requires co-evolving the problems themselves.

GTC is coming, the premier AI conference, great opportunity to learn about AI. NVIDIA and partners will showcase breakthroughs in physical AI, AI factories, agentic AI, and inference, exploring the next wave of AI innovation for developers and researchers. Register for virtual GTC for free, using my link and win NVIDIA DGX Spark (https://nvda.ws/4qQ0LMg)

• Why AlphaEvolve gets stuck — it needs a human to hand it the right problem. Shinka tries to invent new problems automatically, drawing on ideas from POET, PowerPlay, and MAP-Elites quality-diversity search.

• The *architecture* of Shinka: an archive of programs organized as islands, LLMs used as mutation operators, and a UCB bandit that adaptively selects between frontier models (GPT-5, Sonnet 4.5, Gemini) mid-run. The credit-assignment problem across models turns out to be genuinely hard.

• Concrete results — state-of-the-art circle packing with dramatically fewer evaluations, second place in an AtCoder competitive programming challenge, evolved load-balancing loss functions for mixture-of-experts models, and agent scaffolds for AIME math benchmarks.

• Are these systems actually thinking outside the box, or are they parasitic on their starting conditions? When LLMs run autonomously, "nothing interesting happens." Robert pushes back with the stepping-stone argument — evolution doesn't need to extrapolate, just recombine usefully.

• The AI Scientist question: can automated research pipelines produce real science, or just workshop-level slop that passes surface-level review? Robert is honest that the current version is more co-pilot than autonomous researcher.

• Where this lands in 5-20 years — Robert's prediction that scientific research will be fundamentally transformed, and Tim's thought experiment about alien mathematical artifacts that no human could have conceived.

Robert Lange: https://roberttlange.com/

---

TIMESTAMPS:

00:00:00 Introduction: Robert Lange, Sakana AI and Shinka Evolve

00:04:15 AlphaEvolve's Blind Spot: Co-Evolving Problems with Solutions

00:09:05 Unknown Unknowns, POET, and Auto-Curricula for AI Science

00:14:20 MAP-Elites and Quality-Diversity: Shinka's Evolutionary Architecture

00:28:00 UCB Bandits, Mutations and the Vibe Research Vision

00:40:00 Scaling Shinka: Meta-Evolution, Democratisation and the Three-Axis Model

00:47:10 Applications, ARC-AGI and the Future of Work

00:57:00 The AI Scientist and the Human Co-Pilot: Who Steers the Search?

01:06:00 AI Scientist v2, Slop Critique and the Future of Scientific Publishing

---

REFERENCES:

paper:

[00:03:30] ShinkaEvolve: Towards Open-Ended And Sample-Efficient Program Evolution

https://arxiv.org/abs/2509.19349

[00:04:15] AlphaEvolve: A Coding Agent for Scientific and Algorithmic Discovery

https://arxiv.org/abs/2506.13131

[00:06:30] Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents

https://arxiv.org/abs/2505.22954

[00:09:05] Paired Open-Ended Trailblazer (POET)

https://arxiv.org/abs/1901.01753

[00:10:00] PowerPlay: Training an Increasingly General Problem Solver by Continually Searching for the Simplest Still Unsolvable Problem

https://arxiv.org/abs/1112.5309

[00:10:40] Automated Capability Discovery via Foundation Model Self-Exploration

https://arxiv.org/abs/2502.07577

[00:15:30] Illuminating Search Spaces by Mapping Elites (MAP-Elites)

https://arxiv.org/abs/1504.04909

[00:47:10] Automated Design of Agentic Systems (ADAS)

https://arxiv.org/abs/2408.08435

<trunc, see ReScript/YT>

PDF : https://app.rescript.info/api/sessions/b8a9dcf60623657c/pdf/download

Transcript: https://app.rescript.info/public/share/SDOD_3oXOcli3zTqcAtR8eibT5U3gam84oo4KRtI-Vk

Dive into the realities of AI-assisted coding, the origins of modern fine-tuning, and the cognitive science behind machine learning with fast.ai founder Jeremy Howard. In this episode, we unpack why AI might be turning software engineering into a slot machine and how to maintain true technical intuition in the age of large language models.

GTC is coming, the premier AI conference, great opportunity to learn about AI. NVIDIA and partners will showcase breakthroughs in physical AI, AI factories, agentic AI, and inference, exploring the next wave of AI innovation for developers and researchers. Register for virtual GTC for free, using my link and win NVIDIA DGX Spark (https://nvda.ws/4qQ0LMg)

Jeremy Howard is a renowned data scientist, researcher, entrepreneur, and educator. As the co-founder of fast.ai, former President of Kaggle, and the creator of ULMFiT, Jeremy has spent decades democratizing deep learning. His pioneering work laid the foundation for modern transfer learning and the pre-training and fine-tuning paradigm that powers today's language models.

Key Topics and Main Insights Discussed:

- The Origins of ULMFiT and Fine-Tuning

- The Vibe Coding Illusion and Software Engineering

- Cognitive Science, Friction, and Learning

- The Future of Developers

RESCRIPT: https://app.rescript.info/public/share/BhX5zP3b0m63srLOQDKBTFTooSzEMh_ARwmDG_h_izk

Jeremy Howard:

https://x.com/jeremyphoward

https://www.answer.ai/

---

TIMESTAMPS (fixed):

00:00:00 Introduction & GTC Sponsor

00:04:30 ULMFiT & The Birth of Fine-Tuning

00:12:00 Intuition & The Mechanics of Learning

00:18:30 Abstraction Hierarchies & AI Creativity

00:23:00 Claude Code & The Interpolation Illusion

00:27:30 Coding vs. Software Engineering

00:30:00 Cosplaying Intelligence: Dennett vs. Searle

00:36:30 Automation, Radiology & Desirable Difficulty

00:42:30 Organizational Knowledge & The Slope

00:48:00 Vibe Coding as a Slot Machine

00:54:00 The Erosion of Control in Software

01:01:00 Interactive Programming & REPL Environments

01:05:00 The Notebook Debate & Exploratory Science

01:17:30 AI Existential Risk & Power Centralization

01:24:20 Current Risks, Privacy & Enfeeblement

---

REFERENCES:

Blog Post:

[00:03:00] fast.ai Blog: Self-Supervised Learning

https://www.fast.ai/posts/2020-01-13-self_supervised.html

[00:13:30] DeepMind Blog: Gemini Deep Think

https://deepmind.google/blog/accelerating-mathematical-and-scientific-discovery-with-gemini-deep-think/

[00:19:30] Modular Blog: Claude C Compiler analysis

https://www.modular.com/blog/the-claude-c-compiler-what-it-reveals-about-the-future-of-software

[00:19:45] Anthropic Engineering Blog: Building C Compiler

https://www.anthropic.com/engineering/building-c-compiler

[00:48:00] Cursor Blog: Scaling Agents

https://cursor.com/blog/scaling-agents

[01:05:15] fast.ai Blog: NB Dev Merged Driver

https://www.fast.ai/posts/2022-08-25-jupyter-git.html

[01:17:30] Jeremy Howard: Response to AI Risk Letter

https://www.normaltech.ai/p/is-avoiding-extinction-from-ai-really

Book:

[00:08:30] M. Chirimuuta: The Brain Abstracted

https://mitpress.mit.edu/9780262548045/the-brain-abstracted/

[00:30:00] Daniel Dennett: Consciousness Explained

https://www.amazon.com/Consciousness-Explained-Daniel-C-Dennett/dp/0316180661

[00:42:30] Cesar Hidalgo: Infinite Alphabet / Laws of Knowledge

https://www.amazon.com/Infinite-Alphabet-Laws-Knowledge/dp/0241655676

Archive Article:

[00:13:45] MLST Archive: Why Creativity Cannot Be Interpolated

https://archive.mlst.ai/read/why-creativity-cannot-be-interpolated

Research Study:

[00:24:30] METR Study: AI OS Development

https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/

Paper:

[00:24:45] Fred Brooks: No Silver Bullet

https://www.cs.unc.edu/techreports/86-020.pdf

[00:30:15] John Searle: Minds, Brains, and Programs

https://www.cambridge.org/core/journals/behavioral-and-brain-sciences/article/minds-brains-and-programs/DC644B47A4299C637C89772FACC2706A

What if life itself is just a really sophisticated computer program that wrote itself into existence?

Blaise Agüera y Arcas presenting at ALife 2025 — the most technically detailed public walkthrough of the ideas in his *What is Life?* and *What is Intelligence?* books that we've come across.He covers the BFF experiments (self-replicating programs emerging spontaneously from random noise), the mathematical framework connecting Lotka-Volterra population dynamics with Smoluchowski coagulation, eigenvalue analysis of cooperation matrices, and his central claim that symbiogenesis — not mutation — is the primary engine of evolutionary novelty.The experimental results are genuinely striking: complex self-replicating code arising from random byte strings with zero mutation, a sharp phase transition that looks like gelation, and a proof that blocking deep symbiogenetic ancestry trees prevents the transition entirely.A few things worth flagging for critical viewers:— The substrate is more carefully engineered than the framing sometimes suggests. The choice of language, tape length, interaction protocol, and step limits all shape what emerges. Their own SUBLEQ counterexample (where self-replicators *don't* arise despite being theoretically possible) highlights that these design choices matter substantially — and a general theory of which substrates support this transition is still missing.— The leap from "self-replicating programs on fixed-length tapes" to "life was computational and intelligent from the start" involves significant philosophical extrapolation beyond what the experiments directly demonstrate.— The Bedau et al. (2000) open problems paper he references at the start actually sets a higher bar for Challenge 3.2 than BFF currently meets: it asks that "the internal organization of these 'organisms' and the boundaries separating them from their environment arise and be sustained through the activities of lower-level primitives" — whereas BFF's tape boundaries are fixed by design, not emergent.

---

TIMESTAMPS:

00:00:00 Introduction: From Noise to Programs & ALife History

00:03:15 Defining Life: Function as the "Spirit"

00:05:45 Von Neumann's Insight: Life is Embodied Computation

00:09:15 Physics of Computation: Irreversibility & Fallacies

00:15:00 The BFF Experiment: Spontaneous Generation of Code

00:23:45 The Mystery: Complexity Growth Without Mutation

00:27:00 Symbiogenesis: The Engine of Novelty

00:33:15 Mathematical Proof: Blocking Symbiosis Stops Life

00:40:15 Evolutionary Implications: It's Symbiogenesis All The Way Down

00:44:30 Intelligence as Modeling Others

00:46:49 Q&A: Levels of Abstraction & Definitions

---

REFERENCES:

Paper:

[00:01:16] Open Problems in Artificial Life

https://direct.mit.edu/artl/article/6/4/363/2354/Open-Problems-in-Artificial-Life

[00:09:30] When does a physical system compute?

https://arxiv.org/abs/1309.7979

[00:15:00] Computational Life

https://arxiv.org/abs/2406.19108

[00:27:30] On the Origin of Mitosing Cells

https://pubmed.ncbi.nlm.nih.gov/11541392/

[00:42:00] The Major Evolutionary Transitions

https://www.nature.com/articles/374227a0

[00:44:00] The ARC gene

https://www.nih.gov/news-events/news-releases/memory-gene-goes-viral

Person:

[00:05:45] Alan Turing

https://plato.stanford.edu/entries/turing/

[00:07:30] John von Neumann

https://en.wikipedia.org/wiki/John_von_Neumann

[00:11:15] Hector Zenil

https://hectorzenil.net/

[00:12:00] Robert Sapolsky

https://profiles.stanford.edu/robert-sapolsky

---

LINKS:

RESCRIPT: https://app.rescript.info/public/share/ff7gb6HpezOR3DF-gr9-rCoMFzzEgUjLQK6voV5XVWY

What makes something truly *intelligent?* Is a rock an agent? Could a perfect simulation of your brain actually *be* you? In this fascinating conversation, Dr. Jeff Beck takes us on a journey through the philosophical and technical foundations of agency, intelligence, and the future of AI.

Jeff doesn't hold back on the big questions. He argues that from a purely mathematical perspective, there's no structural difference between an agent and a rock – both execute policies that map inputs to outputs. The real distinction lies in *sophistication* – how complex are the internal computations? Does the system engage in planning and counterfactual reasoning, or is it just a lookup table that happens to give the right answers?

*Key topics explored in this conversation:*

*The Black Box Problem of Agency* – How can we tell if something is truly planning versus just executing a pre-computed response? Jeff explains why this question is nearly impossible to answer from the outside, and why the best we can do is ask which model gives us the simplest explanation.

*Energy-Based Models Explained* – A masterclass on how EBMs differ from standard neural networks. The key insight: traditional networks only optimize weights, while energy-based models optimize *both* weights and internal states – a subtle but profound distinction that connects to Bayesian inference.

*Why Your Brain Might Have Evolved from Your Nose* – One of the most surprising moments in the conversation. Jeff proposes that the complex, non-smooth nature of olfactory space may have driven the evolution of our associative cortex and planning abilities.

*The JEPA Revolution* – A deep dive into Yann LeCun's Joint Embedding Prediction Architecture and why learning in latent space (rather than predicting every pixel) might be the key to more robust AI representations.

*AI Safety Without Skynet Fears* – Jeff takes a refreshingly grounded stance on AI risk. He's less worried about rogue superintelligences and more concerned about humans becoming "reward function selectors" – couch potatoes who just approve or reject AI outputs. His proposed solution? Use inverse reinforcement learning to derive AI goals from observed human behavior, then make *small* perturbations rather than naive commands like "end world hunger."

Whether you're interested in the philosophy of mind, the technical details of modern machine learning, or just want to understand what makes intelligence *tick,* this conversation delivers insights you won't find anywhere else.

---

TIMESTAMPS:

00:00:00 Geometric Deep Learning & Physical Symmetries

00:00:56 Defining Agency: From Rocks to Planning

00:05:25 The Black Box Problem & Counterfactuals

00:08:45 Simulated Agency vs. Physical Reality

00:12:55 Energy-Based Models & Test-Time Training

00:17:30 Bayesian Inference & Free Energy

00:20:07 JEPA, Latent Space, & Non-Contrastive Learning

00:27:07 Evolution of Intelligence & Modular Brains

00:34:00 Scientific Discovery & Automated Experimentation

00:38:04 AI Safety, Enfeeblement & The Future of Work

---

REFERENCES:

Concept:

[00:00:58] Free Energy Principle (FEP)

https://en.wikipedia.org/wiki/Free_energy_principle

[00:06:00] Monte Carlo Tree Search

https://en.wikipedia.org/wiki/Monte_Carlo_tree_search

Book:

[00:09:00] The Intentional Stance

https://mitpress.mit.edu/9780262540537/the-intentional-stance/

Paper:

[00:13:00] A Tutorial on Energy-Based Learning (LeCun 2006)

http://yann.lecun.com/exdb/publis/pdf/lecun-06.pdf

[00:15:00] Auto-Encoding Variational Bayes (VAE)

https://arxiv.org/abs/1312.6114

[00:20:15] JEPA (Joint Embedding Prediction Architecture)

https://openreview.net/forum?id=BZ5a1r-kVsf

[00:22:30] The Wake-Sleep Algorithm

https://www.cs.toronto.edu/~hinton/absps/ws.pdf

<trunc, see rescript>

---

RESCRIPT:

https://app.rescript.info/public/share/DJlSbJ_Qx080q315tWaqMWn3PixCQsOcM4Kf1IW9_Eo

PDF:

https://app.rescript.info/api/public/sessions/0efec296b9b6e905/pdf

Professor Mazviita Chirimuuta joins us for a fascinating deep dive into the philosophy of neuroscience and what it really means to understand the mind.*What can neuroscience actually tell us about how the mind works?* In this thought-provoking conversation, we explore the hidden assumptions behind computational theories of the brain, the limits of scientific abstraction, and why the question of machine consciousness might be more complicated than AI researchers assume.Mazviita, author of *The Brain Abstracted,* brings a unique perspective shaped by her background in both neuroscience research and philosophy. She challenges us to think critically about the metaphors we use to understand cognition — from the reflex theory of the late 19th century to today's dominant view of the brain as a computer.*Key topics explored:**The problem of oversimplification* — Why scientific models necessarily leave things out, and how this can sometimes lead entire fields astray. The cautionary tale of reflex theory shows how elegant explanations can blind us to biological complexity.*Is the brain really a computer?* — Mazviita unpacks the philosophical assumptions behind computational neuroscience and asks: if we can model anything computationally, what makes brains special? The answer might challenge everything you thought you knew about AI.*Haptic realism* — A fresh way of thinking about scientific knowledge that emphasizes interaction over passive observation. Knowledge isn't about reading the "source code of the universe" — it's something we actively construct through engagement with the world.*Why embodiment matters for understanding* — Can a disembodied language model truly understand? Mazviita makes a compelling case that human cognition is deeply entangled with our sensory-motor engagement and biological existence in ways that can't simply be abstracted away.*Technology and human finitude* — Drawing on Heidegger, we discuss how the dream of transcending our physical limitations through technology might reflect a fundamental misunderstanding of what it means to be a knower.This conversation is essential viewing for anyone interested in AI, consciousness, philosophy of mind, or the future of cognitive science. Whether you're skeptical of strong AI claims or a true believer in machine consciousness, Mazviita's careful philosophical analysis will give you new tools for thinking through these profound questions.---TIMESTAMPS:00:00:00 The Problem of Generalizing Neuroscience00:02:51 Abstraction vs. Idealization: The "Kaleidoscope"00:05:39 Platonism in AI: Discovering or Inventing Patterns?00:09:42 When Simplification Fails: The Reflex Theory00:12:23 Behaviorism and the "Black Box" Trap00:14:20 Haptic Realism: Knowledge Through Interaction00:20:23 Is Nature Protean? The Myth of Converging Truth00:23:23 The Computational Theory of Mind: A Useful Fiction?00:27:25 Biological Constraints: Why Brains Aren't Just Neural Nets00:31:01 Agency, Distal Causes, and Dennett's Stances00:37:13 Searle's Challenge: Causal Powers and Understanding00:41:58 Heidegger's Warning & The Experiment on Children---REFERENCES:Book:[00:01:28] The Brain Abstractedhttps://mitpress.mit.edu/9780262548045/the-brain-abstracted/[00:11:05] The Integrated Action of the Nervous Systemhttps://www.amazon.sg/integrative-action-nervous-system/dp/9354179029[00:18:15] The Quest for Certainty (Dewey)https://www.amazon.com/Quest-Certainty-Relation-Knowledge-Lectures/dp/0399501916[00:19:45] Realism for Realistic People (Chang)https://www.cambridge.org/core/books/realism-for-realistic-people/ACC93A7F03B15AA4D6F3A466E3FC5AB7<truncated, see ReScript>---RESCRIPT:https://app.rescript.info/public/share/A6cZ1TY35p8ORMmYCWNBI0no9ChU3-Kx7dPXGJURvZ0PDF Transcript:https://app.rescript.info/api/public/sessions/0fb7767e066cf712/pdf

What if everything we think we know about the brain is just a really good metaphor that we forgot was a metaphor?This episode takes you on a journey through the history of scientific simplification, from a young Karl Friston watching wood lice in his garden to the bold claims that your mind is literally software running on biological hardware.We bring together some of the most brilliant minds we've interviewed — Professor Mazviita Chirimuuta, Francois Chollet, Joscha Bach, Professor Luciano Floridi, Professor Noam Chomsky, Nobel laureate John Jumper, and more — to wrestle with a deceptively simple question: *When scientists simplify reality to study it, what gets captured and what gets lost?**Key ideas explored:**The Spherical Cow Problem* — Science requires simplification. We're limited creatures trying to understand systems far more complex than our working memory can hold. But when does a useful model become a dangerous illusion?*The Kaleidoscope Hypothesis* — Francois Chollet's beautiful idea that beneath all the apparent chaos of reality lies simple, repeating patterns — like bits of colored glass in a kaleidoscope creating infinite complexity. Is this profound truth or Platonic wishful thinking?*Is Software Really Spirit?* — Joscha Bach makes the provocative claim that software is literally spirit, not metaphorically. We push back on this, asking whether the "sameness" we see across different computers running the same program exists in nature or only in our descriptions.*The Cultural Illusion of AGI* — Why does artificial general intelligence seem so inevitable to people in Silicon Valley? Professor Chirimuuta suggests we might be caught in a "cultural historical illusion" — our mechanistic assumptions about minds making AI seem like destiny when it might just be a bet.*Prediction vs. Understanding* — Nobel Prize winner John Jumper: AI can predict and control, but understanding requires a human in the loop. Throughout history, we've described the brain as hydraulic pumps, telegraph networks, telephone switchboards, and now computers. Each metaphor felt obviously true at the time. This episode asks: what will we think was naive about our current assumptions in fifty years?Featuring insights from *The Brain Abstracted* by Mazviita Chirimuuta — possibly the most influential book on how we think about thinking in 2025.---TIMESTAMPS:00:00:00 The Wood Louse & The Spherical Cow00:02:04 The Necessity of Abstraction00:04:42 Simplicius vs. Ignorantio: The Boxing Match00:06:39 The Kaleidoscope Hypothesis00:08:40 Is the Mind Software?00:13:15 Critique of Causal Patterns00:14:40 Temperature is Not a Thing00:18:24 The Ship of Theseus & Ontology00:23:45 Metaphors Hardening into Reality00:25:41 The Illusion of AGI Inevitability00:27:45 Prediction vs. Understanding00:32:00 Climbing the Mountain vs. The Helicopter00:34:53 Haptic Realism & The Limits of Knowledge---REFERENCES:Person:[00:00:00] Karl Friston (UCL)https://profiles.ucl.ac.uk/1236-karl-friston[00:06:30] Francois Chollethttps://fchollet.com/[00:14:41] Cesar Hidalgo, MLST interview.https://www.youtube.com/watch?v=vzpFOJRteeI[00:30:30] Terence Tao's Bloghttps://terrytao.wordpress.com/Book:[00:02:25] The Brain Abstractedhttps://mitpress.mit.edu/9780262548045/the-brain-abstracted/[00:06:00] On Learned Ignorancehttps://www.amazon.com/Nicholas-Cusa-learned-ignorance-translation/dp/0938060236[00:24:15] Science and the Modern Worldhttps://amazon.com/dp/0684836394<truncated, see ReScript>

RESCRIPT:https://app.rescript.info/public/share/CYy0ex2M2kvcVRdMnSUky5O7H7hB7v2u_nVhoUiuKD4PDF Transcript: https://app.rescript.info/api/public/sessions/6c44c41e1e0fa6dd/pdf

Thank you to Dr. Maxwell Ramstead for early script work on this show (Ph.D student of Friston) and the woodlice story came from him!

Dr. Jeff Beck, mathematician turned computational neuroscientist, joins us for a fascinating deep dive into why the future of AI might look less like ChatGPT and more like your own brain.

**SPONSOR MESSAGES START**

—

Prolific - Quality data. From real people. For faster breakthroughs.

https://www.prolific.com/?utm_source=mlst

—

**END**

*What if the key to building truly intelligent machines isn't bigger models, but smarter ones?*

In this conversation, Jeff makes a compelling case that we've been building AI backwards. While the tech industry races to scale up transformers and language models, Jeff argues we're missing something fundamental: the brain doesn't work like a giant prediction engine. It works like a scientist, constantly testing hypotheses about a world made of *objects* that interact through *forces* — not pixels and tokens.

*The Bayesian Brain* — Jeff explains how your brain is essentially running the scientific method on autopilot. When you combine what you see with what you hear, you're doing optimal Bayesian inference without even knowing it. This isn't just philosophy — it's backed by decades of behavioral experiments showing humans are surprisingly efficient at handling uncertainty.

*AutoGrad Changed Everything* — Forget transformers for a moment. Jeff argues the real hero of the AI boom was automatic differentiation, which turned AI from a math problem into an engineering problem. But in the process, we lost sight of what actually makes intelligence work.

*The Cat in the Warehouse Problem* — Here's where it gets practical. Imagine a warehouse robot that's never seen a cat. Current AI would either crash or make something up. Jeff's approach? Build models that *know what they don't know*, can phone a friend to download new object models on the fly, and keep learning continuously. It's like giving robots the ability to say "wait, what IS that?" instead of confidently being wrong.

*Why Language is a Terrible Model for Thought* — In a provocative twist, Jeff argues that grounding AI in language (like we do with LLMs) is fundamentally misguided. Self-report is the least reliable data in psychology — people routinely explain their own behavior incorrectly. We should be grounding AI in physics, not words.

*The Future is Lots of Little Models* — Instead of one massive neural network, Jeff envisions AI systems built like video game engines: thousands of small, modular object models that can be combined, swapped, and updated independently. It's more efficient, more flexible, and much closer to how we actually think.

Rescript: https://app.rescript.info/public/share/D-b494t8DIV-KRGYONJghvg-aelMmxSDjKthjGdYqsE

---

TIMESTAMPS:

00:00:00 Introduction & The Bayesian Brain

00:01:25 Bayesian Inference & Information Processing

00:05:17 The Brain Metaphor: From Levers to Computers

00:10:13 Micro vs. Macro Causation & Instrumentalism

00:16:59 The Active Inference Community & AutoGrad

00:22:54 Object-Centered Models & The Grounding Problem

00:35:50 Scaling Bayesian Inference & Architecture Design

00:48:05 The Cat in the Warehouse: Solving Generalization

00:58:17 Alignment via Belief Exchange

01:05:24 Deception, Emergence & Cellular Automata

---

REFERENCES:

Paper:

[00:00:24] Zoubin Ghahramani (Google DeepMind)

https://pmc.ncbi.nlm.nih.gov/articles/PMC3538441/pdf/rsta201

[00:19:20] Mamba: Linear-Time Sequence Modeling

https://arxiv.org/abs/2312.00752

[00:27:36] xLSTM: Extended Long Short-Term Memory

https://arxiv.org/abs/2405.04517

[00:41:12] 3D Gaussian Splatting

https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/

[01:07:09] Lenia: Biology of Artificial Life

https://arxiv.org/abs/1812.05433

[01:08:20] Growing Neural Cellular Automata

https://distill.pub/2020/growing-ca/

[01:14:05] DreamCoder

https://arxiv.org/abs/2006.08381

[01:14:58] The Genomic Bottleneck

https://www.nature.com/articles/s41467-019-11786-6

Person:

[00:16:42] Karl Friston (UCL)

https://www.youtube.com/watch?v=PNYWi996Beg

Tim sits down with Max Bennett to explore how our brains evolved over 600 million years—and what that means for understanding both human intelligence and AI.

Max isn't a neuroscientist by training. He's a tech entrepreneur who got curious, started reading, and ended up weaving together three fields that rarely talk to each other: comparative psychology (what different animals can actually do), evolutionary neuroscience (how brains changed over time), and AI (what actually works in practice).

*Your Brain Is a Guessing Machine*

You don't actually "see" the world. Your brain builds a simulation of what it *thinks* is out there and just uses your eyes to check if it's right. That's why optical illusions work—your brain is filling in a triangle that isn't there, or can't decide if it's looking at a duck or a rabbit.

*Rats Have Regrets*

*Chimps Are Machiavellian*

*Language Is the Human Superpower*

*Does ChatGPT Think?*

(truncated description, more on rescript)

Understanding how the brain evolved isn't just about the past. It gives us clues about:

- What's actually different between human intelligence and AI

- Why we're so easily fooled by status games and tribal thinking

- What features we might want to build into—or leave out of—future AI systems

Get Max's book:

https://www.amazon.com/Brief-History-Intelligence-Humans-Breakthroughs/dp/0063286343

Rescript: https://app.rescript.info/public/share/R234b7AXyDXZusqQ_43KMGsUSvJ2TpSz2I3emnI6j9A

---

TIMESTAMPS:

00:00:00 Introduction: Outsider's Advantage & Neocortex Theories

00:11:34 Perception as Inference: The Filling-In Machine

00:19:11 Understanding, Recognition & Generative Models

00:36:39 How Mice Plan: Vicarious Trial & Error

00:46:15 Evolution of Self: The Layer 4 Mystery

00:58:31 Ancient Minds & The Social Brain: Machiavellian Apes

01:19:36 AI Alignment, Instrumental Convergence & Status Games

01:33:07 Metacognition & The IQ Paradox

01:48:40 Does GPT Have Theory of Mind?

02:00:40 Memes, Language Singularity & Brain Size Myths

02:16:44 Communication, Language & The Cyborg Future

02:44:25 Shared Fictions, World Models & The Reality Gap

---

REFERENCES:Person:

[00:00:05] Karl Friston (UCL)

https://www.youtube.com/watch?v=PNYWi996Beg

[00:00:06] Jeff Hawkins

https://www.youtube.com/watch?v=6VQILbDqaI4

[00:12:19] Hermann von Helmholtz

https://plato.stanford.edu/entries/hermann-helmholtz/

[00:38:34] David Redish (U. Minnesota)

https://redishlab.umn.edu/

[01:10:19] Robin Dunbar

https://www.psy.ox.ac.uk/people/robin-dunbar

[01:15:04] Emil Menzel

https://www.sciencedirect.com/bookseries/behavior-of-nonhuman-primates/vol/5/suppl/C

[01:19:49] Nick Bostrom

https://nickbostrom.com/

[02:28:25] Noam Chomsky

https://linguistics.mit.edu/user/chomsky/

[03:01:22] Judea Pearl

https://samueli.ucla.edu/people/judea-pearl/

Concept/Framework:

[00:05:04] Active Inference

https://www.youtube.com/watch?v=KkR24ieh5Ow

Paper:

[00:35:59] Predictions not commands [Rick A Adams]

https://pubmed.ncbi.nlm.nih.gov/23129312/

Book:

[01:25:42] The Elephant in the Brain

https://www.amazon.com/Elephant-Brain-Hidden-Motives-Everyday/dp/0190495995

[01:28:27] The Status Game

https://www.goodreads.com/book/show/58642436-the-status-game

[02:00:40] The Selfish Gene

https://amazon.com/dp/0198788606

[02:14:25] The Language Game

https://www.amazon.com/Language-Game-Improvisation-Created-Changed/dp/1541674987

[02:54:40] The Evolution of Language

https://www.amazon.com/Evolution-Language-Approaches/dp/052167736X

[03:09:37] The Three-Body Problem

https://amazon.com/dp/0765377063

César Hidalgo has spent years trying to answer a deceptively simple question: What is knowledge, and why is it so hard to move around?

We all have this intuition that knowledge is just... information. Write it down in a book, upload it to GitHub, train an AI on it—done. But César argues that's completely wrong. Knowledge isn't a thing you can copy and paste. It's more like a living organism that needs the right environment, the right people, and constant exercise to survive.

Guest: César Hidalgo, Director of the Center for Collective Learning

1. Knowledge Follows Laws (Like Physics)

2. You Can't Download Expertise

3. Why Big Companies Fail to Adapt

4. The "Infinite Alphabet" of Economies

If you think AI can just "copy" human knowledge, or that development is just about throwing money at poor countries, or that writing things down preserves them forever—this conversation will change your mind. Knowledge is fragile, specific, and collective. It decays fast if you don't use it.

The Infinite Alphabet [César A. Hidalgo]

https://www.penguin.co.uk/books/458054/the-infinite-alphabet-by-hidalgo-cesar-a/9780241655672

https://x.com/cesifoti

Rescript link.

https://app.rescript.info/public/share/eaBHbEo9xamwbwpxzcVVm4NQjMh7lsOQKeWwNxmw0JQ

---

TIMESTAMPS:

00:00:00 The Three Laws of Knowledge

00:02:28 Rival vs. Non-Rival: The Economics of Ideas

00:05:43 Why You Can't Just 'Download' Knowledge

00:08:11 The Detective Novel Analogy

00:11:54 Collective Learning & Organizational Networks

00:16:27 Architectural Innovation: Amazon vs. Barnes & Noble

00:19:15 The First Law: Learning Curves

00:23:05 The Samuel Slater Story: Treason & Memory

00:28:31 Physics of Knowledge: Joule's Cannon

00:32:33 Extensive vs. Intensive Properties

00:35:45 Knowledge Decay: Ise Temple & Polaroid

00:41:20 Absorptive Capacity: Sony & Donetsk

00:47:08 Disruptive Innovation & S-Curves

00:51:23 Team Size & The Cost of Innovation

00:57:13 Geography of Knowledge: Vespa's Origin

01:04:34 Migration, Diversity & 'Planet China'

01:12:02 Institutions vs. Knowledge: The China Story

01:21:27 Economic Complexity & The Infinite Alphabet

01:32:27 Do LLMs Have Knowledge?

---

REFERENCES:

Book:

[00:47:45] The Innovator's Dilemma (Christensen)

https://www.amazon.com/Innovators-Dilemma-Revolutionary-Change-Business/dp/0062060244

[00:55:15] Why Greatness Cannot Be Planned

https://amazon.com/dp/3319155237

[01:35:00] Why Information Grows

https://amazon.com/dp/0465048994

Paper:

[00:03:15] Endogenous Technological Change (Romer, 1990)

https://web.stanford.edu/~klenow/Romer_1990.pdf

[00:03:30] A Model of Growth Through Creative Destruction (Aghion & Howitt, 1992)

https://dash.harvard.edu/server/api/core/bitstreams/7312037d-2b2d-6bd4-e053-0100007fdf3b/content

[00:14:55] Organizational Learning: From Experience to Knowledge (Argote & Miron-Spektor, 2011)

https://www.researchgate.net/publication/228754233_Organizational_Learning_From_Experience_to_Knowledge

[00:17:05] Architectural Innovation (Henderson & Clark, 1990)

https://www.researchgate.net/publication/200465578_Architectural_Innovation_The_Reconfiguration_of_Existing_Product_Technologies_and_the_Failure_of_Established_Firms

[00:19:45] The Learning Curve Equation (Thurstone, 1916)

https://dn790007.ca.archive.org/0/items/learningcurveequ00thurrich/learningcurveequ00thurrich.pdf

[00:21:30] Factors Affecting the Cost of Airplanes (Wright, 1936)

https://pdodds.w3.uvm.edu/research/papers/others/1936/wright1936a.pdf

[00:52:45] Are Ideas Getting Harder to Find? (Bloom et al.)

https://web.stanford.edu/~chadj/IdeaPF.pdf

[01:33:00] LLMs/ Emergence

https://arxiv.org/abs/2506.11135

Person:

[00:25:30] Samuel Slater

https://en.wikipedia.org/wiki/Samuel_Slater

[00:42:05] Masaru Ibuka (Sony)

https://www.sony.com/en/SonyInfo/CorporateInfo/History/SonyHistory/1-02.html

This is a lively, no-holds-barred debate about whether AI can truly be intelligent, conscious, or understand anything at all — and what happens when (or if) machines become smarter than us.

Dr. Mike Israetel is a sports scientist, entrepreneur, and co-founder of RP Strength (a fitness company). He describes himself as a "dilettante" in AI but brings a fascinating outsider's perspective.

Jared Feather (IFBB Pro bodybuilder and exercise physiologist)

The Big Questions:

1. When is superintelligence coming?

2. Does AI actually understand anything?

3. The Simulation Debate (The Spiciest Part)

4. Will AI kill us all? (The Doomer Debate)

5. What happens to human jobs and purpose?

6. Do we need suffering?

Mikes channel: https://www.youtube.com/channel/UCfQgsKhHjSyRLOp9mnffqVg

RESCRIPT INTERACTIVE PLAYER: https://app.rescript.info/public/share/GVMUXHCqctPkXH8WcYtufFG7FQcdJew_RL_MLgMKU1U

---

TIMESTAMPS:

00:00:00 Introduction & Workout Demo

00:04:15 ASI Timelines & Definitions

00:10:24 The Embodiment Debate

00:18:28 Neutrinos & Abstract Knowledge

00:25:56 Can AI Learn From YouTube?

00:31:25 Diversity of Intelligence

00:36:00 AI Slop & Understanding

00:45:18 The Simulation Argument: Fire & Water

00:58:36 Consciousness & Zombies

01:04:30 Do Reasoning Models Actually Reason?

01:12:00 The Live Learning Problem

01:19:15 Superintelligence & Benevolence

01:28:59 What is True Agency?

01:37:20 Game Theory & The &quot;Kill All Humans&quot; Fallacy

01:48:05 Regulation & The China Factor

01:55:52 Mind Uploading & The Future of Love

02:04:41 Economics of ASI: Will We Be Useless?

02:13:35 The Matrix & The Value of Suffering

02:17:30 Transhumanism & Inequality

02:21:28 Debrief: AI Medical Advice & Final Thoughts

---

REFERENCES:

Paper:

[00:10:45] Alchemy and Artificial Intelligence (Dreyfus)

https://www.rand.org/content/dam/rand/pubs/papers/2006/P3244.pdf

[00:10:55] The Chinese Room Argument (John Searle)

https://home.csulb.edu/~cwallis/382/readings/482/searle.minds.brains.programs.bbs.1980.pdf

[00:11:05] The Symbol Grounding Problem (Stephen Harnad)

https://arxiv.org/html/cs/9906002

[00:23:00] Attention Is All You Need

https://arxiv.org/abs/1706.03762

[00:45:00] GPT-4 Technical Report

https://arxiv.org/abs/2303.08774

[01:45:00] Anthropic Agentic Misalignment Paper

https://www.anthropic.com/research/agentic-misalignment

[02:17:45] Retatrutide

https://pubmed.ncbi.nlm.nih.gov/37366315/

Organization:

[00:15:50] CERN

https://home.cern/

[01:05:00] METR Long Horizon Evaluations

https://evaluations.metr.org/

MLST Episode:

[00:23:10] MLST: Llion Jones - Inventors' Remorse

https://www.youtube.com/watch?v=DtePicx_kFY

[00:50:30] MLST: Blaise Agüera y Arcas Interview

https://www.youtube.com/watch?v=rMSEqJ_4EBk

[01:10:00] MLST: David Krakauer

https://www.youtube.com/watch?v=dY46YsGWMIc

Event:

[00:23:40] ARC Prize/Challenge

https://arcprize.org/

Book:

[00:24:45] The Brain Abstracted

https://www.amazon.com/Brain-Abstracted-Simplification-Philosophy-Neuroscience/dp/0262548046

[00:47:55] Pamela McCorduck

https://www.amazon.com/Machines-Who-Think-Artificial-Intelligence/dp/1568812051

[01:23:15] The Singularity Is Nearer (Ray Kurzweil)

https://www.amazon.com/Singularity-Nearer-Ray-Kurzweil-ebook/dp/B08Y6FYJVY

[01:27:35] A Fire Upon The Deep (Vernor Vinge)

https://www.amazon.com/Fire-Upon-Deep-S-F-MASTERWORKS-ebook/dp/B00AVUMIZE/

[02:04:50] Deep Utopia (Nick Bostrom)

https://www.amazon.com/Deep-Utopia-Meaning-Solved-World/dp/1646871642

[02:05:00] Technofeudalism (Yanis Varoufakis)

https://www.amazon.com/Technofeudalism-Killed-Capitalism-Yanis-Varoufakis/dp/1685891241

Visual Context Needed:

[00:29:40] AT-AT Walker (Star Wars)

https://starwars.fandom.com/wiki/All_Terrain_Armored_Transport

Person:

[00:33:15] Andrej Karpathy

https://karpathy.ai/

Video:

[01:40:00] Mike Israetel vs Liron Shapira AI Doom Debate

https://www.youtube.com/watch?v=RaDWSPMdM4o

Company:

[02:26:30] Examine.com

https://examine.com/

We often think of Large Language Models (LLMs) as all-knowing, but as the team reveals, they still struggle with the logic of a second-grader. Why can’t ChatGPT reliably add large numbers? Why does it "hallucinate" the laws of physics? The answer lies in the architecture. This episode explores how *Category Theory* —an ultra-abstract branch of mathematics—could provide the "Periodic Table" for neural networks, turning the "alchemy" of modern AI into a rigorous science.

In this deep-dive exploration, *Andrew Dudzik*, *Petar Velichkovich*, *Taco Cohen*, *Bruno Gavranović*, and *Paul Lessard* join host *Tim Scarfe* to discuss the fundamental limitations of today’s AI and the radical mathematical framework that might fix them.

TRANSCRIPT:

https://app.rescript.info/public/share/LMreunA-BUpgP-2AkuEvxA7BAFuA-VJNAp2Ut4MkMWk

---

Key Insights in This Episode:

* *The "Addition" Problem:* *Andrew Dudzik* explains why LLMs don't actually "know" math—they just recognize patterns. When you change a single digit in a long string of numbers, the pattern breaks because the model lacks the internal "machinery" to perform a simple carry operation.

* *Beyond Alchemy:* deep learning is currently in its "alchemy" phase—we have powerful results, but we lack a unifying theory. Category Theory is proposed as the framework to move AI from trial-and-error to principled engineering. [00:13:49]

* *Algebra with Colors:* To make Category Theory accessible, the guests use brilliant analogies—like thinking of matrices as *magnets with colors* that only snap together when the types match. This "partial compositionality" is the secret to building more complex internal reasoning. [00:09:17]

* *Synthetic vs. Analytic Math:* *Paul Lessard* breaks down the philosophical shift needed in AI research: moving from "Analytic" math (what things are made of) to "Synthetic" math [00:23:41]

---

Why This Matters for AGI

If we want AI to solve the world's hardest scientific problems, it can't just be a "stochastic parrot." It needs to internalize the rules of logic and computation. By imbuing neural networks with categorical priors, researchers are attempting to build a future where AI doesn't just predict the next word—it understands the underlying structure of the universe.

---

TIMESTAMPS:

00:00:00 The Failure of LLM Addition & Physics

00:01:26 Tool Use vs Intrinsic Model Quality

00:03:07 Efficiency Gains via Internalization

00:04:28 Geometric Deep Learning & Equivariance

00:07:05 Limitations of Group Theory

00:09:17 Category Theory: Algebra with Colors

00:11:25 The Systematic Guide of Lego-like Math

00:13:49 The Alchemy Analogy & Unifying Theory

00:15:33 Information Destruction & Reasoning

00:18:00 Pathfinding & Monoids in Computation

00:20:15 System 2 Reasoning & Error Awareness

00:23:31 Analytic vs Synthetic Mathematics

00:25:52 Morphisms & Weight Tying Basics

00:26:48 2-Categories & Weight Sharing Theory

00:28:55 Higher Categories & Emergence

00:31:41 Compositionality & Recursive Folds

00:34:05 Syntax vs Semantics in Network Design

00:36:14 Homomorphisms & Multi-Sorted Syntax

00:39:30 The Carrying Problem & Hopf Fibrations

Petar Veličković (GDM)

https://petar-v.com/

Paul Lessard

https://www.linkedin.com/in/paul-roy-lessard/

Bruno Gavranović

https://www.brunogavranovic.com/

Andrew Dudzik (GDM)

https://www.linkedin.com/in/andrew-dudzik-222789142/

---

REFERENCES:

Model:

[00:01:05] Veo

https://deepmind.google/models/veo/

[00:01:10] Genie

https://deepmind.google/blog/genie-3-a-new-frontier-for-world-models/

Paper:

[00:04:30] Geometric Deep Learning Blueprint

https://arxiv.org/abs/2104.13478

https://www.youtube.com/watch?v=bIZB1hIJ4u8

[00:16:45] AlphaGeometry

https://arxiv.org/abs/2401.08312

[00:16:55] AlphaCode

https://arxiv.org/abs/2203.07814

[00:17:05] FunSearch

https://www.nature.com/articles/s41586-023-06924-6

[00:37:00] Attention Is All You Need

https://arxiv.org/abs/1706.03762

[00:43:00] Categorical Deep Learning

https://arxiv.org/abs/2402.15332

Is a car that wins a Formula 1 race the best choice for your morning commute? Probably not. In this sponsored deep dive with Prolific, we explore why the same logic applies to Artificial Intelligence. While models are currently shattering records on technical exams, they often fail the most important test of all: **the human experience.**

Why High Benchmark Scores Don’t Mean Better AI

Joining us are **Andrew Gordon** (Staff Researcher in Behavioral Science) and **Nora Petrova** (AI Researcher) from **Prolific**. They reveal the hidden flaws in how we currently rank AI and introduce a more rigorous, "humane" way to measure whether these models are actually helpful, safe, and relatable for real people.

---

Key Insights in This Episode:

* *The F1 Car Analogy:* Andrew explains why a model that excels at the "Humanities Last Exam" might be a nightmare for daily use. Technical benchmarks often ignore the nuances of human communication and adaptability.

* *The "Wild West" of AI Safety:* As users turn to AI for sensitive topics like mental health, Nora highlights the alarming lack of oversight and the "thin veneer" of safety training—citing recent controversial incidents like Grok-3’s "Mecha Hitler."

* *Fixing the "Leaderboard Illusion":* The team critiques current popular rankings like Chatbot Arena, discussing how anonymous, unstratified voting can lead to biased results and how companies can "game" the system.

* *The Xbox Secret to AI Ranking:* Discover how Prolific uses *TrueSkill*—the same algorithm Microsoft developed for Xbox Live matchmaking—to create a fairer, more statistically sound leaderboard for LLMs.

* *The Personality Gap:* Early data from the **Humane Leaderboard** suggests that while AI is getting smarter, it is actually performing *worse* on metrics like personality, culture, and "sycophancy" (the tendency for models to become annoying "people-pleasers").

---

About the HUMAINE Leaderboard

Moving beyond simple "A vs. B" testing, the researchers discuss their new framework that samples participants based on *census data* (Age, Ethnicity, Political Alignment). By using a representative sample of the general public rather than just tech enthusiasts, they are building a standard that reflects the values of the real world.

*Are we building models for benchmarks, or are we building them for humans? It’s time to change the scoreboard.*

Rescript link:

https://app.rescript.info/public/share/IDqwjY9Q43S22qSgL5EkWGFymJwZ3SVxvrfpgHZLXQc

---

TIMESTAMPS:

00:00:00 Introduction & The Benchmarking Problem

00:01:58 The Fractured State of AI Evaluation

00:03:54 AI Safety & Interpretability

00:05:45 Bias in Chatbot Arena

00:06:45 Prolific&apos;s Three Pillars Approach

00:09:01 TrueSkill Ranking & Efficient Sampling

00:12:04 Census-Based Representative Sampling

00:13:00 Key Findings: Culture, Personality & Sycophancy

---

REFERENCES:

Paper:

[00:00:15] MMLU

https://arxiv.org/abs/2009.03300

[00:05:10] Constitutional AI

https://arxiv.org/abs/2212.08073

[00:06:45] The Leaderboard Illusion

https://arxiv.org/abs/2504.20879

[00:09:41] HUMAINE Framework Paper

https://huggingface.co/blog/ProlificAI/humaine-framework

Company:

[00:00:30] Prolific

https://www.prolific.com

[00:01:45] Chatbot Arena

https://lmarena.ai/

Person:

[00:00:35] Andrew Gordon

https://www.linkedin.com/in/andrew-gordon-03879919a/

[00:00:45] Nora Petrova

https://www.linkedin.com/in/nora-petrova/

Event:

Algorithm:

[00:09:01] Microsoft TrueSkill

https://www.microsoft.com/en-us/research/project/trueskill-ranking-system/

Leaderboard:

[00:09:21] Prolific HUMAINE Leaderboard

https://www.prolific.com/humaine

[00:09:31] HUMAINE HuggingFace Space

https://huggingface.co/spaces/ProlificAI/humaine-leaderboard

[00:10:21] Prolific AI Leaderboard Portal

https://www.prolific.com/leaderboard

Dataset:

[00:09:51] Prolific Social Reasoning RLHF Dataset

https://huggingface.co/datasets/ProlificAI/social-reasoning-rlhf

Organization:

[00:10:31] MLCommons

https://mlcommons.org/

What if everything we think we know about AI understanding is wrong? Is compression the key to intelligence? Or is there something more—a leap from memorization to true abstraction?

In this fascinating conversation, we sit down with **Professor Yi Ma**—world-renowned expert in deep learning, IEEE/ACM Fellow, and author of the groundbreaking new book *Learning Deep Representations of Data Distributions*. Professor Ma challenges our assumptions about what large language models actually do, reveals why 3D reconstruction isn't the same as understanding, and presents a unified mathematical theory of intelligence built on just two principles: **parsimony** and **self-consistency**.

**SPONSOR MESSAGES START**

—

Prolific - Quality data. From real people. For faster breakthroughs.

https://www.prolific.com/?utm_source=mlst

—

cyber•Fund https://cyber.fund/?utm_source=mlst is a founder-led investment firm accelerating the cybernetic economy

Hiring a SF VC Principal: https://talent.cyber.fund/companies/cyber-fund-2/jobs/57674170-ai-investment-principal#content?utm_source=mlst

Submit investment deck: https://cyber.fund/contact?utm_source=mlst

—

**END**

Key Insights:

**LLMs Don't Understand—They Memorize**

Language models process text (*already* compressed human knowledge) using the same mechanism we use to learn from raw data.

**The Illusion of 3D Vision**

Sora and NeRFs etc that can reconstruct 3D scenes still fail miserably at basic spatial reasoning

**"All Roads Lead to Rome"**

Why adding noise is *necessary* for discovering structure.

**Why Gradient Descent Actually Works**

Natural optimization landscapes are surprisingly smooth—a "blessing of dimensionality"

**Transformers from First Principles**

Transformer architectures can be mathematically derived from compression principles

—

INTERACTIVE AI TRANSCRIPT PLAYER w/REFS (ReScript):

https://app.rescript.info/public/share/Z-dMPiUhXaeMEcdeU6Bz84GOVsvdcfxU_8Ptu6CTKMQ

About Professor Yi Ma

Yi Ma is the inaugural director of the School of Computing and Data Science at Hong Kong University and a visiting professor at UC Berkeley.

https://people.eecs.berkeley.edu/~yima/

https://scholar.google.com/citations?user=XqLiBQMAAAAJ&hl=en

https://x.com/YiMaTweets

**Slides from this conversation:**

https://www.dropbox.com/scl/fi/sbhbyievw7idup8j06mlr/slides.pdf?rlkey=7ptovemezo8bj8tkhfi393fh9&dl=0

**Related Talks by Professor Ma:**

- Pursuing the Nature of Intelligence (ICLR): https://www.youtube.com/watch?v=LT-F0xSNSjo

- Earlier talk at Berkeley: https://www.youtube.com/watch?v=TihaCUjyRLM

TIMESTAMPS:

00:00:00 Introduction

00:02:08 The First Principles Book & Research Vision

00:05:21 Two Pillars: Parsimony & Consistency

00:09:50 Evolution vs. Learning: The Compression Mechanism

00:14:36 LLMs: Memorization Masquerading as Understanding

00:19:55 The Leap to Abstraction: Empirical vs. Scientific

00:27:30 Platonism, Deduction & The ARC Challenge

00:35:57 Specialization & The Cybernetic Legacy

00:41:23 Deriving Maximum Rate Reduction

00:48:21 The Illusion of 3D Understanding: Sora & NeRF

00:54:26 All Roads Lead to Rome: The Role of Noise

00:59:56 All Roads Lead to Rome: The Role of Noise

01:00:14 Benign Non-Convexity: Why Optimization Works

01:06:35 Double Descent & The Myth of Overfitting

01:14:26 Self-Consistency: Closed-Loop Learning

01:21:03 Deriving Transformers from First Principles

01:30:11 Verification & The Kevin Murphy Question

01:34:11 CRATE vs. ViT: White-Box AI & Conclusion

REFERENCES:

Book:

[00:03:04] Learning Deep Representations of Data Distributions

https://ma-lab-berkeley.github.io/deep-representation-learning-book/

[00:18:38] A Brief History of Intelligence

https://www.amazon.co.uk/BRIEF-HISTORY-INTELLIGEN-HB-Evolution/dp/0008560099

[00:38:14] Cybernetics

https://mitpress.mit.edu/9780262730099/cybernetics/

Book (Yi Ma):

[00:03:14] 3-D Vision book

https://link.springer.com/book/10.1007/978-0-387-21779-6

<TRUNC> refs on ReScript link/YT

Pedro Domingos, author of the bestselling book "The Master Algorithm," introduces his latest work: Tensor Logic - a new programming language he believes could become the fundamental language for artificial intelligence.

Think of it like this: Physics found its language in calculus. Circuit design found its language in Boolean logic. Pedro argues that AI has been missing its language - until now.

**SPONSOR MESSAGES START**

—

Build your ideas with AI Studio from Google - http://ai.studio/build

—

Prolific - Quality data. From real people. For faster breakthroughs.

https://www.prolific.com/?utm_source=mlst

—

cyber•Fund https://cyber.fund/?utm_source=mlst is a founder-led investment firm accelerating the cybernetic economy

Hiring a SF VC Principal: https://talent.cyber.fund/companies/cyber-fund-2/jobs/57674170-ai-investment-principal#content?utm_source=mlst

Submit investment deck: https://cyber.fund/contact?utm_source=mlst

—

**END**

Current AI is split between two worlds that don't play well together:

Deep Learning (neural networks, transformers, ChatGPT) - great at learning from data, terrible at logical reasoning

Symbolic AI (logic programming, expert systems) - great at logical reasoning, terrible at learning from messy real-world data

Tensor Logic unifies both. It's a single language where you can:

Write logical rules that the system can actually learn and modify

Do transparent, verifiable reasoning (no hallucinations)

Mix "fuzzy" analogical thinking with rock-solid deduction

INTERACTIVE TRANSCRIPT:

https://app.rescript.info/public/share/NP4vZQ-GTETeN_roB2vg64vbEcN7isjJtz4C86WSOhw

TOC:

00:00:00 - Introduction

00:04:41 - What is Tensor Logic?

00:09:59 - Tensor Logic vs PyTorch & Einsum

00:17:50 - The Master Algorithm Connection

00:20:41 - Predicate Invention & Learning New Concepts

00:31:22 - Symmetries in AI & Physics

00:35:30 - Computational Reducibility & The Universe

00:43:34 - Technical Details: RNN Implementation

00:45:35 - Turing Completeness Debate

00:56:45 - Transformers vs Turing Machines

01:02:32 - Reasoning in Embedding Space

01:11:46 - Solving Hallucination with Deductive Modes

01:16:17 - Adoption Strategy & Migration Path

01:21:50 - AI Education & Abstraction

01:24:50 - The Trillion-Dollar Waste

REFS

Tensor Logic: The Language of AI [Pedro Domingos]

https://arxiv.org/abs/2510.12269

The Master Algorithm [Pedro Domingos]

https://www.amazon.co.uk/Master-Algorithm-Ultimate-Learning-Machine/dp/0241004543

Einsum is All you Need (TIM ROCKTÄSCHEL)

https://rockt.ai/2018/04/30/einsum

https://www.youtube.com/watch?v=6DrCq8Ry2cw

Autoregressive Large Language Models are Computationally Universal (Dale Schuurmans et al - GDM)

https://arxiv.org/abs/2410.03170

Memory Augmented Large Language Models are Computationally Universal [Dale Schuurmans]

https://arxiv.org/pdf/2301.04589

On the computational power of NNs [95/Siegelmann]

https://binds.cs.umass.edu/papers/1995_Siegelmann_JComSysSci.pdf

Sebastian Bubeck

https://www.reddit.com/r/OpenAI/comments/1oacp38/openai_researcher_sebastian_bubeck_falsely_claims/

I am a strange loop - Hofstadter

https://www.amazon.co.uk/Am-Strange-Loop-Douglas-Hofstadter/dp/0465030793

Stephen Wolfram

https://www.youtube.com/watch?v=dkpDjd2nHgo

The Complex World: An Introduction to the Foundations of Complexity Science [David C. Krakauer]

https://www.amazon.co.uk/Complex-World-Introduction-Foundations-Complexity/dp/1947864629

Geometric Deep Learning

https://www.youtube.com/watch?v=bIZB1hIJ4u8

Andrew Wilson (NYU)

https://www.youtube.com/watch?v=M-jTeBCEGHc

Yi Ma

https://www.patreon.com/posts/yi-ma-scientific-141953348

Roger Penrose - road to reality

https://www.amazon.co.uk/Road-Reality-Complete-Guide-Universe/dp/0099440687

Artificial Intelligence: A Modern Approach [Russel and Norvig]

https://www.amazon.co.uk/Artificial-Intelligence-Modern-Approach-Global/dp/1292153962

The Transformer architecture (which powers ChatGPT and nearly all modern AI) might be trapping the industry in a localized rut, preventing us from finding true intelligent reasoning, according to the person who co-invented it. Llion Jones and Luke Darlow, key figures at the research lab Sakana AI, join the show to make this provocative argument, and also introduce new research which might lead the way forwards.

**SPONSOR MESSAGES START**

—

Build your ideas with AI Studio from Google - http://ai.studio/build

—

Tufa AI Labs is hiring ML Research Engineers https://tufalabs.ai/

—

cyber•Fund https://cyber.fund/?utm_source=mlst is a founder-led investment firm accelerating the cybernetic economy

Hiring a SF VC Principal: https://talent.cyber.fund/companies/cyber-fund-2/jobs/57674170-ai-investment-principal#content?utm_source=mlst

Submit investment deck: https://cyber.fund/contact?utm_source=mlst

—

**END**

The "Spiral" Problem – Llion uses a striking visual analogy to explain what current AI is missing. If you ask a standard neural network to understand a spiral shape, it solves it by drawing tiny straight lines that just happen to look like a spiral. It "fakes" the shape without understanding the concept of spiraling.

Introducing the Continuous Thought Machine (CTM) Luke Darlow deep dives into their solution: a biology-inspired model that fundamentally changes how AI processes information.

The Maze Analogy: Luke explains that standard AI tries to solve a maze by staring at the whole image and guessing the entire path instantly. Their new machine "walks" through the maze step-by-step.

Thinking Time: This allows the AI to "ponder." If a problem is hard, the model can naturally spend more time thinking about it before answering, effectively allowing it to correct its own mistakes and backtrack—something current Language Models struggle to do genuinely.

https://sakana.ai/

https://x.com/YesThisIsLion

https://x.com/LearningLukeD

TRANSCRIPT:

https://app.rescript.info/public/share/crjzQ-Jo2FQsJc97xsBdfzfOIeMONpg0TFBuCgV2Fu8

TOC:

00:00:00 - Stepping Back from Transformers

00:00:43 - Introduction to Continuous Thought Machines (CTM)

00:01:09 - The Changing Atmosphere of AI Research

00:04:13 - Sakana’s Philosophy: Research Freedom

00:07:45 - The Local Minimum of Large Language Models

00:18:30 - Representation Problems: The Spiral Example

00:29:12 - Technical Deep Dive: CTM Architecture

00:36:00 - Adaptive Computation & Maze Solving

00:47:15 - Model Calibration & Uncertainty

01:00:43 - Sudoku Bench: Measuring True Reasoning

REFS:

Why Greatness Cannot be planned [Kenneth Stanley]

https://www.amazon.co.uk/Why-Greatness-Cannot-Planned-Objective/dp/3319155237

https://www.youtube.com/watch?v=lhYGXYeMq_E

The Hardware Lottery [Sara Hooker]

https://arxiv.org/abs/2009.06489

https://www.youtube.com/watch?v=sQFxbQ7ade0

Continuous Thought Machines [Luke Darlow et al / Sakana]

https://arxiv.org/abs/2505.05522

https://sakana.ai/ctm/

LSTM: The Comeback Story? [Prof. Sepp Hochreiter]

https://www.youtube.com/watch?v=8u2pW2zZLCs

Questioning Representational Optimism in Deep Learning: The Fractured Entangled Representation Hypothesis [Kumar/Stanley]

https://arxiv.org/pdf/2505.11581

A Spline Theory of Deep Networks [Randall Balestriero]

https://proceedings.mlr.press/v80/balestriero18b/balestriero18b.pdf

https://www.youtube.com/watch?v=86ib0sfdFtw

https://www.youtube.com/watch?v=l3O2J3LMxqI

On the Biology of a Large Language Model [Anthropic, Jack Lindsey et al]

https://transformer-circuits.pub/2025/attribution-graphs/biology.html

The ARC Prize 2024 Winning Algorithm [Daniel Franzen and Jan Disselhoff] “The ARChitects”

https://www.youtube.com/watch?v=mTX_sAq--zY

Neural Turing Machine [Graves]

https://arxiv.org/pdf/1410.5401

Adaptive Computation Time for Recurrent Neural Networks [Graves]

https://arxiv.org/abs/1603.08983

Sudoko Bench [Sakana]

https://pub.sakana.ai/sudoku/

Ever wonder where AI models actually get their "intelligence"? We reveal the dirty secret of Silicon Valley: behind every impressive AI system are thousands of real humans providing crucial data, feedback, and expertise.Guest: Phelim Bradley, CEO and Co-founder of ProlificPhelim Bradley runs Prolific, a platform that connects AI companies with verified human experts who help train and evaluate their models. Think of it as a sophisticated marketplace matching the right human expertise to the right AI task - whether that's doctors evaluating medical chatbots or coders reviewing AI-generated software.Prolific: https://prolific.com/?utm_source=mlsthttps://uk.linkedin.com/in/phelim-bradley-84300826The discussion dives into:**The human data pipeline**: How AI companies rely on human intelligence to train, refine, and validate their models - something rarely discussed openly**Quality over quantity**: Why paying humans well and treating them as partners (not commodities) produces better AI training data**The matching challenge**: How Prolific solves the complex problem of finding the right expert for each specific task, similar to matching Uber drivers to riders but with deep expertise requirements**Future of work**: What it means when human expertise becomes an on-demand service, and why this might actually create more opportunities rather than fewer**Geopolitical implications**: Why the centralization of AI development in US tech companies should concern Europe and the UK

"What is life?" - asks Chris Kempes, a professor at the Santa Fe Institute.

Chris explains that scientists are moving beyond a purely Earth-based, biological view and are searching for a universal theory of life that could apply to anything, anywhere in the universe. He proposes that things we don't normally consider "alive"—like human culture, language, or even artificial intelligence; could be seen as life forms existing on different "substrates".

To understand this, Chris presents a fascinating three-level framework:

- Materials: The physical stuff life is made of. He argues this could be incredibly diverse across the universe, and we shouldn't expect alien life to share our biochemistry.

- Constraints: The universal laws of physics (like gravity or diffusion) that all life must obey, regardless of what it's made of. This is where different life forms start to look more similar.

- Principles: At the highest level are abstract principles like evolution and learning. Chris suggests these computational or "optimization" rules are what truly define a living system.

A key idea is "convergence" – using the example of the eye. It's such a complex organ that you'd think it evolved only once. However, eyes evolved many separate times across different species. This is because the physics of light provides a clear "target", and evolution found similar solutions to the problem of seeing, even with different starting materials.

**SPONSOR MESSAGES**

—

Prolific - Quality data. From real people. For faster breakthroughs.

https://www.prolific.com/?utm_source=mlst

—

Check out NotebookLM from Google here - https://notebooklm.google.com/ - it’s really good for doing research directly from authoritative source material, minimising hallucinations.

—

cyber•Fund https://cyber.fund/?utm_source=mlst is a founder-led investment firm accelerating the cybernetic economy

Hiring a SF VC Principal: https://talent.cyber.fund/companies/cyber-fund-2/jobs/57674170-ai-investment-principal#content?utm_source=mlst

Submit investment deck: https://cyber.fund/contact?utm_source=mlst

—

Prof. Chris Kempes:

https://www.santafe.edu/people/profile/chris-kempes

TRANSCRIPT:

https://app.rescript.info/public/share/Y2cI1i0nX_-iuZitvlguHvaVLQTwPX1Y_E1EHxV0i9I

TOC:

00:00:00 - Introduction to Chris Kempes and the Santa Fe Institute

00:02:28 - The Three Cultures of Science

00:05:08 - What Makes a Good Scientific Theory?

00:06:50 - The Universal Theory of Life

00:09:40 - The Role of Material in Life

00:12:50 - A Hierarchy for Understanding Life

00:13:55 - How Life Diversifies and Converges

00:17:53 - Adaptive Processes and Defining Life

00:19:28 - Functionalism, Memes, and Phylogenies

00:22:58 - Convergence at Multiple Levels

00:25:45 - The Possibility of Simulating Life

00:28:16 - Intelligence, Parasitism, and Spectrums of Life

00:32:39 - Phase Changes in Evolution

00:36:16 - The Separation of Matter and Logic

00:37:21 - Assembly Theory and Quantifying Complexity

REFS:

Developing a predictive science of the biosphere requires the integration of scientific cultures [Kempes et al]

https://www.pnas.org/doi/10.1073/pnas.2209196121

Seeing with an extra sense (“Dangerous prediction”) [Rob Phillips]

https://www.sciencedirect.com/science/article/pii/S0960982224009035

The Multiple Paths to Multiple Life [Christopher P. Kempes & David C. Krakauer]

https://link.springer.com/article/10.1007/s00239-021-10016-2

The Information Theory of Individuality [David Krakauer et al]

https://arxiv.org/abs/1412.2447

Minds, Brains and Programs [Searle]

https://home.csulb.edu/~cwallis/382/readings/482/searle.minds.brains.programs.bbs.1980.pdf

The error threshold

https://www.sciencedirect.com/science/article/abs/pii/S0168170204003843

Assembly theory and its relationship with computational complexity [Kempes et al]

https://arxiv.org/abs/2406.12176

Blaise Agüera y Arcas explores some mind-bending ideas about what intelligence and life really are—and why they might be more similar than we think (filmed at ALIFE conference, 2025 - https://2025.alife.org/).

Life and intelligence are both fundamentally computational (he says). From the very beginning, living things have been running programs. Your DNA? It's literally a computer program, and the ribosomes in your cells are tiny universal computers building you according to those instructions.

**SPONSOR MESSAGES**

—

Prolific - Quality data. From real people. For faster breakthroughs.

https://www.prolific.com/?utm_source=mlst

—

cyber•Fund https://cyber.fund/?utm_source=mlst is a founder-led investment firm accelerating the cybernetic economy

Oct SF conference - https://dagihouse.com/?utm_source=mlst - Joscha Bach keynoting(!) + OAI, Anthropic, NVDA,++

Hiring a SF VC Principal: https://talent.cyber.fund/companies/cyber-fund-2/jobs/57674170-ai-investment-principal#content?utm_source=mlst

Submit investment deck: https://cyber.fund/contact?utm_source=mlst

—

Blaise argues that there is more to evolution than random mutations (like most people think). The secret to increasing complexity is *merging* i.e. when different organisms or systems come together and combine their histories and capabilities.

Blaise describes his "BFF" experiment where random computer code spontaneously evolved into self-replicating programs, showing how purpose and complexity can emerge from pure randomness through computational processes.

https://en.wikipedia.org/wiki/Blaise_Ag%C3%BCera_y_Arcas

https://x.com/blaiseaguera?lang=en

TRANSCRIPT:

https://app.rescript.info/public/share/VX7Gktfr3_wIn4Bj7cl9StPBO1MN4R5lcJ11NE99hLg

TOC:

00:00:00 Introduction - New book "What is Intelligence?"

00:01:45 Life as computation - Von Neumann's insights

00:12:00 BFF experiment - How purpose emerges

00:26:00 Symbiogenesis and evolutionary complexity

00:40:00 Functionalism and consciousness

00:49:45 AI as part of collective human intelligence

00:57:00 Comparing AI and human cognition

REFS:

What is intelligence [Blaise Agüera y Arcas]

https://whatisintelligence.antikythera.org/ [Read free online, interactive rich media]

https://mitpress.mit.edu/9780262049955/what-is-intelligence/ [MIT Press]

Large Language Models and Emergence: A Complex Systems Perspective

https://arxiv.org/abs/2506.11135

Our first Noam Chomsky MLST interview

https://www.youtube.com/watch?v=axuGfh4UR9Q

Chance and Necessity [Jacques Monod]

https://monoskop.org/images/9/99/Monod_Jacques_Chance_and_Necessity.pdf

Wonderful Life: The Burgess Shale and the History of Nature [Stephen Jay Gould]

https://www.amazon.co.uk/Wonderful-Life-Burgess-Nature-History/dp/0099273454

The major evolutionary transitions [E Szathmáry, J M Smith]

https://wiki.santafe.edu/images/0/0e/Szathmary.MaynardSmith_1995_Nature.pdf

Don't Sleep, There Are Snakes: Life and Language in the Amazonian Jungle [Dan Everett]

https://www.amazon.com/Dont-Sleep-There-Are-Snakes/dp/0307386120

The Nature of Technology: What It Is and How It Evolves [W. Brian Arthur]

https://www.amazon.com/Nature-Technology-What-How-Evolves-ebook/dp/B002RI9W16/

The MANIAC [Benjamin Labatut]

https://www.amazon.com/MANIAC-Benjam%C3%ADn-Labatut/dp/1782279814

When We Cease to Understand the World [Benjamin Labatut]

https://www.amazon.com/When-We-Cease-Understand-World/dp/1681375664/

The Boys in the Boat [Dan Brown]

https://www.amazon.com/Boys-Boat-Americans-Berlin-Olympics/dp/0143125478

[Petter Johansson] (Split brain)

https://www.lucs.lu.se/fileadmin/user_upload/lucs/2011/01/Johansson-et-al.-2006-How-Something-Can-Be-Said-About-Telling-More-Than-We-Can-Know.pdf

If Anyone Builds It, Everyone Dies [Eliezer Yudkowsky, Nate Soares]

https://www.amazon.com/Anyone-Builds-Everyone-Dies-Superhuman/dp/0316595640

The science of cycology

https://link.springer.com/content/pdf/10.3758/bf03195929.pdf

<trunc, see YT desc for more>

We sat down with Sara Saab (VP of Product at Prolific) and Enzo Blindow (VP of Data and AI at Prolific) to explore the critical role of human evaluation in AI development and the challenges of aligning AI systems with human values. Prolific is a human annotation and orchestration platform for AI used by many of the major AI labs. This is a sponsored show in partnership with Prolific.

**SPONSOR MESSAGES**

—

cyber•Fund https://cyber.fund/?utm_source=mlst is a founder-led investment firm accelerating the cybernetic economy

Oct SF conference - https://dagihouse.com/?utm_source=mlst - Joscha Bach keynoting(!) + OAI, Anthropic, NVDA,++

Hiring a SF VC Principal: https://talent.cyber.fund/companies/cyber-fund-2/jobs/57674170-ai-investment-principal#content?utm_source=mlst

Submit investment deck: https://cyber.fund/contact?utm_source=mlst

—

While technologists want to remove humans from the loop for speed and efficiency, these non-deterministic AI systems actually require more human oversight than ever before. Prolific's approach is to put "well-treated, verified, diversely demographic humans behind an API" - making human feedback as accessible as any other infrastructure service.

When AI models like Grok 4 achieve top scores on technical benchmarks but feel awkward or problematic to use in practice, it exposes the limitations of our current evaluation methods. The guests argue that optimizing for benchmarks may actually weaken model performance in other crucial areas, like cultural sensitivity or natural conversation.

We also discuss Anthropic's research showing that frontier AI models, when given goals and access to information, independently arrived at solutions involving blackmail - without any prompting toward unethical behavior. Even more concerning, the more sophisticated the model, the more susceptible it was to this "agentic misalignment."

Enzo and Sarah present Prolific's "Humane" leaderboard as an alternative to existing benchmarking systems. By stratifying evaluations across diverse demographic groups, they reveal that different populations have vastly different experiences with the same AI models.

Looking ahead, the guests imagine a world where humans take on coaching and teaching roles for AI systems - similar to how we might correct a child or review code. This also raises important questions about working conditions and the evolution of labor in an AI-augmented world. Rather than replacing humans entirely, we may be moving toward more sophisticated forms of human-AI collaboration.

As AI tech becomes more powerful and general-purpose, the quality of human evaluation becomes more critical, not less. We need more representative evaluation frameworks that capture the messy reality of human values and cultural diversity.

Visit Prolific:

https://www.prolific.com/

Sara Saab (VP Product):

https://uk.linkedin.com/in/sarasaab

Enzo Blindow (VP Data & AI):

https://uk.linkedin.com/in/enzoblindow

TRANSCRIPT:

https://app.rescript.info/public/share/xZ31-0kJJ_xp4zFSC-bunC8-hJNkHpbm7Lg88RFcuLE

TOC:

[00:00:00] Intro & Background

[00:03:16] Human-in-the-Loop Challenges

[00:17:19] Can AIs Understand?

[00:32:02] Benchmarking & Vibes

[00:51:00] Agentic Misalignment Study

[01:03:00] Data Quality vs Quantity

[01:16:00] Future of AI Oversight

REFS:

Anthropic Agentic Misalignment

https://www.anthropic.com/research/agentic-misalignment

Value Compass

https://arxiv.org/pdf/2409.09586

Reasoning Models Don’t Always Say What They Think (Anthropic)

https://www.anthropic.com/research/reasoning-models-dont-say-think

https://assets.anthropic.com/m/71876fabef0f0ed4/original/reasoning_models_paper.pdf

Apollo research - science of evals blog post

https://www.apolloresearch.ai/blog/we-need-a-science-of-evals

Leaderboard Illusion

https://www.youtube.com/watch?v=9W_OhS38rIE MLST video

The Leaderboard Illusion [2025]

Shivalika Singh et al

https://arxiv.org/abs/2504.20879

(Truncated, full list on YT)

Dr. Ilia Shumailov - Former DeepMind AI Security Researcher, now building security tools for AI agents

Ever wondered what happens when AI agents start talking to each other—or worse, when they start breaking things? Ilia Shumailov spent years at DeepMind thinking about exactly these problems, and he's here to explain why securing AI is way harder than you think.

**SPONSOR MESSAGES**

—Check out notebooklm for your research project, it's really powerfulhttps://notebooklm.google.com/

—

Take the Prolific human data survey - https://www.prolific.com/humandatasurvey?utm_source=mlst and be the first to see the results and benchmark their practices against the wider community!

—

cyber•Fund https://cyber.fund/?utm_source=mlst is a founder-led investment firm accelerating the cybernetic economy

Oct SF conference - https://dagihouse.com/?utm_source=mlst - Joscha Bach keynoting(!) + OAI, Anthropic, NVDA,++

Hiring a SF VC Principal: https://talent.cyber.fund/companies/cyber-fund-2/jobs/57674170-ai-investment-principal#content?utm_source=mlst

Submit investment deck: https://cyber.fund/contact?utm_source=mlst

—

We're racing toward a world where AI agents will handle our emails, manage our finances, and interact with sensitive data 24/7. But there is a problem. These agents are nothing like human employees. They never sleep, they can touch every endpoint in your system simultaneously, and they can generate sophisticated hacking tools in seconds. Traditional security measures designed for humans simply won't work.

Dr. Ilia Shumailov

https://x.com/iliaishacked

https://iliaishacked.github.io/

https://sequrity.ai/

TRANSCRIPT:

https://app.rescript.info/public/share/dVGsk8dz9_V0J7xMlwguByBq1HXRD6i4uC5z5r7EVGM

TOC:

00:00:00 - Introduction & Trusted Third Parties via ML

00:03:45 - Background & Career Journey

00:06:42 - Safety vs Security Distinction

00:09:45 - Prompt Injection & Model Capability

00:13:00 - Agents as Worst-Case Adversaries

00:15:45 - Personal AI & CAML System Defense

00:19:30 - Agents vs Humans: Threat Modeling

00:22:30 - Calculator Analogy & Agent Behavior

00:25:00 - IMO Math Solutions & Agent Thinking

00:28:15 - Diffusion of Responsibility & Insider Threats

00:31:00 - Open Source Security Concerns

00:34:45 - Supply Chain Attacks & Trust Issues

00:39:45 - Architectural Backdoors

00:44:00 - Academic Incentives & Defense Work

00:48:30 - Semantic Censorship & Halting Problem

00:52:00 - Model Collapse: Theory & Criticism

00:59:30 - Career Advice & Ross Anderson Tribute

REFS:

Lessons from Defending Gemini Against Indirect Prompt Injections

https://arxiv.org/abs/2505.14534

Defeating Prompt Injections by Design.

Debenedetti, E., Shumailov, I., Fan, T., Hayes, J., Carlini, N., Fabian, D., Kern, C., Shi, C., Terzis, A., & Tramèr, F.

https://arxiv.org/pdf/2503.18813

Agentic Misalignment: How LLMs could be insider threats

https://www.anthropic.com/research/agentic-misalignment

STOP ANTHROPOMORPHIZING INTERMEDIATE TOKENS AS REASONING/THINKING TRACES!

Subbarao Kambhampati et al

https://arxiv.org/pdf/2504.09762

Meiklejohn, S., Blauzvern, H., Maruseac, M., Schrock, S., Simon, L., & Shumailov, I. (2025).

Machine learning models have a supply chain problem.

https://arxiv.org/abs/2505.22778

Gao, Y., Shumailov, I., & Fawaz, K. (2025).

Supply-chain attacks in machine learning frameworks.

https://openreview.net/pdf?id=EH5PZW6aCr

Apache Log4j Vulnerability Guidance

https://www.cisa.gov/news-events/news/apache-log4j-vulnerability-guidance

Bober-Irizar, M., Shumailov, I., Zhao, Y., Mullins, R., & Papernot, N. (2022).

Architectural backdoors in neural networks.

https://arxiv.org/pdf/2206.07840

Position: Fundamental Limitations of LLM Censorship Necessitate New Approaches

David Glukhov, Ilia Shumailov, ...

https://proceedings.mlr.press/v235/glukhov24a.html

AlphaEvolve MLST interview [Matej Balog, Alexander Novikov]

https://www.youtube.com/watch?v=vC9nAosXrJw

We need AI systems to synthesise new knowledge, not just compress the data they see. Jeremy Berman, is a research scientist at Reflection AI and recent winner of the ARC-AGI v2 public leaderboard.**SPONSOR MESSAGES**—Take the Prolific human data survey - https://www.prolific.com/humandatasurvey?utm_source=mlst and be the first to see the results and benchmark their practices against the wider community!—cyber•Fund https://cyber.fund/?utm_source=mlst is a founder-led investment firm accelerating the cybernetic economyOct SF conference - https://dagihouse.com/?utm_source=mlst - Joscha Bach keynoting(!) + OAI, Anthropic, NVDA,++Hiring a SF VC Principal: https://talent.cyber.fund/companies/cyber-fund-2/jobs/57674170-ai-investment-principal#content?utm_source=mlstSubmit investment deck: https://cyber.fund/contact?utm_source=mlst— Imagine trying to teach an AI to think like a human i.e. solving puzzles that are easy for us but stump even the smartest models. Jeremy's evolutionary approach—evolving natural language descriptions instead of python code like his last version—landed him at the top with about 30% accuracy on the ARCv2.We discuss why current AIs are like "stochastic parrots" that memorize but struggle to truly reason or innovate as well as big ideas like building "knowledge trees" for real understanding, the limits of neural networks versus symbolic systems, and whether we can train models to synthesize new ideas without forgetting everything else. Jeremy Berman:https://x.com/jerber888TRANSCRIPT:https://app.rescript.info/public/share/qvCioZeZJ4Q_NlR66m-hNUZnh-qWlUJcS15Wc2OGwD0TOC:Introduction and Overview [00:00:00]ARC v1 Solution [00:07:20]Evolutionary Python Approach [00:08:00]Trade-offs in Depth vs. Breadth [00:10:33]ARC v2 Improvements [00:11:45]Natural Language Shift [00:12:35]Model Thinking Enhancements [00:13:05]Neural Networks vs. Symbolism Debate [00:14:24]Turing Completeness Discussion [00:15:24]Continual Learning Challenges [00:19:12]Reasoning and Intelligence [00:29:33]Knowledge Trees and Synthesis [00:50:15]Creativity and Invention [00:56:41]Future Directions and Closing [01:02:30]REFS:Jeremy’s 2024 article on winning ARCAGI1-pubhttps://jeremyberman.substack.com/p/how-i-got-a-record-536-on-arc-agiGetting 50% (SoTA) on ARC-AGI with GPT-4o [Greenblatt]https://blog.redwoodresearch.org/p/getting-50-sota-on-arc-agi-with-gpt https://www.youtube.com/watch?v=z9j3wB1RRGA [his MLST interview]A Thousand Brains: A New Theory of Intelligence [Hawkins]https://www.amazon.com/Thousand-Brains-New-Theory-Intelligence/dp/1541675819https://www.youtube.com/watch?v=6VQILbDqaI4 [MLST interview]Francois Chollet + Mike Knoop’s labhttps://ndea.com/On the Measure of Intelligence [Chollet]https://arxiv.org/abs/1911.01547On the Biology of a Large Language Model [Anthropic]https://transformer-circuits.pub/2025/attribution-graphs/biology.html The ARChitects [won 2024 ARC-AGI-1-private]https://www.youtube.com/watch?v=mTX_sAq--zY Connectionism critique 1998 [Fodor/Pylshyn]https://uh.edu/~garson/F&P1.PDF Questioning Representational Optimism in Deep Learning: The Fractured Entangled Representation Hypothesis [Kumar/Stanley]https://arxiv.org/pdf/2505.11581 AlphaEvolve interview (also program synthesis)https://www.youtube.com/watch?v=vC9nAosXrJw ShinkaEvolve: Evolving New Algorithms with LLMs, Orders of Magnitude More Efficiently [Lange et al]https://sakana.ai/shinka-evolve/ Deep learning with Python Rev 3 [Chollet] - READ CHAPTER 19 NOW!https://deeplearningwithpython.io/

Professor Andrew Wilson from NYU explains why many common-sense ideas in artificial intelligence might be wrong. For decades, the rule of thumb in machine learning has been to fear complexity. The thinking goes: if your model has too many parameters (is "too complex") for the amount of data you have, it will "overfit" by essentially memorizing the data instead of learning the underlying patterns. This leads to poor performance on new, unseen data. This is known as the classic "bias-variance trade-off" i.e. a balancing act between a model that's too simple and one that's too complex.

**SPONSOR MESSAGES**

—

Tufa AI Labs is an AI research lab based in Zurich. **They are hiring ML research engineers!**

This is a once in a lifetime opportunity to work with one of the best labs in Europe

Contact Benjamin Crouzier - https://tufalabs.ai/

—

Take the Prolific human data survey - https://www.prolific.com/humandatasurvey?utm_source=mlst and be the first to see the results and benchmark their practices against the wider community!

—

cyber•Fund https://cyber.fund/?utm_source=mlst is a founder-led investment firm accelerating the cybernetic economy

Oct SF conference - https://dagihouse.com/?utm_source=mlst - Joscha Bach keynoting(!) + OAI, Anthropic, NVDA,++

Hiring a SF VC Principal: https://talent.cyber.fund/companies/cyber-fund-2/jobs/57674170-ai-investment-principal#content?utm_source=mlst

Submit investment deck: https://cyber.fund/contact?utm_source=mlst

—

Description Continued:

Professor Wilson challenges this fundamental belief (fearing complexity). He makes a few surprising points:

**Bigger Can Be Better**: massive models don't just get more flexible; they also develop a stronger "simplicity bias". So, if your model is overfitting, the solution might paradoxically be to make it even bigger.

**The "Bias-Variance Trade-off" is a Misnomer**: Wilson claims you don't actually have to trade one for the other. You can have a model that is incredibly expressive and flexible while also being strongly biased toward simple solutions. He points to the "double descent" phenomenon, where performance first gets worse as models get more complex, but then surprisingly starts getting better again.

**Honest Beliefs and Bayesian Thinking**: His core philosophy is that we should build models that honestly represent our beliefs about the world. We believe the world is complex, so our models should be expressive. But we also believe in Occam's razor—that the simplest explanation is often the best. He champions Bayesian methods, which naturally balance these two ideas through a process called marginalization, which he describes as an automatic Occam's razor.

TOC:

[00:00:00] Introduction and Thesis

[00:04:19] Challenging Conventional Wisdom

[00:11:17] The Philosophy of a Scientist-Engineer

[00:16:47] Expressiveness, Overfitting, and Bias

[00:28:15] Understanding, Compression, and Kolmogorov Complexity

[01:05:06] The Surprising Power of Generalization

[01:13:21] The Elegance of Bayesian Inference

[01:33:02] The Geometry of Learning

[01:46:28] Practical Advice and The Future of AI

Prof. Andrew Gordon Wilson:

https://x.com/andrewgwils

https://cims.nyu.edu/~andrewgw/

https://scholar.google.com/citations?user=twWX2LIAAAAJ&hl=en

https://www.youtube.com/watch?v=Aja0kZeWRy4

https://www.youtube.com/watch?v=HEp4TOrkwV4

TRANSCRIPT:

https://app.rescript.info/public/share/H4Io1Y7Rr54MM05FuZgAv4yphoukCfkqokyzSYJwCK8

Hosts:

Dr. Tim Scarfe / Dr. Keith Duggar (MIT Ph.D)

REFS:

Deep Learning is Not So Mysterious or Different [Andrew Gordon Wilson]

https://arxiv.org/abs/2503.02113

Bayesian Deep Learning and a Probabilistic Perspective of Generalization [Andrew Gordon Wilson, Pavel Izmailov]

https://arxiv.org/abs/2002.08791

Compute-Optimal LLMs Provably Generalize Better With Scale [Marc Finzi, Sanyam Kapoor, Diego Granziol, Anming Gu, Christopher De Sa, J. Zico Kolter, Andrew Gordon Wilson]

https://arxiv.org/abs/2504.15208

In this episode, hosts Tim and Keith finally realize their long-held dream of sitting down with their hero, the brilliant neuroscientist Professor Karl Friston. The conversation is a fascinating and mind-bending journey into Professor Friston's life's work, the Free Energy Principle, and what it reveals about life, intelligence, and consciousness itself.

**SPONSORS**

Gemini CLI is an open-source AI agent that brings the power of Gemini directly into your terminal - https://github.com/google-gemini/gemini-cli

---

Take the Prolific human data survey - https://www.prolific.com/humandatasurvey?utm_source=mlst and be the first to see the results and benchmark their practices against the wider community!

---

cyber•Fund https://cyber.fund/?utm_source=mlst is a founder-led investment firm accelerating the cybernetic economy

Oct SF conference - https://dagihouse.com/?utm_source=mlst - Joscha Bach keynoting(!) + OAI, Anthropic, NVDA,++

Hiring a SF VC Principal: https://talent.cyber.fund/companies/cyber-fund-2/jobs/57674170-ai-investment-principal#content?utm_source=mlst

Submit investment deck: https://cyber.fund/contact?utm_source=mlst

***

They kick things off by looking back on the 20-year journey of the Free Energy Principle. Professor Friston explains it as a fundamental rule for survival: all living things, from a single cell to a human being, are constantly trying to make sense of the world and reduce unpredictability. It’s this drive to minimize surprise that allows things to exist and maintain their structure.

This leads to a bigger question: What does it truly mean to be "intelligent"? The group debates whether intelligence is everywhere, even in a virus or a plant, or if it requires a certain level of complexity.

Professor Friston introduces the idea of different "kinds" of things, suggesting that creatures like us, who can model themselves and think about the future, possess a unique and "strange" kind of agency that sets us apart.

From intelligence, the discussion naturally flows to the even trickier concept of consciousness. Is it the same as intelligence? Professor Friston argues they are different. He explains that consciousness might emerge from deep, layered self-awareness—not just acting, but understanding that you are the one causing your actions and thinking about your place in the world.

They also explore intelligence at different sizes. Is a corporation intelligent? What about the entire planet? Professor Friston suggests there might be a "Goldilocks zone" for intelligence. It doesn't seem to exist at the super-tiny atomic level or at the massive scale of planets and solar systems, but thrives in the complex middle-ground where we live.

Finally, they tackle one of the most pressing topics of our time: Can we build a truly conscious AI? Professor Friston shares his doubts about whether our current computers are capable of a feat like that. He suggests that genuine consciousness might require a different kind of "mortal" computation, where the machine's physical body and its "mind" are inseparable, much like in biological creatures.

TRANSCRIPT:

https://app.rescript.info/public/share/FZkF8BO7HMt9aFfu2_q69WGT_ZbYZ1VVkC6RtU3eeOI

TOC:

00:00:00: Introduction & Retrospective on the Free Energy Principle

00:09:34: Strange Particles, Agency, and Consciousness

00:37:45: The Scale of Intelligence: From Viruses to the Biosphere

01:01:35: Modelling, Boundaries, and Practical Application

01:21:12: Conclusion

We are joined by Cristopher Moore, a professor at the Santa Fe Institute with a diverse background in physics, computer science, and machine learning.

The conversation begins with Cristopher, who calls himself a "frog" explaining that he prefers to dive deep into specific, concrete problems rather than taking a high-level "bird's-eye view".

They explore why current AI models, like transformers, are so surprisingly effective. Cristopher argues it's because the real world isn't random; it's full of rich structures, patterns, and hierarchies that these models can learn to exploit, even if we don't fully understand how.

**SPONSORS**

Take the Prolific human data survey - https://www.prolific.com/humandatasurvey?utm_source=mlst and be the first to see the results and benchmark their practices against the wider community!

---

cyber•Fund https://cyber.fund/?utm_source=mlst is a founder-led investment firm accelerating the cybernetic economy.

Oct SF conference - https://dagihouse.com/?utm_source=mlst - Joscha Bach keynoting(!) + OAI, Anthropic, NVDA,++

Hiring a SF VC Principal: https://talent.cyber.fund/companies/cyber-fund-2/jobs/57674170-ai-investment-principal#content?utm_source=mlst

Submit investment deck: https://cyber.fund/contact?utm_source=mlst

***

Cristopher Moore:

https://sites.santafe.edu/~moore/

TOC:

00:00:00 - Introduction

00:02:05 - Meet Christopher Moore: A Frog in the World of Science

00:05:14 - The Limits of Transformers and Real-World Data

00:11:19 - Intelligence as Creative Problem-Solving

00:23:30 - Grounding, Meaning, and Shared Reality

00:31:09 - The Nature of Creativity and Aesthetics

00:44:31 - Computational Irreducibility and Universality

00:53:06 - Turing Completeness, Recursion, and Intelligence

01:11:26 - The Universe Through a Computational Lens

01:26:45 - Algorithmic Justice and the Need for Transparency

TRANSCRIPT: https://app.rescript.info/public/share/VRe2uQSvKZOm0oIBoDsrNwt46OMCqRnShVnUF3qyoFk

Filmed at DISI (Diverse Intelligences Summer Institute)

https://disi.org/

REFS:

The Nature of computation [Chris Moore]

https://nature-of-computation.org/

Birds and Frogs [Freeman Dyson]

https://www.ams.org/notices/200902/rtx090200212p.pdf

Replica Theory [Parisi et al]

https://arxiv.org/pdf/1409.2722

Janossy pooling [Fabian Fuchs]

https://fabianfuchsml.github.io/equilibriumaggregation/

Cracking the cryptic [YT channel]

https://www.youtube.com/c/CrackingTheCryptic

Sudoko Bench [Sakana]

https://sakana.ai/sudoku-bench/

Fractured entangled representations “phylogenetic locking in comment” [Kumar/Stanley]

https://arxiv.org/pdf/2505.11581 (see our shows on this)

The War Against Cliché: [Martin Amis]

https://www.amazon.com/War-Against-Cliche-Reviews-1971-2000/dp/0375727167

Rule 110 (CA)

https://mathworld.wolfram.com/Rule150.html

Universality in Elementary Cellular Automata [Matt Cooke]

https://wpmedia.wolfram.com/sites/13/2018/02/15-1-1.pdf

Small Semi-Weakly Universal Turing Machines [Damien Woods]

https://tilde.ini.uzh.ch/users/tneary/public_html/WoodsNeary-FI09.pdf

COMPUTING MACHINERY AND INTELLIGENCE [Turing, 1950]

https://courses.cs.umbc.edu/471/papers/turing.pdf

Comment on Space Time as a causal set [Moore, 88]

https://sites.santafe.edu/~moore/comment.pdf

Recursion Theory on the Reals and Continuous-time Computation [Moore, 96]

Dr. Michael Timothy Bennett is a computer scientist who's deeply interested in understanding artificial intelligence, consciousness, and what it means to be alive. He's known for his provocative paper "What the F*** is Artificial Intelligence" which challenges conventional thinking about AI and intelligence.**SPONSOR MESSAGES***Prolific: Quality data. From real people. For faster breakthroughs.https://prolific.com/mlst?utm_campaign=98404559-MLST&utm_source=youtube&utm_medium=podcast&utm_content=mb***Michael takes us on a journey through some of the biggest questions in AI and consciousness. He starts by exploring what intelligence actually is - settling on the idea that it's about "adaptation with limited resources" (a definition from researcher Pei Wang that he particularly likes).The discussion ranges from technical AI concepts to philosophical questions about consciousness, with Michael offering fresh perspectives that challenge Silicon Valley's "just scale it up" approach to AI. He argues that true intelligence isn't just about having more parameters or data - it's about being able to adapt efficiently, like biological systems do.TOC:1. Introduction & Paper Overview [00:01:34]2. Definitions of Intelligence [00:02:54]3. Formal Models (AIXI, Active Inference) [00:07:06]4. Causality, Abstraction & Embodiment [00:10:45]5. Computational Dualism & Mortal Computation [00:25:51]6. Modern AI, AGI Progress & Benchmarks [00:31:30]7. Hybrid AI Approaches [00:35:00]8. Consciousness & The Hard Problem [00:39:35]9. The Diverse Intelligences Summer Institute (DISI) [00:53:20]10. Living Systems & Self-Organization [00:54:17]11. Closing Thoughts [01:04:24]Michaels socials:https://michaeltimothybennett.com/https://x.com/MiTiBennettTranscript:https://app.rescript.info/public/share/4jSKbcM77Sf6Zn-Ms4hda7C4krRrMcQt0qwYqiqPTPIReferences:Bennett, M.T. "What the F*** is Artificial Intelligence"https://arxiv.org/abs/2503.23923Bennett, M.T. "Are Biological Systems More Intelligent Than Artificial Intelligence?" https://arxiv.org/abs/2405.02325Bennett, M.T. PhD Thesis "How To Build Conscious Machines"https://osf.io/preprints/thesiscommons/wehmg_v1Legg, S. & Hutter, M. (2007). "Universal Intelligence: A Definition of Machine Intelligence"Wang, P. "Defining Artificial Intelligence" - on non-axiomatic reasoning systems (NARS)Chollet, F. (2019). "On the Measure of Intelligence" - introduces the ARC benchmark and developer-aware generalizationHutter, M. (2005). "Universal Artificial Intelligence: Sequential Decisions Based on Algorithmic Probability"Chalmers, D. "The Hard Problem of Consciousness"Descartes, R. - Cartesian dualism and the pineal gland theory (historical context)Friston, K. - Free Energy Principle and Active Inference frameworkLevin, M. - Work on collective intelligence, cancer as information isolation, and "mind blindness"Hinton, G. (2022). "The Forward-Forward Algorithm" - introduces mortal computation conceptAlexander Ororbia & Friston - Formal treatment of mortal computationSutton, R. "The Bitter Lesson" - on search and learning in AIPearl, J. "The Book of Why" - causal inference and reasoningAlternative AGI ApproachesWang, P. - NARS (Non-Axiomatic Reasoning System)Goertzel, B. - Hyperon system and modular AGI architecturesBenchmarks & EvaluationHendrycks, D. - Humanities Last Exam benchmark (mentioned re: saturation)Filmed at:Diverse Intelligences Summer Institute (DISI) https://disi.org/

Deep dive with Dan Hendrycks, a leading AI safety researcher and co-author of the "Superintelligence Strategy" paper with former Google CEO Eric Schmidt and Scale AI CEO Alexandr Wang.

*** SPONSOR MESSAGES

Gemini CLI is an open-source AI agent that brings the power of Gemini directly into your terminal - https://github.com/google-gemini/gemini-cli

Prolific: Quality data. From real people. For faster breakthroughs.

https://prolific.com/mlst?utm_campaign=98404559-MLST&utm_source=youtube&utm_medium=podcast&utm_content=script-gen

***

Hendrycks argues that society is making a fundamental mistake in how it views artificial intelligence. We often compare AI to transformative but ultimately manageable technologies like electricity or the internet. He contends a far better and more realistic analogy is nuclear technology. Like nuclear power, AI has the potential for immense good, but it is also a dual-use technology that carries the risk of unprecedented catastrophe.

The Problem with an AI "Manhattan Project":

A popular idea is for the U.S. to launch a "Manhattan Project" for AI—a secret, all-out government race to build a superintelligence before rivals like China. Hendrycks argues this strategy is deeply flawed and dangerous for several reasons:

- It wouldn’t be secret. You cannot hide a massive, heat-generating data center from satellite surveillance.

- It would be destabilizing. A public race would alarm rivals, causing them to start their own desperate, corner-cutting projects, dramatically increasing global risk.

- It’s vulnerable to sabotage. An AI project can be crippled in many ways, from cyberattacks that poison its training data to physical attacks on its power plants. This is what the paper refers to as a "maiming attack."

This vulnerability leads to the paper's central concept: Mutual Assured AI Malfunction (MAIM). This is the AI-era version of the nuclear-era's Mutual Assured Destruction (MAD). In this dynamic, any nation that makes an aggressive, destabilizing bid for a world-dominating AI must expect its rivals to sabotage the project to ensure their own survival.

This deterrence, Hendrycks argues, is already the default reality we live in.

A Better Strategy: The Three Pillars

Instead of a reckless race, the paper proposes a more stable, three-part strategy modeled on Cold War principles:

- Deterrence: Acknowledge the reality of MAIM. The goal should not be to "win" the race to superintelligence, but to deter anyone from starting such a race in the first place through the credible threat of sabotage.

- Nonproliferation: Just as we work to keep fissile materials for nuclear bombs out of the hands of terrorists and rogue states, we must control the key inputs for catastrophic AI. The most critical input is advanced AI chips (GPUs). Hendrycks makes the powerful claim that building cutting-edge GPUs is now more difficult than enriching uranium, making this strategy viable.

- Competitiveness: The race between nations like the U.S. and China should not be about who builds superintelligence first. Instead, it should be about who can best use existing AI to build a stronger economy, a more effective military, and more resilient supply chains (for example, by manufacturing more chips domestically).

Dan says the stakes are high if we fail to manage this transition:

- Erosion of Control

- Intelligence Recursion

- Worthless Labor

Hendrycks maintains that while the risks are existential, the future is not set.

TOC:

1 Measuring the Beast [00:00:00]

2 Defining the Beast [00:11:34]

3 The Core Strategy [00:38:20]

4 Ideological Battlegrounds [00:53:12]

5 Mechanisms of Control [01:34:45]

TRANSCRIPT:

https://app.rescript.info/public/share/cOKcz4pWRPjh7BTIgybd7PUr_vChUaY6VQW64No8XMs

<truncated, see refs and larger description on YT version>

This episode features Shlomi Fuchter and Jack Parker Holder from Google DeepMind, who are unveiling a new AI called Genie 3. The host, Tim Scarfe, describes it as the most mind-blowing technology he has ever seen. We were invited to their offices to conduct the interview (not sponsored).Imagine you could create a video game world just by describing it. That's what Genie 3 does. It's an AI "world model" that learns how the real world works by watching massive amounts of video. Unlike a normal video game engine (like Unreal or the one for Doom) that needs to be programmed manually, Genie generates a realistic, interactive, 3D world from a simple text prompt.**SPONSOR MESSAGES***Prolific: Quality data. From real people. For faster breakthroughs.https://prolific.com/mlst?utm_campaign=98404559-MLST&utm_source=youtube&utm_medium=podcast&utm_content=script-gen***Here’s a breakdown of what makes it so revolutionary:From Text to a Virtual World: You can type "a drone flying by a beautiful lake" or "a ski slope," and Genie 3 creates that world for you in about three seconds. You can then navigate and interact with it in real-time.It's Consistent: The worlds it creates have a reliable memory. If you look away from an object and then look back, it will still be there, just as it was. The guests explain that this consistency isn't explicitly programmed in; it's a surprising, "emergent" capability of the powerful AI model.A Huge Leap Forward: The previous version, Genie 2, was a major step, but it wasn't fast enough for real-time interaction and was much lower resolution. Genie 3 is 720p, interactive, and photorealistic, running smoothly for several minutes at a time.The Killer App - Training Robots: Beyond entertainment, the team sees Genie 3 as a game-changer for training AI. Instead of training a self-driving car or a robot in the real world (which is slow and dangerous), you can create infinite simulations. You can even prompt rare events to happen, like a deer running across the road, to teach an AI how to handle unexpected situations safely.The Future of Entertainment: this could lead to a "YouTube version 2" or a new form of VR, where users can create and explore endless, interconnected worlds together, like the experience machine from philosophy.While the technology is still a research prototype and not yet available to the public, it represents a monumental step towards creating true artificial worlds from the ground up.Jack Parker Holder [Research Scientist at Google DeepMind in the Open-Endedness Team]https://jparkerholder.github.io/Shlomi Fruchter [Research Director, Google DeepMind]https://shlomifruchter.github.io/TOC:[00:00:00] - Introduction: "The Most Mind-Blowing Technology I've Ever Seen"[00:02:30] - The Evolution from Genie 1 to Genie 2[00:04:30] - Enter Genie 3: Photorealistic, Interactive Worlds from Text[00:07:00] - Promptable World Events & Training Self-Driving Cars[00:14:21] - Guest Introductions: Shlomi Fuchter & Jack Parker Holder[00:15:08] - Core Concepts: What is a "World Model"?[00:19:30] - The Challenge of Consistency in a Generated World[00:21:15] - Context: The Neural Network Doom Simulation[00:25:25] - How Do You Measure the Quality of a World Model?[00:28:09] - The Vision: Using Genie to Train Advanced Robots[00:32:21] - Open-Endedness: Human Skill and Prompting Creativity[00:38:15] - The Future: Is This the Next YouTube or VR?[00:42:18] - The Next Step: Multi-Agent Simulations[00:52:51] - Limitations: Thinking, Computation, and the Sim-to-Real Gap[00:58:07] - Conclusion & The Future of Game EnginesREFS:World Models [David Ha, Jürgen Schmidhuber]https://arxiv.org/abs/1803.10122POEThttps://arxiv.org/abs/1901.01753[Akarsh Kumar, Jeff Clune, Joel Lehman, Kenneth O. Stanley]The Fractured Entangled Representation Hypothesishttps://arxiv.org/pdf/2505.11581TRANSCRIPT:https://app.rescript.info/public/share/Zk5tZXk6mb06yYOFh6nSja7Lg6_qZkgkuXQ-kl5AJqM

Prof. David Krakauer, President of the Santa Fe Institute argues that we are fundamentally confusing knowledge with intelligence, especially when it comes to AI.

He defines true intelligence as the ability to do more with less—to solve novel problems with limited information. This is contrasted with current AI models, which he describes as doing less with more; they require astounding amounts of data to perform tasks that don't necessarily demonstrate true understanding or adaptation. He humorously calls this "really shit programming".

David challenges the popular notion of "emergence" in Large Language Models (LLMs). He explains that the tech community's definition—seeing a sudden jump in a model's ability to perform a task like three-digit math—is superficial. True emergence, from a complex systems perspective, involves a fundamental change in the system's internal organization, allowing for a new, simpler, and more powerful level of description. He gives the example of moving from tracking individual water molecules to using the elegant laws of fluid dynamics. For LLMs to be truly emergent, we'd need to see them develop new, efficient internal representations, not just get better at memorizing patterns as they scale.

Drawing on his background in evolutionary theory, David explains that systems like brains, and later, culture, evolved to process information that changes too quickly for genetic evolution to keep up. He calls culture "evolution at light speed" because it allows us to store our accumulated knowledge externally (in books, tools, etc.) and build upon it without corrupting the original.

This leads to his concept of "exbodiment," where we outsource our cognitive load to the world through things like maps, abacuses, or even language itself.

We create these external tools, internalize the skills they teach us, improve them, and create a feedback loop that enhances our collective intelligence.

However, he ends with a warning. While technology has historically complemented our deficient abilities, modern AI presents a new danger. Because we have an evolutionary drive to conserve energy, we will inevitably outsource our thinking to AI if we can. He fears this is already leading to a "diminution and dilution" of human thought and creativity. Just as our muscles atrophy without use, he argues our brains will too, and we risk becoming mentally dependent on these systems.

TOC:

[00:00:00] Intelligence: Doing more with less

[00:02:10] Why brains evolved: The limits of evolution

[00:05:18] Culture as evolution at light speed

[00:08:11] True meaning of emergence: "More is Different"

[00:10:41] Why LLM capabilities are not true emergence

[00:15:10] What real emergence would look like in AI

[00:19:24] Symmetry breaking: Physics vs. Life

[00:23:30] Two types of emergence: Knowledge In vs. Out

[00:26:46] Causality, agency, and coarse-graining

[00:32:24] "Exbodiment": Outsourcing thought to objects

[00:35:05] Collective intelligence & the boundary of the mind

[00:39:45] Mortal vs. Immortal forms of computation

[00:42:13] The risk of AI: Atrophy of human thought

David Krakauer

President and William H. Miller Professor of Complex Systems

https://www.santafe.edu/people/profile/david-krakauer

REFS:

Large Language Models and Emergence: A Complex Systems Perspective

David C. Krakauer, John W. Krakauer, Melanie Mitchell

https://arxiv.org/abs/2506.11135

Filmed at the Diverse Intelligences Summer Institute:

https://disi.org/

Dr. Maxwell Ramstead grills Guillaume Verdon (AKA “Beff Jezos”) who's the founder of Thermodynamic computing startup Extropic.

Guillaume shares his unique path – from dreaming about space travel as a kid to becoming a physicist, then working on quantum computing at Google, to developing a radically new form of computing hardware for machine learning. He explains how he hit roadblocks with traditional physics and computing, leading him to start his company – building "thermodynamic computers." These are based on a new design for super-efficient chips that use the natural chaos of electrons (think noise and heat) to power AI tasks, which promises to speed up AND lower the costs of modern probabilistic techniques like sampling. He is driven by the pursuit of building computers that work more like your brain, which (by the way) runs on a banana and a glass of water! 

Guillaume talks about his alter ego, Beff Jezos, and the "Effective Accelerationism" (e/acc) movement that he initiated. Its objective is to speed up tech progress in order to “grow civilization” (as measured by energy use and innovation), rather than “slowing down out of fear”. Guillaume argues we need to embrace variance, exploration, and optimism to avoid getting stuck or outpaced by competitors like China. He and Maxwell discuss big ideas like merging humans with AI, decentralizing intelligence, and why boundless growth (with smart constraints) is “key to humanity's future”.

REFS:

1. John Archibald Wheeler - "It From Bit" Concept

00:04:45 - Foundational work proposing that physical reality emerges from information at the quantum level

Learn more: https://cqi.inf.usi.ch/qic/wheeler.pdf 

2. AdS/CFT Correspondence (Holographic Principle)

00:05:15 - Theoretical physics duality connecting quantum gravity in Anti-de Sitter space with conformal field theory

https://en.wikipedia.org/wiki/Holographic_principle 

3. Renormalization Group Theory

00:06:15 - Mathematical framework for analyzing physical systems across different length scales

https://www.damtp.cam.ac.uk/user/dbs26/AQFT/Wilsonchap.pdf 

4. Maxwell's Demon and Information Theory

00:21:15 - Thought experiment linking information processing to thermodynamics and entropy

https://plato.stanford.edu/entries/information-entropy/ 

5. Landauer's Principle

00:29:45 - Fundamental limit establishing minimum energy required for information erasure

https://en.wikipedia.org/wiki/Landauer%27s_principle 

6. Free Energy Principle and Active Inference

01:03:00 - Mathematical framework for understanding self-organizing systems and perception-action loops

https://www.nature.com/articles/nrn2787 

7. Max Tegmark - Information Bottleneck Principle

01:07:00 - Connections between information theory and renormalization in machine learning

https://arxiv.org/abs/1907.07331 

8. Fisher's Fundamental Theorem of Natural Selection

01:11:45 - Mathematical relationship between genetic variance and evolutionary fitness

https://en.wikipedia.org/wiki/Fisher%27s_fundamental_theorem_of_natural_selection 

9. Tensor Networks in Quantum Systems

00:06:45 - Computational framework for simulating many-body quantum systems

https://arxiv.org/abs/1912.10049 

10. Quantum Neural Networks

00:09:30 - Hybrid quantum-classical models for machine learning applications

https://en.wikipedia.org/wiki/Quantum_neural_network 

11. Energy-Based Models (EBMs)

00:40:00 - Probabilistic framework for unsupervised learning based on energy functions

https://www.researchgate.net/publication/200744586_A_tutorial_on_energy-based_learning 

12. Markov Chain Monte Carlo (MCMC)

00:20:00 - Sampling algorithm fundamental to modern AI and statistical physics

https://en.wikipedia.org/wiki/Markov_chain_Monte_Carlo 

13. Metropolis-Hastings Algorithm

00:23:00 - Core sampling method for probability distributions

https://arxiv.org/abs/1504.01896

***SPONSOR MESSAGE***

Google Gemini 2.5 Flash is a state-of-the-art language model in the Gemini app. Sign up at https://gemini.google.com

Are the AI models you use today imposters?

Please watch the intro video we did before this: https://www.youtube.com/watch?v=o1q6Hhz0MAg

In this episode, hosts Dr. Tim Scarfe and Dr. Duggar are joined by AI researcher Prof. Kenneth Stanley and MIT PhD student Akash Kumar to discuss their fascinating paper, "Questioning Representational Optimism in Deep Learning."

Imagine you ask two people to draw a perfect skull. One is a brilliant artist who understands anatomy, the other is a machine that just traces the image. Both drawings look identical, but the artist understands what a skull is—they know where the mouth is, how the jaw works, and that it's symmetrical. The machine just has a tangled mess of lines that happens to form the right picture.

An AI with an elegant representation, has the building blocks to generate truly new ideas.

The Path Is the Goal: As Kenneth Stanley puts it, "it matters not just where you get, but how you got there". Two students can ace a math test, but the one who truly understands the concepts—instead of just memorizing formulas—is the one who will go on to make new discoveries.

The show is a mixture of 3 separate recordings we have done, the original Patreon warmup with Tim/Kenneth, the Tim/Keith "Steakhouse" recorded after the main interview, then the main interview with Kenneth/Akarsh/Keith/Tim. Feel free to skip around. We had to edit this in a rush as we are travelling next week but it's reasonably cleaned up.

TOC:

00:00:00 Intro: Garbage vs. Amazing Representations

00:05:42 How Good Representations Form

00:11:14 Challenging the "Bitter Lesson"

00:18:04 AI Creativity & Representation Types

00:22:13 Steakhouse: Critiques & Alternatives

00:28:30 Steakhouse: Key Concepts & Goldilocks Zone

00:39:42 Steakhouse: A Sober View on AI Risk

00:43:46 Steakhouse: The Paradox of Open-Ended Search

00:47:58 Main Interview: Paper Intro & Core Concepts

00:56:44 Main Interview: Deception and Evolvability

01:36:30 Main Interview: Reinterpreting Evolution

01:56:16 Main Interview: Impostor Intelligence

02:11:15 Main Interview: Recommendations for AI Research

REFS:

Questioning Representational Optimism in Deep Learning:

The Fractured Entangled Representation Hypothesis

Akarsh Kumar, Jeff Clune, Joel Lehman, Kenneth O. Stanley

https://arxiv.org/pdf/2505.11581

Kenneth O. Stanley, Joel Lehman

Why Greatness Cannot Be Planned: The Myth of the Objective

https://amzn.to/44xLaXK

Original show with Kenneth from 4 years ago:

https://www.youtube.com/watch?v=lhYGXYeMq_E

Kenneth Stanley is SVP Open Endedness at Lila Sciences

https://x.com/kenneth0stanley

Akarsh Kumar (MIT)

https://akarshkumar.com/

AND... Kenneth is HIRING (this is an OPPORTUNITY OF A LIFETIME!)

Research Engineer: https://job-boards.greenhouse.io/lila/jobs/7890007002

Research Scientist: https://job-boards.greenhouse.io/lila/jobs/8012245002

TRANSCRIPT:

https://app.rescript.info/public/share/W_T7E1OC2Wj49ccqlIOOztg2MJWaaVbovTeyxcFEQdU

What if today's incredible AI is just a brilliant "impostor"? This episode features host Dr. Tim Scarfe in conversation with guests Prof. Kenneth Stanley (ex-OpenAI), Dr. Keith Duggar (MIT), and Arkash Kumar (MIT).While AI today produces amazing results on the surface, its internal understanding is a complete mess, described as "total spaghetti" [00:00:49]. This is because it's trained with a brute-force method (SGD) that’s like building a sandcastle: it looks right from a distance, but has no real structure holding it together [00:01:45].To explain the difference, Keith Duggar shares a great analogy about his high school physics classes [00:03:18]. One class was about memorizing lots of formulas for specific situations (like the "impostor" AI). The other used calculus to derive the answers from a deeper understanding, which was much easier and more powerful. This is the core difference: one method memorizes, the other truly understands.The episode then introduces a different, more powerful way to build AI, based on Kenneth Stanley's old experiment, "Picbreeder" [00:04:45]. This method creates AI with a shockingly clean and intuitive internal model of the world. For example, it might develop a model of a skull where it understands the "mouth" as a separate component it can open and close, without ever being explicitly trained on that action [00:06:15]. This deep understanding emerges bottom-up, without massive datasets.The secret is to abandon a fixed goal and embrace "deception" [00:08:42]—the idea that the stepping stones to a great discovery often don't look anything like the final result. Instead of optimizing for a target, the AI is built through an open-ended process of exploring what's "interesting" [00:09:15]. This creates a more flexible and adaptable foundation, a bit like how evolvability wins out in nature [00:10:30].The show concludes by arguing that this choice matters immensely. The "impostor" path may be hitting a wall, requiring insane amounts of money and energy for progress and failing to deliver true creativity or continual learning [00:13:00]. The ultimate message is a call to not put all our eggs in one basket [00:14:25]. We should explore these open-ended, creative paths to discover a more genuine form of intelligence, which may be found where we least expect it.REFS:Questioning Representational Optimism in Deep Learning:The Fractured Entangled Representation HypothesisAkarsh Kumar, Jeff Clune, Joel Lehman, Kenneth O. Stanleyhttps://arxiv.org/pdf/2505.11581Kenneth O. Stanley, Joel LehmanWhy Greatness Cannot Be Planned: The Myth of the Objectivehttps://amzn.to/44xLaXKOriginal show with Kenneth from 4 years ago:https://www.youtube.com/watch?v=lhYGXYeMq_EKenneth Stanley is SVP Open Endedness at Lila Scienceshttps://x.com/kenneth0stanleyAkarsh Kumar (MIT)https://akarshkumar.com/AND... Kenneth is HIRING (this is an OPPORTUNITY OF A LIFETIME!)Research Engineer: https://job-boards.greenhouse.io/lila/jobs/7890007002Research Scientist: https://job-boards.greenhouse.io/lila/jobs/8012245002Tim's Code visualisation of FER based on Akarsh repo: https://github.com/ecsplendid/ferTRANSCRIPT: https://app.rescript.info/public/share/YKAZzZ6lwZkjTLRpVJreOOxGhLI8y4m3fAyU8NSavx0

What if the most powerful technology in human history is being built by people who openly admit they don't trust each other? In this explosive 2-hour debate, three AI experts pull back the curtain on the shocking psychology driving the race to Artificial General Intelligence—and why the people building it might be the biggest threat of all. Kokotajlo predicts AGI by 2028 based on compute scaling trends. Marcus argues we haven't solved basic cognitive problems from his 2001 research. The stakes? If Kokotajlo is right and Marcus is wrong about safety progress, humanity may have already lost control.

Sponsor messages:

========

Google Gemini: Google Gemini features Veo3, a state-of-the-art AI video generation model in the Gemini app. Sign up at https://gemini.google.com

Tufa AI Labs are hiring for ML Engineers and a Chief Scientist in Zurich/SF. They are top of the ARCv2 leaderboard!

https://tufalabs.ai/

========

Guest Powerhouse

Gary Marcus - Cognitive scientist, author of "Taming Silicon Valley," and AI's most prominent skeptic who's been warning about the same fundamental problems for 25 years (https://garymarcus.substack.com/)

Daniel Kokotajlo - Former OpenAI insider turned whistleblower who reveals the disturbing rationalizations of AI lab leaders in his viral "AI 2027" scenario (https://ai-2027.com/)

Dan Hendrycks - Director of the Center for AI Safety who created the benchmarks used to measure AI progress and argues we have only years, not decades, to prevent catastrophe (https://danhendrycks.com/)

Transcript:

http://app.rescript.info/public/share/tEcx4UkToi-2jwS1cN51CW70A4Eh6QulBRxDILoXOno

TOC:

Introduction: The AI Arms Race

00:00:04 - The Danger of Automated AI R&D

00:00:43 - The Rationalization: "If we don't, someone else will"

00:01:56 - Sponsor Reads (Tufa AI Labs & Google Gemini)

00:02:55 - Guest Introductions

The Philosophical Stakes

00:04:13 - What is the Positive Vision for AGI?

00:07:00 - The Abundance Scenario: Superintelligent Economy

00:09:06 - Differentiating AGI and Superintelligence (ASI)

00:11:41 - Sam Altman: "A Decade in a Month"

00:14:47 - Economic Inequality & The UBI Problem

Policy and Red Lines

00:17:13 - The Pause Letter: Stopping vs. Delaying AI

00:20:03 - Defining Three Concrete Red Lines for AI Development

00:25:24 - Racing Towards Red Lines & The Myth of "Durable Advantage"

00:31:15 - Transparency and Public Perception

00:35:16 - The Rationalization Cascade: Why AI Labs Race to "Win"

Forecasting AGI: Timelines and Methodologies

00:42:29 - The Case for Short Timelines (Median 2028)

00:47:00 - Scaling Limits: Compute, Data, and Money

00:49:36 - Forecasting Models: Bio-Anchors and Agentic Coding

00:53:15 - The 10^45 FLOP Thought Experiment

The Great Debate: Cognitive Gaps vs. Scaling

00:58:41 - Gary Marcus's Counterpoint: The Unsolved Problems of Cognition

01:00:46 - Current AI Can't Play Chess Reliably

01:08:23 - Can Tools and Neurosymbolic AI Fill the Gaps?

01:16:13 - The Multi-Dimensional Nature of Intelligence

01:24:26 - The Benchmark Debate: Data Contamination and Reliability

01:31:15 - The Superhuman Coder Milestone Debate

01:37:45 - The Driverless Car Analogy

The Alignment Problem

01:39:45 - Has Any Progress Been Made on Alignment?

01:42:43 - "Fairly Reasonably Scares the Sh*t Out of Me"

01:46:30 - Distinguishing Model vs. Process Alignment

Scenarios and Conclusions

01:49:26 - Gary's Alternative Scenario: The Neurosymbolic Shift

01:53:35 - Will AI Become Jeff Dean?

01:58:41 - Takeoff Speeds and Exceeding Human Intelligence

02:03:19 - Final Disagreements and Closing Remarks

REFS:

Gary Marcus (2001) - The Algebraic Mind

https://mitpress.mit.edu/9780262632683/the-algebraic-mind/

00:59:00

Gary Marcus & Ernest Davis (2019) - Rebooting AI

https://www.penguinrandomhouse.com/books/566677/rebooting-ai-by-gary-marcus-and-ernest-davis/

01:31:59

Gary Marcus (2024) - Taming SV

https://www.hachettebookgroup.com/titles/gary-marcus/taming-silicon-valley/9781541704091/

00:03:01

We interview Professor Christopher Summerfield from Oxford University about his new book "These Strange New Minds: How AI Learned to Talk and What It". AI learned to understand the world just by reading text - something scientists thought was impossible. You don't need to see a cat to know what one is; you can learn everything from words alone. This is "the most astonishing scientific discovery of the 21st century."People are split: some refuse to call what AI does "thinking" even when it outperforms humans, while others believe if it acts intelligent, it is intelligent. Summerfield takes the middle ground - AI does something genuinely like human reasoning, but that doesn't make it human.Sponsor messages:========Google Gemini: Google Gemini features Veo3, a state-of-the-art AI video generation model in the Gemini app. Sign up at https://gemini.google.comTufa AI Labs are hiring for ML Engineers and a Chief Scientist in Zurich/SF. They are top of the ARCv2 leaderboard! https://tufalabs.ai/========Prof. Christopher Summerfieldhttps://www.psy.ox.ac.uk/people/christopher-summerfieldThese Strange New Minds: How AI Learned to Talk and What It Meanshttps://amzn.to/4e26BVaTable of Contents:Introduction & Setup00:00:00 Superman 3 Metaphor - Humans Absorbed by Machines00:02:01 Book Introduction & AI Debate Context00:03:45 Sponsor Segments (Google Gemini, Tufa Labs)Philosophical Foundations00:04:48 The Fractured AI Discourse00:08:21 Ancient Roots: Aristotle vs Plato (Empiricism vs Rationalism)00:10:14 Historical AI: Symbolic Logic and Its LimitsThe Language Revolution00:12:11 ChatGPT as the Rubicon Moment00:14:00 The Astonishing Discovery: Learning Reality from Words Alone00:15:47 Equivalentists vs Exceptionalists DebateCognitive Science Perspectives00:19:12 Functionalism and the Duck Test00:21:48 Brain-AI Similarities and Computational Principles00:24:53 Reconciling Chomsky: Evolution vs Learning00:28:15 Lamarckian AI vs Darwinian Human LearningThe Reality of AI Capabilities00:30:29 Anthropomorphism and the Clever Hans Effect00:32:56 The Intentional Stance and Nature of Thinking00:37:56 Three Major AI Worries: Agency, Personalization, DynamicsSocietal Risks and Complex Systems00:37:56 AI Agents and Flash Crash Scenarios00:42:50 Removing Frictions: The Lawfare Example00:46:15 Gradual Disempowerment Theory00:49:18 The Faustian Pact of TechnologyHuman Agency and Control00:51:18 The Crisis of Authenticity00:56:22 Psychology of Control vs Reward01:00:21 Dopamine Hacking and Variable ReinforcementFuture Directions01:02:27 Evolution as Goal-less Optimization01:03:31 Open-Endedness and Creative Evolution01:06:46 Writing, Creativity, and AI-Generated Content01:08:18 Closing RemarksREFS:Academic References (Abbreviated)Essential Books"These Strange New Minds" - C. Summerfield [00:02:01] - Main discussion topic"The Mind is Flat" - N. Chater [00:33:45] - Summerfield's favorite on cognitive illusions"AI: A Guide for Thinking Humans" - M. Mitchell [00:04:58] - Host's previous favorite"Principia Mathematica" - Russell & Whitehead [00:11:00] - Logic Theorist reference"Syntactic Structures" - N. Chomsky (1957) [00:13:30] - Generative grammar foundation"Why Greatness Cannot Be Planned" - Stanley & Lehman [01:04:00] - Open-ended evolutionKey Papers & Studies"Gradual Disempowerment" - D. Duvenaud [00:46:45] - AI threat model"Counterfeit People" - D. Dennett (Atlantic) [00:52:45] - AI societal risks"Open-Endedness is Essential..." - DeepMind/Rocktäschel/Hughes [01:03:42]Heider & Simmel (1944) [00:30:45] - Agency attribution to shapesWhitehall Studies - M. Marmot [00:59:32] - Control and health outcomes"Clever Hans" - O. Pfungst (1911) [00:31:47] - Animal intelligence illusionHistorical References

<trunc, see https://youtu.be/35r0iSajXjA>

"Blurring Reality" - Chai's Social AI Platform - sponsored

This episode of MLST explores the groundbreaking work of Chai, a social AI platform that quietly built one of the world's largest AI companion ecosystems before ChatGPT's mainstream adoption. With over 10 million active users and just 13 engineers serving 2 trillion tokens per day, Chai discovered the massive appetite for AI companionship through serendipity while searching for product-market fit.

CHAI sponsored this show *because they want to hire amazing engineers* --

CAREER OPPORTUNITIES AT CHAI

Chai is actively hiring in Palo Alto with competitive compensation ($300K-$800K+ equity) for roles including AI Infrastructure Engineers, Software Engineers, Applied AI Researchers, and more. Fast-track qualification available for candidates with significant product launches, open source contributions, or entrepreneurial success.

https://www.chai-research.com/jobs/

The conversation with founder William Beauchamp and engineers Tom Lu and Nischay Dhankhar covers Chai's innovative technical approaches including reinforcement learning from human feedback (RLHF), model blending techniques that combine smaller models to outperform larger ones, and their unique infrastructure challenges running exaflop-class compute.

SPONSOR MESSAGES:

***

Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. They are hiring a Chief Engineer and ML engineers in Zurich and SF.

Goto https://tufalabs.ai/

***

Key themes explored include:

- The ethics of AI engagement optimization and attention hacking

- Content moderation at scale with a lean engineering team

- The shift from AI as utility tool to AI as social companion

- How users form deep emotional bonds with artificial intelligence

- The broader implications of AI becoming a social medium

We also examine OpenAI's recent pivot toward companion AI with April's new GPT-4o, suggesting a fundamental shift in how we interact with artificial intelligence - from utility-focused tools to companion-like experiences that blur the lines between human and artificial intimacy.

The episode also covers Chai's unconventional approach to hiring only top-tier engineers, their bootstrap funding strategy focused on user revenue over VC funding, and their rapid experimentation culture where one in five experiments succeed.

TOC:

00:00:00 - Introduction: Steve Jobs' AI Vision & Chai's Scale

00:04:02 - Chapter 1: Simulators - The Birth of Social AI

00:13:34 - Chapter 2: Engineering at Chai - RLHF & Model Blending

00:21:49 - Chapter 3: Social Impact of GenAI - Ethics & Safety

00:33:55 - Chapter 4: The Lean Machine - 13 Engineers, Millions of Users

00:42:38 - Chapter 5: GPT-4o Becoming a Companion - OpenAI's Pivot

00:50:10 - Chapter 6: What Comes Next - The Future of AI Intimacy

TRANSCRIPT: https://www.dropbox.com/scl/fi/yz2ewkzmwz9rbbturfbap/CHAI.pdf?rlkey=uuyk2nfhjzezucwdgntg5ubqb&dl=0

Today GoogleDeepMind released AlphaEvolve: a Gemini coding agent for algorithm discovery. It beat the famous Strassen algorithm for matrix multiplication set 56 years ago. Google has been killing it recently. We had early access to the paper and interviewed the researchers behind the work.

AlphaEvolve: A Gemini-powered coding agent for designing advanced algorithms

https://deepmind.google/discover/blog/alphaevolve-a-gemini-powered-coding-agent-for-designing-advanced-algorithms/

Authors: Alexander Novikov*, Ngân Vũ*, Marvin Eisenberger*, Emilien Dupont*, Po-Sen Huang*, Adam Zsolt Wagner*, Sergey Shirobokov*, Borislav Kozlovskii*, Francisco J. R. Ruiz, Abbas Mehrabian, M. Pawan Kumar, Abigail See, Swarat Chaudhuri, George Holland, Alex Davies, Sebastian Nowozin, Pushmeet Kohli, Matej Balog*

(* indicates equal contribution or special designation, if defined elsewhere)

SPONSOR MESSAGES:

***

Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. They are hiring a Chief Engineer and ML engineers. Events in Zurich.

Goto https://tufalabs.ai/

***

AlphaEvolve works like a very smart, tireless programmer. It uses powerful AI language models (like Gemini) to generate ideas for computer code. Then, it uses an "evolutionary" process – like survival of the fittest for programs. It tries out many different program ideas, automatically tests how well they solve a problem, and then uses the best ones to inspire new, even better programs.

Beyond this mathematical breakthrough, AlphaEvolve has already been used to improve real-world systems at Google, such as making their massive data centers run more efficiently and even speeding up the training of the AI models that power AlphaEvolve itself. The discussion also covers how humans work with AlphaEvolve, the challenges of making AI discover things, and the exciting future of AI helping scientists make new discoveries.

In short, AlphaEvolve is a powerful new AI tool that can invent new algorithms and solve complex problems, showing how AI can be a creative partner in science and engineering.

Guests:

Matej Balog: https://x.com/matejbalog

Alexander Novikov: https://x.com/SashaVNovikov

REFS:

MAP Elites [Jean-Baptiste Mouret, Jeff Clune]

https://arxiv.org/abs/1504.04909

FunSearch [Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M. Pawan Kumar, Emilien Dupont, Francisco J. R. Ruiz, Jordan S. Ellenberg, Pengming Wang, Omar Fawzi, Pushmeet Kohli & Alhussein Fawzi]

https://www.nature.com/articles/s41586-023-06924-6

TOC:

[00:00:00] Introduction: Alpha Evolve's Breakthroughs, DeepMind's Lineage, and Real-World Impact

[00:12:06] Introducing AlphaEvolve: Concept, Evolutionary Algorithms, and Architecture

[00:16:56] Search Challenges: The Halting Problem and Enabling Creative Leaps

[00:23:20] Knowledge Augmentation: Self-Generated Data, Meta-Prompting, and Library Learning

[00:29:08] Matrix Multiplication Breakthrough: From Strassen to AlphaEvolve's 48 Multiplications

[00:39:11] Problem Representation: Direct Solutions, Constructors, and Search Algorithms

[00:46:06] Developer Reflections: Surprising Outcomes and Superiority over Simple LLM Sampling

[00:51:42] Algorithmic Improvement: Hill Climbing, Program Synthesis, and Intelligibility

[01:00:24] Real-World Application: Complex Evaluations and Robotics

[01:05:39] Role of LLMs & Future: Advanced Models, Recursive Self-Improvement, and Human-AI Collaboration

[01:11:22] Resource Considerations: Compute Costs of AlphaEvolve

This is a trial of posting videos on Spotify, thoughts? Email me or chat in our Discord

Randall Balestriero joins the show to discuss some counterintuitive findings in AI. He shares research showing that huge language models, even when started from scratch (randomly initialized) without massive pre-training, can learn specific tasks like sentiment analysis surprisingly well, train stably, and avoid severe overfitting, sometimes matching the performance of costly pre-trained models. This raises questions about when giant pre-training efforts are truly worth it.

He also talks about how self-supervised learning (where models learn from data structure itself) and traditional supervised learning (using labeled data) are fundamentally similar, allowing researchers to apply decades of supervised learning theory to improve newer self-supervised methods.

Finally, Randall touches on fairness in AI models used for Earth data (like climate prediction), revealing that these models can be biased, performing poorly in specific locations like islands or coastlines even if they seem accurate overall, which has important implications for policy decisions based on this data.

SPONSOR MESSAGES:

***

Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. They are hiring a Chief Engineer and ML engineers. Events in Zurich.

Goto https://tufalabs.ai/

***

TRANSCRIPT + SHOWNOTES:

https://www.dropbox.com/scl/fi/n7yev71nsjso71jyjz1fy/RANDALLNEURIPS.pdf?rlkey=0dn4injp1sc4ts8njwf3wfmxv&dl=0

TOC:

1. Model Training Efficiency and Scale

[00:00:00] 1.1 Training Stability of Large Models on Small Datasets

[00:04:09] 1.2 Pre-training vs Random Initialization Performance Comparison

[00:07:58] 1.3 Task-Specific Models vs General LLMs Efficiency

2. Learning Paradigms and Data Distribution

[00:10:35] 2.1 Fair Language Model Paradox and Token Frequency Issues

[00:12:02] 2.2 Pre-training vs Single-task Learning Spectrum

[00:16:04] 2.3 Theoretical Equivalence of Supervised and Self-supervised Learning

[00:19:40] 2.4 Self-Supervised Learning and Supervised Learning Relationships

[00:21:25] 2.5 SSL Objectives and Heavy-tailed Data Distribution Challenges

3. Geographic Representation in ML Systems

[00:25:20] 3.1 Geographic Bias in Earth Data Models and Neural Representations

[00:28:10] 3.2 Mathematical Limitations and Model Improvements

[00:30:24] 3.3 Data Quality and Geographic Bias in ML Datasets

REFS:

[00:01:40] Research on training large language models from scratch on small datasets, Randall Balestriero et al.

https://openreview.net/forum?id=wYGBWOjq1Q

[00:10:35] The Fair Language Model Paradox (2024), Andrea Pinto, Tomer Galanti, Randall Balestriero

https://arxiv.org/abs/2410.11985

[00:12:20] Muppet: Massive Multi-task Representations with Pre-Finetuning (2021), Armen Aghajanyan et al.

https://arxiv.org/abs/2101.11038

[00:14:30] Dissociating language and thought in large language models (2023), Kyle Mahowald et al.

https://arxiv.org/abs/2301.06627

[00:16:05] The Birth of Self-Supervised Learning: A Supervised Theory, Randall Balestriero et al.

https://openreview.net/forum?id=NhYAjAAdQT

[00:21:25] VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning, Adrien Bardes, Jean Ponce, Yann LeCun

https://arxiv.org/abs/2105.04906

[00:25:20] No Location Left Behind: Measuring and Improving the Fairness of Implicit Representations for Earth Data (2025), Daniel Cai, Randall Balestriero, et al.

https://arxiv.org/abs/2502.06831

[00:33:45] Mark Ibrahim et al.'s work on geographic bias in computer vision datasets, Mark Ibrahim

https://arxiv.org/pdf/2304.12210

Prof. Kevin Ellis and Dr. Zenna Tavares talk about making AI smarter, like humans. They want AI to learn from just a little bit of information by actively trying things out, not just by looking at tons of data.

They discuss two main ways AI can "think": one way is like following specific rules or steps (like a computer program), and the other is more intuitive, like guessing based on patterns (like modern AI often does). They found combining both methods works well for solving complex puzzles like ARC.

A key idea is "compositionality" - building big ideas from small ones, like LEGOs. This is powerful but can also be overwhelming. Another important idea is "abstraction" - understanding things simply, without getting lost in details, and knowing there are different levels of understanding.

Ultimately, they believe the best AI will need to explore, experiment, and build models of the world, much like humans do when learning something new.

SPONSOR MESSAGES:

***

Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. They are hiring a Chief Engineer and ML engineers. Events in Zurich.

Goto https://tufalabs.ai/

***

TRANSCRIPT:

https://www.dropbox.com/scl/fi/3ngggvhb3tnemw879er5y/BASIS.pdf?rlkey=lr2zbj3317mex1q5l0c2rsk0h&dl=0

Zenna Tavares:

http://www.zenna.org/

Kevin Ellis:

https://www.cs.cornell.edu/~ellisk/

TOC:

1. Compositionality and Learning Foundations

[00:00:00] 1.1 Compositional Search and Learning Challenges

[00:03:55] 1.2 Bayesian Learning and World Models

[00:12:05] 1.3 Programming Languages and Compositionality Trade-offs

[00:15:35] 1.4 Inductive vs Transductive Approaches in AI Systems

2. Neural-Symbolic Program Synthesis

[00:27:20] 2.1 Integration of LLMs with Traditional Programming and Meta-Programming

[00:30:43] 2.2 Wake-Sleep Learning and DreamCoder Architecture

[00:38:26] 2.3 Program Synthesis from Interactions and Hidden State Inference

[00:41:36] 2.4 Abstraction Mechanisms and Resource Rationality

[00:48:38] 2.5 Inductive Biases and Causal Abstraction in AI Systems

3. Abstract Reasoning Systems

[00:52:10] 3.1 Abstract Concepts and Grid-Based Transformations in ARC

[00:56:08] 3.2 Induction vs Transduction Approaches in Abstract Reasoning

[00:59:12] 3.3 ARC Limitations and Interactive Learning Extensions

[01:06:30] 3.4 Wake-Sleep Program Learning and Hybrid Approaches

[01:11:37] 3.5 Project MARA and Future Research Directions

REFS:

[00:00:25] DreamCoder, Kevin Ellis et al.

https://arxiv.org/abs/2006.08381

[00:01:10] Mind Your Step, Ryan Liu et al.

https://arxiv.org/abs/2410.21333

[00:06:05] Bayesian inference, Griffiths, T. L., Kemp, C., & Tenenbaum, J. B.

https://psycnet.apa.org/record/2008-06911-003

[00:13:00] Induction and Transduction, Wen-Ding Li, Zenna Tavares, Yewen Pu, Kevin Ellis

https://arxiv.org/abs/2411.02272

[00:23:15] Neurosymbolic AI, Garcez, Artur d'Avila et al.

https://arxiv.org/abs/2012.05876

[00:33:50] Induction and Transduction (II), Wen-Ding Li, Kevin Ellis et al.

https://arxiv.org/abs/2411.02272

[00:38:35] ARC, François Chollet

https://arxiv.org/abs/1911.01547

[00:39:20] Causal Reactive Programs, Ria Das, Joshua B. Tenenbaum, Armando Solar-Lezama, Zenna Tavares

http://www.zenna.org/publications/autumn2022.pdf

[00:42:50] MuZero, Julian Schrittwieser et al.

http://arxiv.org/pdf/1911.08265

[00:43:20] VisualPredicator, Yichao Liang

https://arxiv.org/abs/2410.23156

[00:48:55] Bayesian models of cognition, Joshua B. Tenenbaum

https://mitpress.mit.edu/9780262049412/bayesian-models-of-cognition/

[00:49:30] The Bitter Lesson, Rich Sutton

http://www.incompleteideas.net/IncIdeas/BitterLesson.html

[01:06:35] Program induction, Kevin Ellis, Wen-Ding Li

https://arxiv.org/pdf/2411.02272

[01:06:50] DreamCoder (II), Kevin Ellis et al.

https://arxiv.org/abs/2006.08381

[01:11:55] Project MARA, Zenna Tavares, Kevin Ellis

https://www.basis.ai/blog/mara/

Eiso Kant, CTO of poolside AI, discusses the company's approach to building frontier AI foundation models, particularly focused on software development. Their unique strategy is reinforcement learning from code execution feedback which is an important axis for scaling AI capabilities beyond just increasing model size or data volume. Kant predicts human-level AI in knowledge work could be achieved within 18-36 months, outlining poolside's vision to dramatically increase software development productivity and accessibility.

SPONSOR MESSAGES:

***

Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. They are hiring a Chief Engineer and ML engineers. Events in Zurich.

Goto https://tufalabs.ai/

***

Eiso Kant:

https://x.com/eisokant

https://poolside.ai/

TRANSCRIPT:

https://www.dropbox.com/scl/fi/szepl6taqziyqie9wgmk9/poolside.pdf?rlkey=iqar7dcwshyrpeoz0xa76k422&dl=0

TOC:

1. Foundation Models and AI Strategy

[00:00:00] 1.1 Foundation Models and Timeline Predictions for AI Development

[00:02:55] 1.2 Poolside AI's Corporate History and Strategic Vision

[00:06:48] 1.3 Foundation Models vs Enterprise Customization Trade-offs

2. Reinforcement Learning and Model Economics

[00:15:42] 2.1 Reinforcement Learning and Code Execution Feedback Approaches

[00:22:06] 2.2 Model Economics and Experimental Optimization

3. Enterprise AI Implementation

[00:25:20] 3.1 Poolside's Enterprise Deployment Strategy and Infrastructure

[00:26:00] 3.2 Enterprise-First Business Model and Market Focus

[00:27:05] 3.3 Foundation Models and AGI Development Approach

[00:29:24] 3.4 DeepSeek Case Study and Infrastructure Requirements

4. LLM Architecture and Performance

[00:30:15] 4.1 Distributed Training and Hardware Architecture Optimization

[00:33:01] 4.2 Model Scaling Strategies and Chinchilla Optimality Trade-offs

[00:36:04] 4.3 Emergent Reasoning and Model Architecture Comparisons

[00:43:26] 4.4 Balancing Creativity and Determinism in AI Models

[00:50:01] 4.5 AI-Assisted Software Development Evolution

5. AI Systems Engineering and Scalability

[00:58:31] 5.1 Enterprise AI Productivity and Implementation Challenges

[00:58:40] 5.2 Low-Code Solutions and Enterprise Hiring Trends

[01:01:25] 5.3 Distributed Systems and Engineering Complexity

[01:01:50] 5.4 GenAI Architecture and Scalability Patterns

[01:01:55] 5.5 Scaling Limitations and Architectural Patterns in AI Code Generation

6. AI Safety and Future Capabilities

[01:06:23] 6.1 Semantic Understanding and Language Model Reasoning Approaches

[01:12:42] 6.2 Model Interpretability and Safety Considerations in AI Systems

[01:16:27] 6.3 AI vs Human Capabilities in Software Development

[01:33:45] 6.4 Enterprise Deployment and Security Architecture

CORE REFS (see shownotes for URLs/more refs):

[00:15:45] Research demonstrating how training on model-generated content leads to distribution collapse in AI models, Ilia Shumailov et al. (Key finding on synthetic data risk)

[00:20:05] Foundational paper introducing Word2Vec for computing word vector representations, Tomas Mikolov et al. (Seminal NLP technique)

[00:22:15] OpenAI O3 model's breakthrough performance on ARC Prize Challenge, OpenAI (Significant AI reasoning benchmark achievement)

[00:22:40] Seminal paper proposing a formal definition of intelligence as skill-acquisition efficiency, François Chollet (Influential AI definition/philosophy)

[00:30:30] Technical documentation of DeepSeek's V3 model architecture and capabilities, DeepSeek AI (Details on a major new model)

[00:34:30] Foundational paper establishing optimal scaling laws for LLM training, Jordan Hoffmann et al. (Key paper on LLM scaling)

[00:45:45] Seminal essay arguing that scaling computation consistently trumps human-engineered solutions in AI, Richard S. Sutton (Influential "Bitter Lesson" perspective)

<trunc - see PDF shownotes>

Connor Leahy and Gabriel Alfour, AI researchers from Conjecture and authors of "The Compendium," joinus for a critical discussion centered on Artificial Superintelligence (ASI) safety and governance. Drawing from their comprehensive analysis in "The Compendium," they articulate a stark warning about the existential risks inherent in uncontrolled AI development, framing it through the lens of "intelligence domination"—where a sufficiently advanced AI could subordinate humanity, much like humans dominate less intelligent species.

SPONSOR MESSAGES:

***

Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. They are hiring a Chief Engineer and ML engineers. Events in Zurich.

Goto https://tufalabs.ai/

***

TRANSCRIPT + REFS + NOTES:

https://www.dropbox.com/scl/fi/p86l75y4o2ii40df5t7no/Compendium.pdf?rlkey=tukczgf3flw133sr9rgss0pnj&dl=0

https://www.thecompendium.ai/

https://en.wikipedia.org/wiki/Connor_Leahy

https://www.conjecture.dev/about

https://substack.com/@gabecc​

TOC:

1. AI Intelligence and Safety Fundamentals

[00:00:00] 1.1 Understanding Intelligence and AI Capabilities

[00:06:20] 1.2 Emergence of Intelligence and Regulatory Challenges

[00:10:18] 1.3 Human vs Animal Intelligence Debate

[00:18:00] 1.4 AI Regulation and Risk Assessment Approaches

[00:26:14] 1.5 Competing AI Development Ideologies

2. Economic and Social Impact

[00:29:10] 2.1 Labor Market Disruption and Post-Scarcity Scenarios

[00:32:40] 2.2 Institutional Frameworks and Tech Power Dynamics

[00:37:40] 2.3 Ethical Frameworks and AI Governance Debates

[00:40:52] 2.4 AI Alignment Evolution and Technical Challenges

3. Technical Governance Framework

[00:55:07] 3.1 Three Levels of AI Safety: Alignment, Corrigibility, and Boundedness

[00:55:30] 3.2 Challenges of AI System Corrigibility and Constitutional Models

[00:57:35] 3.3 Limitations of Current Boundedness Approaches

[00:59:11] 3.4 Abstract Governance Concepts and Policy Solutions

4. Democratic Implementation and Coordination

[00:59:20] 4.1 Governance Design and Measurement Challenges

[01:00:10] 4.2 Democratic Institutions and Experimental Governance

[01:14:10] 4.3 Political Engagement and AI Safety Advocacy

[01:25:30] 4.4 Practical AI Safety Measures and International Coordination

CORE REFS:

[00:01:45] The Compendium (2023), Leahy et al.

https://pdf.thecompendium.ai/the_compendium.pdf

[00:06:50] Geoffrey Hinton Leaves Google, BBC News

https://www.bbc.com/news/world-us-canada-65452940

[00:10:00] ARC-AGI, Chollet

https://arcprize.org/arc-agi

[00:13:25] A Brief History of Intelligence, Bennett

https://www.amazon.com/Brief-History-Intelligence-Humans-Breakthroughs/dp/0063286343

[00:25:35] Statement on AI Risk, Center for AI Safety

https://www.safe.ai/work/statement-on-ai-risk

[00:26:15] Machines of Love and Grace, Amodei

https://darioamodei.com/machines-of-loving-grace

[00:26:35] The Techno-Optimist Manifesto, Andreessen

https://a16z.com/the-techno-optimist-manifesto/

[00:31:55] Techno-Feudalism, Varoufakis

https://www.amazon.co.uk/Technofeudalism-Killed-Capitalism-Yanis-Varoufakis/dp/1847927270

[00:42:40] Introducing Superalignment, OpenAI

https://openai.com/index/introducing-superalignment/

[00:47:20] Three Laws of Robotics, Asimov

https://www.britannica.com/topic/Three-Laws-of-Robotics

[00:50:00] Symbolic AI (GOFAI), Haugeland

https://en.wikipedia.org/wiki/Symbolic_artificial_intelligence

[00:52:30] Intent Alignment, Christiano

https://www.alignmentforum.org/posts/HEZgGBZTpT4Bov7nH/mapping-the-conceptual-territory-in-ai-existential-safety

[00:55:10] Large Language Model Alignment: A Survey, Jiang et al.

http://arxiv.org/pdf/2309.15025

[00:55:40] Constitutional Checks and Balances, Bok

https://plato.stanford.edu/entries/montesquieu/

<trunc, see PDF>

We are joined by Francois Chollet and Mike Knoop, to launch the new version of the ARC prize! In version 2, the challenges have been calibrated with humans such that at least 2 humans could solve each task in a reasonable task, but also adversarially selected so that frontier reasoning models can't solve them. The best LLMs today get negligible performance on this challenge.

https://arcprize.org/

SPONSOR MESSAGES:

***

Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. They are hiring a Chief Engineer and ML engineers. Events in Zurich.

Goto https://tufalabs.ai/

***

TRANSCRIPT:

https://www.dropbox.com/scl/fi/0v9o8xcpppdwnkntj59oi/ARCv2.pdf?rlkey=luqb6f141976vra6zdtptv5uj&dl=0

TOC:

1. ARC v2 Core Design & Objectives

[00:00:00] 1.1 ARC v2 Launch and Benchmark Architecture

[00:03:16] 1.2 Test-Time Optimization and AGI Assessment

[00:06:24] 1.3 Human-AI Capability Analysis

[00:13:02] 1.4 OpenAI o3 Initial Performance Results

2. ARC Technical Evolution

[00:17:20] 2.1 ARC-v1 to ARC-v2 Design Improvements

[00:21:12] 2.2 Human Validation Methodology

[00:26:05] 2.3 Task Design and Gaming Prevention

[00:29:11] 2.4 Intelligence Measurement Framework

3. O3 Performance & Future Challenges

[00:38:50] 3.1 O3 Comprehensive Performance Analysis

[00:43:40] 3.2 System Limitations and Failure Modes

[00:49:30] 3.3 Program Synthesis Applications

[00:53:00] 3.4 Future Development Roadmap

REFS:

[00:00:15] On the Measure of Intelligence, François Chollet

https://arxiv.org/abs/1911.01547

[00:06:45] ARC Prize Foundation, François Chollet, Mike Knoop

https://arcprize.org/

[00:12:50] OpenAI o3 model performance on ARC v1, ARC Prize Team

https://arcprize.org/blog/oai-o3-pub-breakthrough

[00:18:30] Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, Jason Wei et al.

https://arxiv.org/abs/2201.11903

[00:21:45] ARC-v2 benchmark tasks, Mike Knoop

https://arcprize.org/blog/introducing-arc-agi-public-leaderboard

[00:26:05] ARC Prize 2024: Technical Report, Francois Chollet et al.

https://arxiv.org/html/2412.04604v2

[00:32:45] ARC Prize 2024 Technical Report, Francois Chollet, Mike Knoop, Gregory Kamradt

https://arxiv.org/abs/2412.04604

[00:48:55] The Bitter Lesson, Rich Sutton

http://www.incompleteideas.net/IncIdeas/BitterLesson.html

[00:53:30] Decoding strategies in neural text generation, Sina Zarrieß

https://www.mdpi.com/2078-2489/12/9/355/pdf

Mohamed Osman joins to discuss MindsAI's highest scoring entry to the ARC challenge 2024 and the paradigm of test-time fine-tuning. They explore how the team, now part of Tufa Labs in Zurich, achieved state-of-the-art results using a combination of pre-training techniques, a unique meta-learning strategy, and an ensemble voting mechanism. Mohamed emphasizes the importance of raw data input and flexibility of the network.

SPONSOR MESSAGES:

***

Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. They are hiring a Chief Engineer and ML engineers. Events in Zurich.

Goto https://tufalabs.ai/

***

TRANSCRIPT + REFS:

https://www.dropbox.com/scl/fi/jeavyqidsjzjgjgd7ns7h/MoFInal.pdf?rlkey=cjjmo7rgtenxrr3b46nk6yq2e&dl=0

Mohamed Osman (Tufa Labs)

https://x.com/MohamedOsmanML

Jack Cole (Tufa Labs)

https://x.com/MindsAI_Jack

How and why deep learning for ARC paper:

https://github.com/MohamedOsman1998/deep-learning-for-arc/blob/main/deep_learning_for_arc.pdf

TOC:

1. Abstract Reasoning Foundations

[00:00:00] 1.1 Test-Time Fine-Tuning and ARC Challenge Overview

[00:10:20] 1.2 Neural Networks vs Programmatic Approaches to Reasoning

[00:13:23] 1.3 Code-Based Learning and Meta-Model Architecture

[00:20:26] 1.4 Technical Implementation with Long T5 Model

2. ARC Solution Architectures

[00:24:10] 2.1 Test-Time Tuning and Voting Methods for ARC Solutions

[00:27:54] 2.2 Model Generalization and Function Generation Challenges

[00:32:53] 2.3 Input Representation and VLM Limitations

[00:36:21] 2.4 Architecture Innovation and Cross-Modal Integration

[00:40:05] 2.5 Future of ARC Challenge and Program Synthesis Approaches

3. Advanced Systems Integration

[00:43:00] 3.1 DreamCoder Evolution and LLM Integration

[00:50:07] 3.2 MindsAI Team Progress and Acquisition by Tufa Labs

[00:54:15] 3.3 ARC v2 Development and Performance Scaling

[00:58:22] 3.4 Intelligence Benchmarks and Transformer Limitations

[01:01:50] 3.5 Neural Architecture Optimization and Processing Distribution

REFS:

[00:01:32] Original ARC challenge paper, François Chollet

https://arxiv.org/abs/1911.01547

[00:06:55] DreamCoder, Kevin Ellis et al.

https://arxiv.org/abs/2006.08381

[00:12:50] Deep Learning with Python, François Chollet

https://www.amazon.com/Deep-Learning-Python-Francois-Chollet/dp/1617294438

[00:13:35] Deep Learning with Python, François Chollet

https://www.amazon.com/Deep-Learning-Python-Francois-Chollet/dp/1617294438

[00:13:35] Influence of pretraining data for reasoning, Laura Ruis

https://arxiv.org/abs/2411.12580

[00:17:50] Latent Program Networks, Clement Bonnet

https://arxiv.org/html/2411.08706v1

[00:20:50] T5, Colin Raffel et al.

https://arxiv.org/abs/1910.10683

[00:30:30] Combining Induction and Transduction for Abstract Reasoning, Wen-Ding Li, Kevin Ellis et al.

https://arxiv.org/abs/2411.02272

[00:34:15] Six finger problem, Chen et al.

https://openaccess.thecvf.com/content/CVPR2024/papers/Chen_SpatialVLM_Endowing_Vision-Language_Models_with_Spatial_Reasoning_Capabilities_CVPR_2024_paper.pdf

[00:38:15] DeepSeek-R1-Distill-Llama, DeepSeek AI

https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama-70B

[00:40:10] ARC Prize 2024 Technical Report, François Chollet et al.

https://arxiv.org/html/2412.04604v2

[00:45:20] LLM-Guided Compositional Program Synthesis, Wen-Ding Li and Kevin Ellis

https://arxiv.org/html/2503.15540

[00:54:25] Abstraction and Reasoning Corpus, François Chollet

https://github.com/fchollet/ARC-AGI

[00:57:10] O3 breakthrough on ARC-AGI, OpenAI

https://arcprize.org/

[00:59:35] ConceptARC Benchmark, Arseny Moskvichev, Melanie Mitchell

https://arxiv.org/abs/2305.07141

[01:02:05] Mixtape: Breaking the Softmax Bottleneck Efficiently, Yang, Zhilin and Dai, Zihang and Salakhutdinov, Ruslan and Cohen, William W.

http://papers.neurips.cc/paper/9723-mixtape-breaking-the-softmax-bottleneck-efficiently.pdf

Iman Mirzadeh from Apple, who recently published the GSM-Symbolic paper discusses the crucial distinction between intelligence and achievement in AI systems. He critiques current AI research methodologies, highlighting the limitations of Large Language Models (LLMs) in reasoning and knowledge representation.

SPONSOR MESSAGES:

***

Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. They are hiring a Chief Engineer and ML engineers. Events in Zurich.

Goto https://tufalabs.ai/

***

TRANSCRIPT + RESEARCH:

https://www.dropbox.com/scl/fi/mlcjl9cd5p1kem4l0vqd3/IMAN.pdf?rlkey=dqfqb74zr81a5gqr8r6c8isg3&dl=0

TOC:

1. Intelligence vs Achievement in AI Systems

[00:00:00] 1.1 Intelligence vs Achievement Metrics in AI Systems

[00:03:27] 1.2 AlphaZero and Abstract Understanding in Chess

[00:10:10] 1.3 Language Models and Distribution Learning Limitations

[00:14:47] 1.4 Research Methodology and Theoretical Frameworks

2. Intelligence Measurement and Learning

[00:24:24] 2.1 LLM Capabilities: Interpolation vs True Reasoning

[00:29:00] 2.2 Intelligence Definition and Measurement Approaches

[00:34:35] 2.3 Learning Capabilities and Agency in AI Systems

[00:39:26] 2.4 Abstract Reasoning and Symbol Understanding

3. LLM Performance and Evaluation

[00:47:15] 3.1 Scaling Laws and Fundamental Limitations

[00:54:33] 3.2 Connectionism vs Symbolism Debate in Neural Networks

[00:58:09] 3.3 GSM-Symbolic: Testing Mathematical Reasoning in LLMs

[01:08:38] 3.4 Benchmark Evaluation and Model Performance Assessment

REFS:

[00:01:00] AlphaZero chess AI system, Silver et al.

https://arxiv.org/abs/1712.01815

[00:07:10] Game Changer: AlphaZero's Groundbreaking Chess Strategies, Sadler & Regan

https://www.amazon.com/Game-Changer-AlphaZeros-Groundbreaking-Strategies/dp/9056918184

[00:11:35] Cross-entropy loss in language modeling, Voita

http://lena-voita.github.io/nlp_course/language_modeling.html

[00:17:20] GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in LLMs, Mirzadeh et al.

https://arxiv.org/abs/2410.05229

[00:21:25] Connectionism and Cognitive Architecture: A Critical Analysis, Fodor & Pylyshyn

https://www.sciencedirect.com/science/article/pii/001002779090014B

[00:28:55] Brain-to-body mass ratio scaling laws, Sutskever

https://www.theverge.com/2024/12/13/24320811/what-ilya-sutskever-sees-openai-model-data-training

[00:29:40] On the Measure of Intelligence, Chollet

https://arxiv.org/abs/1911.01547

[00:33:30] On definition of intelligence, Gignac et al.

https://www.sciencedirect.com/science/article/pii/S0160289624000266

[00:35:30] Defining intelligence, Wang

https://cis.temple.edu/~wangp/papers.html

[00:37:40] How We Learn: Why Brains Learn Better Than Any Machine... for Now, Dehaene

https://www.amazon.com/How-We-Learn-Brains-Machine/dp/0525559884

[00:39:35] Surfaces and Essences: Analogy as the Fuel and Fire of Thinking, Hofstadter and Sander

https://www.amazon.com/Surfaces-Essences-Analogy-Fuel-Thinking/dp/0465018475

[00:43:15] Chain-of-thought prompting, Wei et al.

https://arxiv.org/abs/2201.11903

[00:47:20] Test-time scaling laws in machine learning, Brown

https://podcasts.apple.com/mv/podcast/openais-noam-brown-ilge-akkaya-and-hunter-lightman-on/id1750736528?i=1000671532058

[00:47:50] Scaling Laws for Neural Language Models, Kaplan et al.

https://arxiv.org/abs/2001.08361

[00:55:15] Tensor product variable binding, Smolensky

https://www.sciencedirect.com/science/article/abs/pii/000437029090007M

[01:08:45] GSM-8K dataset, OpenAI

https://huggingface.co/datasets/openai/gsm8k

Dr. Max Bartolo from Cohere discusses machine learning model development, evaluation, and robustness. Key topics include model reasoning, the DynaBench platform for dynamic benchmarking, data-centric AI development, model training challenges, and the limitations of human feedback mechanisms. The conversation also covers technical aspects like influence functions, model quantization, and the PRISM project.

Max Bartolo (Cohere):

https://www.maxbartolo.com/

https://cohere.com/command

TRANSCRIPT:

https://www.dropbox.com/scl/fi/vujxscaffw37pqgb6hpie/MAXB.pdf?rlkey=0oqjxs5u49eqa2m7uaol64lbw&dl=0

TOC:

1. Model Reasoning and Verification

[00:00:00] 1.1 Model Consistency and Reasoning Verification

[00:03:25] 1.2 Influence Functions and Distributed Knowledge Analysis

[00:10:28] 1.3 AI Application Development and Model Deployment

[00:14:24] 1.4 AI Alignment and Human Feedback Limitations

2. Evaluation and Bias Assessment

[00:20:15] 2.1 Human Evaluation Challenges and Factuality Assessment

[00:27:15] 2.2 Cultural and Demographic Influences on Model Behavior

[00:32:43] 2.3 Adversarial Examples and Model Robustness

3. Benchmarking Systems and Methods

[00:41:54] 3.1 DynaBench and Dynamic Benchmarking Approaches

[00:50:02] 3.2 Benchmarking Challenges and Alternative Metrics

[00:50:33] 3.3 Evolution of Model Benchmarking Methods

[00:51:15] 3.4 Hierarchical Capability Testing Framework

[00:52:35] 3.5 Benchmark Platforms and Tools

4. Model Architecture and Performance

[00:55:15] 4.1 Cohere's Model Development Process

[01:00:26] 4.2 Model Quantization and Performance Evaluation

[01:05:18] 4.3 Reasoning Capabilities and Benchmark Standards

[01:08:27] 4.4 Training Progression and Technical Challenges

5. Future Directions and Challenges

[01:13:48] 5.1 Context Window Evolution and Trade-offs

[01:22:47] 5.2 Enterprise Applications and Future Challenges

REFS:

[00:03:10] Research at Cohere with Laura Ruis et al., Max Bartolo, Laura Ruis et al.

https://cohere.com/research/papers/procedural-knowledge-in-pretraining-drives-reasoning-in-large-language-models-2024-11-20

[00:04:15] Influence functions in machine learning, Koh & Liang

https://arxiv.org/abs/1703.04730

[00:08:05] Studying Large Language Model Generalization with Influence Functions, Roger Grosse et al.

https://storage.prod.researchhub.com/uploads/papers/2023/08/08/2308.03296.pdf

[00:11:10] The LLM ARChitect: Solving ARC-AGI Is A Matter of Perspective, Daniel Franzen, Jan Disselhoff, and David Hartmann

https://github.com/da-fr/arc-prize-2024/blob/main/the_architects.pdf

[00:12:10] Hugging Face model repo for C4AI Command A, Cohere and Cohere For AI

https://huggingface.co/CohereForAI/c4ai-command-a-03-2025

[00:13:30] OpenInterpreter

https://github.com/KillianLucas/open-interpreter

[00:16:15] Human Feedback is not Gold Standard, Tom Hosking, Max Bartolo, Phil Blunsom

https://arxiv.org/abs/2309.16349

[00:27:15] The PRISM Alignment Dataset, Hannah Kirk et al.

https://arxiv.org/abs/2404.16019

[00:32:50] How adversarial examples arise, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, Aleksander Madry

https://arxiv.org/abs/1905.02175

[00:43:00] DynaBench platform paper, Douwe Kiela et al.

https://aclanthology.org/2021.naacl-main.324.pdf

[00:50:15] Sara Hooker's work on compute limitations, Sara Hooker

https://arxiv.org/html/2407.05694v1

[00:53:25] DataPerf: Community-led benchmark suite, Mazumder et al.

https://arxiv.org/abs/2207.10062

[01:04:35] DROP, Dheeru Dua et al.

https://arxiv.org/abs/1903.00161

[01:07:05] GSM8k, Cobbe et al.

https://paperswithcode.com/sota/arithmetic-reasoning-on-gsm8k

[01:09:30] ARC, François Chollet

https://github.com/fchollet/ARC-AGI

[01:15:50] Command A, Cohere

https://cohere.com/blog/command-a

[01:22:55] Enterprise search using LLMs, Cohere

https://cohere.com/blog/commonly-asked-questions-about-search-from-coheres-enterprise-customers

This sponsored episode features mathematician Ohad Asor discussing logical approaches to AI, focusing on the limitations of machine learning and introducing the Tau language for software development and blockchain tech. Asor argues that machine learning cannot guarantee correctness. Tau allows logical specification of software requirements, automatically creating provably correct implementations with potential to revolutionize distributed systems. The discussion highlights program synthesis, software updates, and applications in finance and governance.SPONSOR MESSAGES:***Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. They are hiring a Chief Engineer and ML engineers. Events in Zurich. Goto https://tufalabs.ai/***TRANSCRIPT + RESEARCH:https://www.dropbox.com/scl/fi/t849j6v1juk3gc15g4rsy/TAU.pdf?rlkey=hh11h2mhog3ncdbeapbzpzctc&dl=0Tau:https://tau.net/Tau Language:https://tau.ai/tau-language/Research:https://tau.net/Theories-and-Applications-of-Boolean-Algebras-0.29.pdfTOC:1. Machine Learning Foundations and Limitations [00:00:00] 1.1 Fundamental Limitations of Machine Learning and PAC Learning Theory [00:04:50] 1.2 Transductive Learning and the Three Curses of Machine Learning [00:08:57] 1.3 Language, Reality, and AI System Design [00:12:58] 1.4 Program Synthesis and Formal Verification Approaches2. Logical Programming Architecture [00:31:55] 2.1 Safe AI Development Requirements [00:32:05] 2.2 Self-Referential Language Architecture [00:32:50] 2.3 Boolean Algebra and Logical Foundations [00:37:52] 2.4 SAT Solvers and Complexity Challenges [00:44:30] 2.5 Program Synthesis and Specification [00:47:39] 2.6 Overcoming Tarski's Undefinability with Boolean Algebra [00:56:05] 2.7 Tau Language Implementation and User Control3. Blockchain-Based Software Governance [01:09:10] 3.1 User Control and Software Governance Mechanisms [01:18:27] 3.2 Tau's Blockchain Architecture and Meta-Programming Capabilities [01:21:43] 3.3 Development Status and Token Implementation [01:24:52] 3.4 Consensus Building and Opinion Mapping System [01:35:29] 3.5 Automation and Financial ApplicationsCORE REFS (more in pinned comment):[00:03:45] PAC (Probably Approximately Correct) Learning framework, Leslie Valianthttps://en.wikipedia.org/wiki/Probably_approximately_correct_learning[00:06:10] Boolean Satisfiability Problem (SAT), Varioushttps://en.wikipedia.org/wiki/Boolean_satisfiability_problem[00:13:55] Knowledge as Justified True Belief (JTB), Matthias Steuphttps://plato.stanford.edu/entries/epistemology/[00:17:50] Wittgenstein's concept of the limits of language, Ludwig Wittgensteinhttps://plato.stanford.edu/entries/wittgenstein/[00:21:25] Boolean algebras, Ohad Osorhttps://tau.net/tau-language-research/[00:26:10] The Halting Problemhttps://plato.stanford.edu/entries/turing-machine/#HaltProb[00:30:25] Alfred Tarski (1901-1983), Mario Gómez-Torrentehttps://plato.stanford.edu/entries/tarski/[00:41:50] DPLLhttps://www.cs.princeton.edu/~zkincaid/courses/fall18/readings/SATHandbook-CDCL.pdf[00:49:50] Tarski's undefinability theorem (1936), Alfred Tarskihttps://plato.stanford.edu/entries/tarski-truth/[00:51:45] Boolean Algebra mathematical foundations, J. Donald Monkhttps://plato.stanford.edu/entries/boolalg-math/[01:02:35] Belief Revision Theory and AGM Postulates, Sven Ove Hanssonhttps://plato.stanford.edu/entries/logic-belief-revision/[01:05:35] Quantifier elimination in atomless boolean algebra, H. Jerome Keislerhttps://people.math.wisc.edu/~hkeisler/random.pdf[01:08:35] Quantifier elimination in Tau language specification, Ohad Asorhttps://tau.ai/Theories-and-Applications-of-Boolean-Algebras-0.29.pdf[01:11:50] Tau Net blockchain platformhttps://tau.net/[01:19:20] Tau blockchain's innovative approach treating blockchain code itself as a contracthttps://tau.net/Whitepaper.pdf

John Palazza from CentML joins us in this sponsored interview to discuss the critical importance of infrastructure optimization in the age of Large Language Models and Generative AI. We explore how enterprises can transition from the innovation phase to production and scale, highlighting the significance of efficient GPU utilization and cost management. The conversation covers the open-source versus proprietary model debate, the rise of AI agents, and the need for platform independence to avoid vendor lock-in, as well as emerging trends in AI infrastructure and the pivotal role of strategic partnerships.

SPONSOR MESSAGES:

***

CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments. Check out their super fast DeepSeek R1 hosting!

https://centml.ai/pricing/

Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. They are hiring a Chief Engineer and ML engineers. Events in Zurich.

Goto https://tufalabs.ai/

***

TRANSCRIPT:

https://www.dropbox.com/scl/fi/dnjsygrgdgq5ng5fdlfjg/JOHNPALAZZA.pdf?rlkey=hl9wyydi9mj077rbg5acdmo3a&dl=0

John Palazza:

Vice President of Global Sales @ CentML

https://www.linkedin.com/in/john-p-b34655/

TOC:

1. Enterprise AI Organization and Strategy

[00:00:00] 1.1 Organizational Structure and ML Ownership

[00:02:59] 1.2 Infrastructure Efficiency and GPU Utilization

[00:07:59] 1.3 Platform Centralization vs Team Autonomy

[00:11:32] 1.4 Enterprise AI Adoption Strategy and Leadership

2. MLOps Infrastructure and Resource Management

[00:15:08] 2.1 Technology Evolution and Enterprise Integration

[00:19:10] 2.2 Enterprise MLOps Platform Development

[00:22:15] 2.3 AI Interface Evolution and Agent-Based Solutions

[00:25:47] 2.4 CentML's Infrastructure Solutions

[00:30:00] 2.5 Workload Abstraction and Resource Allocation

3. LLM Infrastructure Optimization and Independence

[00:33:10] 3.1 GPU Optimization and Cost Efficiency

[00:36:47] 3.2 AI Efficiency and Innovation Challenges

[00:41:40] 3.3 Cloud Provider Strategy and Infrastructure Control

[00:46:52] 3.4 Platform Independence and Vendor Lock-in

[00:50:53] 3.5 Technical Innovation and Growth Strategy

REFS:

[00:01:25] Apple Acquires GraphLab, Apple Inc.

https://techcrunch.com/2016/08/05/apple-acquires-turi-a-machine-learning-company/

[00:03:50] Bain Tech Report 2024, Gartner

https://www.bain.com/insights/topics/technology-report/

[00:04:50] PaaS vs IaaS Efficiency, Gartner

https://www.gartner.com/en/newsroom/press-releases/2024-11-19-gartner-forecasts-worldwide-public-cloud-end-user-spending-to-total-723-billion-dollars-in-2025

[00:14:55] Fashion Quote, Oscar Wilde

https://www.amazon.com/Complete-Works-Oscar-Wilde-Collins/dp/0007144369

[00:15:30] PointCast Network, PointCast Inc.

https://en.wikipedia.org/wiki/Push_technology

[00:18:05] AI Bain Report, Bain & Company

https://www.bain.com/insights/how-generative-ai-changes-the-game-in-tech-services-tech-report-2024/

[00:20:40] Uber Michelangelo, Uber Engineering Team

https://www.uber.com/en-SE/blog/michelangelo-machine-learning-platform/

[00:20:50] Algorithmia Acquisition, DataRobot

https://www.datarobot.com/newsroom/press/datarobot-is-acquiring-algorithmia-enhancing-leading-mlops-architecture-for-the-enterprise/

[00:22:55] Fine Tuning vs RAG, Heydar Soudani, Evangelos Kanoulas & Faegheh Hasibi.

https://arxiv.org/html/2403.01432v2

[00:24:40] LLM Agent Survey, Lei Wang et al.

https://arxiv.org/abs/2308.11432

[00:26:30] CentML CServe, CentML

https://docs.centml.ai/apps/llm

[00:29:15] CentML Snowflake, Snowflake

https://www.snowflake.com/en/engineering-blog/optimize-llms-with-llama-snowflake-ai-stack/

[00:30:15] NVIDIA H100 GPU, NVIDIA

https://www.nvidia.com/en-us/data-center/h100/

[00:33:25] CentML\'s 60% savings, CentML

https://centml.ai/platform/

Federico Barbero (DeepMind/Oxford) is the lead author of "Transformers Need Glasses!".

Have you ever wondered why LLMs struggle with seemingly simple tasks like counting or copying long strings of text? We break down the theoretical reasons behind these failures, revealing architectural bottlenecks and the challenges of maintaining information fidelity across extended contexts.

Federico explains how these issues are rooted in the transformer's design, drawing parallels to over-squashing in graph neural networks and detailing how the softmax function limits sharp decision-making.

But it's not all bad news! Discover practical "glasses" that can help transformers see more clearly, from simple input modifications to architectural tweaks.

SPONSOR MESSAGES:

***

CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments. Check out their super fast DeepSeek R1 hosting!

https://centml.ai/pricing/

Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. They are hiring a Chief Engineer and ML engineers. Events in Zurich.

Goto https://tufalabs.ai/

***

https://federicobarbero.com/

TRANSCRIPT + RESEARCH:

https://www.dropbox.com/s/h7ys83ztwktqjje/Federico.pdf?dl=0

TOC:

1. Transformer Limitations: Token Detection & Representation

[00:00:00] 1.1 Transformers fail at single token detection

[00:02:45] 1.2 Representation collapse in transformers

[00:03:21] 1.3 Experiment: LLMs fail at copying last tokens

[00:18:00] 1.4 Attention sharpness limitations in transformers

2. Transformer Limitations: Information Flow & Quantization

[00:18:50] 2.1 Unidirectional information mixing

[00:18:50] 2.2 Unidirectional information flow towards sequence beginning in transformers

[00:21:50] 2.3 Diagonal attention heads as expensive no-ops in LAMA/Gemma

[00:27:14] 2.4 Sequence entropy affects transformer model distinguishability

[00:30:36] 2.5 Quantization limitations lead to information loss & representational collapse

[00:38:34] 2.6 LLMs use subitizing as opposed to counting algorithms

3. Transformers and the Nature of Reasoning

[00:40:30] 3.1 Turing completeness conditions in transformers

[00:43:23] 3.2 Transformers struggle with sequential tasks

[00:45:50] 3.3 Windowed attention as solution to information compression

[00:51:04] 3.4 Chess engines: mechanical computation vs creative reasoning

[01:00:35] 3.5 Epistemic foraging introduced

REFS:

[00:01:05] Transformers Need Glasses!, Barbero et al.

https://proceedings.neurips.cc/paper_files/paper/2024/file/b1d35561c4a4a0e0b6012b2af531e149-Paper-Conference.pdf

[00:05:30] Softmax is Not Enough, Veličković et al.

https://arxiv.org/abs/2410.01104

[00:11:30] Adv Alg Lecture 15, Chawla

https://pages.cs.wisc.edu/~shuchi/courses/787-F09/scribe-notes/lec15.pdf

[00:15:05] Graph Attention Networks, Veličković

https://arxiv.org/abs/1710.10903

[00:19:15] Extract Training Data, Carlini et al.

https://arxiv.org/pdf/2311.17035

[00:31:30] 1-bit LLMs, Ma et al.

https://arxiv.org/abs/2402.17764

[00:38:35] LLMs Solve Math, Nikankin et al.

https://arxiv.org/html/2410.21272v1

[00:38:45] Subitizing, Railo

https://link.springer.com/10.1007/978-1-4419-1428-6_578

[00:43:25] NN & Chomsky Hierarchy, Delétang et al.

https://arxiv.org/abs/2207.02098

[00:51:05] Measure of Intelligence, Chollet

https://arxiv.org/abs/1911.01547

[00:52:10] AlphaZero, Silver et al.

https://pubmed.ncbi.nlm.nih.gov/30523106/

[00:55:10] Golden Gate Claude, Anthropic

https://www.anthropic.com/news/golden-gate-claude

[00:56:40] Chess Positions, Chase & Simon

https://www.sciencedirect.com/science/article/abs/pii/0010028573900042

[01:00:35] Epistemic Foraging, Friston

https://www.frontiersin.org/journals/computational-neuroscience/articles/10.3389/fncom.2016.00056/full

We speak with Sakana AI, who are building nature-inspired methods that could fundamentally transform how we develop AI systems.

The guests include Chris Lu, a researcher who recently completed his DPhil at Oxford University under Prof. Jakob Foerster's supervision, where he focused on meta-learning and multi-agent systems. Chris is the first author of the DiscoPOP paper, which demonstrates how language models can discover and design better training algorithms. Also joining is Robert Tjarko Lange, a founding member of Sakana AI who specializes in evolutionary algorithms and large language models. Robert leads research at the intersection of evolutionary computation and foundation models, and is completing his PhD at TU Berlin on evolutionary meta-learning. The discussion also features Cong Lu, currently a Research Scientist at Google DeepMind's Open-Endedness team, who previously helped develop The AI Scientist and Intelligent Go-Explore.

SPONSOR MESSAGES:

***

CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments. Check out their super fast DeepSeek R1 hosting!

https://centml.ai/pricing/

Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. They are hiring a Chief Engineer and ML engineers. Events in Zurich.

Goto https://tufalabs.ai/

***

* DiscoPOP - A framework where language models discover their own optimization algorithms

* EvoLLM - Using language models as evolution strategies for optimization

The AI Scientist - A fully automated system that conducts scientific research end-to-end

* Neural Attention Memory Models (NAMMs) - Evolved memory systems that make transformers both faster and more accurate

TRANSCRIPT + REFS:

https://www.dropbox.com/scl/fi/gflcyvnujp8cl7zlv3v9d/Sakana.pdf?rlkey=woaoo82943170jd4yyi2he71c&dl=0

Robert Tjarko Lange

https://roberttlange.com/

Chris Lu

https://chrislu.page/

Cong Lu

https://www.conglu.co.uk/

Sakana

https://sakana.ai/blog/

TOC:

1. LLMs for Algorithm Generation and Optimization

[00:00:00] 1.1 LLMs generating algorithms for training other LLMs

[00:04:00] 1.2 Evolutionary black-box optim using neural network loss parameterization

[00:11:50] 1.3 DiscoPOP: Non-convex loss function for noisy data

[00:20:45] 1.4 External entropy Injection for preventing Model collapse

[00:26:25] 1.5 LLMs for black-box optimization using abstract numerical sequences

2. Model Learning and Generalization

[00:31:05] 2.1 Fine-tuning on teacher algorithm trajectories

[00:31:30] 2.2 Transformers learning gradient descent

[00:33:00] 2.3 LLM tokenization biases towards specific numbers

[00:34:50] 2.4 LLMs as evolution strategies for black box optimization

[00:38:05] 2.5 DiscoPOP: LLMs discovering novel optimization algorithms

3. AI Agents and System Architectures

[00:51:30] 3.1 ARC challenge: Induction vs. transformer approaches

[00:54:35] 3.2 LangChain / modular agent components

[00:57:50] 3.3 Debate improves LLM truthfulness

[01:00:55] 3.4 Time limits controlling AI agent systems

[01:03:00] 3.5 Gemini: Million-token context enables flatter hierarchies

[01:04:05] 3.6 Agents follow own interest gradients

[01:09:50] 3.7 Go-Explore algorithm: archive-based exploration

[01:11:05] 3.8 Foundation models for interesting state discovery

[01:13:00] 3.9 LLMs leverage prior game knowledge

4. AI for Scientific Discovery and Human Alignment

[01:17:45] 4.1 Encoding Alignment & Aesthetics via Reward Functions

[01:20:00] 4.2 AI Scientist: Automated Open-Ended Scientific Discovery

[01:24:15] 4.3 DiscoPOP: LLM for Preference Optimization Algorithms

[01:28:30] 4.4 Balancing AI Knowledge with Human Understanding

[01:33:55] 4.5 AI-Driven Conferences and Paper Review

Clement Bonnet discusses his novel approach to the ARC (Abstraction and Reasoning Corpus) challenge. Unlike approaches that rely on fine-tuning LLMs or generating samples at inference time, Clement's method encodes input-output pairs into a latent space, optimizes this representation with a search algorithm, and decodes outputs for new inputs. This end-to-end architecture uses a VAE loss, including reconstruction and prior losses.

SPONSOR MESSAGES:

***

CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments. Check out their super fast DeepSeek R1 hosting!

https://centml.ai/pricing/

Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. They are hiring a Chief Engineer and ML engineers. Events in Zurich.

Goto https://tufalabs.ai/

***

TRANSCRIPT + RESEARCH OVERVIEW:

https://www.dropbox.com/scl/fi/j7m0gaz1126y594gswtma/CLEMMLST.pdf?rlkey=y5qvwq2er5nchbcibm07rcfpq&dl=0

Clem and Matthew-

https://www.linkedin.com/in/clement-bonnet16/

https://github.com/clement-bonnet

https://mvmacfarlane.github.io/

TOC

1. LPN Fundamentals

[00:00:00] 1.1 Introduction to ARC Benchmark and LPN Overview

[00:05:05] 1.2 Neural Networks' Challenges with ARC and Program Synthesis

[00:06:55] 1.3 Induction vs Transduction in Machine Learning

2. LPN Architecture and Latent Space

[00:11:50] 2.1 LPN Architecture and Latent Space Implementation

[00:16:25] 2.2 LPN Latent Space Encoding and VAE Architecture

[00:20:25] 2.3 Gradient-Based Search Training Strategy

[00:23:39] 2.4 LPN Model Architecture and Implementation Details

3. Implementation and Scaling

[00:27:34] 3.1 Training Data Generation and re-ARC Framework

[00:31:28] 3.2 Limitations of Latent Space and Multi-Thread Search

[00:34:43] 3.3 Program Composition and Computational Graph Architecture

4. Advanced Concepts and Future Directions

[00:45:09] 4.1 AI Creativity and Program Synthesis Approaches

[00:49:47] 4.2 Scaling and Interpretability in Latent Space Models

REFS

[00:00:05] ARC benchmark, Chollet

https://arxiv.org/abs/2412.04604

[00:02:10] Latent Program Spaces, Bonnet, Macfarlane

https://arxiv.org/abs/2411.08706

[00:07:45] Kevin Ellis work on program generation

https://www.cs.cornell.edu/~ellisk/

[00:08:45] Induction vs transduction in abstract reasoning, Li et al.

https://arxiv.org/abs/2411.02272

[00:17:40] VAEs, Kingma, Welling

https://arxiv.org/abs/1312.6114

[00:27:50] re-ARC, Hodel

https://github.com/michaelhodel/re-arc

[00:29:40] Grid size in ARC tasks, Chollet

https://github.com/fchollet/ARC-AGI

[00:33:00] Critique of deep learning, Marcus

https://arxiv.org/vc/arxiv/papers/2002/2002.06177v1.pdf

Prof. Jakob Foerster, a leading AI researcher at Oxford University and Meta, and Chris Lu, a researcher at OpenAI -- they explain how AI is moving beyond just mimicking human behaviour to creating truly intelligent agents that can learn and solve problems on their own. Foerster champions open-source AI for responsible, decentralised development. He addresses AI scaling, goal misalignment (Goodhart's Law), and the need for holistic alignment, offering a quick look at the future of AI and how to guide it.

SPONSOR MESSAGES:

***

CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments. Check out their super fast DeepSeek R1 hosting!

https://centml.ai/pricing/

Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. They are hiring a Chief Engineer and ML engineers. Events in Zurich.

Goto https://tufalabs.ai/

***

TRANSCRIPT/REFS:

https://www.dropbox.com/scl/fi/yqjszhntfr00bhjh6t565/JAKOB.pdf?rlkey=scvny4bnwj8th42fjv8zsfu2y&dl=0

Prof. Jakob Foerster

https://x.com/j_foerst

https://www.jakobfoerster.com/

University of Oxford Profile:

https://eng.ox.ac.uk/people/jakob-foerster/

Chris Lu:

https://chrislu.page/

TOC

1. GPU Acceleration and Training Infrastructure

[00:00:00] 1.1 ARC Challenge Criticism and FLAIR Lab Overview

[00:01:25] 1.2 GPU Acceleration and Hardware Lottery in RL

[00:05:50] 1.3 Data Wall Challenges and Simulation-Based Solutions

[00:08:40] 1.4 JAX Implementation and Technical Acceleration

2. Learning Frameworks and Policy Optimization

[00:14:18] 2.1 Evolution of RL Algorithms and Mirror Learning Framework

[00:15:25] 2.2 Meta-Learning and Policy Optimization Algorithms

[00:21:47] 2.3 Language Models and Benchmark Challenges

[00:28:15] 2.4 Creativity and Meta-Learning in AI Systems

3. Multi-Agent Systems and Decentralization

[00:31:24] 3.1 Multi-Agent Systems and Emergent Intelligence

[00:38:35] 3.2 Swarm Intelligence vs Monolithic AGI Systems

[00:42:44] 3.3 Democratic Control and Decentralization of AI Development

[00:46:14] 3.4 Open Source AI and Alignment Challenges

[00:49:31] 3.5 Collaborative Models for AI Development

REFS

[[00:00:05] ARC Benchmark, Chollet

https://github.com/fchollet/ARC-AGI

[00:03:05] DRL Doesn't Work, Irpan

https://www.alexirpan.com/2018/02/14/rl-hard.html

[00:05:55] AI Training Data, Data Provenance Initiative

https://www.nytimes.com/2024/07/19/technology/ai-data-restrictions.html

[00:06:10] JaxMARL, Foerster et al.

https://arxiv.org/html/2311.10090v5

[00:08:50] M-FOS, Lu et al.

https://arxiv.org/abs/2205.01447

[00:09:45] JAX Library, Google Research

https://github.com/jax-ml/jax

[00:12:10] Kinetix, Mike and Michael

https://arxiv.org/abs/2410.23208

[00:12:45] Genie 2, DeepMind

https://deepmind.google/discover/blog/genie-2-a-large-scale-foundation-world-model/

[00:14:42] Mirror Learning, Grudzien, Kuba et al.

https://arxiv.org/abs/2208.01682

[00:16:30] Discovered Policy Optimisation, Lu et al.

https://arxiv.org/abs/2210.05639

[00:24:10] Goodhart's Law, Goodhart

https://en.wikipedia.org/wiki/Goodhart%27s_law

[00:25:15] LLM ARChitect, Franzen et al.

https://github.com/da-fr/arc-prize-2024/blob/main/the_architects.pdf

[00:28:55] AlphaGo, Silver et al.

https://arxiv.org/pdf/1712.01815.pdf

[00:30:10] Meta-learning, Lu, Towers, Foerster

https://direct.mit.edu/isal/proceedings-pdf/isal2023/35/67/2354943/isal_a_00674.pdf

[00:31:30] Emergence of Pragmatics, Yuan et al.

https://arxiv.org/abs/2001.07752

[00:34:30] AI Safety, Amodei et al.

https://arxiv.org/abs/1606.06565

[00:35:45] Intentional Stance, Dennett

https://plato.stanford.edu/entries/ethics-ai/

[00:39:25] Multi-Agent RL, Zhou et al.

https://arxiv.org/pdf/2305.10091

[00:41:00] Open Source Generative AI, Foerster et al.

https://arxiv.org/abs/2405.08597

<trunc, see PDF/YT>

Daniel Franzen and Jan Disselhoff, the "ARChitects" are the official winners of the ARC Prize 2024. Filmed at Tufa Labs in Zurich - they revealed how they achieved a remarkable 53.5% accuracy by creatively utilising large language models (LLMs) in new ways. Discover their innovative techniques, including depth-first search for token selection, test-time training, and a novel augmentation-based validation system. Their results were extremely surprising.

SPONSOR MESSAGES:

***

CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments. Check out their super fast DeepSeek R1 hosting!

https://centml.ai/pricing/

Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. They are hiring a Chief Engineer and ML engineers. Events in Zurich.

Goto https://tufalabs.ai/

***

Jan Disselhoff

https://www.linkedin.com/in/jan-disselhoff-1423a2240/

Daniel Franzen

https://github.com/da-fr

ARC Prize: http://arcprize.org/

TRANSCRIPT AND BACKGROUND READING:

https://www.dropbox.com/scl/fi/utkn2i1ma79fn6an4yvjw/ARCHitects.pdf?rlkey=67pe38mtss7oyhjk2ad0d2aza&dl=0

TOC

1. Solution Architecture and Strategy Overview

[00:00:00] 1.1 Initial Solution Overview and Model Architecture

[00:04:25] 1.2 LLM Capabilities and Dataset Approach

[00:10:51] 1.3 Test-Time Training and Data Augmentation Strategies

[00:14:08] 1.4 Sampling Methods and Search Implementation

[00:17:52] 1.5 ARC vs Language Model Context Comparison

2. LLM Search and Model Implementation

[00:21:53] 2.1 LLM-Guided Search Approaches and Solution Validation

[00:27:04] 2.2 Symmetry Augmentation and Model Architecture

[00:30:11] 2.3 Model Intelligence Characteristics and Performance

[00:37:23] 2.4 Tokenization and Numerical Processing Challenges

3. Advanced Training and Optimization

[00:45:15] 3.1 DFS Token Selection and Probability Thresholds

[00:49:41] 3.2 Model Size and Fine-tuning Performance Trade-offs

[00:53:07] 3.3 LoRA Implementation and Catastrophic Forgetting Prevention

[00:56:10] 3.4 Training Infrastructure and Optimization Experiments

[01:02:34] 3.5 Search Tree Analysis and Entropy Distribution Patterns

REFS

[00:01:05] Winning ARC 2024 solution using 12B param model, Franzen, Disselhoff, Hartmann

https://github.com/da-fr/arc-prize-2024/blob/main/the_architects.pdf

[00:03:40] Robustness of analogical reasoning in LLMs, Melanie Mitchell

https://arxiv.org/html/2411.14215

[00:07:50] Re-ARC dataset generator for ARC task variations, Michael Hodel

https://github.com/michaelhodel/re-arc

[00:15:00] Analysis of search methods in LLMs (greedy, beam, DFS), Chen et al.

https://arxiv.org/html/2408.00724v2

[00:16:55] Language model reachability space exploration, University of Toronto

https://www.youtube.com/watch?v=Bpgloy1dDn0

[00:22:30] GPT-4 guided code solutions for ARC tasks, Ryan Greenblatt

https://redwoodresearch.substack.com/p/getting-50-sota-on-arc-agi-with-gpt

[00:41:20] GPT tokenization approach for numbers, OpenAI

https://platform.openai.com/docs/guides/text-generation/tokenizer-examples

[00:46:25] DFS in AI search strategies, Russell & Norvig

https://www.amazon.com/Artificial-Intelligence-Modern-Approach-4th/dp/0134610997

[00:53:10] Paper on catastrophic forgetting in neural networks, Kirkpatrick et al.

https://www.pnas.org/doi/10.1073/pnas.1611835114

[00:54:00] LoRA for efficient fine-tuning of LLMs, Hu et al.

https://arxiv.org/abs/2106.09685

[00:57:20] NVIDIA H100 Tensor Core GPU specs, NVIDIA

https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/

[01:04:55] Original MCTS in computer Go, Yifan Jin

https://stanford.edu/~rezab/classes/cme323/S15/projects/montecarlo_search_tree_report.pdf

Sepp Hochreiter, the inventor of LSTM (Long Short-Term Memory) networks – a foundational technology in AI. Sepp discusses his journey, the origins of LSTM, and why he believes his latest work, XLSTM, could be the next big thing in AI, particularly for applications like robotics and industrial simulation. He also shares his controversial perspective on Large Language Models (LLMs) and why reasoning is a critical missing piece in current AI systems.

SPONSOR MESSAGES:

***

CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments. Check out their super fast DeepSeek R1 hosting!

https://centml.ai/pricing/

Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. They are hiring a Chief Engineer and ML engineers. Events in Zurich.

Goto https://tufalabs.ai/

***

TRANSCRIPT AND BACKGROUND READING:

https://www.dropbox.com/scl/fi/n1vzm79t3uuss8xyinxzo/SEPPH.pdf?rlkey=fp7gwaopjk17uyvgjxekxrh5v&dl=0

Prof. Sepp Hochreiter

https://www.nx-ai.com/

https://x.com/hochreitersepp

https://scholar.google.at/citations?user=tvUH3WMAAAAJ&hl=en

TOC:

1. LLM Evolution and Reasoning Capabilities

[00:00:00] 1.1 LLM Capabilities and Limitations Debate

[00:03:16] 1.2 Program Generation and Reasoning in AI Systems

[00:06:30] 1.3 Human vs AI Reasoning Comparison

[00:09:59] 1.4 New Research Initiatives and Hybrid Approaches

2. LSTM Technical Architecture

[00:13:18] 2.1 LSTM Development History and Technical Background

[00:20:38] 2.2 LSTM vs RNN Architecture and Computational Complexity

[00:25:10] 2.3 xLSTM Architecture and Flash Attention Comparison

[00:30:51] 2.4 Evolution of Gating Mechanisms from Sigmoid to Exponential

3. Industrial Applications and Neuro-Symbolic AI

[00:40:35] 3.1 Industrial Applications and Fixed Memory Advantages

[00:42:31] 3.2 Neuro-Symbolic Integration and Pi AI Project

[00:46:00] 3.3 Integration of Symbolic and Neural AI Approaches

[00:51:29] 3.4 Evolution of AI Paradigms and System Thinking

[00:54:55] 3.5 AI Reasoning and Human Intelligence Comparison

[00:58:12] 3.6 NXAI Company and Industrial AI Applications

REFS:

[00:00:15] Seminal LSTM paper establishing Hochreiter's expertise (Hochreiter & Schmidhuber)

https://direct.mit.edu/neco/article-abstract/9/8/1735/6109/Long-Short-Term-Memory

[00:04:20] Kolmogorov complexity and program composition limitations (Kolmogorov)

https://link.springer.com/article/10.1007/BF02478259

[00:07:10] Limitations of LLM mathematical reasoning and symbolic integration (Various Authors)

https://www.arxiv.org/pdf/2502.03671

[00:09:05] AlphaGo’s Move 37 demonstrating creative AI (Google DeepMind)

https://deepmind.google/research/breakthroughs/alphago/

[00:10:15] New AI research lab in Zurich for fundamental LLM research (Benjamin Crouzier)

https://tufalabs.ai

[00:19:40] Introduction of xLSTM with exponential gating (Beck, Hochreiter, et al.)

https://arxiv.org/abs/2405.04517

[00:22:55] FlashAttention: fast & memory-efficient attention (Tri Dao et al.)

https://arxiv.org/abs/2205.14135

[00:31:00] Historical use of sigmoid/tanh activation in 1990s (James A. McCaffrey)

https://visualstudiomagazine.com/articles/2015/06/01/alternative-activation-functions.aspx

[00:36:10] Mamba 2 state space model architecture (Albert Gu et al.)

https://arxiv.org/abs/2312.00752

[00:46:00] Austria’s Pi AI project integrating symbolic & neural AI (Hochreiter et al.)

https://www.jku.at/en/institute-of-machine-learning/research/projects/

[00:48:10] Neuro-symbolic integration challenges in language models (Diego Calanzone et al.)

https://openreview.net/forum?id=7PGluppo4k

[00:49:30] JKU Linz’s historical and neuro-symbolic research (Sepp Hochreiter)

https://www.jku.at/en/news-events/news/detail/news/bilaterale-ki-projekt-unter-leitung-der-jku-erhaelt-fwf-cluster-of-excellence/

YT: https://www.youtube.com/watch?v=8u2pW2zZLCs

<truncated, see show notes/YT>

Professor Randall Balestriero joins us to discuss neural network geometry, spline theory, and emerging phenomena in deep learning, based on research presented at ICML. Topics include the delayed emergence of adversarial robustness in neural networks ("grokking"), geometric interpretations of neural networks via spline theory, and challenges in reconstruction learning. We also cover geometric analysis of Large Language Models (LLMs) for toxicity detection and the relationship between intrinsic dimensionality and model control in RLHF.

SPONSOR MESSAGES:

***

CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments.

https://centml.ai/pricing/

Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. Are you interested in working on reasoning, or getting involved in their events?

Goto https://tufalabs.ai/

***

Randall Balestriero

https://x.com/randall_balestr

https://randallbalestriero.github.io/

Show notes and transcript: https://www.dropbox.com/scl/fi/3lufge4upq5gy0ug75j4a/RANDALLSHOW.pdf?rlkey=nbemgpa0jhawt1e86rx7372e4&dl=0

TOC:

- Introduction

- 00:00:00: Introduction

- Neural Network Geometry and Spline Theory

- 00:01:41: Neural Network Geometry and Spline Theory

- 00:07:41: Deep Networks Always Grok

- 00:11:39: Grokking and Adversarial Robustness

- 00:16:09: Double Descent and Catastrophic Forgetting

- Reconstruction Learning

- 00:18:49: Reconstruction Learning

- 00:24:15: Frequency Bias in Neural Networks

- Geometric Analysis of Neural Networks

- 00:29:02: Geometric Analysis of Neural Networks

- 00:34:41: Adversarial Examples and Region Concentration

- LLM Safety and Geometric Analysis

- 00:40:05: LLM Safety and Geometric Analysis

- 00:46:11: Toxicity Detection in LLMs

- 00:52:24: Intrinsic Dimensionality and Model Control

- 00:58:07: RLHF and High-Dimensional Spaces

- Conclusion

- 01:02:13: Neural Tangent Kernel

- 01:08:07: Conclusion

REFS:

[00:01:35] Humayun – Deep network geometry & input space partitioning

https://arxiv.org/html/2408.04809v1

[00:03:55] Balestriero & Paris – Linking deep networks to adaptive spline operators

https://proceedings.mlr.press/v80/balestriero18b/balestriero18b.pdf

[00:13:55] Song et al. – Gradient-based white-box adversarial attacks

https://arxiv.org/abs/2012.14965

[00:16:05] Humayun, Balestriero & Baraniuk – Grokking phenomenon & emergent robustness

https://arxiv.org/abs/2402.15555

[00:18:25] Humayun – Training dynamics & double descent via linear region evolution

https://arxiv.org/abs/2310.12977

[00:20:15] Balestriero – Power diagram partitions in DNN decision boundaries

https://arxiv.org/abs/1905.08443

[00:23:00] Frankle & Carbin – Lottery Ticket Hypothesis for network pruning

https://arxiv.org/abs/1803.03635

[00:24:00] Belkin et al. – Double descent phenomenon in modern ML

https://arxiv.org/abs/1812.11118

[00:25:55] Balestriero et al. – Batch normalization’s regularization effects

https://arxiv.org/pdf/2209.14778

[00:29:35] EU – EU AI Act 2024 with compute restrictions

https://www.lw.com/admin/upload/SiteAttachments/EU-AI-Act-Navigating-a-Brave-New-World.pdf

[00:39:30] Humayun, Balestriero & Baraniuk – SplineCam: Visualizing deep network geometry

https://openaccess.thecvf.com/content/CVPR2023/papers/Humayun_SplineCam_Exact_Visualization_and_Characterization_of_Deep_Network_Geometry_and_CVPR_2023_paper.pdf

[00:40:40] Carlini – Trade-offs between adversarial robustness and accuracy

https://arxiv.org/pdf/2407.20099

[00:44:55] Balestriero & LeCun – Limitations of reconstruction-based learning methods

https://openreview.net/forum?id=ez7w0Ss4g9

(truncated, see shownotes PDF)

Nicholas Carlini from Google DeepMind offers his view of AI security, emergent LLM capabilities, and his groundbreaking model-stealing research. He reveals how LLMs can unexpectedly excel at tasks like chess and discusses the security pitfalls of LLM-generated code.

SPONSOR MESSAGES:

***

CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments.

https://centml.ai/pricing/

Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. Are you interested in working on reasoning, or getting involved in their events?

Goto https://tufalabs.ai/

***

Transcript: https://www.dropbox.com/scl/fi/lat7sfyd4k3g5k9crjpbf/CARLINI.pdf?rlkey=b7kcqbvau17uw6rksbr8ccd8v&dl=0

TOC:

1. ML Security Fundamentals

[00:00:00] 1.1 ML Model Reasoning and Security Fundamentals

[00:03:04] 1.2 ML Security Vulnerabilities and System Design

[00:08:22] 1.3 LLM Chess Capabilities and Emergent Behavior

[00:13:20] 1.4 Model Training, RLHF, and Calibration Effects

2. Model Evaluation and Research Methods

[00:19:40] 2.1 Model Reasoning and Evaluation Metrics

[00:24:37] 2.2 Security Research Philosophy and Methodology

[00:27:50] 2.3 Security Disclosure Norms and Community Differences

3. LLM Applications and Best Practices

[00:44:29] 3.1 Practical LLM Applications and Productivity Gains

[00:49:51] 3.2 Effective LLM Usage and Prompting Strategies

[00:53:03] 3.3 Security Vulnerabilities in LLM-Generated Code

4. Advanced LLM Research and Architecture

[00:59:13] 4.1 LLM Code Generation Performance and O(1) Labs Experience

[01:03:31] 4.2 Adaptation Patterns and Benchmarking Challenges

[01:10:10] 4.3 Model Stealing Research and Production LLM Architecture Extraction

REFS:

[00:01:15] Nicholas Carlini’s personal website & research profile (Google DeepMind, ML security) - https://nicholas.carlini.com/

[00:01:50] CentML AI compute platform for language model workloads - https://centml.ai/

[00:04:30] Seminal paper on neural network robustness against adversarial examples (Carlini & Wagner, 2016) - https://arxiv.org/abs/1608.04644

[00:05:20] Computer Fraud and Abuse Act (CFAA) – primary U.S. federal law on computer hacking liability - https://www.justice.gov/jm/jm-9-48000-computer-fraud

[00:08:30] Blog post: Emergent chess capabilities in GPT-3.5-turbo-instruct (Nicholas Carlini, Sept 2023) - https://nicholas.carlini.com/writing/2023/chess-llm.html

[00:16:10] Paper: “Self-Play Preference Optimization for Language Model Alignment” (Yue Wu et al., 2024) - https://arxiv.org/abs/2405.00675

[00:18:00] GPT-4 Technical Report: development, capabilities, and calibration analysis - https://arxiv.org/abs/2303.08774

[00:22:40] Historical shift from descriptive to algebraic chess notation (FIDE) - https://en.wikipedia.org/wiki/Descriptive_notation

[00:23:55] Analysis of distribution shift in ML (Hendrycks et al.) - https://arxiv.org/abs/2006.16241

[00:27:40] Nicholas Carlini’s essay “Why I Attack” (June 2024) – motivations for security research - https://nicholas.carlini.com/writing/2024/why-i-attack.html

[00:34:05] Google Project Zero’s 90-day vulnerability disclosure policy - https://googleprojectzero.blogspot.com/p/vulnerability-disclosure-policy.html

[00:51:15] Evolution of Google search syntax & user behavior (Daniel M. Russell) - https://www.amazon.com/Joy-Search-Google-Master-Information/dp/0262042878

[01:04:05] Rust’s ownership & borrowing system for memory safety - https://doc.rust-lang.org/book/ch04-00-understanding-ownership.html

[01:10:05] Paper: “Stealing Part of a Production Language Model” (Carlini et al., March 2024) – extraction attacks on ChatGPT, PaLM-2 - https://arxiv.org/abs/2403.06634

[01:10:55] First model stealing paper (Tramèr et al., 2016) – attacking ML APIs via prediction - https://arxiv.org/abs/1609.02943

Join Prof. Subbarao Kambhampati and host Tim Scarfe for a deep dive into OpenAI's O1 model and the future of AI reasoning systems.

* How O1 likely uses reinforcement learning similar to AlphaGo, with hidden reasoning tokens that users pay for but never see

* The evolution from traditional Large Language Models to more sophisticated reasoning systems

* The concept of "fractal intelligence" in AI - where models work brilliantly sometimes but fail unpredictably

* Why O1's improved performance comes with substantial computational costs

* The ongoing debate between single-model approaches (OpenAI) vs hybrid systems (Google)

* The critical distinction between AI as an intelligence amplifier vs autonomous decision-maker

SPONSOR MESSAGES:

***

CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments.

https://centml.ai/pricing/

Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. Are you interested in working on reasoning, or getting involved in their events?

Goto https://tufalabs.ai/

***

TOC:

1. **O1 Architecture and Reasoning Foundations**

[00:00:00] 1.1 Fractal Intelligence and Reasoning Model Limitations

[00:04:28] 1.2 LLM Evolution: From Simple Prompting to Advanced Reasoning

[00:14:28] 1.3 O1's Architecture and AlphaGo-like Reasoning Approach

[00:23:18] 1.4 Empirical Evaluation of O1's Planning Capabilities

2. **Monte Carlo Methods and Model Deep-Dive**

[00:29:30] 2.1 Monte Carlo Methods and MARCO-O1 Implementation

[00:31:30] 2.2 Reasoning vs. Retrieval in LLM Systems

[00:40:40] 2.3 Fractal Intelligence Capabilities and Limitations

[00:45:59] 2.4 Mechanistic Interpretability of Model Behavior

[00:51:41] 2.5 O1 Response Patterns and Performance Analysis

3. **System Design and Real-World Applications**

[00:59:30] 3.1 Evolution from LLMs to Language Reasoning Models

[01:06:48] 3.2 Cost-Efficiency Analysis: LLMs vs O1

[01:11:28] 3.3 Autonomous vs Human-in-the-Loop Systems

[01:16:01] 3.4 Program Generation and Fine-Tuning Approaches

[01:26:08] 3.5 Hybrid Architecture Implementation Strategies

Transcript: https://www.dropbox.com/scl/fi/d0ef4ovnfxi0lknirkvft/Subbarao.pdf?rlkey=l3rp29gs4hkut7he8u04mm1df&dl=0

REFS:

[00:02:00] Monty Python (1975)

Witch trial scene: flawed logical reasoning.

https://www.youtube.com/watch?v=zrzMhU_4m-g

[00:04:00] Cade Metz (2024)

Microsoft–OpenAI partnership evolution and control dynamics.

https://www.nytimes.com/2024/10/17/technology/microsoft-openai-partnership-deal.html

[00:07:25] Kojima et al. (2022)

Zero-shot chain-of-thought prompting ('Let's think step by step').

https://arxiv.org/pdf/2205.11916

[00:12:50] DeepMind Research Team (2023)

Multi-bot game solving with external and internal planning.

https://deepmind.google/research/publications/139455/

[00:15:10] Silver et al. (2016)

AlphaGo's Monte Carlo Tree Search and Q-learning.

https://www.nature.com/articles/nature16961

[00:16:30] Kambhampati, S. et al. (2023)

Evaluates O1's planning in "Strawberry Fields" benchmarks.

https://arxiv.org/pdf/2410.02162

[00:29:30] Alibaba AIDC-AI Team (2023)

MARCO-O1: Chain-of-Thought + MCTS for improved reasoning.

https://arxiv.org/html/2411.14405

[00:31:30] Kambhampati, S. (2024)

Explores LLM "reasoning vs retrieval" debate.

https://arxiv.org/html/2403.04121v2

[00:37:35] Wei, J. et al. (2022)

Chain-of-thought prompting (introduces last-letter concatenation).

https://arxiv.org/pdf/2201.11903

[00:42:35] Barbero, F. et al. (2024)

Transformer attention and "information over-squashing."

https://arxiv.org/html/2406.04267v2

[00:46:05] Ruis, L. et al. (2023)

Influence functions to understand procedural knowledge in LLMs.

https://arxiv.org/html/2411.12580v1

(truncated - continued in shownotes/transcript doc)

Laura Ruis, a PhD student at University College London and researcher at Cohere, explains her groundbreaking research into how large language models (LLMs) perform reasoning tasks, the fundamental mechanisms underlying LLM reasoning capabilities, and whether these models primarily rely on retrieval or develop procedural knowledge.

SPONSOR MESSAGES:

***

CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments.

https://centml.ai/pricing/

Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. Are you interested in working on reasoning, or getting involved in their events?

Goto https://tufalabs.ai/

***

TOC

1. LLM Foundations and Learning

1.1 Scale and Learning in Language Models [00:00:00]

1.2 Procedural Knowledge vs Fact Retrieval [00:03:40]

1.3 Influence Functions and Model Analysis [00:07:40]

1.4 Role of Code in LLM Reasoning [00:11:10]

1.5 Semantic Understanding and Physical Grounding [00:19:30]

2. Reasoning Architectures and Measurement

2.1 Measuring Understanding and Reasoning in Language Models [00:23:10]

2.2 Formal vs Approximate Reasoning and Model Creativity [00:26:40]

2.3 Symbolic vs Subsymbolic Computation Debate [00:34:10]

2.4 Neural Network Architectures and Tensor Product Representations [00:40:50]

3. AI Agency and Risk Assessment

3.1 Agency and Goal-Directed Behavior in Language Models [00:45:10]

3.2 Defining and Measuring Agency in AI Systems [00:49:50]

3.3 Core Knowledge Systems and Agency Detection [00:54:40]

3.4 Language Models as Agent Models and Simulator Theory [01:03:20]

3.5 AI Safety and Societal Control Mechanisms [01:07:10]

3.6 Evolution of AI Capabilities and Emergent Risks [01:14:20]

REFS:

[00:01:10] Procedural Knowledge in Pretraining & LLM Reasoning

Ruis et al., 2024

https://arxiv.org/abs/2411.12580

[00:03:50] EK-FAC Influence Functions in Large LMs

Grosse et al., 2023

https://arxiv.org/abs/2308.03296

[00:13:05] Surfaces and Essences: Analogy as the Core of Cognition

Hofstadter & Sander

https://www.amazon.com/Surfaces-Essences-Analogy-Fuel-Thinking/dp/0465018475

[00:13:45] Wittgenstein on Language Games

https://plato.stanford.edu/entries/wittgenstein/

[00:14:30] Montague Semantics for Natural Language

https://plato.stanford.edu/entries/montague-semantics/

[00:19:35] The Chinese Room Argument

David Cole

https://plato.stanford.edu/entries/chinese-room/

[00:19:55] ARC: Abstraction and Reasoning Corpus

François Chollet

https://arxiv.org/abs/1911.01547

[00:24:20] Systematic Generalization in Neural Nets

Lake & Baroni, 2023

https://www.nature.com/articles/s41586-023-06668-3

[00:27:40] Open-Endedness & Creativity in AI

Tim Rocktäschel

https://arxiv.org/html/2406.04268v1

[00:30:50] Fodor & Pylyshyn on Connectionism

https://www.sciencedirect.com/science/article/abs/pii/0010027788900315

[00:31:30] Tensor Product Representations

Smolensky, 1990

https://www.sciencedirect.com/science/article/abs/pii/000437029090007M

[00:35:50] DreamCoder: Wake-Sleep Program Synthesis

Kevin Ellis et al.

https://courses.cs.washington.edu/courses/cse599j1/22sp/papers/dreamcoder.pdf

[00:36:30] Compositional Generalization Benchmarks

Ruis, Lake et al., 2022

https://arxiv.org/pdf/2202.10745

[00:40:30] RNNs & Tensor Products

McCoy et al., 2018

https://arxiv.org/abs/1812.08718

[00:46:10] Formal Causal Definition of Agency

Kenton et al.

https://arxiv.org/pdf/2208.08345v2

[00:48:40] Agency in Language Models

Sumers et al.

https://arxiv.org/abs/2309.02427

[00:55:20] Heider & Simmel’s Moving Shapes Experiment

https://www.nature.com/articles/s41598-024-65532-0

[01:00:40] Language Models as Agent Models

Jacob Andreas, 2022

https://arxiv.org/abs/2212.01681

[01:13:35] Pragmatic Understanding in LLMs

Ruis et al.

https://arxiv.org/abs/2210.14986

Jürgen Schmidhuber, the father of generative AI, challenges current AI narratives, revealing that early deep learning work is in his opinion misattributed, where it actually originated in Ukraine and Japan. He discusses his early work on linear transformers and artificial curiosity which preceded modern developments, shares his expansive vision of AI colonising space, and explains his groundbreaking 1991 consciousness model. Schmidhuber dismisses fears of human-AI conflict, arguing that superintelligent AI scientists will be fascinated by their own origins and motivated to protect life rather than harm it, while being more interested in other superintelligent AI and in cosmic expansion than earthly matters. He offers unique insights into how humans and AI might coexist. This was the long-awaited second, unreleased part of our interview we filmed last time. SPONSOR MESSAGES: *** CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments. https://centml.ai/pricing/ Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. Are you interested in working on reasoning, or getting involved in their events? Goto https://tufalabs.ai/ *** Interviewer: Tim Scarfe TOC [00:00:00] The Nature and Motivations of AI [00:02:08] Influential Inventions: 20th vs. 21st Century [00:05:28] Transformer and GPT: A Reflection The revolutionary impact of modern language models, the 1991 linear transformer, linear vs. quadratic scaling, the fast weight controller, and fast weight matrix memory. [00:11:03] Pioneering Contributions to AI and Deep Learning The invention of the transformer, pre-trained networks, the first GANs, the role of predictive coding, and the emergence of artificial curiosity. [00:13:58] AI's Evolution and Achievements The role of compute, breakthroughs in handwriting recognition and computer vision, the rise of GPU-based CNNs, achieving superhuman results, and Japanese contributions to CNN development. [00:15:40] The Hardware Lottery and GPUs GPUs as a serendipitous advantage for AI, the gaming-AI parallel, and Nvidia's strategic shift towards AI. [00:19:58] AI Applications and Societal Impact AI-powered translation breaking communication barriers, AI in medicine for imaging and disease prediction, and AI's potential for human enhancement and sustainable development. [00:23:26] The Path to AGI and Current Limitations Distinguishing large language models from AGI, challenges in replacing physical world workers, and AI's difficulty in real-world versus board games. [00:25:56] AI and Consciousness Simulating consciousness through unsupervised learning, chunking and automatizing neural networks, data compression, and self-symbols in predictive world models. [00:30:50] The Future of AI and Humanity Transition from AGIs as tools to AGIs with their own goals, the role of humans in an AGI-dominated world, and the concept of Homo Ludens. [00:38:05] The AI Race: Europe, China, and the US Europe's historical contributions, current dominance of the US and East Asia, and the role of venture capital and industrial policy. [00:50:32] Addressing AI Existential Risk The obsession with AI existential risk, commercial pressure for friendly AIs, AI vs. hydrogen bombs, and the long-term future of AI. [00:58:00] The Fermi Paradox and Extraterrestrial Intelligence Expanding AI bubbles as an explanation for the Fermi paradox, dark matter and encrypted civilizations, and Earth as the first to spawn an AI bubble. [01:02:08] The Diversity of AI and AI Ecologies The unrealism of a monolithic super intelligence, diverse AIs with varying goals, and intense competition and collaboration in AI ecologies. [01:12:21] Final Thoughts and Closing Remarks REFERENCES: See pinned comment on YT: https://youtu.be/fZYUqICYCAk

Professor Yoshua Bengio is a pioneer in deep learning and Turing Award winner. Bengio talks about AI safety, why goal-seeking “agentic” AIs might be dangerous, and his vision for building powerful AI tools without giving them agency. Topics include reward tampering risks, instrumental convergence, global AI governance, and how non-agent AIs could revolutionize science and medicine while reducing existential threats. Perfect for anyone curious about advanced AI risks and how to manage them responsibly.

SPONSOR MESSAGES:

***

CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments.

https://centml.ai/pricing/

Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. Are you interested in working on reasoning, or getting involved in their events?

They are hosting an event in Zurich on January 9th with the ARChitects, join if you can.

Goto https://tufalabs.ai/

***

Interviewer: Tim Scarfe

Yoshua Bengio:

https://x.com/Yoshua_Bengio

https://scholar.google.com/citations?user=kukA0LcAAAAJ&hl=en

https://yoshuabengio.org/

https://en.wikipedia.org/wiki/Yoshua_Bengio

TOC:

1. AI Safety Fundamentals

[00:00:00] 1.1 AI Safety Risks and International Cooperation

[00:03:20] 1.2 Fundamental Principles vs Scaling in AI Development

[00:11:25] 1.3 System 1/2 Thinking and AI Reasoning Capabilities

[00:15:15] 1.4 Reward Tampering and AI Agency Risks

[00:25:17] 1.5 Alignment Challenges and Instrumental Convergence

2. AI Architecture and Safety Design

[00:33:10] 2.1 Instrumental Goals and AI Safety Fundamentals

[00:35:02] 2.2 Separating Intelligence from Goals in AI Systems

[00:40:40] 2.3 Non-Agent AI as Scientific Tools

[00:44:25] 2.4 Oracle AI Systems and Mathematical Safety Frameworks

3. Global Governance and Security

[00:49:50] 3.1 International AI Competition and Hardware Governance

[00:51:58] 3.2 Military and Security Implications of AI Development

[00:56:07] 3.3 Personal Evolution of AI Safety Perspectives

[01:00:25] 3.4 AI Development Scaling and Global Governance Challenges

[01:12:10] 3.5 AI Regulation and Corporate Oversight

4. Technical Innovations

[01:23:00] 4.1 Evolution of Neural Architectures: From RNNs to Transformers

[01:26:02] 4.2 GFlowNets and Symbolic Computation

[01:30:47] 4.3 Neural Dynamics and Consciousness

[01:34:38] 4.4 AI Creativity and Scientific Discovery

SHOWNOTES (Transcript, references, best clips etc):

https://www.dropbox.com/scl/fi/ajucigli8n90fbxv9h94x/BENGIO_SHOW.pdf?rlkey=38hi2m19sylnr8orb76b85wkw&dl=0

CORE REFS (full list in shownotes and pinned comment):

[00:00:15] Bengio et al.: "AI Risk" Statement

https://www.safe.ai/work/statement-on-ai-risk

[00:23:10] Bengio on reward tampering & AI safety (Harvard Data Science Review)

https://hdsr.mitpress.mit.edu/pub/w974bwb0

[00:40:45] Munk Debate on AI existential risk, featuring Bengio

https://munkdebates.com/debates/artificial-intelligence

[00:44:30] "Can a Bayesian Oracle Prevent Harm from an Agent?" (Bengio et al.) on oracle-to-agent safety

https://arxiv.org/abs/2408.05284

[00:51:20] Bengio (2024) memo on hardware-based AI governance verification

https://yoshuabengio.org/wp-content/uploads/2024/08/FlexHEG-Memo_August-2024.pdf

[01:12:55] Bengio’s involvement in EU AI Act code of practice

https://digital-strategy.ec.europa.eu/en/news/meet-chairs-leading-development-first-general-purpose-ai-code-practice

[01:27:05] Complexity-based compositionality theory (Elmoznino, Jiralerspong, Bengio, Lajoie)

https://arxiv.org/abs/2410.14817

[01:29:00] GFlowNet Foundations (Bengio et al.) for probabilistic inference

https://arxiv.org/pdf/2111.09266

[01:32:10] Discrete attractor states in neural systems (Nam, Elmoznino, Bengio, Lajoie)

https://arxiv.org/pdf/2302.06403

François Chollet discusses the outcomes of the ARC-AGI (Abstraction and Reasoning Corpus) Prize competition in 2024, where accuracy rose from 33% to 55.5% on a private evaluation set.

SPONSOR MESSAGES:

***

CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments.

https://centml.ai/pricing/

Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. Are you interested in working on reasoning, or getting involved in their events?

They are hosting an event in Zurich on January 9th with the ARChitects, join if you can.

Goto https://tufalabs.ai/

***

Read about the recent result on o3 with ARC here (Chollet knew about it at the time of the interview but wasn't allowed to say):

https://arcprize.org/blog/oai-o3-pub-breakthrough

TOC:

1. Introduction and Opening

[00:00:00] 1.1 Deep Learning vs. Symbolic Reasoning: François’s Long-Standing Hybrid View

[00:00:48] 1.2 “Why Do They Call You a Symbolist?” – Addressing Misconceptions

[00:01:31] 1.3 Defining Reasoning

3. ARC Competition 2024 Results and Evolution

[00:07:26] 3.1 ARC Prize 2024: Reflecting on the Narrative Shift Toward System 2

[00:10:29] 3.2 Comparing Private Leaderboard vs. Public Leaderboard Solutions

[00:13:17] 3.3 Two Winning Approaches: Deep Learning–Guided Program Synthesis and Test-Time Training

4. Transduction vs. Induction in ARC

[00:16:04] 4.1 Test-Time Training, Overfitting Concerns, and Developer-Aware Generalization

[00:19:35] 4.2 Gradient Descent Adaptation vs. Discrete Program Search

5. ARC-2 Development and Future Directions

[00:23:51] 5.1 Ensemble Methods, Benchmark Flaws, and the Need for ARC-2

[00:25:35] 5.2 Human-Level Performance Metrics and Private Test Sets

[00:29:44] 5.3 Task Diversity, Redundancy Issues, and Expanded Evaluation Methodology

6. Program Synthesis Approaches

[00:30:18] 6.1 Induction vs. Transduction

[00:32:11] 6.2 Challenges of Writing Algorithms for Perceptual vs. Algorithmic Tasks

[00:34:23] 6.3 Combining Induction and Transduction

[00:37:05] 6.4 Multi-View Insight and Overfitting Regulation

7. Latent Space and Graph-Based Synthesis

[00:38:17] 7.1 Clément Bonnet’s Latent Program Search Approach

[00:40:10] 7.2 Decoding to Symbolic Form and Local Discrete Search

[00:41:15] 7.3 Graph of Operators vs. Token-by-Token Code Generation

[00:45:50] 7.4 Iterative Program Graph Modifications and Reusable Functions

8. Compute Efficiency and Lifelong Learning

[00:48:05] 8.1 Symbolic Process for Architecture Generation

[00:50:33] 8.2 Logarithmic Relationship of Compute and Accuracy

[00:52:20] 8.3 Learning New Building Blocks for Future Tasks

9. AI Reasoning and Future Development

[00:53:15] 9.1 Consciousness as a Self-Consistency Mechanism in Iterative Reasoning

[00:56:30] 9.2 Reconciling Symbolic and Connectionist Views

[01:00:13] 9.3 System 2 Reasoning - Awareness and Consistency

[01:03:05] 9.4 Novel Problem Solving, Abstraction, and Reusability

10. Program Synthesis and Research Lab

[01:05:53] 10.1 François Leaving Google to Focus on Program Synthesis

[01:09:55] 10.2 Democratizing Programming and Natural Language Instruction

11. Frontier Models and O1 Architecture

[01:14:38] 11.1 Search-Based Chain of Thought vs. Standard Forward Pass

[01:16:55] 11.2 o1’s Natural Language Program Generation and Test-Time Compute Scaling

[01:19:35] 11.3 Logarithmic Gains with Deeper Search

12. ARC Evaluation and Human Intelligence

[01:22:55] 12.1 LLMs as Guessing Machines and Agent Reliability Issues

[01:25:02] 12.2 ARC-2 Human Testing and Correlation with g-Factor

[01:26:16] 12.3 Closing Remarks and Future Directions

SHOWNOTES PDF:

https://www.dropbox.com/scl/fi/ujaai0ewpdnsosc5mc30k/CholletNeurips.pdf?rlkey=s68dp432vefpj2z0dp5wmzqz6&st=hazphyx5&dl=0

AI professor Jeff Clune ruminates on open-ended evolutionary algorithms—systems designed to generate novel and interesting outcomes forever. Drawing inspiration from nature’s boundless creativity, Clune and his collaborators aim to build “Darwin Complete” search spaces, where any computable environment can be simulated. By harnessing the power of large language models and reinforcement learning, these AI agents continuously develop new skills, explore uncharted domains, and even cooperate with one another in complex tasks.

SPONSOR MESSAGES:

***

CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments.

https://centml.ai/pricing/

Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on reasoning and AGI. Are you interested in working on reasoning, or getting involved in their events?

They are hosting an event in Zurich on January 9th with the ARChitects, join if you can.

Goto https://tufalabs.ai/

***

A central theme throughout Clune’s work is “interestingness”: an elusive quality that nudges AI agents toward genuinely original discoveries. Rather than rely on narrowly defined metrics—which often fail due to Goodhart’s Law—Clune employs language models to serve as proxies for human judgment. In doing so, he ensures that “interesting” always reflects authentic novelty, opening the door to unending innovation.

Yet with these extraordinary possibilities come equally significant risks. Clune says we need AI safety measures—particularly as the technology matures into powerful, open-ended forms. Potential pitfalls include agents inadvertently causing harm or malicious actors subverting AI’s capabilities for destructive ends. To mitigate this, Clune advocates for prudent governance involving democratic coalitions, regulation of cutting-edge models, and global alignment protocols.

Jeff Clune:

https://x.com/jeffclune

http://jeffclune.com/

(Interviewer: Tim Scarfe)

TOC:

1. Introduction

[00:00:00] 1.1 Overview and Opening Thoughts

2. Sponsorship

[00:03:00] 2.1 TufaAI Labs and CentML

3. Evolutionary AI Foundations

[00:04:12] 3.1 Open-Ended Algorithm Development and Abstraction Approaches

[00:07:56] 3.2 Novel Intelligence Forms and Serendipitous Discovery

[00:11:46] 3.3 Frontier Models and the 'Interestingness' Problem

[00:30:36] 3.4 Darwin Complete Systems and Evolutionary Search Spaces

4. System Architecture and Learning

[00:37:35] 4.1 Code Generation vs Neural Networks Comparison

[00:41:04] 4.2 Thought Cloning and Behavioral Learning Systems

[00:47:00] 4.3 Language Emergence in AI Systems

[00:50:23] 4.4 AI Interpretability and Safety Monitoring Techniques

5. AI Safety and Governance

[00:53:56] 5.1 Language Model Consistency and Belief Systems

[00:57:00] 5.2 AI Safety Challenges and Alignment Limitations

[01:02:07] 5.3 Open Source AI Development and Value Alignment

[01:08:19] 5.4 Global AI Governance and Development Control

6. Advanced AI Systems and Evolution

[01:16:55] 6.1 Agent Systems and Performance Evaluation

[01:22:45] 6.2 Continuous Learning Challenges and In-Context Solutions

[01:26:46] 6.3 Evolution Algorithms and Environment Generation

[01:35:36] 6.4 Evolutionary Biology Insights and Experiments

[01:48:08] 6.5 Personal Journey from Philosophy to AI Research

Shownotes:

We craft detailed show notes for each episode with high quality transcript and references and best parts bolded.

https://www.dropbox.com/scl/fi/fz43pdoc5wq5jh7vsnujl/JEFFCLUNE.pdf?rlkey=uu0e70ix9zo6g5xn6amykffpm&st=k2scxteu&dl=0

Neel Nanda, a senior research scientist at Google DeepMind, leads their mechanistic interpretability team. In this extensive interview, he discusses his work trying to understand how neural networks function internally. At just 25 years old, Nanda has quickly become a prominent voice in AI research after completing his pure mathematics degree at Cambridge in 2020.

Nanda reckons that machine learning is unique because we create neural networks that can perform impressive tasks (like complex reasoning and software engineering) without understanding how they work internally. He compares this to having computer programs that can do things no human programmer knows how to write. His work focuses on "mechanistic interpretability" - attempting to uncover and understand the internal structures and algorithms that emerge within these networks.

SPONSOR MESSAGES:

***

CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments.

https://centml.ai/pricing/

Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on ARC and AGI, they just acquired MindsAI - the current winners of the ARC challenge. Are you interested in working on ARC, or getting involved in their events? Goto https://tufalabs.ai/

***

SHOWNOTES, TRANSCRIPT, ALL REFERENCES (DONT MISS!):

https://www.dropbox.com/scl/fi/36dvtfl3v3p56hbi30im7/NeelShow.pdf?rlkey=pq8t7lyv2z60knlifyy17jdtx&st=kiutudhc&dl=0

We riff on:

* How neural networks develop meaningful internal representations beyond simple pattern matching

* The effectiveness of chain-of-thought prompting and why it improves model performance

* The importance of hands-on coding over extensive paper reading for new researchers

* His journey from Cambridge to working with Chris Olah at Anthropic and eventually Google DeepMind

* The role of mechanistic interpretability in AI safety

NEEL NANDA:

https://www.neelnanda.io/

https://scholar.google.com/citations?user=GLnX3MkAAAAJ&hl=en

https://x.com/NeelNanda5

Interviewer - Tim Scarfe

TOC:

1. Part 1: Introduction

[00:00:00] 1.1 Introduction and Core Concepts Overview

2. Part 2: Outside Interview

[00:06:45] 2.1 Mechanistic Interpretability Foundations

3. Part 3: Main Interview

[00:32:52] 3.1 Mechanistic Interpretability

4. Neural Architecture and Circuits

[01:00:31] 4.1 Biological Evolution Parallels

[01:04:03] 4.2 Universal Circuit Patterns and Induction Heads

[01:11:07] 4.3 Entity Detection and Knowledge Boundaries

[01:14:26] 4.4 Mechanistic Interpretability and Activation Patching

5. Model Behavior Analysis

[01:30:00] 5.1 Golden Gate Claude Experiment and Feature Amplification

[01:33:27] 5.2 Model Personas and RLHF Behavior Modification

[01:36:28] 5.3 Steering Vectors and Linear Representations

[01:40:00] 5.4 Hallucinations and Model Uncertainty

6. Sparse Autoencoder Architecture

[01:44:54] 6.1 Architecture and Mathematical Foundations

[02:22:03] 6.2 Core Challenges and Solutions

[02:32:04] 6.3 Advanced Activation Functions and Top-k Implementations

[02:34:41] 6.4 Research Applications in Transformer Circuit Analysis

7. Feature Learning and Scaling

[02:48:02] 7.1 Autoencoder Feature Learning and Width Parameters

[03:02:46] 7.2 Scaling Laws and Training Stability

[03:11:00] 7.3 Feature Identification and Bias Correction

[03:19:52] 7.4 Training Dynamics Analysis Methods

8. Engineering Implementation

[03:23:48] 8.1 Scale and Infrastructure Requirements

[03:25:20] 8.2 Computational Requirements and Storage

[03:35:22] 8.3 Chain-of-Thought Reasoning Implementation

[03:37:15] 8.4 Latent Structure Inference in Language Models

Jonas Hübotter, PhD student at ETH Zurich's Institute for Machine Learning, discusses his groundbreaking research on test-time computation and local learning. He demonstrates how smaller models can outperform larger ones by 30x through strategic test-time computation and introduces a novel paradigm combining inductive and transductive learning approaches.

Using Bayesian linear regression as a surrogate model for uncertainty estimation, Jonas explains how models can efficiently adapt to specific tasks without massive pre-training. He draws an analogy to Google Earth's variable resolution system to illustrate dynamic resource allocation based on task complexity.

The conversation explores the future of AI architecture, envisioning systems that continuously learn and adapt beyond current monolithic models. Jonas concludes by proposing hybrid deployment strategies combining local and cloud computation, suggesting a future where compute resources are allocated based on task complexity rather than fixed model size.

This research represents a significant shift in machine learning, prioritizing intelligent resource allocation and adaptive learning over traditional scaling approaches.

SPONSOR MESSAGES:

CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments.

https://centml.ai/pricing/

Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on ARC and AGI, they just acquired MindsAI - the current winners of the ARC challenge. Are you interested in working on ARC, or getting involved in their events? Goto https://tufalabs.ai/

Transcription, references and show notes PDF download:

https://www.dropbox.com/scl/fi/cxg80p388snwt6qbp4m52/JonasFinal.pdf?rlkey=glk9mhpzjvesanlc14rtpvk4r&st=6qwi8n3x&dl=0

Jonas Hübotter

https://jonhue.github.io/

https://scholar.google.com/citations?user=pxi_RkwAAAAJ

Transductive Active Learning: Theory and Applications (NeurIPS 2024)

https://arxiv.org/pdf/2402.15898

EFFICIENTLY LEARNING AT TEST-TIME: ACTIVE FINE-TUNING OF LLMS (SIFT)

https://arxiv.org/pdf/2410.08020

TOC:

1. Test-Time Computation Fundamentals

[00:00:00] Intro

[00:03:10] 1.1 Test-Time Computation and Model Performance Comparison

[00:05:52] 1.2 Retrieval Augmentation and Machine Teaching Strategies

[00:09:40] 1.3 In-Context Learning vs Fine-Tuning Trade-offs

2. System Architecture and Intelligence

[00:15:58] 2.1 System Architecture and Intelligence Emergence

[00:23:22] 2.2 Active Inference and Constrained Agency in AI

[00:29:52] 2.3 Evolution of Local Learning Methods

[00:32:05] 2.4 Vapnik's Contributions to Transductive Learning

3. Resource Optimization and Local Learning

[00:34:35] 3.1 Computational Resource Allocation in ML Models

[00:35:30] 3.2 Historical Context and Traditional ML Optimization

[00:37:55] 3.3 Variable Resolution Processing and Active Inference in ML

[00:43:01] 3.4 Local Learning and Base Model Capacity Trade-offs

[00:48:04] 3.5 Active Learning vs Local Learning Approaches

4. Information Retrieval and Model Interpretability

[00:51:08] 4.1 Information Retrieval and Nearest Neighbor Limitations

[01:03:07] 4.2 Model Interpretability and Surrogate Models

[01:15:03] 4.3 Bayesian Uncertainty Estimation and Surrogate Models

5. Distributed Systems and Deployment

[01:23:56] 5.1 Memory Architecture and Controller Systems

[01:28:14] 5.2 Evolution from Static to Distributed Learning Systems

[01:38:03] 5.3 Transductive Learning and Model Specialization

[01:41:58] 5.4 Hybrid Local-Cloud Deployment Strategies

Professor Swarat Chaudhuri from the University of Texas at Austin and visiting researcher at Google DeepMind discusses breakthroughs in AI reasoning, theorem proving, and mathematical discovery. Chaudhuri explains his groundbreaking work on COPRA (a GPT-based prover agent), shares insights on neurosymbolic approaches to AI.

Professor Swarat Chaudhuri:

https://www.cs.utexas.edu/~swarat/

SPONSOR MESSAGES:

CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments.

https://centml.ai/pricing/

Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on ARC and AGI, they just acquired MindsAI - the current winners of the ARC challenge. Are you interested in working on ARC, or getting involved in their events? Goto https://tufalabs.ai/

TOC:

[00:00:00] 0. Introduction / CentML ad, Tufa ad

1. AI Reasoning: From Language Models to Neurosymbolic Approaches

[00:02:27] 1.1 Defining Reasoning in AI

[00:09:51] 1.2 Limitations of Current Language Models

[00:17:22] 1.3 Neuro-symbolic Approaches and Program Synthesis

[00:24:59] 1.4 COPRA and In-Context Learning for Theorem Proving

[00:34:39] 1.5 Symbolic Regression and LLM-Guided Abstraction

2. AI in Mathematics: Theorem Proving and Concept Discovery

[00:43:37] 2.1 AI-Assisted Theorem Proving and Proof Verification

[01:01:37] 2.2 Symbolic Regression and Concept Discovery in Mathematics

[01:11:57] 2.3 Scaling and Modularizing Mathematical Proofs

[01:21:53] 2.4 COPRA: In-Context Learning for Formal Theorem-Proving

[01:28:22] 2.5 AI-driven theorem proving and mathematical discovery

3. Formal Methods and Challenges in AI Mathematics

[01:30:42] 3.1 Formal proofs, empirical predicates, and uncertainty in AI mathematics

[01:34:01] 3.2 Characteristics of good theoretical computer science research

[01:39:16] 3.3 LLMs in theorem generation and proving

[01:42:21] 3.4 Addressing contamination and concept learning in AI systems

REFS:

00:04:58 The Chinese Room Argument, https://plato.stanford.edu/entries/chinese-room/

00:11:42 Software 2.0, https://medium.com/@karpathy/software-2-0-a64152b37c35

00:11:57 Solving Olympiad Geometry Without Human Demonstrations, https://www.nature.com/articles/s41586-023-06747-5

00:13:26 Lean, https://lean-lang.org/

00:15:43 A General Reinforcement Learning Algorithm That Masters Chess, Shogi, and Go Through Self-Play, https://www.science.org/doi/10.1126/science.aar6404

00:19:24 DreamCoder (Ellis et al., PLDI 2021), https://arxiv.org/abs/2006.08381

00:24:37 The Lambda Calculus, https://plato.stanford.edu/entries/lambda-calculus/

00:26:43 Neural Sketch Learning for Conditional Program Generation, https://arxiv.org/pdf/1703.05698

00:28:08 Learning Differentiable Programs With Admissible Neural Heuristics, https://arxiv.org/abs/2007.12101

00:31:03 Symbolic Regression With a Learned Concept Library (Grayeli et al., NeurIPS 2024), https://arxiv.org/abs/2409.09359

00:41:30 Formal Verification of Parallel Programs, https://dl.acm.org/doi/10.1145/360248.360251

01:00:37 Training Compute-Optimal Large Language Models, https://arxiv.org/abs/2203.15556

01:18:19 Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, https://arxiv.org/abs/2201.11903

01:18:42 Draft, Sketch, and Prove: Guiding Formal Theorem Provers With Informal Proofs, https://arxiv.org/abs/2210.12283

01:19:49 Learning Formal Mathematics From Intrinsic Motivation, https://arxiv.org/pdf/2407.00695

01:20:19 An In-Context Learning Agent for Formal Theorem-Proving (Thakur et al., CoLM 2024), https://arxiv.org/pdf/2310.04353

01:23:58 Learning to Prove Theorems via Interacting With Proof Assistants, https://arxiv.org/abs/1905.09381

01:39:58 An In-Context Learning Agent for Formal Theorem-Proving (Thakur et al., CoLM 2024), https://arxiv.org/pdf/2310.04353

01:42:24 Programmatically Interpretable Reinforcement Learning (Verma et al., ICML 2018), https://arxiv.org/abs/1804.02477

Nora Belrose, Head of Interpretability Research at EleutherAI, discusses critical challenges in AI safety and development. The conversation begins with her technical work on concept erasure in neural networks through LEACE (LEAst-squares Concept Erasure), while highlighting how neural networks' progression from simple to complex learning patterns could have important implications for AI safety.

Many fear that advanced AI will pose an existential threat -- pursuing its own dangerous goals once it's powerful enough. But Belrose challenges this popular doomsday scenario with a fascinating breakdown of why it doesn't add up.

Belrose also provides a detailed critique of current AI alignment approaches, particularly examining "counting arguments" and their limitations when applied to AI safety. She argues that the Principle of Indifference may be insufficient for addressing existential risks from advanced AI systems. The discussion explores how emergent properties in complex AI systems could lead to unpredictable and potentially dangerous behaviors that simple reductionist approaches fail to capture.

The conversation concludes by exploring broader philosophical territory, where Belrose discusses her growing interest in Buddhism's potential relevance to a post-automation future. She connects concepts of moral anti-realism with Buddhist ideas about emptiness and non-attachment, suggesting these frameworks might help humans find meaning in a world where AI handles most practical tasks. Rather than viewing this automated future with alarm, she proposes that Zen Buddhism's emphasis on spontaneity and presence might complement a society freed from traditional labor.

SPONSOR MESSAGES:

CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments.

https://centml.ai/pricing/

Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on ARC and AGI, they just acquired MindsAI - the current winners of the ARC challenge. Are you interested in working on ARC, or getting involved in their events? Goto https://tufalabs.ai/

Nora Belrose:

https://norabelrose.com/

https://scholar.google.com/citations?user=p_oBc64AAAAJ&hl=en

https://x.com/norabelrose

SHOWNOTES:

https://www.dropbox.com/scl/fi/38fhsv2zh8gnubtjaoq4a/NORA_FINAL.pdf?rlkey=0e5r8rd261821g1em4dgv0k70&st=t5c9ckfb&dl=0

TOC:

1. Neural Network Foundations

[00:00:00] 1.1 Philosophical Foundations and Neural Network Simplicity Bias

[00:02:20] 1.2 LEACE and Concept Erasure Fundamentals

[00:13:16] 1.3 LISA Technical Implementation and Applications

[00:18:50] 1.4 Practical Implementation Challenges and Data Requirements

[00:22:13] 1.5 Performance Impact and Limitations of Concept Erasure

2. Machine Learning Theory

[00:32:23] 2.1 Neural Network Learning Progression and Simplicity Bias

[00:37:10] 2.2 Optimal Transport Theory and Image Statistics Manipulation

[00:43:05] 2.3 Grokking Phenomena and Training Dynamics

[00:44:50] 2.4 Texture vs Shape Bias in Computer Vision Models

[00:45:15] 2.5 CNN Architecture and Shape Recognition Limitations

3. AI Systems and Value Learning

[00:47:10] 3.1 Meaning, Value, and Consciousness in AI Systems

[00:53:06] 3.2 Global Connectivity vs Local Culture Preservation

[00:58:18] 3.3 AI Capabilities and Future Development Trajectory

4. Consciousness Theory

[01:03:03] 4.1 4E Cognition and Extended Mind Theory

[01:09:40] 4.2 Thompson's Views on Consciousness and Simulation

[01:12:46] 4.3 Phenomenology and Consciousness Theory

[01:15:43] 4.4 Critique of Illusionism and Embodied Experience

[01:23:16] 4.5 AI Alignment and Counting Arguments Debate

(TRUNCATED, TOC embedded in MP3 file with more information)

Prof. Gennady Pekhimenko (CEO of CentML, UofT) joins us in this *sponsored episode* to dive deep into AI system optimization and enterprise implementation. From NVIDIA's technical leadership model to the rise of open-source AI, Pekhimenko shares insights on bridging the gap between academic research and industrial applications. Learn about "dark silicon," GPU utilization challenges in ML workloads, and how modern enterprises can optimize their AI infrastructure. The conversation explores why some companies achieve only 10% GPU efficiency and practical solutions for improving AI system performance. A must-watch for anyone interested in the technical foundations of enterprise AI and hardware optimization.

CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments. Cheaper, faster, no commitments, pay as you go, scale massively, simple to setup. Check it out!

https://centml.ai/pricing/

SPONSOR MESSAGES:

MLST is also sponsored by Tufa AI Labs - https://tufalabs.ai/

They are hiring cracked ML engineers/researchers to work on ARC and build AGI!

SHOWNOTES (diarised transcript, TOC, references, summary, best quotes etc)

https://www.dropbox.com/scl/fi/w9kbpso7fawtm286kkp6j/Gennady.pdf?rlkey=aqjqmncx3kjnatk2il1gbgknk&st=2a9mccj8&dl=0

TOC:

1. AI Strategy and Leadership

[00:00:00] 1.1 Technical Leadership and Corporate Structure

[00:09:55] 1.2 Open Source vs Proprietary AI Models

[00:16:04] 1.3 Hardware and System Architecture Challenges

[00:23:37] 1.4 Enterprise AI Implementation and Optimization

[00:35:30] 1.5 AI Reasoning Capabilities and Limitations

2. AI System Development

[00:38:45] 2.1 Computational and Cognitive Limitations of AI Systems

[00:42:40] 2.2 Human-LLM Communication Adaptation and Patterns

[00:46:18] 2.3 AI-Assisted Software Development Challenges

[00:47:55] 2.4 Future of Software Engineering Careers in AI Era

[00:49:49] 2.5 Enterprise AI Adoption Challenges and Implementation

3. ML Infrastructure Optimization

[00:54:41] 3.1 MLOps Evolution and Platform Centralization

[00:55:43] 3.2 Hardware Optimization and Performance Constraints

[01:05:24] 3.3 ML Compiler Optimization and Python Performance

[01:15:57] 3.4 Enterprise ML Deployment and Cloud Provider Partnerships

4. Distributed AI Architecture

[01:27:05] 4.1 Multi-Cloud ML Infrastructure and Optimization

[01:29:45] 4.2 AI Agent Systems and Production Readiness

[01:32:00] 4.3 RAG Implementation and Fine-Tuning Considerations

[01:33:45] 4.4 Distributed AI Systems Architecture and Ray Framework

5. AI Industry Standards and Research

[01:37:55] 5.1 Origins and Evolution of MLPerf Benchmarking

[01:43:15] 5.2 MLPerf Methodology and Industry Impact

[01:50:17] 5.3 Academic Research vs Industry Implementation in AI

[01:58:59] 5.4 AI Research History and Safety Concerns

Eliezer Yudkowsky and Stephen Wolfram discuss artificial intelligence and its potential existen‑

tial risks. They traversed fundamental questions about AI safety, consciousness, computational irreducibility, and the nature of intelligence.

The discourse centered on Yudkowsky’s argument that advanced AI systems pose an existential threat to humanity, primarily due to the challenge of alignment and the potential for emergent goals that diverge from human values. Wolfram, while acknowledging potential risks, approached the topic from a his signature measured perspective, emphasizing the importance of understanding computational systems’ fundamental nature and questioning whether AI systems would necessarily develop the kind of goal‑directed behavior Yudkowsky fears.

***

MLST IS SPONSORED BY TUFA AI LABS!

The current winners of the ARC challenge, MindsAI are part of Tufa AI Labs. They are hiring ML engineers. Are you interested?! Please goto https://tufalabs.ai/

***

TOC:

1. Foundational AI Concepts and Risks

[00:00:01] 1.1 AI Optimization and System Capabilities Debate

[00:06:46] 1.2 Computational Irreducibility and Intelligence Limitations

[00:20:09] 1.3 Existential Risk and Species Succession

[00:23:28] 1.4 Consciousness and Value Preservation in AI Systems

2. Ethics and Philosophy in AI

[00:33:24] 2.1 Moral Value of Human Consciousness vs. Computation

[00:36:30] 2.2 Ethics and Moral Philosophy Debate

[00:39:58] 2.3 Existential Risks and Digital Immortality

[00:43:30] 2.4 Consciousness and Personal Identity in Brain Emulation

3. Truth and Logic in AI Systems

[00:54:39] 3.1 AI Persuasion Ethics and Truth

[01:01:48] 3.2 Mathematical Truth and Logic in AI Systems

[01:11:29] 3.3 Universal Truth vs Personal Interpretation in Ethics and Mathematics

[01:14:43] 3.4 Quantum Mechanics and Fundamental Reality Debate

4. AI Capabilities and Constraints

[01:21:21] 4.1 AI Perception and Physical Laws

[01:28:33] 4.2 AI Capabilities and Computational Constraints

[01:34:59] 4.3 AI Motivation and Anthropomorphization Debate

[01:38:09] 4.4 Prediction vs Agency in AI Systems

5. AI System Architecture and Behavior

[01:44:47] 5.1 Computational Irreducibility and Probabilistic Prediction

[01:48:10] 5.2 Teleological vs Mechanistic Explanations of AI Behavior

[02:09:41] 5.3 Machine Learning as Assembly of Computational Components

[02:29:52] 5.4 AI Safety and Predictability in Complex Systems

6. Goal Optimization and Alignment

[02:50:30] 6.1 Goal Specification and Optimization Challenges in AI Systems

[02:58:31] 6.2 Intelligence, Computation, and Goal-Directed Behavior

[03:02:18] 6.3 Optimization Goals and Human Existential Risk

[03:08:49] 6.4 Emergent Goals and AI Alignment Challenges

7. AI Evolution and Risk Assessment

[03:19:44] 7.1 Inner Optimization and Mesa-Optimization Theory

[03:34:00] 7.2 Dynamic AI Goals and Extinction Risk Debate

[03:56:05] 7.3 AI Risk and Biological System Analogies

[04:09:37] 7.4 Expert Risk Assessments and Optimism vs Reality

8. Future Implications and Economics

[04:13:01] 8.1 Economic and Proliferation Considerations

SHOWNOTES (transcription, references, summary, best quotes etc):

https://www.dropbox.com/scl/fi/3st8dts2ba7yob161dchd/EliezerWolfram.pdf?rlkey=b6va5j8upgqwl9s2muc924vtt&st=vemwqx7a&dl=0

Francois Chollet, a prominent AI expert and creator of ARC-AGI, discusses intelligence, consciousness, and artificial intelligence.

Chollet explains that real intelligence isn't about memorizing information or having lots of knowledge - it's about being able to handle new situations effectively. This is why he believes current large language models (LLMs) have "near-zero intelligence" despite their impressive abilities. They're more like sophisticated memory and pattern-matching systems than truly intelligent beings.

***

MLST IS SPONSORED BY TUFA AI LABS!

The current winners of the ARC challenge, MindsAI are part of Tufa AI Labs. They are hiring ML engineers. Are you interested?! Please goto https://tufalabs.ai/

***

He introduced his "Kaleidoscope Hypothesis," which suggests that while the world seems infinitely complex, it's actually made up of simpler patterns that repeat and combine in different ways. True intelligence, he argues, involves identifying these basic patterns and using them to understand new situations.

Chollet also talked about consciousness, suggesting it develops gradually in children rather than appearing all at once. He believes consciousness exists in degrees - animals have it to some extent, and even human consciousness varies with age and circumstances (like being more conscious when learning something new versus doing routine tasks).

On AI safety, Chollet takes a notably different stance from many in Silicon Valley. He views AGI development as a scientific challenge rather than a religious quest, and doesn't share the apocalyptic concerns of some AI researchers. He argues that intelligence itself isn't dangerous - it's just a tool for turning information into useful models. What matters is how we choose to use it.

ARC-AGI Prize:

https://arcprize.org/

Francois Chollet:

https://x.com/fchollet

Shownotes:

https://www.dropbox.com/scl/fi/j2068j3hlj8br96pfa7bi/CHOLLET_FINAL.pdf?rlkey=xkbr7tbnrjdl66m246w26uc8k&st=0a4ec4na&dl=0

TOC:

1. Intelligence and Model Building

[00:00:00] 1.1 Intelligence Definition and ARC Benchmark

[00:05:40] 1.2 LLMs as Program Memorization Systems

[00:09:36] 1.3 Kaleidoscope Hypothesis and Abstract Building Blocks

[00:13:39] 1.4 Deep Learning Limitations and System 2 Reasoning

[00:29:38] 1.5 Intelligence vs. Skill in LLMs and Model Building

2. ARC Benchmark and Program Synthesis

[00:37:36] 2.1 Intelligence Definition and LLM Limitations

[00:41:33] 2.2 Meta-Learning System Architecture

[00:56:21] 2.3 Program Search and Occam's Razor

[00:59:42] 2.4 Developer-Aware Generalization

[01:06:49] 2.5 Task Generation and Benchmark Design

3. Cognitive Systems and Program Generation

[01:14:38] 3.1 System 1/2 Thinking Fundamentals

[01:22:17] 3.2 Program Synthesis and Combinatorial Challenges

[01:31:18] 3.3 Test-Time Fine-Tuning Strategies

[01:36:10] 3.4 Evaluation and Leakage Problems

[01:43:22] 3.5 ARC Implementation Approaches

4. Intelligence and Language Systems

[01:50:06] 4.1 Intelligence as Tool vs Agent

[01:53:53] 4.2 Cultural Knowledge Integration

[01:58:42] 4.3 Language and Abstraction Generation

[02:02:41] 4.4 Embodiment in Cognitive Systems

[02:09:02] 4.5 Language as Cognitive Operating System

5. Consciousness and AI Safety

[02:14:05] 5.1 Consciousness and Intelligence Relationship

[02:20:25] 5.2 Development of Machine Consciousness

[02:28:40] 5.3 Consciousness Prerequisites and Indicators

[02:36:36] 5.4 AGI Safety Considerations

[02:40:29] 5.5 AI Regulation Framework

Anil Ananthaswamy is an award-winning science writer and former staff writer and deputy news editor for the London-based New Scientist magazine.

Machine learning systems are making life-altering decisions for us: approving mortgage loans, determining whether a tumor is cancerous, or deciding if someone gets bail. They now influence developments and discoveries in chemistry, biology, and physics—the study of genomes, extrasolar planets, even the intricacies of quantum systems. And all this before large language models such as ChatGPT came on the scene.

We are living through a revolution in machine learning-powered AI that shows no signs of slowing down. This technology is based on relatively simple mathematical ideas, some of which go back centuries, including linear algebra and calculus, the stuff of seventeenth- and eighteenth-century mathematics. It took the birth and advancement of computer science and the kindling of 1990s computer chips designed for video games to ignite the explosion of AI that we see today. In this enlightening book, Anil Ananthaswamy explains the fundamental math behind machine learning, while suggesting intriguing links between artificial and natural intelligence. Might the same math underpin them both?

As Ananthaswamy resonantly concludes, to make safe and effective use of artificial intelligence, we need to understand its profound capabilities and limitations, the clues to which lie in the math that makes machine learning possible.

Why Machines Learn: The Elegant Math Behind Modern AI:

https://amzn.to/3UAWX3D

https://anilananthaswamy.com/

Sponsor message:

DO YOU WANT WORK ON ARC with the MindsAI team (current ARC winners)?

Interested? Apply for an ML research position: benjamin@tufa.ai

Shownotes:

https://www.dropbox.com/scl/fi/wpv22m5jxyiqr6pqfkzwz/anil.pdf?rlkey=9c233jo5armr548ctwo419n6p&st=xzhahtje&dl=0

Chapters:

1. ML Fundamentals and Prerequisites

[00:00:00] 1.1 Differences Between Human and Machine Learning

[00:00:35] 1.2 Mathematical Prerequisites and Societal Impact of ML

[00:02:20] 1.3 Author's Journey and Book Background

[00:11:30] 1.4 Mathematical Foundations and Core ML Concepts

[00:21:45] 1.5 Bias-Variance Tradeoff and Modern Deep Learning

2. Deep Learning Architecture

[00:29:05] 2.1 Double Descent and Overparameterization in Deep Learning

[00:32:40] 2.2 Mathematical Foundations and Self-Supervised Learning

[00:40:05] 2.3 High-Dimensional Spaces and Model Architecture

[00:52:55] 2.4 Historical Development of Backpropagation

3. AI Understanding and Limitations

[00:59:13] 3.1 Pattern Matching vs Human Reasoning in ML Models

[01:00:20] 3.2 Mathematical Foundations and Pattern Recognition in AI

[01:04:08] 3.3 LLM Reliability and Machine Understanding Debate

[01:12:50] 3.4 Historical Development of Deep Learning Technologies

[01:15:21] 3.5 Alternative AI Approaches and Bio-inspired Methods

4. Ethical and Neurological Perspectives

[01:24:32] 4.1 Neural Network Scaling and Mathematical Limitations

[01:31:12] 4.2 AI Ethics and Societal Impact

[01:38:30] 4.3 Consciousness and Neurological Conditions

[01:46:17] 4.4 Body Ownership and Agency in Neuroscience

Professor Michael Levin explores the revolutionary concept of diverse intelligence, demonstrating how cognitive capabilities extend far beyond traditional brain-based intelligence. Drawing from his groundbreaking research, he explains how even simple biological systems like gene regulatory networks exhibit learning, memory, and problem-solving abilities. Levin introduces key concepts like "cognitive light cones" - the scope of goals a system can pursue - and shows how these ideas are transforming our approach to cancer treatment and biological engineering. His insights challenge conventional views of intelligence and agency, with profound implications for both medicine and artificial intelligence development. This deep discussion reveals how understanding intelligence as a spectrum, from molecular networks to human minds, could be crucial for humanity's future technological development. Contains technical discussion of biological systems, cybernetics, and theoretical frameworks for understanding emergent cognition.

Prof. Michael Levin

https://as.tufts.edu/biology/people/faculty/michael-levin

https://x.com/drmichaellevin

Sponsor message:

DO YOU WANT WORK ON ARC with the MindsAI team (current ARC winners)?

Interested? Apply for an ML research position: benjamin@tufa.ai

TOC

1. Intelligence Fundamentals and Evolution

[00:00:00] 1.1 Future Evolution of Human Intelligence and Consciousness

[00:03:00] 1.2 Science Fiction's Role in Exploring Intelligence Possibilities

[00:08:15] 1.3 Essential Characteristics of Human-Level Intelligence and Relationships

[00:14:20] 1.4 Biological Systems Architecture and Intelligence

2. Biological Computing and Cognition

[00:24:00] 2.1 Agency and Intelligence in Biological Systems

[00:30:30] 2.2 Learning Capabilities in Gene Regulatory Networks

[00:35:37] 2.3 Biological Control Systems and Competency Architecture

[00:39:58] 2.4 Scientific Metaphors and Polycomputing Paradigm

3. Systems and Collective Intelligence

[00:43:26] 3.1 Embodiment and Problem-Solving Spaces

[00:44:50] 3.2 Perception-Action Loops and Biological Intelligence

[00:46:55] 3.3 Intelligence, Wisdom and Collective Systems

[00:53:07] 3.4 Cancer and Cognitive Light Cones

[00:57:09] 3.5 Emergent Intelligence and AI Agency

Shownotes:

https://www.dropbox.com/scl/fi/i2vl1vs009thg54lxx5wc/LEVIN.pdf?rlkey=dtk8okhbsejryiu2vrht19qp6&st=uzi0vo45&dl=0

REFS:

[0:05:30] A Fire Upon the Deep - Vernor Vinge sci-fi novel on AI and consciousness

[0:05:35] Maria Chudnovsky - MacArthur Fellow, Princeton mathematician, graph theory expert

[0:14:20] Bow-tie architecture in biological systems - Network structure research by Csete & Doyle

[0:15:40] Richard Watson - Southampton Professor, evolution and learning systems expert

[0:17:00] Levin paper on human issues in AI and evolution

[0:19:00] Bow-tie architecture in Darwin's agential materialism - Levin

[0:22:55] Philip Goff - Work on panpsychism and consciousness in Galileo's Error

[0:23:30] Strange Loop - Hofstadter's work on self-reference and consciousness

[0:25:00] The Hard Problem of Consciousness - Van Gulick

[0:26:15] Daniel Dennett - Theories on consciousness and intentional systems

[0:29:35] Principle of Least Action - Light path selection in physics

[0:29:50] Free Energy Principle - Friston's unified behavioral framework

[0:30:35] Gene regulatory networks - Learning capabilities in biological systems

[0:36:55] Minimal networks with learning capacity - Levin

[0:38:50] Multi-scale competency in biological systems - Levin

[0:41:40] Polycomputing paradigm - Biological computation by Bongard & Levin

[0:45:40] Collective intelligence in biology - Levin et al.

[0:46:55] Niche construction and stigmergy - Torday

[0:53:50] Tasmanian Devil Facial Tumor Disease - Transmissible cancer research

[0:55:05] Cognitive light cone - Computational boundaries of self - Levin

[0:58:05] Cognitive properties in sorting algorithms - Zhang, Goldstein & Levin

Will Williams is CTO of Speechmatics in Cambridge. In this sponsored episode - he shares deep technical insights into modern speech recognition technology and system architecture. The episode covers several key technical areas:

* Speechmatics' hybrid approach to ASR, which focusses on unsupervised learning methods, achieving comparable results with 100x less data than fully supervised approaches. Williams explains why this is more efficient and generalizable than end-to-end models like Whisper.

* Their production architecture implementing multiple operating points for different latency-accuracy trade-offs, with careful latency padding (up to 1.8 seconds) to ensure consistent user experience. The system uses lattice-based decoding with language model integration for improved accuracy.

* The challenges and solutions in real-time ASR, including their approach to diarization (speaker identification), handling cross-talk, and implicit source separation. Williams explains why these problems remain difficult even with modern deep learning approaches.

* Their testing and deployment infrastructure, including the use of mirrored environments for catching edge cases in production, and their strategy of maintaining global models rather than allowing customer-specific fine-tuning.

* Technical evolution in ASR, from early days of custom CUDA kernels and manual memory management to modern frameworks, with Williams offering interesting critiques of current PyTorch memory management approaches and arguing for more efficient direct memory allocation in production systems.

Get coding with their API! This is their URL:

https://www.speechmatics.com/

DO YOU WANT WORK ON ARC with the MindsAI team (current ARC winners)?

MLST is sponsored by Tufa Labs:

Focus: ARC, LLMs, test-time-compute, active inference, system2 reasoning, and more.

Interested? Apply for an ML research position: benjamin@tufa.ai

TOC

1. ASR Core Technology & Real-time Architecture

[00:00:00] 1.1 ASR and Diarization Fundamentals

[00:05:25] 1.2 Real-time Conversational AI Architecture

[00:09:21] 1.3 Neural Network Streaming Implementation

[00:12:49] 1.4 Multi-modal System Integration

2. Production System Optimization

[00:29:38] 2.1 Production Deployment and Testing Infrastructure

[00:35:40] 2.2 Model Architecture and Deployment Strategy

[00:37:12] 2.3 Latency-Accuracy Trade-offs

[00:39:15] 2.4 Language Model Integration

[00:40:32] 2.5 Lattice-based Decoding Architecture

3. Performance Evaluation & Ethical Considerations

[00:44:00] 3.1 ASR Performance Metrics and Capabilities

[00:46:35] 3.2 AI Regulation and Evaluation Methods

[00:51:09] 3.3 Benchmark and Testing Challenges

[00:54:30] 3.4 Real-world Implementation Metrics

[01:00:51] 3.5 Ethics and Privacy Considerations

4. ASR Technical Evolution

[01:09:00] 4.1 WER Calculation and Evaluation Methodologies

[01:10:21] 4.2 Supervised vs Self-Supervised Learning Approaches

[01:21:02] 4.3 Temporal Learning and Feature Processing

[01:24:45] 4.4 Feature Engineering to Automated ML

5. Enterprise Implementation & Scale

[01:27:55] 5.1 Future AI Systems and Adaptation

[01:31:52] 5.2 Technical Foundations and History

[01:34:53] 5.3 Infrastructure and Team Scaling

[01:38:05] 5.4 Research and Talent Strategy

[01:41:11] 5.5 Engineering Practice Evolution

Shownotes:

https://www.dropbox.com/scl/fi/d94b1jcgph9o8au8shdym/Speechmatics.pdf?rlkey=bi55wvktzomzx0y5sic6jz99y&st=6qwofv8t&dl=0

Dr. Sanjeev Namjoshi, a machine learning engineer who recently submitted a book on Active Inference to MIT Press, discusses the theoretical foundations and practical applications of Active Inference, the Free Energy Principle (FEP), and Bayesian mechanics. He explains how these frameworks describe how biological and artificial systems maintain stability by minimizing uncertainty about their environment.

DO YOU WANT WORK ON ARC with the MindsAI team (current ARC winners)?

MLST is sponsored by Tufa Labs:

Focus: ARC, LLMs, test-time-compute, active inference, system2 reasoning, and more.

Future plans: Expanding to complex environments like Warcraft 2 and Starcraft 2.

Interested? Apply for an ML research position: benjamin@tufa.ai

Namjoshi traces the evolution of these fields from early 2000s neuroscience research to current developments, highlighting how Active Inference provides a unified framework for perception and action through variational free energy minimization. He contrasts this with traditional machine learning approaches, emphasizing Active Inference's natural capacity for exploration and curiosity through epistemic value.

He sees Active Inference as being at a similar stage to deep learning in the early 2000s - poised for significant breakthroughs but requiring better tools and wider adoption. While acknowledging current computational challenges, he emphasizes Active Inference's potential advantages over reinforcement learning, particularly its principled approach to exploration and planning.

Dr. Sanjeev Namjoshi

https://snamjoshi.github.io/

TOC:

1. Theoretical Foundations: AI Agency and Sentience

[00:00:00] 1.1 Intro

[00:02:45] 1.2 Free Energy Principle and Active Inference Theory

[00:11:16] 1.3 Emergence and Self-Organization in Complex Systems

[00:19:11] 1.4 Agency and Representation in AI Systems

[00:29:59] 1.5 Bayesian Mechanics and Systems Modeling

2. Technical Framework: Active Inference and Free Energy

[00:38:37] 2.1 Generative Processes and Agent-Environment Modeling

[00:42:27] 2.2 Markov Blankets and System Boundaries

[00:44:30] 2.3 Bayesian Inference and Prior Distributions

[00:52:41] 2.4 Variational Free Energy Minimization Framework

[00:55:07] 2.5 VFE Optimization Techniques: Generalized Filtering vs DEM

3. Implementation and Optimization Methods

[00:58:25] 3.1 Information Theory and Free Energy Concepts

[01:05:25] 3.2 Surprise Minimization and Action in Active Inference

[01:15:58] 3.3 Evolution of Active Inference Models: Continuous to Discrete Approaches

[01:26:00] 3.4 Uncertainty Reduction and Control Systems in Active Inference

4. Safety and Regulatory Frameworks

[01:32:40] 4.1 Historical Evolution of Risk Management and Predictive Systems

[01:36:12] 4.2 Agency and Reality: Philosophical Perspectives on Models

[01:39:20] 4.3 Limitations of Symbolic AI and Current System Design

[01:46:40] 4.4 AI Safety Regulation and Corporate Governance

5. Socioeconomic Integration and Modeling

[01:52:55] 5.1 Economic Policy and Public Sentiment Modeling

[01:55:21] 5.2 Free Energy Principle: Libertarian vs Collectivist Perspectives

[01:58:53] 5.3 Regulation of Complex Socio-Technical Systems

[02:03:04] 5.4 Evolution and Current State of Active Inference Research

6. Future Directions and Applications

[02:14:26] 6.1 Active Inference Applications and Future Development

[02:22:58] 6.2 Cultural Learning and Active Inference

[02:29:19] 6.3 Hierarchical Relationship Between FEP, Active Inference, and Bayesian Mechanics

[02:33:22] 6.4 Historical Evolution of Free Energy Principle

[02:38:52] 6.5 Active Inference vs Traditional Machine Learning Approaches

Transcript and shownotes with refs and URLs:

https://www.dropbox.com/scl/fi/qj22a660cob1795ej0gbw/SanjeevShow.pdf?rlkey=w323r3e8zfsnve22caayzb17k&st=el1fdgfr&dl=0

Dr. Joscha Bach discusses advanced AI, consciousness, and cognitive modeling. He presents consciousness as a virtual property emerging from self-organizing software patterns, challenging panpsychism and materialism. Bach introduces "Cyberanima," reinterpreting animism through information processing, viewing spirits as self-organizing software agents.

He addresses limitations of current large language models and advocates for smaller, more efficient AI models capable of reasoning from first principles. Bach describes his work with Liquid AI on novel neural network architectures for improved expressiveness and efficiency.

The interview covers AI's societal implications, including regulation challenges and impact on innovation. Bach argues for balancing oversight with technological progress, warning against overly restrictive regulations.

Throughout, Bach frames consciousness, intelligence, and agency as emergent properties of complex information processing systems, proposing a computational framework for cognitive phenomena and reality.

SPONSOR MESSAGE:

DO YOU WANT WORK ON ARC with the MindsAI team (current ARC winners)? MLST is sponsored by Tufa Labs: Focus: ARC, LLMs, test-time-compute, active inference, system2 reasoning, and more. Future plans: Expanding to complex environments like Warcraft 2 and Starcraft 2. Interested? Apply for an ML research position: benjamin@tufa.ai

TOC

[00:00:00] 1.1 Consciousness and Intelligence in AI Development

[00:07:44] 1.2 Agency, Intelligence, and Their Relationship to Physical Reality

[00:13:36] 1.3 Virtual Patterns and Causal Structures in Consciousness

[00:25:49] 1.4 Reinterpreting Concepts of God and Animism in Information Processing Terms

[00:32:50] 1.5 Animism and Evolution as Competition Between Software Agents

2. Self-Organizing Systems and Cognitive Models in AI

[00:37:59] 2.1 Consciousness as self-organizing software

[00:45:49] 2.2 Critique of panpsychism and alternative views on consciousness

[00:50:48] 2.3 Emergence of consciousness in complex systems

[00:52:50] 2.4 Neuronal motivation and the origins of consciousness

[00:56:47] 2.5 Coherence and Self-Organization in AI Systems

3. Advanced AI Architectures and Cognitive Processes

[00:57:50] 3.1 Second-Order Software and Complex Mental Processes

[01:01:05] 3.2 Collective Agency and Shared Values in AI

[01:05:40] 3.3 Limitations of Current AI Agents and LLMs

[01:06:40] 3.4 Liquid AI and Novel Neural Network Architectures

[01:10:06] 3.5 AI Model Efficiency and Future Directions

[01:19:00] 3.6 LLM Limitations and Internal State Representation

4. AI Regulation and Societal Impact

[01:31:23] 4.1 AI Regulation and Societal Impact

[01:49:50] 4.2 Open-Source AI and Industry Challenges

Refs in shownotes and MP3 metadata

Shownotes:

https://www.dropbox.com/scl/fi/g28dosz19bzcfs5imrvbu/JoschaInterview.pdf?rlkey=s3y18jy192ktz6ogd7qtvry3d&st=10z7q7w9&dl=0

Alessandro Palmarini is a post-baccalaureate researcher at the Santa Fe Institute working under the supervision of Melanie Mitchell. He completed his undergraduate degree in Artificial Intelligence and Computer Science at the University of Edinburgh. Palmarini's current research focuses on developing AI systems that can efficiently acquire new skills from limited data, inspired by François Chollet's work on measuring intelligence. His work builds upon the DreamCoder program synthesis system, introducing a novel approach called "dream decompiling" to improve library learning in inductive program synthesis. Palmarini is particularly interested in addressing the Abstraction and Reasoning Corpus (ARC) challenge, aiming to create AI systems that can perform abstract reasoning tasks more efficiently than current approaches. His research explores the balance between computational efficiency and data efficiency in AI learning processes.

DO YOU WANT WORK ON ARC with the MindsAI team (current ARC winners)? MLST is sponsored by Tufa Labs: Focus: ARC, LLMs, test-time-compute, active inference, system2 reasoning, and more. Future plans: Expanding to complex environments like Warcraft 2 and Starcraft 2. Interested? Apply for an ML research position: benjamin@tufa.ai

TOC:

1. Intelligence Measurement in AI Systems

[00:00:00] 1.1 Defining Intelligence in AI Systems

[00:02:00] 1.2 Research at Santa Fe Institute

[00:04:35] 1.3 Impact of Gaming on AI Development

[00:05:10] 1.4 Comparing AI and Human Learning Efficiency

2. Efficient Skill Acquisition in AI

[00:06:40] 2.1 Intelligence as Skill Acquisition Efficiency

[00:08:25] 2.2 Limitations of Current AI Systems in Generalization

[00:09:45] 2.3 Human vs. AI Cognitive Processes

[00:10:40] 2.4 Measuring AI Intelligence: Chollet's ARC Challenge

3. Program Synthesis and ARC Challenge

[00:12:55] 3.1 Philosophical Foundations of Program Synthesis

[00:17:14] 3.2 Introduction to Program Induction and ARC Tasks

[00:18:49] 3.3 DreamCoder: Principles and Techniques

[00:27:55] 3.4 Trade-offs in Program Synthesis Search Strategies

[00:31:52] 3.5 Neural Networks and Bayesian Program Learning

4. Advanced Program Synthesis Techniques

[00:32:30] 4.1 DreamCoder and Dream Decompiling Approach

[00:39:00] 4.2 Beta Distribution and Caching in Program Synthesis

[00:45:10] 4.3 Performance and Limitations of Dream Decompiling

[00:47:45] 4.4 Alessandro's Approach to ARC Challenge

[00:51:12] 4.5 Conclusion and Future Discussions

Refs:

Full reflist on YT VD, Show Notes and MP3 metadata

Show Notes: https://www.dropbox.com/scl/fi/x50201tgqucj5ba2q4typ/Ale.pdf?rlkey=0ubvk7p5gtyx1gpownpdadim8&st=5pniu3nq&dl=0

François Chollet discusses the limitations of Large Language Models (LLMs) and proposes a new approach to advancing artificial intelligence. He argues that current AI systems excel at pattern recognition but struggle with logical reasoning and true generalization.

This was Chollet's keynote talk at AGI-24, filmed in high-quality. We will be releasing a full interview with him shortly. A teaser clip from that is played in the intro!

Chollet introduces the Abstraction and Reasoning Corpus (ARC) as a benchmark for measuring AI progress towards human-like intelligence. He explains the concept of abstraction in AI systems and proposes combining deep learning with program synthesis to overcome current limitations. Chollet suggests that breakthroughs in AI might come from outside major tech labs and encourages researchers to explore new ideas in the pursuit of artificial general intelligence.

TOC

1. LLM Limitations and Intelligence Concepts

[00:00:00] 1.1 LLM Limitations and Composition

[00:12:05] 1.2 Intelligence as Process vs. Skill

[00:17:15] 1.3 Generalization as Key to AI Progress

2. ARC-AGI Benchmark and LLM Performance

[00:19:59] 2.1 Introduction to ARC-AGI Benchmark

[00:20:05] 2.2 Introduction to ARC-AGI and the ARC Prize

[00:23:35] 2.3 Performance of LLMs and Humans on ARC-AGI

3. Abstraction in AI Systems

[00:26:10] 3.1 The Kaleidoscope Hypothesis and Abstraction Spectrum

[00:30:05] 3.2 LLM Capabilities and Limitations in Abstraction

[00:32:10] 3.3 Value-Centric vs Program-Centric Abstraction

[00:33:25] 3.4 Types of Abstraction in AI Systems

4. Advancing AI: Combining Deep Learning and Program Synthesis

[00:34:05] 4.1 Limitations of Transformers and Need for Program Synthesis

[00:36:45] 4.2 Combining Deep Learning and Program Synthesis

[00:39:59] 4.3 Applying Combined Approaches to ARC Tasks

[00:44:20] 4.4 State-of-the-Art Solutions for ARC

Shownotes (new!): https://www.dropbox.com/scl/fi/i7nsyoahuei6np95lbjxw/CholletKeynote.pdf?rlkey=t3502kbov5exsdxhderq70b9i&st=1ca91ewz&dl=0

[0:01:15] Abstraction and Reasoning Corpus (ARC): AI benchmark (François Chollet)

https://arxiv.org/abs/1911.01547

[0:05:30] Monty Hall problem: Probability puzzle (Steve Selvin)

https://www.tandfonline.com/doi/abs/10.1080/00031305.1975.10479121

[0:06:20] LLM training dynamics analysis (Tirumala et al.)

https://arxiv.org/abs/2205.10770

[0:10:20] Transformer limitations on compositionality (Dziri et al.)

https://arxiv.org/abs/2305.18654

[0:10:25] Reversal Curse in LLMs (Berglund et al.)

https://arxiv.org/abs/2309.12288

[0:19:25] Measure of intelligence using algorithmic information theory (François Chollet)

https://arxiv.org/abs/1911.01547

[0:20:10] ARC-AGI: GitHub repository (François Chollet)

https://github.com/fchollet/ARC-AGI

[0:22:15] ARC Prize: $1,000,000+ competition (François Chollet)

https://arcprize.org/

[0:33:30] System 1 and System 2 thinking (Daniel Kahneman)

https://www.amazon.com/Thinking-Fast-Slow-Daniel-Kahneman/dp/0374533555

[0:34:00] Core knowledge in infants (Elizabeth Spelke)

https://www.harvardlds.org/wp-content/uploads/2017/01/SpelkeKinzler07-1.pdf

[0:34:30] Embedding interpretive spaces in ML (Tennenholtz et al.)

https://arxiv.org/abs/2310.04475

[0:44:20] Hypothesis Search with LLMs for ARC (Wang et al.)

https://arxiv.org/abs/2309.05660

[0:44:50] Ryan Greenblatt's high score on ARC public leaderboard

https://arcprize.org/

Ivan Zhang, co-founder of Cohere, discusses the company's enterprise-focused AI solutions. He explains Cohere's early emphasis on embedding technology and training models for secure environments.

Zhang highlights their implementation of Retrieval-Augmented Generation in healthcare, significantly reducing doctor preparation time. He explores the shift from monolithic AI models to heterogeneous systems and the importance of improving various AI system components. Zhang shares insights on using synthetic data to teach models reasoning, the democratization of software development through AI, and how his gaming skills transfer to running an AI company.

He advises young developers to fully embrace AI technologies and offers perspectives on AI reliability, potential risks, and future model architectures.

https://cohere.com/

https://ivanzhang.ca/

https://x.com/1vnzh

TOC:

00:00:00 Intro

00:03:20 AI & Language Model Evolution

00:06:09 Future AI Apps & Development

00:09:29 Impact on Software Dev Practices

00:13:03 Philosophical & Societal Implications

00:16:30 Compute Efficiency & RAG

00:20:39 Adoption Challenges & Solutions

00:22:30 GPU Optimization & Kubernetes Limits

00:24:16 Cohere's Implementation Approach

00:28:13 Gaming's Professional Influence

00:34:45 Transformer Optimizations

00:36:45 Future Models & System-Level Focus

00:39:20 Inference-Time Computation & Reasoning

00:42:05 Capturing Human Thought in AI

00:43:15 Research, Hiring & Developer Advice

REFS:

00:02:31 Cohere, https://cohere.com/

00:02:40 The Transformer architecture, https://arxiv.org/abs/1706.03762

00:03:22 The Innovator's Dilemma, https://www.amazon.com/Innovators-Dilemma-Technologies-Management-Innovation/dp/1633691780

00:09:15 The actor model, https://en.wikipedia.org/wiki/Actor_model

00:14:35 John Searle's Chinese Room Argument, https://plato.stanford.edu/entries/chinese-room/

00:18:00 Retrieval-Augmented Generation, https://arxiv.org/abs/2005.11401

00:18:40 Retrieval-Augmented Generation, https://docs.cohere.com/v2/docs/retrieval-augmented-generation-rag

00:35:39 Let’s Verify Step by Step, https://arxiv.org/pdf/2305.20050

00:39:20 Adaptive Inference-Time Compute, https://arxiv.org/abs/2410.02725

00:43:20 Ryan Greenblatt ARC entry, https://redwoodresearch.substack.com/p/getting-50-sota-on-arc-agi-with-gpt

Disclaimer: This show is part of our Cohere partnership series

Prof. Tim Rocktäschel, AI researcher at UCL and Google DeepMind, talks about open-ended AI systems. These systems aim to keep learning and improving on their own, like evolution does in nature.

Ad: Are you a hardcore ML engineer who wants to work for Daniel Cahn at SlingshotAI building AI for mental health? Give him an email! - danielc@slingshot.xyz

TOC:

00:00:00 Introduction to Open-Ended AI and Key Concepts

00:01:37 Tim Rocktäschel's Background and Research Focus

00:06:25 Defining Open-Endedness in AI Systems

00:10:39 Subjective Nature of Interestingness and Learnability

00:16:22 Open-Endedness in Practice: Examples and Limitations

00:17:50 Assessing Novelty in Open-ended AI Systems

00:20:05 Adversarial Attacks and AI Robustness

00:24:05 Rainbow Teaming and LLM Safety

00:25:48 Open-ended Research Approaches in AI

00:29:05 Balancing Long-term Vision and Exploration in AI Research

00:37:25 LLMs in Program Synthesis and Open-Ended Learning

00:37:55 Transition from Human-Based to Novel AI Strategies

00:39:00 Expanding Context Windows and Prompt Evolution

00:40:17 AI Intelligibility and Human-AI Interfaces

00:46:04 Self-Improvement and Evolution in AI Systems

Show notes (New!) https://www.dropbox.com/scl/fi/5avpsyz8jbn4j1az7kevs/TimR.pdf?rlkey=pqjlcqbtm3undp4udtgfmie8n&st=x50u1d1m&dl=0

REFS:

00:01:47 - UCL DARK Lab (Rocktäschel) - AI research lab focusing on RL and open-ended learning - https://ucldark.com/

00:02:31 - GENIE (Bruce) - Generative interactive environment from unlabelled videos - https://arxiv.org/abs/2402.15391

00:02:42 - Promptbreeder (Fernando) - Self-referential LLM prompt evolution - https://arxiv.org/abs/2309.16797

00:03:05 - Picbreeder (Secretan) - Collaborative online image evolution - https://dl.acm.org/doi/10.1145/1357054.1357328

00:03:14 - Why Greatness Cannot Be Planned (Stanley) - Book on open-ended exploration - https://www.amazon.com/Why-Greatness-Cannot-Planned-Objective/dp/3319155237

00:04:36 - NetHack Learning Environment (Küttler) - RL research in procedurally generated game - https://arxiv.org/abs/2006.13760

00:07:35 - Open-ended learning (Clune) - AI systems for continual learning and adaptation - https://arxiv.org/abs/1905.10985

00:07:35 - OMNI (Zhang) - LLMs modeling human interestingness for exploration - https://arxiv.org/abs/2306.01711

00:10:42 - Observer theory (Wolfram) - Computationally bounded observers in complex systems - https://writings.stephenwolfram.com/2023/12/observer-theory/

00:15:25 - Human-Timescale Adaptation (Rocktäschel) - RL agent adapting to novel 3D tasks - https://arxiv.org/abs/2301.07608

00:16:15 - Open-Endedness for AGI (Hughes) - Importance of open-ended learning for AGI - https://arxiv.org/abs/2406.04268

00:16:35 - POET algorithm (Wang) - Open-ended approach to generate and solve challenges - https://arxiv.org/abs/1901.01753

00:17:20 - AlphaGo (Silver) - AI mastering the game of Go - https://deepmind.google/technologies/alphago/

00:20:35 - Adversarial Go attacks (Dennis) - Exploiting weaknesses in Go AI systems - https://www.ifaamas.org/Proceedings/aamas2024/pdfs/p1630.pdf

00:22:00 - Levels of AGI (Morris) - Framework for categorizing AGI progress - https://arxiv.org/abs/2311.02462

00:24:30 - Rainbow Teaming (Samvelyan) - LLM-based adversarial prompt generation - https://arxiv.org/abs/2402.16822

00:25:50 - Why Greatness Cannot Be Planned (Stanley) - 'False compass' and 'stepping stone collection' concepts - https://www.amazon.com/Why-Greatness-Cannot-Planned-Objective/dp/3319155237

00:27:45 - AI Debate (Khan) - Improving LLM truthfulness through debate - https://proceedings.mlr.press/v235/khan24a.html

00:29:40 - Gemini (Google DeepMind) - Advanced multimodal AI model - https://deepmind.google/technologies/gemini/

00:30:15 - How to Take Smart Notes (Ahrens) - Effective note-taking methodology - https://www.amazon.com/How-Take-Smart-Notes-Nonfiction/dp/1542866502

(truncated, see shownotes)