Knowledge Distillation with Helen Byrne

Emily Mackevicius is a co-founder and director of Basis, a nonprofit applied research organization focused on understanding and building intelligence while advancing society’s ability to solve intractable problems. Emily is a member of the Simons Society of Fellows, and a postdoc in the Aronov lab and the Center for Theoretical Neuroscience at Columbia’s Zuckerman Institute. Her research uncovers how complex cognitive behaviors are generated by networks of neurons through local interactions and learning mechanisms. Links to work mentioned in this episode: basis.aihttps://elifesciences.org/articles/80680https://www.basis.ai/blog/sr-fang2023/https://www.nature.com/articles/nn.4650https://www.sciencedirect.com/science/article/abs/pii/S0959438817302349https://www.cell.com/cell/fulltext/S0092-8674(24)00235-6https://www.npr.org/2024/04/05/1198909635/chickadee-bird-brain-memory-brain-pattern-foodhttps://github.com/BasisResearch/collab-creatureshttps://basisresearch.github.io/chirho/getting_started.htmlhttp://polis.basis.ai/

Stability AI’s Stable Diffusion model is one of the best known and most widely used text-to-image systems. The decision to open-source both the model weights and code has ensured its mass adoption, with the company claiming more than 330 million downloads. Details of the latest version - Stable Diffusion 3 - were revealed in a paper, published by the company in March 2024. In this episode, Stability AI’s Kate Hodesdon joins Helen to discuss some of SD3’s new features, including improved capabilities for generating text within images and overall image quality. Kate also talks about developments to the underlying model structure of Stable Diffusion, as well as the challenges associated with creating models that deliver more efficient inference. The Stable Diffusion 3 paper can be found here: https://arxiv.org/pdf/2403.03206.pdf

No organisation in the AI world is under more intense scrutiny than OpenAI. The maker of Dall-E, GPT4, ChatGPT and Sora is constantly pushing the boundaries of artificial intelligence and has supercharged the enthusiasm of the general public for AI technologies. With that elevated position come questions about how OpenAI can ensure its models are not used for malign purposes. In this interview we talk to Rosie Campbell from OpenAI’s policy research team about the many processes and safeguards in place to prevent abuse. Rosie also talks about the forward-looking work of the policy research team, anticipating longer-term risks that might emerge with more advanced AI systems. Helen and Rosie discuss the challenges associated with agentic systems (AI that can interface with the wider world via APIs and other technologies), red-teaming new models, and whether advanced AIs should have ‘rights’ in the same way that humans or animals do. You can read the paper referenced in this episode ‘Practices for Governing Agentic AI Systems’ co-written by Rosie and her colleagues: https://cdn.openai.com/papers/practices-for-governing-agentic-ai-systems.pdf Watch the video of the interview here: https://www.youtube.com/watch?v=81LNrlEqgcM

Nina Schick is a leading commentator on Artificial Intelligence and its impact on business, geopolitics and humanity. Her book ‘Deepfakes and the Infocalypse’ charts the early use of gen AI to create deepfake pornography and the technology’s subsequent use as a tool of political manipulation. With over two decades of geopolitical experience, Nina has long been focused on macro-trends for society. She has advised global leaders, including Joe Biden, the President of the United States, and Anders Fogh Rasmussen, the former Secretary General of NATO. She has also worked with some of the world’s premier companies and organisations, including Microsoft, Adobe, DARPA, and the UN. A familiar face at technology conferences such as CES, TEDx, CogX and WebSummit, Nina is also a regular contributor to discussions about AI on the BBC, CNN, Sky News, Bloomberg and more. In her conversation with Helen, Nina outlines the continuing risks posed by deepfake technologies and the technological counter-measures that can be used to safeguard against them. You can watch the video of her interview on YouTube: https://youtu.be/f4zTbGWYan8

Charlie Blake from Graphcore’s research team discusses their AI Papers of the Month for January 2024. Graphcore research has been collating and sharing a review of the most consequential AI papers internally, every month, for a number of years. Now – for the first time – the research team is making this valuable resource public, to help the wider AI community keep up-to-date with the most exciting breakthroughs. Papers of the Month for January 2024 (with some work from December 2023) includes: Bad Students Make Great Teachers: Active Learning Accelerates Large-Scale Visual Understanding https://arxiv.org/abs/2312.05328 Authors: Talfan Evans, Shreya Pathak, Hamza Merzic, et al. (Google DeepMind, UCL) Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws https://arxiv.org/abs/2401.00448 Authors: Nikhil Sardana and Jonathan Frankle (MosaicML) Analyzing and Improving the Training Dynamics of Diffusion Models https://arxiv.org/abs/2312.02696 Authors: Tero Karras et al. (Nvidia, Aalto University) Solving olympiad geometry without human demonstrations https://www.nature.com/articles/s41586-023-06747-5 Authors: Trieu H. Trinh, Yuhuai Wu, Quoc V. Le, He He and Thang Luong (Google DeepMind, New York University) To read about January’s Papers of the Month, visit the Graphcore blog. https://www.graphcore.ai/posts/great-teachers-and-beyond-chinchilla-papers-of-the-month-jan-2024

Data is the fuel that is powering the AI revolution - but what do we do when there's just not enough data to satisfy the insatiable appetite of new model training? In this episode, Florian Hönicke, Principal AI Engineer at Jina AI, discusses the use of LLMs to generate synthetic data to help solve the data bottleneck. He also addresses the potential risks associated with an over-reliance on synthetic data. German startup Jina AI is one of the many exciting companies coming out of Europe, supporting the development and commercialisation of generative AI. The team at Jina AI gained widespread attention in late 2023 for the release of the first open-source text embedding model with an 8192 token length. Jina-embeddings-v2 achieves state-of-the-art performance on a range of embedding-related tasks and matches the performance of OpenAI's proprietary ada-002 model. Watch the video of our interview: https://youtu.be/AP80hZajk5w

NeurIPS is the world’s largest AI conference, where leading AI practitioners come together to share the latest research and debate the way forward for artificial intelligence. In this special episode, Helen examines some of the big themes of NeurIPS 2023 and talks to a range of attendees about their work, the big issues of the day, and what they’ve seen at NeurIPS that caught their attention. It’s fair to say that LLMs loomed large over this year’s conference, but there’s plenty more to discuss – from AI’s potential to combat climate change to new techniques for computational efficiency. Helen’s guests are: Sofia Liguori – Research Engineer at Google Deepmind, specialising in the application of AI to sustainability and climate change. Priya Donti – Assistant Professor in Electrical Engineering and Computer Science at MIT and Co-founder of Climate Change AI. Priya discusses the challenges associated with introducing leading-edge AI systems into highly complex real-world power generation and delivery systems. Irene Chen – Assistant Professor at UC Berkeley and UCSF’s Computational Precision Health program. Irene talks about her goal of delivering more equitable healthcare at a time when AI is set to disrupt the field. She also discusses the potential to make use of commercial LLMs in a way that protects sensitive user data. James Briggs – AI engineer at Graphcore. James and colleagues were presenting their paper ‘Training and inference of large language models using 8-bit floating point’ at this year’s NeurIPS. James explains their work and the importance of using smaller numerical representations to unlock computational efficiency in AI. Abhinav (Abhi) Venigalla – is a member of the technical staff at Databricks. The company provides a range of products to help organisations unlock the potential of enterprise-grade AI. Abhi talks about the increasing emphasis on inference tools and computational efficiency as AI moves out of the research lab and into commercial deployment.

Miranda Mowbray is one of Britain’s leading thinkers on the ethics of Artificial Intelligence. After a long and distinguished career as a research scientist with HP, she is now an Honorary Lecturer in Computer Science at the University of Bristol where she specialises in ethics for AI, and data science for cybersecurity. In our wide-ranging conversation, Miranda breaks down the definition of AI ethics into its many constituent parts – including safety, transparency, non-discrimination and fairness. She tells us that there’s probably too much focus on the dire predictions of AI ‘doomers’ and not enough on the more immediate, but less apocalyptic outcomes. On a lighter note, Miranda reveals her personal mission to change the world, and shows off a sculpture that she had commissioned, based on the imaginings of generative AI. You can watch a video of our interview with Miranda here: https://youtu.be/tbnHxbM5ZR8

Danijela Horak explains how the BBC is making use of AI and its plans for the future, including detecting deepfakes as well as using deepfake technology as part of its production process. Danijela and Helen discuss the Corporation's use of open source models and its view on closed source technologies such as the GPT family of models from OpenAI. We find out how the BBC uses AI for recommendation, while taking cautious approach to user data, and Helen and Danjela reflect on why there needs to be more rigour in AI research so that the field doesn't end up on a par with 'social sciences'! Watch the video of our conversation with Danijela at: https://www.youtube.com/watch?v=QOwecs8KRLg

First episode coming soon... Join Graphcore's Helen Byrne and leading figures in the world of Artificial Intelligence as they discuss advances in AI research, commercial deployment, ethics, and occasional weird and wonderful applications.