Oxide and Friends

RFDs--Requests for Discussion--are how we at Oxide discuss... just about everything! Technical design, hardware component selection, changes in process, culture, interview systems, (even) chat--we have RFDs for all of these, over 500 in a bit under 5 years. Bryan and Adam were joined by Oxide colleagues instrumental to RFDs, from their most prolific author to those making them more consumable. In addition to Bryan Cantrill and Adam Leventhal, we were joined by Oxide colleagues, Robert Mustacchi, David Crespo, Ben Leonard, and Augustus Mayo. Some of the topics we hit on, in the order that we hit them: Oxide RFD siteRFD 1: Requests for DiscussionA Tool for DiscussionSun PSARC casesThe Queen's DuckThe Hairy ArmJoyent RFDsRFC-3AsciiDocJoyent RFD 77OxF: Hiring Processes with Gergely OroszOxide RFD APIprogenitorherehere"Own your strategic weirdness"RFD 113: Engineering DeterminationIf we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

Lots of engineering decisions get made on vibes. Popularity, anecdotes—they can lead to expedient decisions rather than rigorous ones. At Oxide, our choice to go with CockroachDB was hardly hasty! Dave Pacheco joins Bryan and Adam to talk about why we choose CRDB… and how Cockroach Lab’s recent switch to a proprietary license impacts that. In addition to Bryan Cantrill and Adam Leventhal, our special guest was Dave Pacheco. Some of the topics we hit on, in the order that we hit them: TechCrunch: Cockroach Labs shakes up its licensing to force bigger companies to payKelsey's TweetOxide RFD 53: Control plane data storage requirementsOxide RFD 110: CockroachDB for the control plane databaseOxide RFD 508: Whither CockroachDBJoyent blog post on the outage due to postgres autovacuumJepsenDave's CRDB exploration repoChronyOxF: A Debugging OdysseyThe Liberation of RethinkDBIf we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

The Oxide control plane coordinates multiple services to do complex, compound operations. Early on, we knew we wanted to provide a robust structure for these multi-part workflows. We stumbled onto Distributed Sagas and built our own implementation in Steno. Bryan and Adam are joined by several members of the Oxide team who built and use Steno to drive the complex operation of the control plane. In addition to Bryan Cantrill and Adam Leventhal, speakers included Dave Pacheco. Eliza Weisman, Andrew Stone, Greg Colombo, and James MacMahon. Some of the topics we hit on, in the order that we hit them: Distributed Sagas: A Protocol for Coordinating Microservices - Caitie McCaffreyOxide RFD 107: Workflows EngineStenotromboneSAGASAutomated saga testingOxide RFD 289: Steno UpgradeFeral Concurrency Control paper from Berkeley and the University of SydneyEliza's PRSteno's description of its divergence from Distributed SagasAWS "constant work" blogOxF on formal methodsA complex bug with sagas: "tl;dr there's TWENTY steps in 5042 that leads to an accounting bug"Oxide RFD 373: Reliable Persistent WorkflowsEliza's novella on updating an instanceIf we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

Nicholas Carlini joined Bryan and Adam to talk about his terrific blog post on his many pragmatic uses of LLMs to solve real problems. He has great advice about when to use them (often!) and what kinds of problems they handle well. LLMs aren't great at many things, but used well they can be an amazing tool. In addition to Bryan Cantrill and Adam Leventhal, we were joined by special guest, Nicholas Carlini as well as by listeners Mike Cafarella, p5commit, and chrisbur. Some of the topics we hit on, in the order that we hit them: How I Use "AI"The McLaughlin GroupSurge 2011 ~ Closing Plenary ~ Theo SchlossnagleMicrosoft's Tay chatbotCurb Your Enthusiasm: Larry vs. SiriSal Khan on LLMsGoogle's awful AI adGoogle pulls adIf we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

Bryan and Adam were joined by security expert, Katie Moussouris, to discuss the largest global IT outage in history. It was an event as broadly impactful as it will be instructive; as Bryan noted, you can see all of computing from here, from crash dumps to antitrust. In addition to Bryan Cantrill and Adam Leventhal, we were joined by special guest Katie Moussouris. If we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

Raja Koduri joined Bryan and Adam to answer a question sent in from a listener: what's are the differences between a CPU, GPU, FPGA, and ASIC? And after a walk through history of hardware, software, their intersection and relevant companies, we ... almost answered it! In addition to Bryan Cantrill and Adam Leventhal, our special guest was Raja Koduri. Some of the topics we hit on, in the order that we hit them: 3dfx Oral History Panel with Ross Smith, Scott Sellers, Gary Tarolli, and Gordon Campbell3dfxOpenGLGlideDirect3DCUDADennard scalingVLIWGPGPUAMD APUEnergy Efficiency and AI HardwareIf we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

Charity Majors joined Bryan, Adam, and the Oxide Friends to talk about the idea of "innovation tokens"--a fixed budget for, so called, "innovative" projects. When is boring better and when is innovation the safer approach? Is Oxide issuing innovation tokens in some sort of hyper-inflationary cycle!? In addition to Bryan Cantrill and Adam Leventhal, we were joined by special guest Charity Majors. Some of the topics we hit on, in the order that we hit them: Glyph: Against Innovation TokensCharity's Twitter ThreadOxF: Let That Sink In! (Whither Twitter?)DruidScuba whitepaperOxide RFD 68: Partnership as Shared ValuesGood Strategy Bad Strategy by Richard RumeltDropshotProgenitorOxF: The Pragmatism of HubrisOxF: HeliosIf we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

Every so often we like to give our Oxide and Friends hot takes (or as Adam puts it "Bryan getting trolled on Twitter"). This time, a viral tweet suggests that NVIDIA is on the same trajectory as Sun Microsystems on its ascent during the Dot Com Bubble. From two alumni of Sun's rise and fall: maaaaybe not. In addition to Bryan Cantrill and Adam Leventhal, speakers included Todd Gamblin. Some of the topics we hit on, in the order that we hit them: The Tweet!OxF: Innovation Stagnation?read the tweetFramework laptop RISC-V mainboardTadpole SPARCbookOxF: A Requiem for SPARC with Tom Lyonpart Ipart IIpart IIIJensenRIVA 128OxF: Steve Jobs & the Next Big ThingIf we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

Bryan and Adam were joined by The Changelog’s Adam Stacoviak for a … wide ranging conversation! Something for everyone—especially fans of HBO’s Silicon Valley! In addition to Bryan Cantrill and Adam Leventhal, we were joined by special guest Adam Stacoviak. Some of the topics we hit on, in the order that we hit them: Bryan on Changelog Changelog: 23 years of Ruby with MatzSWOTBachmanity InsanityStraight outta KubeconBreakmaster CylinderAdam Stac on githubChangelog Dance Party by BMCComputer History Museum: Oral HistoriesBryan's talk on social audioIf we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

Back in May 2014 Joyent accidentally rebooted an entire datacenter (not just the handful of node as intended!). That incident--traumatic was it was--informed many aspects of the Oxide product. Bryan and Adam were joined by members of that former Joyent team to discuss, commiserate, and--perhaps--get some things off their chests. a live show weekly on Mondays at 5p for about an hour, and recording them all; here is the recording. In addition to Bryan Cantrill and Adam Leventhal, speakers included Josh Clulow, Brian Bennett, Robert Mustacchi, and Steve Tuck. Some of the topics we hit on, in the order that we hit them: The Register: Fat-fingered admin downs entire Joyent data centerBryan's talk: Debugging Under FireOxide and Friends on the Oakland BallersThe Ur AgentJoyent post-mortemIf we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

The long-awaited Oxide and Friends bookclub! Bryan and Adam were joined by special guest--and real life biologist--Greg Cost to discuss Philip Ball's terrific book, How Life Works: A User’s Guide to the New Biology. Spoiler: Alan Turing makes a very expected appearance! In addition to Bryan Cantrill and Adam Leventhal, we were joined by special guest Greg Cost. Some of the topics we hit on, in the order that we hit them: The Turing patternRNA as a precursor to DNAXenopus frogXenobotsAnton computerBryan's reading notes Central themes Passages of note:

The Oxide Friends have talked about the Hashicorp license change, the emergence of an open source fork of Terraform in OpenTofu, and other topics in open source. A few weeks ago both InfoWorld and Hashicorp (independently?) accused OpenTofu of stealing Terraform code—a serious claim that turned out to be fully unfounded. We (you!) have been lucky to avoid this topic with a couple of guests lined up to talk about the xz exploit discovery and founding the Oakland Ballers… but we ran out of distractions! Bryan and Adam talk about this FUD and FUD generally. Your hosts were Bryan Cantrill and Adam Leventhal. Some of the topics we hit on, in the order that we hit them: Infoworld: OpenTofu may be showing us the wrong way to forkOpenTofu responseIf we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

Bryan, Adam, Steve, and the Oxide Friends are joined by the founders of the Oakland Ballers, the continuation of a long history of baseball in Oakland. There turns out to be a plenty in common between founding a computer company and founding a baseball team--and we both have our fans supporting us! In addition to Bryan Cantrill and Adam Leventhal, we were joined by very special guests Paul Freedman and Bryan Carmel as well our somewhat-special boss, Steve Tuck. Some of the topics we hit on, in the order that we hit them: The Oakland BallersBryan and Adam at Manaea's no-noThe Munson-Nixon lineThe Pioneer LeagueBaseball's longest gameAdam's neighbor, Bill George, scorer of the longest gameYolo HighwheelersBART's sponsorship of the BallersJ.T. Snow joins the BallersJ.T. saves Dusty's sonIf we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

Andres Freund joined Bryan and Adam to talk about his discovery of the xz backdoor. It’s an incredible story… so great to get into the details with Andres. We started by ranting about the coverage in the New York Times… coverage that explicitly refused to dig into the details! It’s all the more shocking because the big story here is how Andres’ penchant for digging into the details is what saved us all from what would have been a pervasive and damaging attack! In addition to Bryan Cantrill and Adam Leventhal, we were joined by special guest Andres Freund. Our research for this episode: Andres' initial public disclosureNew York Times: Did One Guy Just Stop a Huge Cyberattack?Kevin RooseNew York Times front page from April 4th, 2024How I got started as a developer with Andres Freund & Heikki Linnakangas | Path To Citus Con Ep08The Mystery of ‘Jia Tan,’ the XZ Backdoor Mastermind | WIREDHow one volunteer stopped a backdoor from exposing Linux systems worldwide - The VergeLinux backdoor was a long con, possibly with nation-state support, experts say - Nextgov/FCWresearch!rsc: Timeline of the xz open source attackBrian Krebs thread on mastodonXz/liblzma: Bash-stage Obfuscation ExplainedA Microcosm of the interactions in Open Source projectsRisky Business #743 -- A chat about the xz backdoor with the guy who found itRisky Biz News: F-Droid narrowly avoided XZ-like incident in 2020What we know about the xz Utils backdoor that almost infected the world | Ars TechnicaEverything I know about the XZ backdoorLINUX Unplugged 556: The xz Backdoor Exposed 🚨If we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers! Recorded April 8th, 2024

The Oxide Friends talk about about cultural idiosyncrasies--turns out we have a lot of them at Oxide! Some might even sound good enough for you to try out! Demo Fridays, morning water-cooler, no-meet Wednesdays, recorded meetings, dog-pile debugging (aka CSPAN for debugging), RFDs (requests for discussion), no performance review process... In addition to Bryan Cantrill and Adam Leventhal, we were joined by Oxide colleague Steve Klabnik. Some of the topics we hit on, in the order that we hit them: Engineering a cultureIt's Free Real EstateWho killed the network switch?OxF: Engineering CultureJujutsuOtM: Jeff RothschildPositive Coaching AllianceDORAOxide RFDRFD 68: Partnership as Shared ValuesMatthew Sanabria: Observability Companies to Watch in 2024 If we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

Nicholas Carlini joined Bryan, Adam, and the Oxide Friends to talk about his work with adversarial machine learning. He's found sequences of--seemingly random--tokens that cause LLMs to ignore their restrictions! Also: printf is Turing complete?! In addition to Bryan Cantrill and Adam Leventhal, we were joined by special guest Nicholas Carlini. If we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

Data visualization is an important--and overlooked!--tool in the software engineer's tool belt. Bryan describes a recent journey with gnuplot while Oxide colleague, Charlie Park, shares his own experience with data visualization and Adam offers a visual analysis of Simpsons episodes. Stay tuned to the end to find out about the Oxide and Friends book club coming up in May. In addition to Bryan Cantrill and Adam Leventhal, we were joined by Oxide Colleague, Charlie Park. Some of the topics we hit on, in the order that we hit them: Bryan's rad gnuplotGitHub PR with Bryan's visualizationsPronunciation of "Tufte"/ˈtʌfti/Flame Graphsflamegraph-rsThis American Life: A Little Bit of KnowledgeStatemapsMinard's diagramhttps://twitter.com/thingskatedid/status/1386077306381242371plot.awkregular expressionsBNF grammarsisvgidotDTrace aggregationsratatuignuplotMatplotlibggplot2ParaViewGLVisAbraham Wald and the airplane diagram with red bullet holes – here’s the origin storyKartlyticsHow Life WorksIf we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

Bryan and Adam are joined by members of the Oxide storage team--Josh, Alan, James, and Matt--to talk about Crucible, the service that provides block storage for VM instances running in the Oxide Rack. In addition to Bryan Cantrill and Adam Leventhal, we were joined by Oxide colleagues Josh Clulow, Alan Hanson, James MacMahon, and Matt Keeter. Some of the topics we hit on, in the order that we hit them: Hyper-converged infrastructureFibre ChannelZFSIntroduction to Flexible Data Placement: A New Era of Optimized Data ManagementStorage Architecture ConsiderationsCephRFD 60: Storage Architecture ConsiderationsRFD 177: Implementation of Data StorageRFD 444: Crucible Upstairs RefactoringRFD 445: Crucible Upstairs BackpressureIf we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

Sometimes Bryan gets trolled by a tweet and brings it to Adam and the Oxide Friends. This was a well-crafted troll: is innovation slowing? Are the most interesting problems solved. In a word: no. For many more words, listen in! In addition to Bryan Cantrill and Adam Leventhal, we were joined by Steve Klabnik. Some of the topics we hit on, in the order that we hit them: The TweetNate SilverSecularitySecular stagnationAngela Collier: physics progress in the last 70 years?Haber processCRISPR gene editingCode Breaker by Walter IsaacsonLeonhard EulerDijkstra's algorithmRaftAntibioticAcquired: TSMCEUV lithographyIf we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

Bryan and Adam are joined by Oxide colleagues Josh Clulow, Patrick Mooney, and Steve Klabnik to discuss Helios, the operating system that runs on the Oxide Rack. Helios is a distro of illumos (derived from OpenSolaris, derived from Solaris, etc.). What's a distro? Why did Oxide choose illumos? Plenty of cross-generational appeal in this episode! Some of the topics we hit on, in the order that we hit them: Helios github repoHacker News thread its releaseOmniOSRust Tier 2 supportBryan's talk on holistic bootOxide and Friends: Holistic BootOxide and Friends: Shipping Rack 1The Quality Death SpiralOxide's "St. Louis" branch of illumosBryan's sleeper bug from 1991illumos booksIf we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!