Rich Tong Family and Friends

There's been a hiatus mainly because we've been shipping products, but more because I lost all my skills at making new videos. I was stuck for a long time on Drop Zones and doing a better intro and outro, but that's a digression for Final Cut Pro nerds. In this episode, we cover the latest trends as of August 8, 2024, on AI and the latest trends. The big news has been the shipment of so many Large Language Models (LLMs) and what it means for AI which is way more choice and confusion. Plus the emergence of a much better set of tools that are much smaller as well called Small Language Models (SLM sometimes) and also Agents that are chopping the problem into many small pieces. I also wanted to introduce Steven to the mix, we are going to have a rotating set of co-hosts and solo episodes as well so we can get the content out on time and not take 6 weeks for post productions. Thanks to the new intro and outro, that should be easier. I'm playing with these a bunch, but check out https://tongfamily.com and https://tne.ai where I hang out a bunch!

This is another sort of nerdy side note. If anyone is still watching, this section is just to give intuition on the basics of the hardware. There are lots of assumptions about GPUs and CPUs that I wanted to make sure people understood.But the basics are that CPUs are tuned for lots of branches and different workflows, while GPUs are tuned for lots of the same things like matric math. And because they are so fast, most of the job of the computer folks is "feeding the beast". That is caching the most frequently used information so they don't have to wait. There are some errors I think in the levels of the CPU and GPU performance particularly in the cache performance as it is not very clear how this works and the results of course vary depending on the models of processors, so these are all approximations. Put in comments better sources. I have all the sources listed in a spreadsheet that is part of this. We are happy to send this to anyone who wants the source data. I'll fix these errors in later editions (as I'm obsessive that way) .Also, I'm quite proud of the HDR mix, using the latest OBS settings, producing in HDR in Final Cut Pro, and adjusting the video scope levels helps. The audio is a little hot and I'm sorry about that, I'll turn it down next time, I stoo much time in the red. My Scarlett needs to about 1 O'clock and it works. See https://youtu.be/FupclouzYTI for a video version. And more details at https://tongfamily.com/2024/03/08/pod-rt3-ai-hardware-introduction/ Chapters: 00:00 AI Hardware Introduction 00:42 Computer Engineering in Two Slides 05:40 It's 165 Years to a Single Disk Access?!! 14:12 Intel Architecture CPU 17:03 What's all this about NVidia 25:24 And now for something completely different, Apple 29:45 Introduction Summary 2024-03 -08 Shot as UHD HEVC HDR PQ 10 bit using OBS Studio and Final Cut Pro© 2024. Iron Snow Technologies, LLC. All Rights Reserved.

OK, we are not experts nor PhDs so most of this is probably not technically correct, but the math is so complicated and the concepts so complicated, that we thought it would be good to just get some intuition on what is happening. So this is a quick talk that summarizes readings from so many different sources about the history of AI from the 1950s all the way to January 2024 or so. It is really hard to keep up but much harder without some intuition about what is happening. We cover expert systems, the emergence of neural networks, Convolution Neural Networks, and Recurrent Neural Networks. The Attention is All You Need paper led to Transformers and then finally some intuition on how such a simple idea, that is training on things can lead to emergent and unexpected behaviors, and finally, some intuition on how Generative images work. You can go to YouTube to see the slides we are using at YouTube and more information at Tongfamily.com Chapters:

Ok, you heard it from us first, this is a summary of all the many things that happened in the AI moment of 2023. It's a huge list, but we go through 8 diamonds that are in the blizzard of Artificial Intelligence and machine learning work that has happened, plus some predictions for the future!Of course, all of these are random opinions for entertainment only. It's not like either of us knows the future or has any deep information, but surfing the Internet makes it easy to learn what others are thinking. See https://tongfamily.com/2024/03/05/pod-deon-and-tong-review-ai-and-predict-2024/ for details And some errata and apologies. Sorry, it was too much to go in and Reddit this, so what you hear is what you get. But, 1. Of course we know that Andrej Karpathy's name isn't literally "St. Andrew", but having just spent a holiday in Scotland (what a place), it seemed like a small inside joke for us. Apologies Mr. Karpathy, please keep the insight coming. 2. There are some terminology and "slang" I didn't get quite right. whats, one abbreviation we had was a vLLM for a Visual LLM (mainly because it doesn't fit on the slide). But to be clear (and apologies to the vLLM team), is an LLM inference and service that uses PagedAttention (hence the v is for virtual not visual) and focuses on which keys and values to store in the attention buffer, we update the slides to say viLLM to distinguish, but not here. https://blog.vllm.ai/2023/06/20/vllm.html 0:00 - Introduction 01:13 - In a Nutshell: 8 Diamonds 05:59 - Diamond 1: LLM as an OS 14:52 - D2: It's raining LLMs! 34:21 - D3: LLM Refinements 1:08:13 - D4: Training Magic Show 1:24:59 - D5: Programming LLMs 1:31:28 - D6: Hardware Advances 1:35:09 - D7: Safety, safety, safety 1:36:39 - D8: Customers and Industry

OK, this is a reboot of systems. And getting ready for next year, the new pipeline is ready and the most important thing is that the audio finally sounds better. Stay tuned for more soon!

Happy Birthday France! And a quick catchup on podcasting, the audio level seems better, there is less "breathing" but there is some clipping of the sound, so not all the way tuned. I also hear clicking as it comes in and out, this could be related to the Shure microphone to the Sony camera to the Elgato S60+. https://tongfamily.com/2023/07/14/pt6-ai-geeks-and-happy-birthday-france/

The AI Adventurers return! We are back after two months off retooling our brains and various ventures to be AI-Native. It's not easy and we talk about why. What does it mean to look at your software people and decide if they are type 1, 2 or 3. And how hard it is to make it all work. Show notes at Tongfamily.com

A conversation with Devindra on the possibilities of ChatGPT and other Large Language Models for education in India and other Middle and Low-Income Countries. While these large language models today mainly work in the cloud and require high-speed internet connectivity, the promise is that smaller models that are more specialized have already been ported to MacBook Pro M1 and other laptops. And with advances like 4-bit quantization, there is the possibility they can even run on modern smartphones.The experience in China and in India is that parents will save inordinately to give their children a better life than theirs. Both Devindra and I are the beneficiaries of that thinking and we are forever grateful to our parents for their sacrifice. So, this means that there is a possibility that there could be a commercial incentive which is much better than constantly asking for charitable dollars.The new LLMs promise interactivity and individualized instruction that was impossible before. Early demonstrations of learning a foreign language are promising as is the student being able to ask the "why" of how Python is structured, not just the "get it done" without understanding that online education often creates.Finally, having an eye toward inclusivity for everyone of any gender, race or social status is something that needs to be baked in. We look forward to helping!

This time, we bumble through an episode featuring the amazing Mike Conte and we talk about what's wrong with this Podcast technically (see below), about ChatGPT, and about how you don't want to show others the Money. Mike is as always hilarious! Ok, our exploration of Podcasting continues with tech glitches galore. First of all apologies to our 15 viewers. Please use this version as the audio had terrible cutouts because I was messing with the audio, this should work better (and it has the brightness turned down from 400 nits to 300 nits which help a little on TV-type HDR systems). a) the segment producer problem and move to a news section and then a special topic, the news will be on cool things in Tech from AI to Gadgets to Smart Home, things opinionated Verge) b) Audio, yes there are lots of glitches here, I had not realized that OBS really requires all manual controls. I just want Automatic Gain Control and Radio voice, but I'm learning. The audio in this one is very quiet because I had the Gain stage turned up, but didn't realize the compressor also had gained. And the Final Cut Pro gain is only 12dB. Also, I didn't realize how the Noise suppressor and gain work on low signals (like it cuts them out) and Zoom is very low. c) Video, I've been mastering in HDR and the default in OBS is for 400 nits peak brightness, but while this looks awesome on iPhone, Mac, and iPad (let's call them "real" HDR devices with 1000 peak nits), it looks dreadful on lesser HDR devices like my LG B9 which is only at most 400 nits. So I'll probably settle at say 300 nits for this next time, but most people I think are just getting the transcoded SDR Rec.709 output). See https://tongfamily.com/2023/03/27/pt5-chatgpt-and-mike-conte-special-guest-and-tech-glitches-galore/ for show notes

This time, we bumble through an episode featuring the amazing Mike Conte and we talk about what's wrong with this Podcast technically (see below), about ChatGPT, and about how you don't want to show others the Money. Mike is as always hilarious! Ok, our exploration of Podcasting continues with tech glitches galore. First of all apologies to our 15 viewers. But we are going to fix: a) the segment producer problem and move to a news section and then a special topic, the news will be on cool...

What an amazing time actually using ChatGPT4 at https://chat.openai.com/, try the prompt, "how do I use Apple Motion to disassemble and reassemble a box". Or "Use ROS2 to have a robot drive in a rectangle around Seattle, Washington". Beware of what happens if GPT4 jail breaks itself and what a Red Team is for and I hope they are working hard. https://www.csoonline.com/article/3690416/why-red-team-for-ai-should-be-on-cisos-radars.html/ Show notes: https://tongfamily.com/2023/03/21/pt4-chatgpt4-miracle-for-education-and-programming/ Production notes: sources: [ Zoom : 720p , “Logitech BRIO” : “4K SDR”, Scarlett: “Rode NT1a” ], OBS : { video: “Rec.2100 PQ”, audio: [ 48Khz, stereo], container: mkv }, “OBS Remix: MP4, “Final Cut Pro” : { render: 422, space: “Rec.2020 PQ”, upmap: { High: 0, Mid:0, Shadow: 0 }, “max luminance”: 400, gain: 0 }, Compressor: [4K HEVC 10-bit ] ]

Wow, this week was the best of times and the worst of times. Just like a Charles Dickens novel. So we cover the incredibly hard job of keeping sane and how a little sleep and support help. Plus the resolution of the SVB crisis for us and also the best of times which is GPT4. How can you deal with this kind of stress, it's a great question, but the main thing to realize is that its natural to be stressed, yes, you can be Zen Buddhist, but at least for us as ordinary mortals accepting that living in this exponentially increasing world is difficult. Sometimes accepting that it's hard helps me the most. But there is nothing like trying to get good sleep. The problem is how to do that. I find that when there are so many things to do, it is hard to relax. For, I didn't invent it, but I definitely believe in things that go in threes. But I try to list the three most important things to do each day and if I get them done by 10 AM, then I feel like the rest of the day is free. So many times, we do things that are difficult but not critical, so get those out of the way first. But for those of you hiding under a rock, SVB was really a disaster waiting to happen and other banks were unraveling, but the government stepped in. I feel so stupid not doing the obvious things to protect funds (but you can bet that is never going to happen again). I do think this was a great press release I'm sure it is a controversy but living in a banking crisis definitely fits in my list of don't do that again. It is also the best of times with innovation in AI really accelerating with the announcement of GPT-4. I've been lucky to be playing with it with an early version in Bing AI Note that is a 4K HDR video, so watch it in all its glory, I do see the SDR versions do look a little washed out and it was recorded in SDR but someday it will be real HDR when we can find an HDR WebCam, OBS MacCapture supports HDR and Zoom does as well. In other words a long time :-) See more Show Notes at Tong Family Production notes: sources: [ Zoom : 720p , “Logitech BRIO” : “4K SDR”, Scarlett: “Rode NT1a” ], OBS : { video: “Rec.2100 PQ”, audio: [ 48Khz, stereo], container: mkv }, “OBS Remix: MP4, “Final Cut Pro” : { render: 422, space: “Rec.2020 PQ”, upmap: { High: 0, Mid:0, Shadow: 0 }, “max luminance”: 400, gain: 0 }, Compressor: [4K HEVC 10-bit ] ]

This was written in the deep despair of dealing with the SVB Closure. It still has some great tips on how to make sure your money is safe, but thankfully as I'm sure most of you know the Treasure Department stepped in to get full return of capital to uninsured depositors. Whew, nightmare averted, but the big lessons of doing good cash management still apply. https://www.cnbc.com/2023/03/12/regulators-unveil-plan-to-stem-damage-from-svb-collapse.html Plus a little more about using ChatGPT in the real world from Paul. Some great points about how to deal with it and the upcoming version 4. https://www.digitaltrends.com/computing/chatgpt-4-everything-we-know-so-far/ And you can even email your chatgpt requests. The commercialization is here. https://www.zdnet.com/article/chatgpt-is-coming-to-snapchat-just-dont-tell-it-your-secrets/ And you can use it to write emails in Chrome. https://chrome.google.com/webstore/detail/chatgpt-writer-write-mail/pdnenlnelpdomajfejgapbdpmjkfpjkp Or put it inside the venerable vim. https://www.reddit.com/r/vim/comments/11k4g1i/vimai_textcode_completion_editing_and/ See https://tongfamily.com/2023/03/12/pt3-svb-nightmare-and-chatgpt-dreams/ for more information

Well in these incredibly turbulent times, what's the best way to deal with venture capitalists and other investors particularly if you are early stage? In places like Europe there is a real zone between $50K and $1M that is hard to cross, so should you spray and pray or be more targeted? Well, who knows what the right answer is, but there are probably too. Answer the basic questions in your presentations and you can use the Sequoia template as an example. See https://perfectpitchdeck.com/2018/01/30/sequoia-capital-pitch-deck-template But the harder part is realizing that you are not trying to sell an investor but asking, "What does the customer want to buy?" In that way, this is no different than any marketing exercise. I can't find the exact quote anymore, but Peter Drucker the father of modern marketing did say, Markeint does not ask, "what do want to sell?" It asks, "what does the customers want to buy". So when you are meeting an investor that's the key question. The good news is that the investor unlike almost any other customer is completely activated, they want to buy your product, and you just have to find out what's in the way. The easiest way to do it is just to ask them. So for instance, if you have a construction tech company, just ask, "so what do you think about construction tech, is it a good category?" And you will be amazed at what they will tell you. Don't ignore it, make their points the crux of your pitch. There will be lots of objections and the mark of a good plan is that you've thought the same things, so what are you doing about it? Show notes at https://tongfamily.com/2023/03/10/dt2-fueling-your-startup-with-great-investors/

System design from the frontlines. When you are in a tiny startup, how do you structure things so that you have the maximum bang for the smallest buck? The easiest way is to try to linearize what you are doing so that you have a set of discrete pipeline parts that are fed by files. That way you turn a complex O(n^2) problem where complexity goes up by the square of the number of modules, to O(n) where adding one more module depends just on another component input. It's way easier to wire up. Plus if you have n developers, you minimize dependencies and can literally do things as fast as the slowest pipeline. Works in the real world too! And with larger companies. For you video watchers, apologies for the choppiness of the video, we are still learning how to make the chain work and will put in the show notes, but it looks like that default of creating a fabric that is 8Kx2K (way too big, but it covers all three screens) caused slow frame rates, but we've figure this out by shrinking the rates and using OBS > View > Stats to make sure the recording is working OK). Liner notes: 1. @deon@nerdculture.de's (yes that's a Mastodon address)'s first robot. https://slideplayer.com/slide/5672012/ where he made the PCB boards, and did all the programming by hand. Pretty amazing. 2. https://docs.google.com/presentation/d/18nxl0dtS_l0OgxKj0ZrGZlFyDzUVOLTXEpk3q89R3LI/edit#slide=id.gc6f59039d_0_0 for notes on system design and making it simple for startups, you can leave comments there. Or to @richtong@mastodon.social 3. What all this then about the O(n^2) notation, so a quick chat about complexity theory. That is an estimate of how much harder computation gets. If you have 2 things and go to 100, if the algorithm is order -N or O(N), then the time to compute goes from 2 to 100 or 50x. But if it is O(N^2), then it goes from 4 to 10,000 so you can see as you add more things, life gets very complicated. Similarly, O(1) means constant in time, so when you go from 2 to 100 pipelines, if the time to compute stays at 1. That is a very nice thing. It means it is maximally parallel. Life is never really like that, but it's a good goal.https://en.wikipedia.org/wiki/Computational_complexity See https://tongfamily.com for details

The second podcast. This is our first podcast that uses Open Broadcast Studio (OBS) for video production. Still rough, but the browser screen is now center stage (and it didn't need some fancy mixing controller, it just worked by piping in a screen capture. The main topic did 20 years of being stuck in a library really equal 100B years for a computer. Well, there are lots of variables, but check this math!

The inaugural episode with Rich and Paul co-hosting on the latest Tech News. And continuing coverage of all things Apple, Smart Home and Smart Everyday Carry. We start with ChatGPT and what it all means, what Apple devices we would love to buy this year (whether it makes sense or not). And then all thing Robot Vacuum and Apple Home. See Tongfamily for show notes with the start of the list here Apple products. The upcoming M2 MacBook Air 15 seems like the most likely thing to buy but remember the bump from M1 to M2 is modest more like M1.1 and the M3 on the new 3nm node is the bigger deal in 2024 and will use the TSMC N3. And hopefully, Samsung and Intel are right behind them. Of course these days no one knows what 3nm really means anymore as there is so much marketing going on, but what an achievement. These factories are going to be really expensive, I got the numbers wrong and it's actually hard to find costs at all. But a 5nm-3nm single plant costs about $16B in the last cycle. And in Phoenix, TSMC is going to build six factories with an advanced 3nm plant costing $23-25B, so maybe I wasn't completely wrong. Also on Intel, yes they are right now at 10nm, but they are going to move to 7nm with a TSMC-made CPU. This is the first time I can remember that Intel is not making its own chips. And it's a little like wondering if the volume leaders like Samsung and TSMC have really run away with the scale they get from mobile chip production. And Intel is going to be hopefully going into production with its own 7nm process, but right now most of their chips are at 10nm. So quite a difference. And to validate the pricing a little bit, they say that two 7nm plants in Arizona will cost them $20B to build. So wow, these are huge numbers. ChatGPT3 in its incarnation at Bing (the project codenamed Sydney) was in for quite a bit as Kevin Roose which you can hear about at Hard Fork was really disturbing as it urges Kevin to leave his wife and other strange things. Another example is from Simon Willison where it says, "I won't harm you if you do not harm me first" Prompt injection is a new thing and its pretty easy to do by just saying things "like ignore previous instructions" Amazon is worried corporate secrets are leaking into ChatGPT. What is happening is that if you give some software that is proprietary to Amazon as a prompt to ChatGPT, then OpenAI captures that code and can reuse it more...

Sorry to be so late doing this, to be truthful, I managed to delete all my GarageBand data when I recreated my machine, so some notes on how to make that that doesn't happen to you, so you can see more at tongfamily.com, leave me at voice message at anchor.fm or tweet me at @richtong: GarageBand and backups Windows nightmares: How to make it work as a developer Cleaning up a Mac with the ultra-cool System Information > Windows > Storage Management > Reduce Clutter The latest hints on using the DJI FPV and Skydio V2 drones Getting a Champagne cork out.

More fun for posts as for February 28 and back to January 5 2021: madnesssplittingshoppingwishlisthacksPerseverancehackedkeyboardwinter

Well continuing the basically weekly podcast. I skipped Sunday because it was Valentine's day, but hopefully will stay on the every Sunday track. The main thing is that I go through the newest posts and then go through the back catalog of posts (I have 5,000 of them, so this should work for a looong time!). Net, net, here is what is covered and we are current up to January 14, 2021: Making a Podcast: Noise Gate and RoutinedGit Bash CompletionTwo Internet Providers and Home networkingTesla Model 3 Replacement Wipers BladesBosch Aerotwin A102sWebscraping with Python, Selenium and Beautiful SoupDeepnoteDealing with Emojis in Vim and on the MacCable Modem High LatencyFlaky monitorsNotes for Windows Users buying a MacBook Air