Plumbers of Data Science

Super happy to have Taylor with me on this stream. She is the VP of Data Labs at Rivery and therefore has a lot of experience with data platforms. We'll talk about the modern data stack and where it's going. I'm excited to hear her experience about the changes that are happening in the data space, and what that means for data engineers & data teams.

In this video I talk to Piyush who's an engineer at Google and has his own YouTube channel: "Tech Tutorials with Piyush". He's a really good guy and I love how he's dedicated to teaching engineering. We are talking about some awesome topics like: Have fun! You can also check this on out on YouTube: https://youtu.be/FZemVaqQcnM If you want to get into Data Engineering check out my Academy at https://learndataengineering.com

I have seen people doing that wrong a few times. Luckily Wouter Trappers who is helping companies as a professional can help. We talked about The steps you need to take from value proposition to dashboards. Wouter is really knowledgeable and it was super fun talking with him and hearing his approach.

Dhruba Borthakur is CTO at Rockset and a passionate Data Engineer. Before co-founding Rockset he played a big role in development of Hadoop HDFS at Yahoo as well as HBase and RocksDB at Facebook. His current project is the serverless Rockset platform where you can gain real time analytics insight into your data. I tried it out before our talk and really liked it.

The Podcast is back!!!! I promise I am going to keep it up to date this time ;) In this episode I talk about my newest Data Engineering course. I think it's the ultimate 1 hour 15 minutes introduction to Data Engineering. There were also a ton of questions from the chat that I answered. Think you really enjoy this.

We are looking into the network communication protocol map. I first saw this like 10 years ago and its awesome. Then we check out the Big Data and Data Science Landscape image. It shows you all the tools available to do data science, machine learning and data engineering. Which is very helpful if you are researching for tools to use. Before using the Twitter API you got to create a developer account. So, I show you how I created one. After that I tried to get Nifi to download Tweets but it...

Today's podcast is a bit of a behind the scenes. What it takes to do a audio podcast. How you can get audio to text transcriptions for free. .Also Github questions on how to work with branches on the Cookbook

Today a case study about OLX with a guest it was super fun! Here are the slides Alexeyand I talked about: https://www.slideshare.net/mobile/AlexeyGrigorev/image-models-infrastructure-at-olx

In this episode we install the Nifi docker container and look into how we can extract the twitter data. We are also talking about the differences between infrastructure as a service, platform as a service and application as a service.

In this episode we look into the Twitter API documentation, which I love by the way. How can we get old tweets for a certain hashtags and how to get current live tweets for these hashtags.

Tips on how you find a job in Germany and two super interesting mails.

The cookbook my Youtube, it will be for free, forever! Check out the data engineering cookbook on GitHub: https://github.com/andkret/Cookbook

Updates of the cookbook and how to collaborate on it

In this episode we talk about the lambda architecture with stream and batch processing as well as a alternative the Kappa Architecture that consists only of streaming. Also Data engineer vs data scientist and we discuss Andrew Ng's AI Transformation Playbook

How do you choose between Cloud vs On-Premise, pros and cons and what you have to think about. Because there are good reasons to not go cloud. Also thoughts on how to choose between the cloud providers by just comparing instance prices. Otherwise the comparison will drive you insane.

In this episode we go over the ideas I have for the data engineering course structure. It was your chance for you to influence what we put in there.

In this video we go over some of the 100+ comments I received on LinkedIn about a data engineering training.

Let's check out how LinkedIn is processing data

How is Twitter doing Data Engineering? Oh man, they have a lot of cool things to share these tweets.

In this episode we are looking at the data engineering at Spotify, my favorite music streaming service. How do they process all that data?