NewsTech News

Software Engineering Daily

softwareengineeringdaily.com

Software Daily: The world through the lens of software.

Play

Open in

2,221 episodes
Avg 53 min
English

Episodes 2,221

Newest Oldest Longest

All20 Unplayed20 In progress0 Played0

Counted on this page — what you have heard stays on this device, so it is not something the list can be paged by.

#754
Mar 13, 2018 · 46 min
Metamask with Dan Finlay
Decentralized applications can be built on the Ethereum blockchain. Just as the Bitcoin blockchain is a distributed, append-only ledger of financial transaction history, Ethereum is a distributed, append-only ledger of computational transaction history. New kinds of applications can be built on the Ethereum blockchain—and just like every new technology, we need an interface to bridge that new technology and our existing technology. We can use a pure Ethereum browser like Mist—or we can use a Chrome extension like Metamask to turn our normal browser into an Ethereum interface. Dan Finlay is the lead developer of Metamask. In today’s episode, we explore why you would want to interface with decentralized applications, and the different ways of doing so. A few examples we explore—simple transactions like transferring Ether from one person to another; or transacting with a smart contract. My personal anecdote: I recently used Metamask for the first time to fund a GitCoin issue. GitCoin is a way to put up financial rewards for people solving open source issues. I locked up $42 in an Ethereum smart contract, and it became the bounty of that issue. The issue was solved, and I released the $42 from the smart contract to be sent to the developer who solved it. In this example, Ethereum served as a simple escrow service. To send my Ether, I used the Metamask plugin on my Chrome browser. If you are a little confused—don’t worry. We explain it all in this episode. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#753
Mar 12, 2018 · 56 min
Monopolies and Proof of Stake with Karl Floersh
Decentralized applications might someday offer alternatives to modern monopolies. Uber, Airbnb, Facebook, Amazon—all of these services could be recreated on a decentralized stack of technologies like Ethereum, IPFS, and Golem. Fully decentralized services could be more transparent, cheaper, and more efficient. But let’s be realistic. Today, even the simplest applications of fully decentralized blockchains don’t work as well as we need them to. Cryptokitties offered a glimpse into how a simple viral application can limit the throughput of Ethereum. And don’t forget that these technologies are in some ways still subject to centralization in their current form. Miners form the decentralized consensus layer—and that mining activity is physically centralized in large server farms. The decentralized future is possible. In order to get there, we need to make progress on the low-level tools that such a world will be built upon. This is the realization that today’s guest Karl Floersh had. Karl is a researcher for the Ethereum Foundation. He was initially excited about the prospect of decentralized apps—such as a decentralized Uber. But as he looked more closely at the space, he realized how early we are, and how much work there is to be done on foundational technologies. Proof of Stake is the central topic of discussion in today’s conversation with Karl. Proof of Stake is a consensus mechanism that is an alternative to Proof of Work. In Proof of Work, miners race to validate blocks of transactions. This results in duplicated effort and perhaps wasted energy. In Proof of Stake, validators are chosen to approve transactions. These validators lock up an amount of currency that they are willing to “stake.” If a validator acts badly, the validator will lose their entire stake. This mechanism could be more efficient—and we will explain why that is in this episode. If Proof of Stake works, it could lead to a faster, truly decentralized Ethereum blockchain. That’s a remarkable potential outcome. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#752
Mar 9, 2018 · 59 min
Proof of Stake with Subhan Nadeem
For a decade, Bitcoin’s proof-of-work system has run without disruption. In a proof-of-work scheme, Bitcoin miners compete to solve a cryptographic puzzle associated with a block of transactions. Every ten minutes, all the Bitcoin miner nodes race to be the first to solve a block of transactions. Only one miner wins each block, meaning the other nodes’ time was ultimately wasted. There is also a massive expense of electricity. Bitcoin is a system with low transaction throughput—about 7 transactions per second. Computer scientists have wondered—is there an alternative way of doing consensus? What if we took all the wasted compute power from proof of work, and allocated it in a way that makes transactions get processed faster? But Bitcoin’s governance tends to be extremely conservative. A change to the consensus mechanism probably won’t happen any time soon in Bitcoin. Ethereum’s consensus mechanism is modeled after that of Bitcoin—proof-of-work mining. But Ethereum’s governance ethos is quite different. Ethereum is in the process of planning and implementing proof of stake, an alternative consensus mechanism in which trusted validators are chosen to validate blocks of transactions. Subhan Nadeem is a student at the University of Waterloo where he studies computer science and business. He is the author of several popular articles on Medium that explain blockchain concepts. He joins the show to talk about crypto from the point of a student—and gives us a great walk through of different consensus mechanisms. To find all of our old episodes about cryptocurrencies, check out our apps in the iOS or Android app store. They have all 700 of our episodes, with recommendations, related links, discussions and more. And it’s all open source–if you are looking for an open source project to contribute to, come check us out at github.com/softwareengineeringdaily. We welcome all kinds of contributors–new developers and experts. Engineers and non-technical people. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#751
Mar 8, 2018 · 50 min
How Aragon Manages DAOs with Luis Cuende
Humans organize into groups. There are lots of group types: religions, corporations, national governments, state governments, citizenries, clubs, musical bands. Every group has governance. Governance defines the rules, and the ways that rules change. The United States requires citizens to pay taxes. A corporation requires you to show up to work, but they have to pay you a salary. Most groups today are managed by people. If you break a law, you have to go to court and sit in front of a judge and jury, who decide how you will be punished. If you work at a corporation, and you have a problem with your manager, you go to HR to arbitrate it. These organizations are centralized. There is a governing body who sets the rules. If there is an ambiguity, the person who happens to be in power gets to decide how the ambiguity is resolved. Power is centralized in that governing body. These organizations are run by people. The governance of these organizations is enforced only to the extent that the human government carries out its duties. A decentralized autonomous organization is a group that can run with neither centralized nor human intervention. It is decentralized and autonomous. It is a DAO. Aragon is a platform for running and managing decentralized autonomous organizations. Luis Cuende is the founder of Aragon, and joins the show to explain what a DAO is and why people want to create them. We also talk about the engineering of Aragon and the structure of its ICO—which raised $25m via token sale. Meetups for Software Engineering Daily are being planned! Go to softwareengineeringdaily.com/meetup if you want to register for an upcoming Meetup. In March, I’ll be visiting Datadog in New York and Hubspot in Boston, and in April I’ll be at Telesign in LA. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#750
Mar 7, 2018 · 48 min
Smart Contracts with Raine Revere
Smart contracts are programs that run on the Ethereum blockchain. A smart contract developer pays Ether to deploy the contract. When a contract is deployed, every full node on the Ethereum blockchain has a copy of the contract code in that node’s address space. Every full node needs to hold a copy of every smart contract. This allows every full node to process every call to any smart contract. If you want to call a smart contract, that contract will execute on every full node. When you call a smart contract, you are initiating a transaction. Like Bitcoin transactions, these Ethereum transactions get batched into blocks. Ethereum full nodes compete to solve the cryptographic puzzle associated with a block. But instead of mere financial transactions, these are computational transactions. Raine Revere is a smart contract engineer and cofounder at Maiden and she joins the show to describe smart contract creation and deployment. It’s a great introduction to some Ethereum fundamentals. To find all of our old episodes about cryptocurrencies, check out our apps in the iOS or Android app store. They have all 700 of our episodes, with recommendations, related links, discussions and more. And it’s all open source–if you are looking for an open source project to contribute to, come check us out at github.com/softwareengineeringdaily. We welcome all kinds of contributors–new developers and experts. Engineers and non-technical people. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#749
Mar 6, 2018 · 54 min
Bitcoin’s Future with Joseph Bonneau
Joseph Bonneau is co-author of Bitcoin and Cryptocurrency Technologies, a popular textbook. At NYU, he works as an assistant professor exploring cryptography and security. His YouTube lessons teaching Bitcoin have hundreds of thousands of views. His material offers clear explanations of how Bitcoin works. Since Joseph has a clear understanding of the objective facts around Bitcoin, he is the perfect person to ask about the more subjective topics: the common misunderstandings of Bitcoin; the governance tradeoffs between Ethereum and Bitcoin; proof of work vs. proof of stake. Joseph believes that the early mainstream cryptocurrency solutions will be largely centralized—and that we are likely to move beyond Bitcoin to more efficient currencies. I enjoyed hearing his reasons behind this perspective. Meetups for Software Engineering Daily are being planned! Go to softwareengineeringdaily.com/meetup if you want to register for an upcoming Meetup. In March, I’ll be visiting Datadog in New York and Hubspot in Boston, and in April I’ll be at Telesign in LA. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#748
Mar 5, 2018 · 49 min
Smart Agriculture with Mike Prorock
Farms have lots of data. A corn farmer needs to monitor the chemical composition of soil. A soybean farmer needs to track crop yield. A chicken farmer needs to count the number of eggs produced. If this data is captured, it can be acted upon—for example, a dry farm can automatically turn up its irrigation system. Or the data can simply be studied. If you work in a pure software business, you might take for granted how easy it is to track your metrics. On the farm, you need to use sensors and drones to gather your data. Mike Prorock is the CTO of Mesur.io, a company that makes sensors and software infrastructure for agriculture. He joins the show to describe the use cases for agriculture technology, and the architecture behind it. Today’s episode is a great complement to our recent episodes on streaming data. Mesur.io offers a case study in how streaming systems can be put into practice. Mike will also be speaking at the upcoming Strata Data Conference in San Jose. Meetups for Software Engineering Daily are being planned! Go to softwareengineeringdaily.com/meetup if you want to register for an upcoming Meetup. In March, I’ll be visiting Datadog in New York and Hubspot in Boston, and in April I’ll be at Telesign in LA. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#747
Mar 2, 2018 · 52 min
Dogecoin with Jackson Palmer
Dogecoin was started in 2013 as a joke. Jackson Palmer forked Bitcoin and created his cryptocurrency as a play off the “doge” meme. The currency became popular as a means of reddit users “tipping” each other. If I made a comment on reddit that you liked, you might send me some Dogecoin. This use case allowed people to share the idea of Dogecoin virally, and Dogecoin became valuable, even though the currency did not have any technical properties that made it significantly different than Bitcoin. As Dogecoin was becoming popular, an experienced Internet scam artist took notice and started a Dogecoin exchange called Moolah. Moolah was used to steal money from its customers and investors, and the CEO was arrested. Jackson Palmer was not involved in this scheme, but it soured his feelings about Dogecoin and the entire Bitcoin space. His coin, which had been created as a joke, had been repurposed as a weapon to steal money. Jackson left the Dogecoin community in 2015 to focus on other things. But as Bitcoin entered the mainstream conversation, Jackson has been pulled back into the world of cryptocurrency. Jackson’s YouTube channel has over 20,000 subscribers, who tune into learn about consensus protocols, new tokens, and cryptocurrency news. In today’s episode, Jackson and I discuss his experiences with Dogecoin, and how that compares with the scams around low-quality ICOs that are pulling in retail investors today. We also discuss more positive things–such as proof-of-stake and newer consensus protocols. If you are looking for an internship, apply to the Software Engineering Daily internship, at softwaredaily.com/jobs. And if you are looking to recruit engineers, you can post jobs for your company there as well–it’s completely free to post jobs and to apply. We are hoping to find interns to contribute to the Software Daily open source project–and if you want to see what we are building, go to SoftwareDaily.com or check out our apps in the iOS or Android app store. They have all 650 of our episodes, with recommendations, related links, discussions and more. Also–Meetups for Software Engineering Daily are being planned! Go to softwareengineeringdaily.com/meetup if you want to register for an upcoming Meetup. In March, I’ll be visiting Datadog in New York and Hubspot in Boston, and in April I’ll be at Telesign in LA. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#746
Mar 1, 2018 · 1 hr
Blockchain Scalability with Peter Ullrich
There are two factors that limit the rate at which transactions are accepted into the Bitcoin blockchain: block time and block size. Block time defines how often a new block is appended onto the blockchain. Block size defines how many transactions fit into a new block. As of March 2018, the current block time and block size allow for about 7 transactions per second to be accepted into the Bitcoin blockchain. In today’s episode, we discuss the technical limitations of the Bitcoin blockchain, and some potential solutions to scalability: SegWit and lightning network. Today’s guest is Peter Ullrich, the host of Explain Blockchain. Explain Blockchain is a podcast I have found tremendously useful as I have started to learn about blockchains. He provides thorough, technical explanations of complicated topics, and I recommend subscribing to his show, and listening to the episodes multiple times, because there is a lot of content condensed into a short amount of time. Over the next month, we will be exploring a variety of blockchain-based technologies. Some interviews will be high-level conversations that assume only a familiarity with cryptocurrencies. Some of them will be deeply technical, and assume a strong understanding of Bitcoin and Ethereum. And some episodes, like today’s episode, will be aimed at the developer who is in the process of “going down the rabbit hole.” If you are looking for an internship, apply to the Software Engineering Daily internship, at softwaredaily.com/jobs. And if you are looking to recruit engineers, you can post jobs for your company there as well–it’s completely free to post jobs and to apply. We are hoping to find interns to contribute to the Software Daily open source project–and if you want to see what we are building, go to SoftwareDaily.com or check out our apps in the iOS or Android app store. They have all 650 of our episodes, with recommendations, related links, discussions and more. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#745
Feb 28, 2018 · 54 min
Bitcoin Transactions with Daniel Van Flymen
Bitcoin is an immutable, append-only blockchain ledger that reaches consensus through proof-of-work. The contents of the ledger are financial transactions–people sending and receiving Bitcoin currency to each other. Since Bitcoin, there have been other cryptocurrencies that have similar properties–like Ethereum and the IPFS/Filecoin system. Similar to Bitcoin, they use a decentralized, proof of work based system with a currency reward system–but the ledger being maintained is not purely financial. A currency is a necessary component to maintaining the blockchain’s validity. Over the next month, we will be exploring a variety of blockchain-based technologies. Some interviews will be high-level conversations that assume only a familiarity with cryptocurrencies. Some of them will be deeply technical, and assume a strong understanding of Bitcoin and Ethereum. And some episodes, like today’s episode, will be aimed at the developer who is in the process of “going down the rabbit hole.” If you are finding yourself reading about Bitcoin and Ethereum a few hours every day, but you are still struggling to grasp the basics, this episode is for you. It is meant to be a complement to other introductory resources, such as “Mastering Bitcoin,” by Andreas Antonopoulos. Cryptocurrency systems are revolutionary–they will unlock completely new applications in the very near future. It’s If you are trying to understand these decentralized currencies and applications, the best place to start is with Bitcoin. That’s why I was happy to have Daniel Van Flymen back on the show. Daniel was previously on for one of our most popular episodes–”Blockchain Building,” in which he talked about how useful it can be to build a blockchain based system for practice. Today, Daniel discusses the basics of Bitcoin transactions. What happens when you send money? How are transactions represented on the blockchain? How do full nodes and light clients interact with each other? These are difficult topics to discuss purely over audio, so this episode is best listened to as a companion resource for someone who is studying cryptocurrencies. If you are looking for an internship, apply to the Software Engineering Daily internship, at softwaredaily.com/jobs. And if you are looking to recruit engineers, you can post jobs for your company there as well–it’s completely free to post jobs and to apply. We are hoping to find interns to contribute to the Software Daily open source project–and if you want to see what we are building, go to SoftwareDaily.com or check out our apps in the iOS or Android app store. They have all 650 of our episodes, with recommendations, related links, discussions and more. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#744
Feb 27, 2018 · 46 min
Scale Self-Driving with Alexandr Wang
The easiest way to train a computer to recognize a picture of cat is to show the computer a million labeled images of cats. The easiest way to train a computer to recognize a stop sign is to show the computer a million labeled stop signs. Supervised machine learning systems require labeled data. Today, most of that labeling needs to be done by humans. When a large tech company decides to “build a machine learning model,” that often requires a massive amount of effort to get labeled data. Hundreds of thousands of knowledge workers around the world earn their income from labeling tasks. An example task might be “label all of the pedestrians in this intersection.” You receive a picture of a crowded intersection, and your task is to circle all the pedestrians. You have now created a piece of labeled data. Scale API is a company that turns API requests into human tasks. Their most recent release is an API for labeling data that has been generated from sensors. As self-driving cars emerge onto our streets, the sensors on these cars generate LIDAR, radar, and camera data. The cars will interpret that data in real time using their machine learning models, and then they will send that data to the cloud so that the data can be processed offline to improve the machine learning models of every car on the road. The first step in that processing pipeline is the labeling–which is the focus of today’s conversation. Alexandr Wang is the CEO of Scale, and he joins the show to discuss self-driving cars, labeling, and the company he co-founded. A few notes before we get started. We just launched the Software Daily job board. To check it out, go to softwaredaily.com/jobs. You can post jobs, you can apply for jobs, and it’s all free. If you are looking to hire, or looking for a job, I recommend checking it out. And if you are looking for an internship, you can use the job board to apply for an internship at Software Engineering Daily. Also, Meetups for Software Engineering Daily are being planned! Go to softwareengineeringdaily.com/meetup if you want to register for an upcoming Meetup. In March, I’ll be visiting Datadog in New York and Hubspot in Boston, and in April I’ll be at Telesign in LA. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#743
Feb 26, 2018 · 56 min
Spark and Streaming with Matei Zaharia
Apache Spark is a system for processing large data sets in parallel. The core abstraction of Spark is the resilient distributed dataset (RDD), a working set of data that sits in memory for fast, iterative processing. Matei Zaharia created Spark with two goals: to provide a composable, high-level set of APIs for performing distributed processing; and to provide a unified engine for running complete apps. High-level APIs like SparkSQL and MLlib enable developers to build ambitious applications quickly. A developer using SparkSQL can work interactively with a huge dataset, which is a significant improvement on batch Hive jobs running on Hadoop. A developer training a machine learning model can put the model through multiple steps in the training process without checkpointing the data to disk. The second goal of Spark–a “unified engine for running complete apps”–was the focus of my conversation with today’s guest Matei Zaharia. Matei is the CTO of Databricks, a company that was started to implement his vision for Spark and to build highly usable products on top of the technology. Databricks Delta is a project that combines a data warehouse, data lake, and streaming system–all sitting on top of Amazon S3 and using Spark for processing. In our recent episodes about streaming, we explored some common streaming architectures. A large volume of data comes into the system and is stored in Apache Kafka. Backend microservices and distributed streaming frameworks read that data and store it in databases and data lakes. A data warehouse allows for fast access to the large volumes of data–so that machine learning systems and business analysts can work with data sets interactively. The goal of Databricks Delta is to condense the streaming system, the data lake, and the data warehouse into a single system that is easy to use. If you listened to the previous episodes, you will have an idea for the level of complexity that is involved in managing these different systems. For some companies, it makes complete sense to manage a Kafka cluster, a Spark cluster, a set of S3 buckets, and a data warehouse like Amazon Redshift. But we probably don’t want all of that management to the lowest barrier to entry. Delta will hopefully reduce that barrier to entry and make it easier for enterprises to set up large systems for processing data. A few notes before we get started. We just launched the Software Daily job board. To check it out, go to softwaredaily.com/jobs. You can post jobs, you can apply for jobs, and it’s all free. If you are looking to hire, or looking for a job, I recommend checking it out. And if you are looking for an internship, you can use the job board to apply for an internship at Software Engineering Daily. Also, Meetups for Software Engineering Daily are being planned! Go to softwareengineeringdaily.com/meetup if you want to register for an upcoming Meetup. In March, I’ll be visiting Datadog in New York and Hubspot in Boston, and in April I’ll be at Telesign in LA. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#742
Feb 23, 2018 · 1 hr 1 min
Cloud and Edge with Steve Herrod
Steve Herrod led engineering at VMWare as the company scaled from 30 engineers to 3,000 engineers. After 11 years, he left to become a managing director for General Catalyst, a venture capital firm. Since he has both operating experience and a wide view of the technology landscape as an investor, he is well-equipped to discuss a topic that we have been covering on Software Engineering Daily: the integration of cloud and edge computing. Today, we think of the cloud as a network of large data centers operated by big players like Google, Amazon, and Microsoft. The cloud is where most of the computation across the world takes place. My smartphone and laptop are “edge” devices. They are lightweight computers that don’t perform much complex processing. I would not be able to run a large production database or a 3 terabyte MapReduce job on my laptop. The current division of labor makes sense in this world of smart clouds and low-power, low-bandwidth devices. But the devices are getting cheaper, smarter, and more proliferate. Cars, drones, security cameras, sensors and other devices can serve as points of computation that are geographically between the edge devices and the cloud. With more devices between you and the cloud, there is an opportunity to put computation on those devices. Everyone knows that cloud and edge computing will become intermingled in the coming years. But predicting just how it will play out is nearly impossible. And as an investor, if you bet on something too early, you get the same result as someone who was wrong altogether. A good analogy for the “cloud and edge” space of investments might be the “smart home.” Everyone knows the smart home is coming eventually, but it’s very hard to tell how long it will be before smart home systems are in widespread use–so it is an open question how to invest in the space. Summer internship applications to Software Engineering Daily are also being accepted. If you are interested in working with us on the Software Engineering Daily open source project full-time this Summer, send an application to internships@softwareengineeringdaily.com. We’d love to hear from you. If you haven’t seen what we are building, check out softwaredaily.com, or download the Software Engineering Daily app for iOS or Android. These apps have all 650 of our episodes in a searchable format–we have recommendations, categories, related links and discussions around the episodes. It’s all free and also open source–if you are interested in getting involved in our open source community, we have lots of people working on the project and we do our best to be friendly and inviting to new people coming in looking for their first open source project. You can find that project at Github.com/softwareengineeringdaily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#741
Feb 22, 2018 · 58 min
Serverless Systems with Eduardo Laureano
On Software Engineering Daily, we have been covering the “serverless” movement in detail. For people who don’t use serverless functions, it seems like a niche. Serverless functions are stateless, auto-scaling, event driven blobs of code. You might say “serverless sounds kind of cool, but why don’t I just use a server? It’s a paradigm I’m used to.” Serverless is exciting not because of what it adds but because of what it subtracts. The potential of serverless technology is to someday not have to worry about scalability at all. Today, we take for granted that if you start a new company, you are building it on cloud infrastructure. The problem of maintaining server hardware disappeared for 99% of startups, which unlocked a wealth of innovation. The cloud also simplified scalability for most startups–but there are still plenty of companies that struggle to scale. Significant mental energy is spent on the following questions: How many database replicas do I need? How do I configure my load balancer? How many nodes should I put in my Kafka cluster? Serverless functions are important because they are an auto-scaling component that sits at a low level. This makes it easy to build auto scaling systems on top of them. Auto scaling databases, queueing systems, machine learning tools, and user applications. And since the problem is being solved at such a low level, the pricing competitions will also take place at the low level, meaning that systems built on serverless functions will probably see steep declines in costs in the coming years. Serverless compute could eventually become free or nearly free, with the major cloud providers using it as a loss leader to onboard developers to higher level services. All of this makes for an exciting topic of discussion, that we will be repeatedly covering. Today’s show is with Eduardo Laureano, the principal program manager of Azure Functions. It was a fantastic conversation and we covered applications of serverless, improvements to the “cold start problem,” and how the Azure Functions platform is built and operated. Full disclosure: Microsoft is a sponsor of Software Engineering Daily. Meetups for Software Engineering Daily are being planned! Go to softwareengineeringdaily.com/meetup if you want to register for an upcoming Meetup. In March, I’ll be visiting Datadog in New York and Hubspot in Boston, and in April I’ll be at Telesign in LA. Summer internship applications to Software Engineering Daily are also being accepted. If you are interested in working with us on the Software Engineering Daily open source project full-time this Summer, send an application to internships@softwareengineeringdaily.com. We’d love to hear from you. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#740
Feb 21, 2018 · 1 hr
Cloud Foundry Overview with Mike Dalessio
Earlier this year we did several shows about Cloud Foundry, followed by several shows about Kubernetes. Both of these projects allow you to build scalable, multi-node applications–but they serve different types of users. Cloud Foundry encompasses a larger scope of the application experience than Kubernetes. Kubernetes is lower level, and is actually being used within newer versions of Cloud Foundry to give Cloud Foundry users access to the Kubernetes abstractions. Recording those shows gave me a wide understanding of how infrastructure is managed and how it has evolved. Today’s episode provides more context on Cloud Foundry–how the project got started, how people use it, and where Cloud Foundry is going. Today’s guest Mike Dalessio is a VP of engineering on Pivotal Cloud Foundry, and we had a great time talking about his work. Engineering leadership is a fine art, and conversations with engineering leaders are consistently interesting–this was no exception. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#739
Feb 20, 2018 · 59 min
Kafka Design Patterns with Gwen Shapira
Kafka is at the center of modern streaming systems. Kafka serves as a database, a pubsub system, a buffer, and a data recovery tool. It’s an extremely flexible tool, and that flexibility has led to its use as a platform for a wide variety of data intensive applications. Today’s guest is Gwen Shapira, a product manager at Confluent. Confluent is a company that was started by the creators of Apache Kafka–Jay Kreps, Neha Narkhede, and Jun Rao, who have all been on the show in previous episodes. In those shows, we discussed the inner workings of Kafka. This episode is more about practical use cases and design patterns. Gwen explores a few use cases. First, reconciling data between different servers. A massive, international, multi-user game like World of Warcraft needs to keep its users in sync despite the fact that those users are pinging different server locations. Kafka can help reconcile data between the multiple servers. This discussion reminded me of the awesome show we did with Yan Cui about scalable multiplayer games. Other examples we discussed include log management, data enrichment, and large scale analytics. It was a great conversation and I think you will enjoy it as well. Summer internship applications to Software Engineering Daily are also being accepted. If you are interested in working with us on the Software Engineering Daily open source project full-time this Summer, send an application to internships@softwareengineeringdaily.com. We’d love to hear from you. If you haven’t seen what we are building, check out softwaredaily.com, or download the Software Engineering Daily app for iOS or Android. These apps have all 650 of our episodes in a searchable format–we have recommendations, categories, related links and discussions around the episodes. It’s all free and also open source–if you are interested in getting involved in our open source community, we have lots of people working on the project and we do our best to be friendly and inviting to new people coming in looking for their first open source project. You can find that project at Github.com/softwareengineeringdaily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#738
Feb 19, 2018 · 56 min
Streaming Architecture with Ted Dunning
Streaming architecture defines how large volumes of data make their way through an organization. Data is created at a user’s smartphone, or on a sensor inside of a conveyor belt at a factory. That data is sent to a set of backend services that aggregate the data, organizing it and making it available to business analysts, application developers, and machine learning algorithms. The velocity at which data is created has led to widespread use of the “stream” abstraction–a never ending, append-only array of data. To deal with this volume, streams need to be buffered, batched, cached, mapreduced, machine learned, and munged until they are in a state where they can provide value to the end user. There are numerous ways that data can travel this path, and in today’s episode we discuss the streaming systems, data lakes, and data warehouses that can be used to build an architecture that makes use of streaming data. Ted Dunning is a chief application architect at MapR, and he joins the show to discuss the patterns that engineering teams are using to build modern streaming architectures. Full disclosure: MapR is a sponsor of Software Engineering Daily. Meetups for Software Engineering Daily are being planned! Go to softwareengineeringdaily.com/meetup if you want to register for an upcoming Meetup. In March, I’ll be visiting Datadog in New York and Hubspot in Boston, and in April I’ll be at Telesign in LA. Summer internship applications to Software Engineering Daily are also being accepted. If you are interested in working with us on the Software Engineering Daily open source project full-time this Summer, send an application to internships@softwareengineeringdaily.com. We’d love to hear from you. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#737
Feb 16, 2018 · 53 min
Streaming Analytics with Scott Kidder
When you go to a website where a video is playing, and your video lags, how does the website know that you are having a bad experience? Problems with video are often not complete failures–maybe part of the video loads, and plays just fine, and then the rest of the video is buffering. You have probably experienced sitting in front of a video, waiting for it to load as the loading wheel mysteriously spins. Since problems with video are often not complete failures, troubleshooting a problem with a user’s video playback is not as straightforward as just logging whenever a crash occurs. You need to continuously monitor the video playback on every client device and aggregate it in a centralized system for analysis. The centralized logging system will allow you to separate problems with a specific user from problems with the video service itself. A single user could have bad wifi, or have 50 tabs open with different videos. To identify problems that are caused by the video player rather than the user, you need to capture the playback from every video and every user. Scott Kidder works at Mux, where he builds a streaming analytics system for video monitoring. In this episode, Scott explains how events make it from a video player onto the backend analytics system running on Kinesis and Apache Flink. Events from the browser are constantly added to Kinesis (which is much like Kafka). Apache Flink reads those events off of Kinesis and map reduces them to discover anomalies. For example, if 100 users watch a 20 minute cat video, and the video stops playing at minute 12 for all 100 users, there is probably some data corruption in that video. You would only be able to discover that by assessing all users. Scott and I discussed the streaming infrastructure that he works on at Mux, as well as other streaming systems like Spark, Apache Beam, and Kafka. This episode is the first in a short series about streaming data infrastructure. I wanted to do some shows in preparation for Strata Data conference in March in San Jose, which I will be attending thanks to a complimentary ticket from O’Reilly. O’Reilly has been kind enough to give me free tickets since Software Engineering Daily started and did not have the money to attend any conferences. If you want to attend Strata, you can use promo code PCSED to get 20% off. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#736
Feb 15, 2018 · 58 min
Streaming Architecture with Tugdual Grall
Tugdual Grall is an engineer with MapR. In today’s episode, we explore use cases and architectural patterns for streaming analytics. Full disclosure: MapR is a sponsor of Software Engineering Daily. In past shows, we have covered data engineering in detail–we’ve looked at Uber’s streaming architecture, talked to Matei Zaharia about the basics of Apache Spark, and explored the history of Hadoop. To find all of our episodes about data engineering , download the Software Engineering Daily app for iOS or Android. These apps have all 650 of our episodes in a searchable format–we have recommendations, categories, related links and discussions around the episodes. It’s all free and also open source–if you are interested in getting involved in our open source community, we have lots of people working on the project and we do our best to be friendly and inviting to new people coming in looking for their first open source project. You can find that project at Github.com/softwareengineeringdaily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#735
Feb 14, 2018 · 42 min
Machine Learning Deployments with Kinnary Jangla
Pinterest is a visual feed of ideas, products, clothing, and recipes. Millions of users browse Pinterest to find images and text that are tailored to their interests. Like most companies, Pinterest started with a large monolithic application that served all requests. As Pinterest’s engineering resources expanded, some of the architecture was broken up into microservices and Dockerized, which make the system easier to reason about. To serve users with better feeds, Pinterest built a machine learning pipeline using Kafka, Spark, and Presto. User events are generated from the frontend, logged onto Kafka, and aggregated to build machine learning models. These models are deployed into Docker containers much like the production microservices. Kinnary Jangla is a senior software engineer at Pinterest, and she joins the show to talk about her experiences at the company–breaking up the monolith, architecting a machine learning pipeline, and deploying those models into production. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Showing 1,521–1,540 of 2,221 episodes