NewsTech News

Software Engineering Daily

softwareengineeringdaily.com

Software Daily: The world through the lens of software.

Play

Open in

2,221 episodes
Avg 53 min
English

Episodes 2,221

Newest Oldest Longest

All20 Unplayed20 In progress0 Played0

Counted on this page — what you have heard stays on this device, so it is not something the list can be paged by.

#694
Dec 20, 2017 · 1 hr 3 min
Managing Engineers with Ron Lichty
“Management is about human beings. Its task is to make people capable of joint performance. To make their strengths effective and their weaknesses irrelevant.” That quote is from Peter Drucker. It is one of the many useful quotes collected in Ron Lichty’s book “Managing the Unmanageable”—and it illustrates why we work in teams. When we collaborate with each other, we make each other’s strengths effective, and our weaknesses become irrelevant. To collaborate effectively, we need leaders. We need management. Ron Lichty spent 6 years managing engineers at Apple, and many more years in management and director roles elsewhere. In his book, Ron lays out the lessons he learned in 30 years of engineering management. Ron also describes concrete strategies for how to manage engineers productively. An engineer who becomes a manager needs to learn new skills. And the hardest skills to master have nothing to do with technology. Prioritizing the right projects, allocating engineering resources, making architectural decisions—all of those skills are important. But the art of relationships—of diplomacy and language—is harder to learn than any technical skill. How do you motivate an engineer to do something that is boring? How do you have a difficult conversation with an engineer who needs to improve? When a conflict between engineers comes up, do you confront the conflict head-on, or do you wait for those engineers to resolve it among themselves? These questions do not have easy answers. The best way to learn how to react to these situations is to live through them. The second best way to learn is to read and listen to people who have seen so much of the management dynamic that they can distill it into anecdotes and aphorisms. In today’s show, Ron shares several stories that changed how I think about management. Ron and I did not have time to discuss everything I wanted to, and I recommend checking out his podcast episode on Software Engineering Radio for more detail. And also check out his book—Managing the Unmanageable. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#693
Dec 19, 2017 · 1 hr 1 min
Hacker Noon with David Smooke
The New York Times makes most of its money off of subscriptions. Facebook makes its money off of native advertising. Hacker News is funded by Y-Combinator. Each of these business models creates biases in the information that gets promoted on the respective platforms. This is why I like to know the origin story and the business models behind the publications that I read. Published content is shaped by the profit motive of the publication. And yet, last month, I repeatedly found myself reading high quality content on a Medium publication that I did not know the origin of: Hacker Noon. Hacker Noon is a popular Medium publication that syndicates curated content written about software. Let me explain “syndication.” Imagine that I just spent three days on a Medium post about functional programming, and I have zero followers on social media. How can I get people to read awesome post? The answer is syndication. I can submit my Medium post to Hacker Noon. This gives me free distribution, and it gives Hacker Noon free content—a win-win relationship. But why was it worth it for Hacker Noon to spend time curating content? That syndication process takes time. You have to read through lots of submissions, sometimes you have to send it back to the author to have it edited. And this is all to build a following on Medium. I have not heard of Medium being a profitable platform to build a business. It’s worth pointing out the difference between Medium and WordPress. On WordPress, this model of curated syndication has worked to massive success—for example, the Huffington Post and TechCrunch. These businesses make millions of dollars from advertising networks, because they are built on WordPress, and WordPress is an open model. A publisher on WordPress can install plugins that serve ads from third party providers like Outbrain and Taboola. A WordPress site can also install any kind of data collection scripts, to gather data on visitors, and sell it to the highest bidder. The lack of third party plugins is the blessing and the curse of Medium. Because there is no third party ecosystem, reading content on Medium is a beautiful experience. The page loads quickly and predictably. There are no random scripts that are blocking the page as they hog your browser’s resources. When you go to close the page, there is never a popup that asks you to subscribe to a newsletter. When I read content on Medium, I am not getting slapped across the face with ads for reverse mortgages and açaí berries. I am not being tagged for retargeting. It’s a beautiful experience. But Medium seems like an ecosystem that would not allow for the content syndication business like Hacker Noon. I wanted to know who was running Hacker Noon, how the business works, and what it says about Medium as a publishing platform. Hacker Noon turns out to be part of a network of Medium publications called AMI. AMI’s network includes sites like Art + Marketing, Future Travel, and Fit Yourself Club–all of which are distinct syndication platforms. David Smooke is the CEO of AMI, and he joins this episode to explain how his business works, how he has scaled the content syndication business, and why he is betting on Medium. It was a detailed look into the state of online publishing and where it might be headed. If you don’t read Hacker Noon already, one article to start with that shows off the quality of content is Learn Blockchains By Building One. I interviewed the author of that article, Daniel Van Flymen, and it has been one of the most popular episodes of Software Engineering Daily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#692
Dec 18, 2017 · 55 min
Protocol Buffers with Kenton Varda
When engineers are writing code, they are manipulating objects. You might have a user object represented on your computer, and that user object has several different fields—a name, a gender, and an age. When you want to send that object across the network to a different computer, the object needs to be turned into a sequence of 1s and 0s that will travel efficiently across the network. This is known as “serialization.” As the user object sits on your computer, it is represented in 1s and 0s. You could just send that same representation over the wire. But we use efficient serialization to send it over the network in a more compact format. We also have to make sure that when we send that object to another service, the other service knows how to deserialize it, and turn it back into a format that we can operate on at the application level. Protocol buffers are a serialization protocol that originated at Google. Protocol buffers created a standardized interface for efficiently passing data between services. When Kenton Varda worked at Google, he was the tech lead for protocol buffers, and he joins the show to explain how protobufs work—and a newer serialization protocol that Kenton led: Cap’n Proto. You can expect to walk away from this episode with an understanding of how serialization protocols work, and the design tradeoffs you can make when creating a serialization protocol. We also touched on a startup that Kenton founded, called Sandstorm, and how he eventually found himself at Cloudflare, where he works on Cloudflare workers. With these topics, we did not go as deep as I would have liked, and I look forward to having Kenton back on in the near future. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#691
Dec 15, 2017 · 58 min
High Volume Logging with Steve Newman
Google Docs is used by millions of people to collaborate on documents together. With today’s technology, you could spend a weekend coding and build a basic version of a collaborative text editor. But in 2004 it was not so easy. In 2004 Steve Newman built a product called Writely, which allowed users to collaborate on documents together. Initially, Writely was hosted on a single server that Steve managed himself. All of the reads and writes to the documents went through that single server. Writely rapidly grew in popularity, and Steve went through a crash course in distributed systems as he tried to keep up with the user base. In 2006, Writely was acquired by Google, and Steve spent his next four years turning Writely into Google Docs. Eventually he moved onto other projects within Google—“Cosmo” and “Megastore Replication.” When Steve left the company in 2010, he took with him the lessons of logging and monitoring that keep Google’s infrastructure observable. Large organizations have terabytes of log data to manage. This data streams off the servers that are running our applications. That log data gets processed in a “metrics pipeline” and turned into monitoring data. Monitoring data aggregates log data in a more presentable format. Most of the log messages that get created will never be seen with human eyes. These logs get aggregated into metrics, then compressed, and (in many cases) eventually thrown away. Different companies have different sensitivity around their logs, so some companies may not garbage collect any of their logs! When a problem occurs in our infrastructure, we need to be able to dig into our terabytes of log data and quickly find the root cause of a problem. If our log data is compressed and stored on disk, it will take longer to access it. But if we keep all of our logs in memory, it could get expensive. To review: if I want to build a logging system from scratch today I need to build: a metrics pipeline for converting log data into monitoring data; a complicated caching system, a way to store and compress logs; a query engine that knows how to ask questions to the log storage system; a user interface so I don’t have to inspect these logs via command line… The list of requirements goes on and on—which is why there is a huge industry around log management. And logging keeps evolving! One example we covered recently is distributed tracing, which is used to diagnose requests that travel through multiple endpoints. After Steve Newman left Google, he started Scalyr, a product that allows developers to consume, store, and query log messages. I was looking forward to talking to Steve about data engineering, and the query engine that Scalyr has architected, but we actually spent most of our conversation talking about the early days of Writely, and his time at Google—particularly the operational challenges of Google’s infrastructure. Full disclosure: Scalyr is a sponsor of Software Engineering Daily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#690
Dec 14, 2017 · 46 min
Scala at Duolingo with Andre Kenji Horie
Duolingo is a language learning platform with over 200 million users. On a daily basis millions of users receive customized language lessons targeted specifically to them. These lessons are generated by a system called the session generator. Andre Kenji Horie is senior engineer at Duolingo. He wrote about the process of rewriting the session generator, moving from Python to Scala and changing architecture at the same time. In this episode Adam Bell talks with him about the reasons for the rewrite, what drove them to move to Scala and the experience of moving from one technology stack to another. Rewriting Doulingo’s Engine in Scala Jobs at Duolingo Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#689
Dec 13, 2017 · 55 min
Engineering Values with Lynne Tye
The values system of a company guides the actions of the engineers who work at that company. Some companies value open communication and a flat organization where anybody can talk to anyone else. Other companies encourage hierarchy and secrecy, so that employees are focused on their specific section of the company. Some companies take themselves seriously, and have a work environment that is as stoic as the military. Other companies pride themselves on having good beer and a friendly, laid back atmosphere. When company values are properly defined, the values can be used as reference points when making decisions. At Amazon, one of the core company values is “bias for action.” As an engineer, you are often in a situation where you can wait for more information, or you can start a project with an incomplete picture for how you will finish it. The “bias for action” lets you know that you should usually start the project despite having an incomplete picture. Another use of a company values system is for hiring. When a company publishes their values, prospective employees can use those stated values as a way to know if they would be a good cultural fit. For example “move fast and break things” was a value that allowed Facebook to ship new products faster than any other company before it. But the speed of movement is not for everyone. Some engineers like to have their code unit tested, and free of all bugs before shipping to production. Every company has values that define their company. And every engineer has values that define how they want to work. Lynne Tye started her company Key Values as a platform to index companies by their values systems. This allows engineers to find companies that are a good cultural fit for their values system. Lynne joins the show today to explain how engineers and companies define their values systems, and how that affects the outcomes of engineering organizations. Lynne also talks about her time at HomeJoy, one of the first companies in the “gig economy”. HomeJoy was an on-demand house cleaning service that grew extremely fast, but ultimately went under due to lawsuits. The challenges of HomeJoy were a predictor of the challenges later faced by Uber and Airbnb, and it was fascinating to hear Lynne reflect on her time spent managing operations at HomeJoy–which was about as operationally intensive a company as you can imagine! Thanks to Courtland Allen for the intro to Lynne, and if you haven’t checked out the Indie Hackers podcast, which is hosted by Courtland, you should subscribe to it. Indie Hackers breaks down the engineering and business models behind small software companies–it’s one of my favorite shows. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#688
Dec 12, 2017 · 59 min
Cloud Marketplace with Zack Bloom
Ten years ago, if you wanted to build software, you probably needed to know how to write code. Today, the line between “technical” and “non-technical” people is blurring. Website designers can make a living building sites for people on WordPress or Squarespace–without knowing how to write code. Salesforce integration experts can help a sales team set up complicated software–without knowing how to write code. Shopify experts can set up an ecommerce store to your exact specifications–without knowing how to write code. WordPress, Squarespace, Salesforce, and Shopify are all fantastic services–but they are not compatible with each other. I can’t install a WordPress plugin on Salesforce. Now imagine this from the point of view of plugin creators. Plugin creators make easy ways to integrate different pieces of software together. Take PayPal as an example. PayPal wants to make it easy for software builders to integrate with their API. One plugin that PayPal has is a button that says “Pay with PayPal.” If I am a developer at PayPal, and I am building a button that people should be able to easily put on their webpage so that their users can pay with PayPal, I have to create a button that is compatible with WordPress, and Squarespace, and Wix, and Weebly, and GoDaddy, and Blogger, and all the other website builders that I might want to integrate with. In 2014, Zack Bloom started a company called Eager. Eager was a cloud app marketplace which allowed app developers to make flexible plugins that non-technical users could drag and drop into their site without technical expertise. In order for these non-technical users to add any apps from the Eager marketplace to their webpage, they had to drop in a line of JavaScript–which is, unfortunately, a significant hurdle for a nontechnical user. Eager proved to be a useful distribution mechanism for plugin developers who could write a plugin once and get distributed to multiple plugin marketplaces. But Eager was not as widely used as a way to directly drag and drop plugins onto sites. The question was: how do you build a marketplace for non-technical users to add plugins to any website without forcing the non-technical user to write code? How do you make editing any website as easy as a WYSIWYG editor? The CDN turns out to be the perfect distribution platform for these kinds of apps. Users already integrate with a CDN, so the CDN can do the work of inserting the code that allows the plugins to be added to a user’s webpage. Because of the opportunity for the integration between a plugin marketplace and a CDN, Eager was acquired by Cloudflare, and Eager became Cloudflare apps. Zack Bloom joins the show today to discuss the motivations for his company, the engineering behind building a cloud app marketplace, and the acquisition process of his company Eager. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#687
Dec 11, 2017 · 1 hr 9 min
Scalable Multiplayer Games with Yan Cui
Remember when the best game you could play on your phone was Snake? In 1998, Snake was preloaded on Nokia phones, and it was massively popular. That same year Half-Life won game of the year on PC. Metal Gear Solid came out for Playstation. The first version of Starcraft also came out in 1998. In 1998, few people would have anticipated that games with as much interactivity as Starcraft would be played on mobile phones twenty years later. Today, mobile phones have the graphics and processing power of a desktop gaming PC from two decades ago. But one thing still separates desktop gaming from mobile gaming: the network. With desktop gaming, users have a reliable wired connection that keeps their packets moving over the network with speeds that let them compete with other users. With mobile gaming, the network can be flaky. How do we architect real-time strategy games that can be played over an intermittent network connection? Yan Cui is an engineer at Space Ape Games, a company that makes interactive multiplayer games for mobile devices. In a previous episode, Yan described his work re-architecting a social networking startup where the costs had gotten out of control. Yan has a skill for describing software architecture and explaining the tradeoffs. When architecting a multiplayer mobile game, there are many tradeoffs to consider. What do you build and what do you buy? Do you centralize your geographical deployment to make it easier to reconcile conflicts, or do you spread your server deployment out globally? What is the interaction between the mobile clients and the server? The question of interaction between client and server for a mobile game has lessons that are important for anyone building a highly interactive mobile application. For example, think about Uber. When I make a request for a car, I can look at my phone and see the car on the map, slowly approaching me. The driver can look at his phone and see if I move across the street. This is accomplished by synchronizing the data from the driver’s phone and my phone in a centralized server, and sending the synchronized state of the world out to me and the driver. How much data does the centralized server need to get from the mobile phones? How often does it need to make those requests? The answers to these questions will vary based on bandwidth, device type, phone battery life, and other factors. There are similar problems in mobile game engineering, when users are in different players on a virtual map. They are fighting each other, trying to avoid enemies, trying to steal power ups from each other. Mobile games can be even more interactive than a ridesharing app like Uber, so the questions of data synchronization can be even harder to answer. On Software Engineering Daily, we have explored the topic of real-time synchronization in our past shows about the infrastructure of Uber and Lyft. To find these old episodes, you can download the Software Engineering Daily app for iOS and for Android. In other podcast players, you can only access the most recent 100 episodes. With these apps, we are building a new way to consume content about software engineering. They are open-sourced at github.com/softwareengineeringdaily. If you are looking for an open source project to get involved with, we would love to get your help. Yan Cui’s new video course: AWS Lambda in Motion Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#686
Dec 8, 2017 · 1 hr 18 min
Decentralized Objects with Martin Kleppman
The Internet was designed as a decentralized system. Theoretically, if Alice wants to send an email to Bob, she can set up an email client on her computer and send that email to Bob’s email server on his computer. In reality, very few people run their own email servers. We all send our emails to centralized services like Gmail, and connect to those centralized services using our own client—a browser on our laptop or a mobile application on our smart phone. Gmail is popular because nobody wants to run their own email server—it’s too much work. With Gmail, our emails our centralized, but centralization comes with convenience. Similar centralization happened with online payments. Decentralization is a desirable feature of computer systems. So how do we make more of our applications decentralized? Martin Kleppman is a distributed systems researcher and the author of Data Intensive Applications. Martin is concerned by the centralization of our computer networks, and he works on CRDT technology in order to make it easier for people to build peer-to-peer applications. Most of the people who know how to build systems with CRDTs are distributed systems PhDs, database experts, and people working at huge internet companies. How do you make developer-friendly CRDTs? How do you allow random hackers to build peer-to-peer applications that avoid conflicts? Start by making a CRDT out of the most widely used, generalizable data structure in modern application development: the JSON object. In today’s episode, Martin and I talk about conflict resolution, CRDTs, and decentralized applications. This is Martin’s second time on the show, and his first interview is the most popular episode to date. You can find a link to that episode in the show notes for this episode, or you can find it in the Software Engineering Daily app for iOS and for Android. In other podcast players, you can only access the most recent 100 episodes. With these apps, we are building a new way to consume content about software engineering. They are open-sourced at github.com/softwareengineeringdaily. If you are looking for an open source project to get involved with, we would love to get your help. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#685
Dec 7, 2017 · 43 min
Serverless Applications with Randall Hunt
Developers can build networked applications today without having to deploy their code to a server. These “serverless” applications are constructed from managed services and functions-as-a-service. Managed services are cloud offerings like database-as-a-service, queueing-as-a-service, or search-as-a-service. These managed services are easy to use. They take care of operational burdens like scalability and outages. But managed services typically solve a narrow use case. You can’t build an application entirely out of managed services. Managed services are scalable and narrow. Functions-as-a-service are scalable and flexible. With managed services, you make remote calls to a service with a well-defined API. With functions-as-a-service, you can deploy your own code. But functions-as-a-service execute against transient, unreliable compute resources. They aren’t a good fit for low latency computation, and the code you run on them should be stateless. Managed services and functions-as-a-service are the perfect complements. Managed services provide you with well-defined server abstractions that every application needs—like databases, search indexes, and queues. Functions as a service offer flexible “glue code” that you can use to create custom interactions between the managed services. The term “serverless” is used to describe the applications that are built entirely with managed services and functions as a service. Serverless applications are dramatically simpler to build and easier to operate than cloud applications of the past. The costs of managed services can get expensive, but the costs of functions as a service can cost 1/10th of what it might take to run a server that is handling your requests. Whether the size of your bill will increase or decrease as your company becomes “serverless” is less of an issue than the fact that your employees will be more productive: serverless applications have less operational burden, so developers spend more time architecting and implementing software. It has been 5 years since the Netflix infrastructure team was talking about the aspirational goal of a “no-ops” software culture. Your software should be so well-defined that you do not need regular intervention of ops staff to reboot your servers and reconfigure your load balancers. Serverless is a newer way of moving operational expense into capital expense. Today’s guest Randall Hunt is a senior technical evangelist with Amazon Web Services. He travels around the world meeting developers and speaking at conferences about AWS Lambda, the functions as a service platform from Amazon. Randall has given some excellent talks about how to architect and build serverless applications (which I will add to the show notes), and today we explore those application patterns further. Serverless Services – Randall Hunt Randall Hunt at AWS Summit Seoul Serverless, What is it Good For? Randall Hunt Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#684
Dec 6, 2017 · 1 hr 13 min
Data Science Mindset with Zacharias Voulgaris
A company’s approach to data can make or break the business. In the past, data was static. There was not much data, it sat in Excel, and it was interacted with on a nightly or monthly basis. Now, data is dynamic, real time and huge. To tap into available data, many industries have oriented themselves to becoming data intensive. With many new industry sectors becoming data driven, a new field called data science emerged. As a new field, data science has attracted a lot of attention from professionals with diverse backgrounds. Describing what is data science and who is a data scientist is not easy. As technologies surrounding the field continue to evolve and new verticals are added, the discourse surrounding the field has attracted different voices putting forward their definition of the field. In this episode, Zacharias Voulgaris joins guest host Sid Ramesh to discuss the developments in the field. He is the author of several data science books, and in today’s conversation Zacharias explains what he means by the data science mindset–including trends and misconceptions that people have on the field. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#683
Dec 5, 2017 · 1 hr
Secure Authentication with Praneet Sharma
When I log into my bank account from my laptop, I first enter my banking password. Then the bank sends a text message to my phone with a unique code, and I enter that code into my computer to finish the login. This login process is two-factor authentication. I am proving my identity by entering my banking password (the first factor) and validating that I am in control of my phone (the second factor) by receiving that text message. But in order to log in from my laptop, I need to be in control of my laptop. The laptop itself is a factor. With the laptop and my password, I have two factors. I might not actually need the phone as a factor. Praneet Sharma is the CEO of Keyless, a product that moves 2-factor authentication into the browser. Praneet joins the show to discuss how all kinds of authentication work: multi-factor authentication, single sign on, and Yubikey. We use this discussion of authentication methods to help explain why it actually could make sense for some people to be doing 2-factor authentication without requiring people to take out their phone. We also explore recent security breaches like Target, Equifax and Yahoo–and the industry of security software sold to developers. I see giant banners for security software companies every time I go into the San Francisco airport, and Praneet explained to me some of the products that these kinds of companies are selling. Praneet has joined the show in a previous episode to talk about advertising fraud. He also works with Shailin Dhar at Method Media Intelligence. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#682
Dec 4, 2017 · 1 hr 7 min
Serverless Scheduling with Rodric Rabbah
Functions as a service are deployable functions that run without an addressable server. Functions as a service scale without any work by the developer. When you deploy a function as a service to a cloud provider, the cloud provider will take care of running that function whenever it is called. You don’t have to worry about spinning up a new machine and monitoring that machine, and spinning the machine down once it becomes idle. You just tell the cloud provider that you want to run a function, and the cloud provider executes it and returns the result. Functions as a service can be more cost effective than running virtual machines or containerized infrastructure, because you are letting the cloud provider decide where to schedule your function, and you are giving the cloud provider flexibility on when to schedule the function. The developer experience for deploying a serverless function can feel mysterious. You send a blob of code into the cloud. Later on, you send a request to call that code in the cloud. The result of the execution of that code gets sent back down to you. What is happening in between? Rodric Rabbah is the principal researcher and technical lead in serverless computing at IBM. He helped design IBM Cloud Functions, the open source functions-as-a-service platform that IBM has deployed and operationalized as IBM Cloud Functions. Rodric joins the show to explain how to build a platform for functions as a service. When a user deploys a function to IBM Cloud Functions, that function gets stored in a database as a blob of text, waiting to be called. When the user makes a call to the function, IBM Cloud Functions takes it from the database and queues the function in Kafka, and eventually schedules the function onto a container for execution. Once the function has executed, IBM Cloud Functions stores the result in a database and sends that result to the user. When you execute a function, the time spent scheduling it and loading it onto a container is known as the “cold start problem”. The steps of executing a serverless function take time, but the resource savings are significant. Your code is just stored as a blob of text in a database, rather than sitting in memory on a server, waiting to execute. In his research for building IBM Cloud Functions, Rodric wrote about some of the tradeoffs for users who build applications with serverless functions. The tradeoffs exist along what Rodric calls “the serverless trilemma.” In today’s episode, we discuss why people are using functions-as-a-service, the architecture of IBM Cloud Functions, and the unsolved challenges of building a serverless platform. Full disclosure: IBM is a sponsor of Software Engineering Daily. OpenWhisk Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#681
Dec 1, 2017 · 51 min
Animating VueJS with Sarah Drasner
Most user interfaces that we interact with are not animated. We click on a button, and a form blinks into view. We click a link and the page abruptly changes. On the other hand, when we interact with an application that has animations, we can feel the difference. The animations are often subtle. If you aren’t sure what I’m talking about, pay attention the next time you use Slack or Facebook Messenger or iMessage. Airbnb values animation so much that they built Lottie, a library for animation. In an animated application, the user interface feels alive. When a software team takes the time to build animations into small interactions, the user perceives the animations as polish and attention to detail. Sarah Drasner has been evangelizing the value of animations for years, and she is an expert at implementing complex and beautiful animations on the web. She works at Microsoft as a developer advocate and joins the show to talk about how to build animations. If you are building a web application and want to create a unique UI, you might find this show useful. JavaScript supports detailed animations, often through the manipulation of SVG files. SVG stands for “scalable vector graphics” a file format that represents an image in its own DOM. SVG is so flexible because of this DOM format, which defines the different parts of the SVG. This is in contrast to a bitmap, which is just a simple matrix of dots, without any rich metadata. You could manipulate SVG with raw JavaScript—but most people use a frontend JavaScript framework like React, Angular, or VueJS. Sarah has been implementing most of her recent web animations using Vue, and she is a member of the Vue core team. Vue has an entertaining story, because it gained popularity in a time when Google was supporting AngularJS and Facebook was supporting ReactJS. The first version of Vue was created from scratch by a single developer, Evan You. If you are a Vue developer looking for an open source project to hack on, you can check out softwaredaily.com, which is an open source platform to consume content about software. In addition to the Vue web app, we also have the Software Engineering Daily app for iOS and for Android. All of these apps are open-sourced at github.com/softwareengineeringdaily. If you are looking for an open source project to get involved with, we would love to get your help. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#680
Nov 30, 2017 · 55 min
React and GraphQL at New York Times
Are we a media company or a technology company? Facebook and the New York Times are both asking themselves this question. Facebook originally intended to focus only on building technology–to be a neutral arbiter of information. This has turned out to be impossible. The Facebook newsfeed is defined by algorithms that are only as neutral as the input data. Even if we could agree on a neutral data set to build a neutral newsfeed, the algorithms that generate this news feed are not public, so we have no way to vet their neutrality. Facebook is such a powerful engine for distribution, it has allowed for a rise in the number of publishers who can get their voice heard. As a result, large media companies have lost market share because Facebook has replaced their distribution. The New York Times has always been a media company–but the standards for media consumption have shot up. Millions of people produce content for free, and that content is distributed through high quality experiences like Twitter, YouTube, Medium, and Facebook. When a page takes too long to load on NewYorkTimes.com, it doesn’t matter how good the content is–the user is going to navigate away before they read anything. Today, the New York Times has built out an experienced engineering team. In a previous episode, we reported how the Times uses Kafka to make its old content more accessible. In today’s show, we talk about how the Times uses React and GraphQL to improve the performance and the developer experience of engineers who are building software at the New York Times. Scott Taylor and James Lawrie are software engineers at the New York Times. In this episode, they explain how the New York Times looks at technology. The user experience on New York Times rivals that of a platform company like Facebook, and this is assisted by technologies originally built at Facebook: React, Relay, and GraphQL. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#679
Nov 29, 2017 · 1 hr 3 min
How IBM Runs Its Cloud with Jason McGee
Functions as a service let developers deploy stateless application logic that is cheap and scalable. Functions as a service still have some problems to overcome in the areas state management, function composition, usability, and developer education. Kubernetes is a tool for managing containerized infrastructure. Developers put their apps into containers on Kubernetes, and Kubernetes provides a control plane for deployment, scalability, load balancing, and monitoring. So–all of the things that you would want out of a managed service become much easier when you put applications into Kubernetes. This is why Kubernetes has become so popular–and it is why Kubernetes itself is being offered as a managed service by many cloud providers–including IBM. For the last decade, IBM has been building out its cloud offerings–and for two of those years, Jason McGee has been CTO of IBM Cloud Platform. In this episode, Jason discusses what it is like to build and manage a cloud, from operations to economics to engineering. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#678
Nov 28, 2017 · 44 min
Thumbtack Infrastructure with Nate Kupp
Thumbtack is a marketplace for real-world services. On Thumbtack, people get their house painted, their dog walked, and their furniture assembled. With 40,000 daily marketplace transactions, the company handles significant traffic. On yesterday’s episode, we explored how one aspect of Thumbtack’s marketplace recently changed, going from asynchronous matching to synchronous “instant” matching. In this episode, we zoom out to the larger architecture of Thumbtack, and how the company has grown through its adoption of managed services from both AWS and Google Cloud. The word “serverless” has a few definitions. In the context of today’s episode, serverless is all about managed services like Google BigQuery, Google Cloud PubSub, and Amazon ECS. The majority of infrastructure at Thumbtack is built using services that automatically scale up and down. Application deployment, data engineering, queueing, and databases are almost entirely handled by cloud providers. For the most part, Thumbtack is a “serverless” company. And it makes sense–if you are building a high-volume marketplace, you are not in the business of keeping servers running. You are in the business of improving your matching algorithms, your user experience, and your overall architecture. Paying for lots of managed services is more expensive than running virtual machines–but Thumbtack saves money from not having to hire site reliability engineers. Nate Kupp leads the technical infrastructure team, and we met at QCon in San Francisco to talk about how to architect a modern marketplace. This was my third time attending QCon and as always I was impressed by the quality of presentations and conversations I had there. They were also kind enough to set up some dedicated space for podcasters like myself. The most widely used cloud provider is AWS, but more and more companies that come on the show are starting to use some of the managed services from Google. The great news for developers is that integration between these managed services is pretty easy. At Thumbtack, the production infrastructure on AWS serves user requests. The log of transactions that occur get pushed from AWS to Google Cloud, where the data engineering occurs. On Google Cloud, the transaction records are queued in Cloud PubSub, a message queueing service. Those transactions are pulled off the queue and stored in BigQuery, a system for storage and querying of high volumes of data. BigQuery is used as the data lake to pull from when orchestrating machine learning jobs. These machine learning jobs are run in Cloud Dataproc, a managed service that runs Apache Spark. After training a model in Google Cloud, that model is deployed on the AWS side, where it serves user traffic. On the Google Cloud side, the orchestration of these different managed services is done by Apache Airflow, an open source tool that is one of the few pieces of infrastructure that Thumbtack does have to manage themselves on Google Cloud. To find out more about the Thumbtack infrastructure, check out the video of the talk Nate gave at QCon San Francisco, or check out the Thumbtack Engineering Blog. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#677
Nov 27, 2017 · 54 min
Marketplace Matching with Xing Chen
The labor market is moving online. Taxi drivers are joining Uber and Lyft. Digital freelancers are selling their services through Fiverr. Experienced software contractors are leaving contract agencies to join Gigster. Online labor marketplaces create market efficiency by improving the communications between buyers and sellers. Workers make their own hours, and their performance is judged by customers and algorithms, rather than the skewed perspective of a human manager. These marketplaces for human labor are in different verticals, but they share a common problem: how do you most efficiently match supply and demand? Perfect marketplace matching is an unsolved problem. Hundreds of computer science papers have been written about the problems of stable matching, which often turn out to be NP-Complete. The stock market has been attempting to automate marketplace matching for decades, and inefficiencies are discovered every year. Today’s show is about matching buyers and sellers on Thumbtack, a marketplace for local services. For the first seven years, Thumbtack was building liquidity in its 2-sided market. During those years, the model for job requests was as follows: let’s say I was on Thumbtack looking for someone to paint my house. I would post a job that would say I am looking for house painters. The workers on Thumbtack that paint houses could see my job and place a bid on it. Then I would choose from the bids and get my house painted. This was the “asynchronous” model. The actions of the buyer and seller were not synchronized. There was a significant delay between the time when the buyer posted a job and the time when a seller places a bid, and then another delay before the buyer selects from the sellers. Thumbtack recently moved to an “instant matching” model. After gathering data about the people selling services on the platform, Thumbtack is now able to avoid the asynchronous bidding process. In the new experience, a buyer goes on the platform, requests a house painter, and is instantly matched to someone who has a history of accepting house painting tasks that fit the parameters of the buyer. From the user’s perspective, this is a simple improvement. From Thumbtack’s perspective, there was significant architectural change required. In the asynchronous model, the user requests lined up in a queue, and were matched with pros who placed bids on the items in that queue. In the instant matching model, a user request became more like a search query–the parameters of that request hit an index of pros and returns a response immediately. Xing Chen is an engineer from Thumbtack, and joins the show to describe the rearchitecture process–how Thumbtack went from an asynchronous matching system to synchronous, instant matching. We also explore some of the other architectural themes of Thumbtack, which we dive into in further detail in tomorrow’s episode about scaling Thumbtack’s infrastructure, which uses both AWS and Google Cloud. On Software Engineering Daily, we have explored the software architecture and business models of different labor marketplaces–from Uber to Fiverr. To find these old episodes, you can download the Software Engineering Daily app for iOS and for Android. In other podcast players, you can only access the most recent 100 episodes. With these apps, we are building a new way to consume content about software engineering. They are open-sourced at github.com/softwareengineeringdaily. If you are looking for an open source project to get involved with, we would love to get your help. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#676
Nov 22, 2017 · 50 min
Load Balancing at Scale with Vivek Panyam
Facebook serves interactive content to billions of users. Google serves query requests on the world’s biggest search engine. Uber handles a significant percentage of the transportation within the United States. These services are handling radically different types of traffic, but many of the techniques they use to balance loads are similar. Vivek Panyam is an engineer with Uber, and he previously interned at Google and Facebook. In a popular blog post about load balancing at scale, he described how a large company scales up a popular service. The methods for scaling up load balancing are simple, but effective–and they help to illustrate how load balancing works at different layers of the networking stack. Let’s say you have a simple service where a user makes a request, and your service sends them a response with a cat picture. Your service starts to get popular, and begins timing out and failing to send a response to users. When your service starts to get overwhelmed, you can scale up load by creating another service instance that is a copy of your cat picture service. Now you have two service instances, and you can use a layer 7 load balancer to route traffic evenly between those two service instances. You can keep adding service instances as the load scales and have the load distributed among those new instances. Eventually, your L7 load balancer is handling so much traffic itself that you can’t put any more service instances in front of it. So you have to set up another L7 load balancer, and put an L4 load balancer in front of those L7 load balancers. You can scale up that tier of L7 load balancers, each of which is balancing traffic across a set of your service instances. But eventually, even your L4 load balancer gets overwhelmed with requests for cat pictures. You have to set up another tier, this time with L3 load balancing… In this episode, Vivek gives a clear description for how load balancing works. We also review the 7 networking layers before discussing why there are different types of load balancers associated with the different networking layers. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#675
Nov 21, 2017 · 54 min
Incident Response with Emil Storlarsky
As a system becomes more complex, the chance of failure increases. At a large enough scale, failures are inevitable. Incident response is the practice of preparing for and effectively recovering from these failures. An engineering team can use checklists and runbooks to minimize failures. They can put a plan in place for responding to failures. And they can use the process of post mortems to reflect on a failure and take full advantage of the lessons of that failure. Emil Storlarsky is a production engineer at Shopify where his role shares many similarities with that of Google’s site reliability engineers. In this episode, Emil argues that the academic study of emergency management and industries such as aerospace and transportation have a lot to teach software engineers about responding to production problems. In this interview with guest host Adam Bell, Emil argues that we need to move beyond tribal knowledge and incorporate practices such as an incident command system and rigorous use of checklists. Emil suggests that we need to move beyond a mindset of “move fast and break things” and toward a place of more deliberate preparation. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript. Incident Response Insights Talk The Human Side Of Post Mortems

Showing 1,581–1,600 of 2,221 episodes