NewsTech News

Software Engineering Daily

softwareengineeringdaily.com

Software Daily: The world through the lens of software.

Play

Open in

2,221 episodes
Avg 53 min
English

Episodes 2,221

Newest Oldest Longest

All20 Unplayed20 In progress0 Played0

Counted on this page — what you have heard stays on this device, so it is not something the list can be paged by.

#1193
Jul 25, 2019 · 57 min
Envoy Mobile with Matt Klein
Envoy is an open source edge and service proxy that was originally developed at Lyft. Envoy is often deployed as a sidecar application that runs alongside a service and helps that service by providing features such as routing, rate limiting, telemetry, and security policy. Envoy has gained significant traction in the open source community, and has formed the backbone of popular service mesh projects such as Istio. Envoy has been mostly used as a backend technology, but the potential applications of Envoy include frontend client applications as well. The goal of Envoy is to make the network easier to work with–and the network includes client applications such as mobile apps running on a phone. Envoy Mobile is a network proxy for mobile applications. Envoy Mobile brings many of the benefits of Envoy to the mobile client ecosystem. It provides mobile developers with a library that can simplify or abstract away many of the modern advances that have been made in networking in recent years, such as HTTP2, gRPC, and QUIC. Matt Klein is the creator of Envoy, and he joins the show to discuss Envoy Mobile. Matt describes how the networking challenges of mobile applications are similar to those of backend systems and cloud infrastructure. We discuss the advances in networking technology that Envoy Mobile helps bring to the mobile ecosystem, and also touch on the scalability challenges that Matt is seeing at Lyft.
#1192
Jul 24, 2019 · 45 min
TechLitAfrica: Computer Literacy in Africa with Tyler Cinnamon
In the developed world, it is easy to take for granted that we grew up with computers. Technology is so pervasive in the United States that we have debates about how early in child development a human should be given a smartphone. Across much of Africa, there is a shortage of access to computers. Children grow up without much exposure to computers at all. Smartphones are starting to proliferate the continent, but the bandwidth limitations prevent the sort of unrestricted mobile Internet usage that many of us have in the west. Tyler Cinnamon is a software engineer and the co-founder of TechLitAfrica, an organization dedicated to improving technology literacy and reducing poverty in Africa. TechLitAfrica takes old computers from the United States which are no longer in use and repurposes those computers with educational software and a downloaded subset of the Internet. Then, TechLitAfrica takes those computers to Africa and sets them up as computer labs. Tyler joins the show to talk through the technical and cultural challenges of building TechLitAfrica.
#1191
Jul 23, 2019 · 1 hr 4 min
Data Engineering with Tobias Macey
The Hadoop ecosystem provided every company with the tools to store and query large amounts of data at a low cost. Since 2005, that ecosystem has expanded with more and more open source applications for data infrastructure. Data infrastructure includes databases, data lakes, distributed queues, data warehouses, query engines, web applications, on-prem software, closed source, open source, cloud platforms, and software-as-a-service. There is so much software, and it is difficult to keep track of everything in the data engineering ecosystem. Tobias Macey is the host of the Data Engineering Podcast, a show about technologies and practices in the world of data engineering. Tobias joins today’s episode as a guest to give his perspective on the evolving landscape of data engineering, and the trends he is seeing on the data engineering podcast.
#1190
Jul 22, 2019 · 57 min
Emerging Markets: Vietnam with Charles Lee
From Africa to India to Asia to South America–computer science and programming are rising in popularity in every emerging market. Each of these markets has regional needs for technology. Just like every culture develops its own food and television, every culture needs different types of applications to run their lives. In Vietnam, the day-to-day life of a citizen is different than it is in the United States. Yes, everyone needs Google and YouTube and Instagram. But the trends in messaging, food delivery, gig economy, and other B2C technology sectors are considerably different than the west. Charles Lee is the founder of CoderSchool a coding school in Vietnam. Before moving to Vietnam to start CoderSchool, he worked as a software engineer in San Francisco for several years. In today’s show, Charles describes the difference between the US and Vietnamese technology sectors, from consumer applications to business software to coding education.
#1189
Jul 19, 2019 · 54 min
Facebook GraphQL with Lee Byron
In 2011, Facebook had begun to focus its efforts on mobile development. Mobile phones did not have access to reliable, high bandwidth connections, and the Facebook engineering team needed to find a solution to improve the request latency between mobile clients and the backend Facebook infrastructure. One source of latency was recursive data fetching. If a mobile application client made a request to the backend for newsfeed, the backend API would return the newsfeed, but some components of that feed would require additional requests to the backend. In practice, this might result in a newsfeed loading partially on a phone, but having a delayed loading time for the comments of a newsfeed item. GraphQL is a solution that came out of this problem of recursive data fetching. A GraphQL server provides middleware to aggregate all of the necessary information to serve a complete request. GraphQL connects to backend data sources and federates the frontend request across these different data sources. GraphQL was open sourced in 2015, and has found many use cases in addition to simplifying backend data fetching for mobile clients. Today, GraphQL is used by PayPal, Shopify, Twitter, and hundreds of other companies. Lee Byron is the co-creator of GraphQL and he joins the show to tell the story of GraphQL, and how it fit into Facebook’s shift to mobile.
#1188
Jul 18, 2019 · 58 min
Facebook Open Source Management with Tom Occhino
Facebook has released open source software projects that have changed the industry. The most impactful projects to date are the React frontend user interface tools: ReactJS and React Native. Before React became popular, there were multiple competing solutions for the dominant frontend JavaScript framework. React became the most prominent because of its invention of JSX, its one-way data flow, and the strength of its community. The React community is led by Facebook engineers. Facebook has a full-time team dedicated to improving React. Facebook also has the benefit of seeing the hardest problems in web development ahead of the rest of the industry, since Facebook is always pushing the leading edge of what software can do. As React evolves, Facebook itself is the first user of updates to the libraries. Engineers on the React team will proactively update the applications on other teams with new React code. In this way, Facebook’s React team is similar to a platform engineering team, except the open source community benefits from the improvements as well. Tom Occhino is an engineering director at Facebook. He joins the show to discuss his ten years of experience at the company and his role today as a senior leader in charge of open source developers.
#1187
Jul 17, 2019 · 32 min
Facebook Data Infrastructure with Dhruba Borthakur
Facebook generates high volumes of data at a rapid pace. Dhruba Borthakur joined Facebook in 2008 to work on data infrastructure. His early projects at Facebook were around Hadoop, the distributed file system and MapReduce computation platform that laid the foundation for the “big data” movement. At the time, Facebook was generating as much data as any other startup, and the company needed to stay at the leading edge of scalability techniques for its Hadoop Distributed File System (HDFS) cluster. Traditionally, Hadoop managed its file system by synchronizing the coordination of the different data nodes with the help of a single master node. At Facebook, the scale of the data was such that the HDFS cluster had thousands of data nodes, which was too much volume for a single master node to handle. Dhruba helped implement redundancy at the master node to create a more resilient system. The early days of the big data movement was focused on batch processing. A company like Facebook would gather large amounts of data into databases and HDFS, and run offline analytics workloads to gather reports on an hourly, daily, or weekly basis. Over time, data infrastructure has moved closer to a “real time” processing model. Data infrastructure does not only support batch offline reporting–it also supports machine learning jobs that need to be run on a more frequent basis. These jobs have lower latency requirements, and have driven the adoption of in-memory stream processing systems like Spark and Flink. Dhruba joins the show to discuss his time at Facebook building data infrastructure. He takes us through the major projects he worked on, including the early Hadoop infrastructure, the refactoring of online user workloads to be more “pull” based than “push” based, and the creation of RocksDB, a storage engine he helped create at Facebook. Today, Dhruba is the CTO and co-founder of Rockset, a company that builds data infrastructure and database APIs on top of RocksDB. Rockset is building infrastructure for modern technology companies–many of which are facing problems that bear significant resemblance to the ones Facebook encountered as it scaled.
#1186
Jul 16, 2019 · 42 min
Facebook Engineering Culture with Raylene Yung
Facebook moves fast because of vision, collaboration, and trust. The fast pace of development is enabled by constantly improving infrastructure and a sense of unity throughout the company. In Facebook’s early days, there was an emphasis on rapidly deploying new code to drive constant improvement and experimentation. Product quality was maintained by engineers closely checking each other’s code reviews, rather than writing detailed unit testing suites. Facebook engineers had a sense for how the product should operate, and they were able to evaluate whether a new feature was working properly by testing a live version of that feature. At Facebook, the vision of the company is clearly communicated to the employees. Every employee within Facebook can articulate the vision for the company, and will use similar language in describing that vision. Since the employees are aligned on strategy, they can also align in their implementation of product features. This reduces conflicts across roles and between teams. Facebook has also shown a willingness to trust its engineers. Trust was exemplified by Facebook’s tolerance for failures in the early days. When an engineer broke a build, or shipped a feature that failed to gain traction, that engineer was usually not punished. They may have even been rewarded, if the company could learn significantly from such an error. Raylene Yung was an engineer at Facebook from 2009 to 2015. As she moved from individual contributor to manager to engineering director, Raylene worked on products including News Feed, Timeline, Privacy, and Sharing. Raylene joins the show to give her reflections on the Facebook product and engineering environment. She explained how Facebook’s culture of collaboration, vision, and trust drive fast product development and minimizes conflict. Raylene left Facebook and joined Stripe, where she worked on payments systems and international expansion for almost four years.
#1185
Jul 15, 2019 · 56 min
Facebook PHP with Keith Adams
Facebook was built using PHP, a programming language that was used widely in the late 90s and early 2000s. PHP allows developers to get web applications built quickly and easily, although PHP has a reputation for being difficult to scale. In the early days of Facebook, the company was scaling rapidly on every dimension. New users were piling into Facebook. Existing users were increasing their interactions and developing new patterns of usage. The Facebook application was rolling out new features quickly, adding them into the Facebook PHP codebase. A common pattern for scaling a large software application is to use a microservices architecture, breaking up the monolithic application into small services which can scale independently. For many applications, this pattern works well. But for some applications, microservices makes less sense. Microsoft Excel is one example. In Excel, a user is making updates to a complex data model using formulas, functions, and other in-app tools that need to be fast, performant, and integrated. The user needs to have a sense that the Excel data model will update quickly in response to changes. A software team working on a spreadsheet product such as Excel might prefer to keep all the application logic in a monolithic application. A monolith can centralize logic and make it easier to reason about. A monolith can reduce the number of network hops, cutting down on distributed systems problems. Testing and deploying a monolithic application can be less complex than doing so in a distributed, microservices system. Facebook chose to scale its PHP monolith rather than breaking it up into distributed microservices. Scaling PHP allowed Facebook to continue moving fast without going through a painful refactoring that would have slowed down the entire company. The first effort to scale PHP involved transpiling the entire PHP application into C++. This C++ version of Facebook ran faster and with a lower memory footprint. But C++ required ahead-of-time compilation: the PHP codebase had to be converted to C++ in one synchronous step. The Hip Hop Virtual Machine (HHVM) was started to allow the Facebook PHP monolith to run on the V8 JavaScript virtual machine. The advantage of running PHP on V8 was that it allowed Facebook to transpile PHP to JavaScript, an interpreted language that fit better with Facebook’s dynamic application build path. HHVM also allows Facebook to take advantage of the rapidly improving JavaScript runtime environment. Keith Adams was an engineer at Facebook for six years, where he helped develop infrastructure to scale PHP effectively. Keith is now the chief architect at Slack, which is also a scaled PHP application. Keith returns to Software Engineering Daily to discuss why and how Facebook scaled PHP.
#1184
Jul 12, 2019 · 56 min
Meltano: Data Engineering Lifecycle with Danielle Morrill
Data engineering allows a company to take advantage of the large quantities of data that the company has generated. In many companies, new data has been produced rapidly for many years, but the company has not been able to take full advantage of it. Creating large data sets does not provide immediate value for a company. A company needs to perform data engineering and data science to take full advantage of it. When data gets generated, it is stored in a database, data lake, or API backend like Google Analytics. In order to manipulate that data, it is often pulled into a data warehouse. A data warehouse provides fast access time to large quantities of data. Pulling data from a source like a database or data lake into a data warehouse requires a process known as extract and load. Once the data is in the data warehouse, it may also undergo a transform, which enriches the data or puts it in a format that is easier to make use of. Once data is in a data warehouse, it can be used to build models, interactive dashboards, and Jupyter Notebooks. The data engineering lifecycle has many different components, which is why data engineering can often be intimidating to a company that is trying to make use of their data. Meltano is a project with the goal of providing a system of conventions for managing the data engineering lifecycle. Meltano was started by GitLab, and the Meltano project has some strategic similarities to GitLab. Danielle Morill is the general manager of Meltano at GitLab. She joins the show to discuss the world of data engineering, and the architecture of Meltano. We touch on the different components of a data engineering pipeline, and the most acute pain points for data engineers.
#1183
Jul 11, 2019 · 57 min
Software Media with Charles Max Wood
Software engineers have a wide variety of media to choose from, including podcasts, blogs, YouTube videos, conferences. The amount of software engineering media that is available is growing and accelerating. Eight years ago, there were not as many options for information about software. Charles Max Wood founded Devchat.tv to create a network of podcasts and other content for software engineers. Today, his podcasts include the popular shows JavaScript Jabber, Ruby Rogues, and Adventures in Angular. Chuck joins the show for a conversation about software media. This was his second time on the show, with his first episode exploring his podcast JavaScript Jabber.
#1182
Jul 10, 2019 · 52 min
gVisor Container Isolation with Michael Pratt and Yoshi Tamura
Software applications running within a host operating system need to be isolated. Isolation prevents security vulnerabilities, such as one application accessing the memory of another. In modern cloud environments, a single physical host might be running multiple virtual machines on top of a hypervisor. Those virtual machines might be divided up into containers. The different virtual machines and containers might be operated by different users, or even different companies. gVisor is a container sandbox runtime open sourced by Google. gVisor runs containers in a new user-space kernel, and provides a container security system with low overhead. gVisor improves on the previous security properties of containers. Michael Pratt and Yoshi Tamura work on gVisor at Google, and they join the show to talk through the purpose of gVisor and the engineering around the project.
#1181
Jul 9, 2019 · 50 min
Software Free Radicals with Lior Kanfi
Creating a software company has never been easier. Software engineers are increasingly seeing entrepreneurship as a viable career path. But the path to being an independent software developer is not always clear. Most engineers spend some of their career working at a software company. Even an engineer who intends to build a company someday can thrive within the right environment. Lior Kanfi is a software engineer and the founder of Tikal, a company he started at the height of the dot com bubble in 1999. His initial vision for the company was to build a product around managing knowledge and people within a company. When the dot com bubble burst in the year 2000, it became much harder to run a product-focused business. Companies were not buying lots of experimental software, and investors were not aggressively funding new software companies. After the market collapsed, Lior shifted Tikal from a product-focused company into consulting, in order to have a more reliable income stream. Today, Tikal is a successful software consultancy based in Israel and the Bay Area. Lior describes the engineers within Tikal as “free radical” software developers: independent people who want to learn about new technologies and build experience interacting with clients. Lior joins the show to talk about his 20 year journey building Tikal and the differences between engineers in the Bay Area and those in Israel. Lior also hosted the Full Stack Tech Radar Day in Tel Aviv, which was a great conference I attended.
#1180
Jul 8, 2019 · 51 min
Edge Storage with Steve Klabnik
Edge computing allows for faster data access and computation. When your client application makes a request, that request might be routed to the edge. Edge servers are more numerous and more widely distributed than normal data centers, but an edge server might not have all of the data or the complete application logic for the backend to serve your request. Edge servers have historically been used for content delivery networks (CDN). CDNs are useful for hosting and serving media files that might otherwise be slow to access over a network. More recently, applications are also using edge servers for computation, as well as storage of resources which are smaller than the movies, music, and images that have traditionally been stored on an edge server. Steve Klabnik is an engineer with Cloudflare, and he returns to the show to discuss storage at the edge. In Steve’s previous appearances we have explored Rust and WebAssembly, and we also touch on those topics in today’s episode.
#1179
Jul 7, 2019 · 29 min
You Are Not A Commodity (Keynote at Tikal Full Stack Tech Radar Day)
Today’s episode is a keynote I gave at Full Stack Tech Radar Day in Tel Aviv. The talk is called “You Are Not a Commodity”. This talk is also available as a YouTube video. The slides can be accessed here. The world of commodity engineering is coming to an end. Developers are becoming more productive, more flexible, and more entrepreneurial. How does this impact you as an engineer? And how does it impact large companies who need a fungible set of engineers to maintain their software?
#1178
Jul 5, 2019 · 59 min
ListenNotes: Podcast Search Engine with Wenbin Fang
ListenNotes allows users to find podcasts by categories, popularity, and search queries. ListenNotes is not a podcast client that you download onto your phone, but it allows users to build playlists. These playlists are automatically published to an RSS feed, so that users can curate a playlist on desktop and subscribe to that playlist from any client. Podcasts are growing in popularity. There are more podcasts than ever before, and a podcast search engine needs to refresh its index regularly enough to capture the updates to the podcast universe. A podcast search engine needs to decide what to index. There are many potential fields to choose from within a podcast: the podcast title, the description, the audio transcription, and other metadata–should all of these factor into the results of a search engine? Wenbin Fang is the founder of ListenNotes, and the engineer who has built most of the application. He joins the show to talk about the world of podcasting and his work on ListenNotes, including the business model. Today, ListenNotes makes its money from advertisements and API requests–there is a growing market for applications that want an API for querying the ListenNotes backend.
#1177
Jul 3, 2019 · 1 hr
Permissionless Innovation with Joseph Jacks
Open source software allows developers to take code from the Internet and modify it for their own use. Open source has allowed innovation to occur on a massive scale. Today, open source software powers our consumer client applications and our backend cloud server infrastructure. Linux powers single node operating systems and Kubernetes is the foundation for new distributed systems. Hadoop created an open source distributed file system, Spark gave us a computational runtime on top of it, and Kafka created a middleware platform for shuttling data from one place to another. There are numerous other examples of how open source has changed the world of software development. Open source has also reshaped the business landscape of infrastructure software companies. A common business structure for a modern infrastructure company is the “open core” model. An open core company maintains an open source project that is free to use, but also sells a product or service around that product. Companies with an open core model include Red Hat, HashiCorp, and GitLab. Many companies are building a thriving business with the open core business model. But these companies do not directly control the most important part of the infrastructure supply chain: the cloud provider. Cloud providers have a fundamental tension with open core companies because the cloud providers offer services that compete with the open core companies. In addition to the issue of cloud providers competing directly with the open core companies, some people have questioned whether Amazon Web Services is capturing an unfair portion of the value that is being created by open source. Amazon Web Services is the biggest cloud provider, and it has built a large catalog of services that are built off of open source software. But AWS has not historically contributed heavily to open source relative to the value it has captured. One example of an open core company which has lost market share to an AWS cloud-hosted offering is Elastic, the open core company which maintains the ElasticSearch open source project. Amazon ElasticSearch Service is a closed-source hosted offering built on top of the ElasticSearch. Elastic (the company) has increasingly intermingled proprietary software with their open source repository, making it less clear how that open source repository can be used by companies that want to deploy it for their commercial use. Open core companies such as MongoDB, Redis Labs, and Cockroach Labs have responded to the competitive pressures of AWS by changing their licenses and making it more expensive for cloud providers to offer a cloud-hosted offering of their open source project. The dynamics between cloud providers and open core companies will continue to evolve in the coming years. The norms around open source are up for debate. Joseph Jacks is the founder of OSS Capital, a venture firm focused on investments in commercial open source software companies. He returns to the show to discuss the changing landscape of open core companies, and the benefits of permissionless innovation.
#1176
Jul 2, 2019 · 54 min
Google Earth WebAssembly with Jordon Mears
Google Earth allows users to explore the imagery of the real world. Imagery for Google Earth is taken from satellites, cars equipped with cameras, and other sources. Google Earth renders a data intensive 3-D model of the world on a client application such as a desktop browser or virtual reality system. WebAssembly is a runtime for executing code other than JavaScript in a browser-based environment. WebAssembly is useful for data-intensive workloads, and developers can use programming languages such as Rust or C++ in the browser by compiling to WebAssembly. Jordon Mears works on Google Earth, and he joins the show to talk about the engineering behind Google Earth and how WebAssembly is being used to improve efficiency. He also discusses the state of tooling around WebAssembly today.
#1175
Jul 1, 2019 · 51 min
FoundationDB with Ryan Worl
FoundationDB is a multi-model distributed key-value store. It is fully ACID compliant and horizontally scalable. FoundationDB is not usually used directly by an application developer–FoundationDB is a foundational building block for higher level distributed systems such as the metadata store for data warehousing tool Snowflake. Ryan Worl is a software engineer who specializes in FoundationDB. He joins the show to discuss the architecture of FoundationDB, including the roles of different server components and the read and write path of FoundationDB. We also talk about applications of FoundationDB, and how it compares to storage engines such as RocksDB and databases such as CockroachDB and Spanner.
#1174
Jun 29, 2019 · 1 hr 3 min
FindCollabs Hackathon Winners: Kitspace and Rivaly
FindCollabs is a platform for finding collaborators and building projects. Three months ago we had our first hackathon, with lots of projects being created and collaborated on. In an earlier episode, we showcased the first place winner ARhythm. Today’s show features two more interviews with winners from the first FindCollabs hackathon. Kitspace is an open source hardware registry, and Rivaly is an app for informal ranked leagues, such as ping pong games at work, or board game clubs among your friends. The second FindCollabs hackathon is going on today. Check it out by going to findcollabs.com/open.

Showing 1,161–1,180 of 2,221 episodes