NewsTech News

Software Engineering Daily

softwareengineeringdaily.com

Software Daily: The world through the lens of software.

Play

Open in

2,221 episodes
Avg 53 min
English

Episodes 2,221

Newest Oldest Longest

All20 Unplayed20 In progress0 Played0

Counted on this page — what you have heard stays on this device, so it is not something the list can be paged by.

#1130
May 2, 2019 · 52 min
Security Monitoring with Jeff Williams
The modern software supply chain contains many different points of distribution: JavaScript frameworks, npm modules, Docker containers, open source repositories, cloud providers, on-prem firmware, IoT, networking proxies, and so much more. With so much attack surface, securing a large enterprise is an uphill battle. Jeff Williams is the CTO at Contrast Security, a company that makes infrastructure monitoring tools. Contrast Security works by intercepting network traffic at a low level and assessing whether that traffic maps to a common threat model. Jeff joins the show to talk about different approaches to monitoring and securing large infrastructure deployments. Contrast Community Edition RECENT UPDATES: FindCollabs is a company I started recently The FindCollabs Podcast is out! FindCollabs is hiring a React developer FindCollabs Hackathon #1 has ended! Congrats to ARhythm, Kitspace, and Rivaly for winning 1st, 2nd, and 3rd place ($4,000, $1000, and a set of SE Daily hoodies, respectively). The most valuable feedback award and the most helpful community member award both go to Vynce Montgomery, who will receive both the SE Daily Towel and the SE Daily Old School Bucket Hat We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/ Podsheets is our open source set of tools for managing podcasts and podcast businesses New version of Software Daily, our app and ad-free subscription service
#1129
May 1, 2019 · 44 min
Software Growth with Greg Kogan
Growing a software business requires an understanding of engineering, sales, and marketing. As we learn software engineering, we also pick up some knowledge about how a business should operate. We know that there are customers, and that our product needs to be scalable to serve more customers. We know that some features are more important than others, and so we focus on building the features that matter the most. But unless we make a deliberate focus, engineers do not learn how to sell and market a software product. Learning how to sell and market software is an important skill to develop. It allows a software engineer to be self-sufficient. If you already know how to write software, sales and marketing are actually the only other pieces you need to be an “entrepreneur”. And the basics of sales and marketing are often easier and more fun to learn than the first painful days of learning basic programming. Greg Kogan is an engineer who has shifted his focus to working as a consultant for companies that are trying to go to market with a technical product. Greg has helped grow companies such as Netlify, Scalyr, and Domino Data Lab. Much of his work is around products targeted toward developers. Greg joins the show to describe his methodical approach to selling and marketing software. RECENT UPDATES: FindCollabs is a company I started recently The FindCollabs Podcast is out! FindCollabs is hiring a React developer FindCollabs Hackathon #1 has ended! Congrats to ARhythm, Kitspace, and Rivaly for winning 1st, 2nd, and 3rd place ($4,000, $1000, and a set of SE Daily hoodies, respectively). The most valuable feedback award and the most helpful community member award both go to Vynce Montgomery, who will receive both the SE Daily Towel and the SE Daily Old School Bucket Hat We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/ Podsheets is our open source set of tools for managing podcasts and podcast businesses New version of Software Daily, our app and ad-free subscription service
#1128
Apr 30, 2019 · 34 min
Container Platform Security with Maya Kaczorowski
A Kubernetes instance occupies a wide footprint of multiple servers, creating an appealing target to an attacker, due to its access to a large pool of compute resources. A common attack against an exposed Kubernetes cluster is to take it over for the purposes of mining cryptocurrency. Thus it is important to keep a cluster secure. The importance of security is magnified for a cloud provider. A cloud provider runs a managed Kubernetes service, which might be running thousands of Kubernetes clusters. If the cloud provider’s chosen distribution of Kubernetes contains a vulnerability, or if the Kubernetes instances are misconfigured, all of these clusters could be exposed to the same vulnerability. Maya Kaczorowski works on the security of Google’s managed Kubernetes service GKE. In today’s show we discuss the attack surface of a managed Kubernetes service. Maya was previously on the show to talk about container security. This episode is a good companion to that one, as well as a previous show with Liz Rice about container security. RECENT UPDATES: FindCollabs is a company I started recently The FindCollabs Podcast is out! FindCollabs is hiring a React developer FindCollabs Hackathon #1 has ended! Congrats to ARhythm, Kitspace, and Rivaly for winning 1st, 2nd, and 3rd place ($4,000, $1000, and a set of SE Daily hoodies, respectively). The most valuable feedback award and the most helpful community member award both go to Vynce Montgomery, who will receive both the SE Daily Towel and the SE Daily Old School Bucket Hat We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/ Podsheets is our open source set of tools for managing podcasts and podcast businesses New version of Software Daily, our app and ad-free subscription service
#1127
Apr 29, 2019 · length unknown
Lyft’s Data Platform with Li Gao
FindCollabs is a company I started recently The FindCollabs Podcast is out! FindCollabs is hiring a React developer FindCollabs Hackathon #1 has ended! Congrats to ARhythm, Kitspace, and Rivaly for winning 1st, 2nd, and 3rd place ($4,000, $1000, and a set of SE Daily hoodies, respectively). The most valuable feedback award and the most helpful community member award both go to Vynce Montgomery, who will receive both the SE Daily Towel and the SE Daily Old School Bucket Hat We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/ Podsheets is our open source set of tools for managing podcasts and podcast businesses New version of Software Daily, our app and ad-free subscription service Lyft generates petabytes of data. Driver and rider behavior, pricing information, the movement of cars through space; all of this data is received by Lyft’s backend services, buffered into Kafka queues, and processed by various stream processing systems. Lyft moves the high volumes of data into a data lake for different users throughout the company to use offline. Machine learning jobs, batch jobs, streaming jobs and materialized databases can be created on top of that data lake. Druid and Superset are used for operational analytics and dashboarding. Li Gao is a data engineer at Lyft. He joins the show to explore the different aspects of Lyft’s data platform. We also talk about the tradeoffs of streaming frameworks, and how to manage machine learning infrastructure. This episode is a great companion to our show about Uber’s data platform, and illustrates some fundamental differences in how the two ridesharing companies operate.
#1126
Apr 26, 2019 · 1 hr 4 min
Cloud with Eric Brewer
To the extent that I am a software engineering journalist, I feel inclined to scrutinize all of the cloud providers. But to the extent that I am an engineer and a business person, I feel only admiration and love for the cloud providers. Cloud computing has brought the cost of starting an Internet business down to zero. Cloud computing has opened up my eyes to a world of creative possibilities that knows no boundaries, and for that I will always be a fan of all of the rivaling cloud companies because they all have played a role in creating the current software landscape. Eric Brewer is a Google Fellow and VP Infrastructure. He is well-known for his work on the CAP theorem, a distributed systems concept that formalized the tradeoffs between consistency, availability, and partition tolerance in a distributed system. At Google, Eric is as much a strategist and product creator as he is a theoretician. He has worked on database systems such as Spanner, machine learning systems such as TensorFlow, and container orchestration systems such as Kubernetes and GKE. Eric joins the show to talk about Google’s philosophy as a cloud provider, and how his understanding of distributed systems has evolved since joining the company.
#1125
Apr 25, 2019 · 58 min
Intricately: Mapping the Internet with Fima Leshinsky
RECENT UPDATES: FindCollabs is a company I started recently The FindCollabs Podcast is out! FindCollabs is hiring a React developer FindCollabs Hackathon #1 has ended! Congrats to ARhythm, Kitspace, and Rivaly for winning 1st, 2nd, and 3rd place ($4,000, $1000, and a set of SE Daily hoodies, respectively). The most valuable feedback award and the most helpful community member award both go to Vynce Montgomery, who will receive both the SE Daily Towel and the SE Daily Old School Bucket Hat We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/ Podsheets is our open source set of tools for managing podcasts and podcast businesses New version of Software Daily, our app and ad-free subscription service Intricately is a company that maps the breadth and depth of cloud infrastructure usage. Using a combination of clever algorithms, data engineering, and web crawlers, Intricately derives information about how different companies spend money on infrastructure. Fima Leshinsky is the CEO and co-founder at Intricately. In his previous job at Akamai, he began to study how a cloud provider such as Akamai could figure out how much its competitors were charging certain customers. Since CDN infrastructure is a commodity with reasonably low switching cost, a provider that can undercut its competitors significantly can have an edge in the marketplace. From his work at Akamai, Fima felt there was a market opportunity to provide this kind of service to the broader market of cloud providers. There are more cloud providers than ever before, and the kind of data that Intricately aggregates is highly useful to this competitive marketplace. Fima joins the show to talk about the modern landscape of cloud providers, and how to build a system that maps the Internet.
#1124
Apr 24, 2019 · 46 min
gVisor: Secure Container Sandbox with Yoshi Tamura
RECENT UPDATES: Podsheets is our open source set of tools for managing podcasts and podcast businesses New version of Software Daily, our app and ad-free subscription service FindCollabs is hiring a React developer FindCollabs Hackathon #1 has ended! Congrats to ARhythm, Kitspace, and Rivaly for winning 1st, 2nd, and 3rd place ($4,000, $1000, and a set of SE Daily hoodies, respectively). The most valuable feedback award and the most helpful community member award both go to Vynce Montgomery, who will receive both the SE Daily Towel and the SE Daily Old School Bucket Hat. The Linux operating system includes user space and kernel space. In user space, the user can create and interact with a variety of applications directly. In kernel space, the Linux kernel provides a stable environment in which device drivers interact with hardware and manage low level resources. A Linux container is a virtualized environment that runs within user space. To perform an operation, a process in a container in user space makes a syscall (system call) into kernel space. This allows the container to have access to resources like memory and disk. Kernel space must be kept secure to ensure operating system integrity–but Linux includes hundreds of syscalls. Each syscall represents an interface between the user space and kernel space. Security vulnerabilities can emerge from this wide attack surface of different syscalls, and most applications only need a small number of syscalls to perform their required functionality. gVisor is a project to restrict the number of syscalls that the kernel and user space need to communicate. gVisor is a runtime layer between the user space container and the kernel space. gVisor reduces the number of syscalls that can be made into kernel space. The security properties of gVisor make it an exciting project today–but it is the portability features of gVisor that hint at a huge future opportunity. By inserting an interpreter interface between containers and the Linux kernel, gVisor presents the container world with the opportunity to run on operating systems other than Linux. There are many reasons why it might be appealing to run containers on an operating system other than Linux. Linux was built many years ago, before the explosion of small devices, smart phones, IoT hubs, voice assistants and smart cars. To be more speculative, Google is working on a secretive new operating system called Fuscia. gVisor could be a layer that allows workloads to be ported from Linux servers to Fuscia servers. Yoshi Tamura is a product manager at Google with a background in containers and virtualization. He joins the show to talk about gVisor and the different kinds of virtualization.
#1123
Apr 23, 2019 · 1 hr 5 min
Observability Engineering with James Burns
RECENT UPDATES: Podsheets is our open source set of tools for managing podcasts and podcast businesses New version of Software Daily, our app and ad-free subscription service FindCollabs is hiring a React developer FindCollabs Hackathon #1 has ended! Congrats to ARhythm, Kitspace, and Rivaly for winning 1st, 2nd, and 3rd place ($4,000, $1000, and a set of SE Daily hoodies, respectively). The most valuable feedback award and the most helpful community member award both go to Vynce Montgomery, who will receive both the SE Daily Towel and the SE Daily Old School Bucket Hat Twilio is a communications infrastructure company with thousands of internal services and thousands of request per second. Each request generates logs, metrics, and distributed traces which can be used to troubleshoot failures and improve latency. Since Twilio is used for 2-factor authentication and text message relaying, Twilio is critical infrastructure for most applications that implement it. The service must remain highly available even in times of peak application traffic, or outages at a particular cloud provider. When he was at Twilio, James Burns worked on platform infrastructure and observability. James was at Twilio from 2014 to 2017, a time in which the company experienced rapid scalability. His work encompassed site reliability, monitoring, cost management and incident response. He also led chaos engineering exercises called “game days”, in which the company deliberately caused infrastructure to fail in order to ensure the reliability of failover systems and to discover problematic dependencies. James joins the show to talk about his time at Twilio and his perspectives on how to instrument and observe complex applications. Full disclosure: James now works at LightStep, which is a sponsor of Software Engineering Daily.
#1122
Apr 22, 2019 · 51 min
Serverless Runtimes with Steren Giannini
RECENT UPDATES: Podsheets is our open source set of tools for managing podcasts and podcast businesses New version of Software Daily, our app and ad-free subscription service FindCollabs is hiring a React developer FindCollabs Hackathon #1 has ended! Congrats to ARhythm, Kitspace, and Rivaly for winning 1st, 2nd, and 3rd place ($4,000, $1000, and a set of SE Daily hoodies, respectively). The most valuable feedback award and the most helpful community member award both go to Vynce Montgomery, who will receive both the SE Daily Towel and the SE Daily Old School Bucket Hat. Google’s options for running serverless workloads started with App Engine. App Engine is a way to deploy an application in a fully managed environment. Since the early days of App Engine, managed infrastructure has matured and become more granular. We now have serverless databases, queueing systems, machine learning tools, and functions as a service. Developers can create fully managed, event-driven, highly scalable systems with less code and less operations. Different cloud providers are taking different approaches to offering serverless runtimes. Google’s approach involves the open source Knative project and a hosted platform for running Knative workloads called Cloud Run. Steren Giannini is a product manager at Google working on serverless tools. He joins the show to discuss Google’s serverless projects and the implementation details in building them.
#1121
Apr 19, 2019 · 57 min
Products with Ryan Hoover
RECENT UPDATES: Podsheets is our open source set of tools for managing podcasts and podcast businesses New version of Software Daily, our app and ad-free subscription service Software Daily is looking for help with Android engineering, QA, machine learning, and more FindCollabs Hackathon has ended–winners will probably be announced by the time this episode airs; we will be announcing our next hackathon in a few weeks, so stay tuned Products are an art form. As with any art, the world of products includes creators, patrons, fans, business people, and investors. Product Hunt is a place where those different people connect to build and discuss products. Products are different from other art forms in that they are measured not only through the lens of design and beauty–but also through utility. From software to books to couches to toiletry–we all have products that have improved our lives so much that we feel a deep sense of connection and hope for that product and the people behind it. Ryan Hoover is the founder of Product Hunt, a product I have found tremendous value and satisfaction from over the years. He is also a host of Product Hunt Radio, a weekly podcast with the people creating and exploring the future. Ryan joins the show to discuss products, the process of creating something useful, and his investing strategy. Ryan runs the Weekend Fund, an early stage investment fund.
#1120
Apr 18, 2019 · 45 min
Facebook OSS License Policy with Joel Marcey, Michael Cheng, and Kathy Kam
RECENT UPDATES: Podsheets is our open source set of tools for managing podcasts and podcast businesses New version of Software Daily, our app and ad-free subscription service Software Daily is looking for help with Android engineering, QA, machine learning, and more FindCollabs Hackathon has ended–winners will probably be announced by the time this episode airs; we will be announcing our next hackathon in a few weeks, so stay tuned Open source policy has become a business issue as well as a political one. Businesses like Elastic, MongoDB (the company), and Redis Labs have started to view the open source licenses of the projects they work on as a means for business defensibility against cloud providers offering similar services. It remains to be seen how viable this strategy will be for the commercial open source vendors. Companies that do not directly sell commercial open source are also grappling with questions around open source licensing. Facebook has become a force in the open source world through projects like React and GraphQL. Facebook leads these projects, but Facebook is not monetizing them other than to the extent that they use the projects to build Facebook.com. Facebook’s incentives are aligned with the rest of the industry on the quality of the GraphQL and React projects. Proper licensing can help Facebook keep those incentives in alignment. Joel Marcey, Michael Cheng, and Kathy Kam from Facebook join me for a discussion of the state of open source licensing, and how that impacts Facebook.
#1119
Apr 17, 2019 · 54 min
Drishti: Deep Learning for Manufacturing with Krish Chaudhury
RECENT UPDATES: Podsheets is our open source set of tools for managing podcasts and podcast businesses New version of Software Daily, our app and ad-free subscription service Software Daily is looking for help with Android engineering, QA, machine learning, and more FindCollabs Hackathon has ended–winners will probably be announced by the time this episode airs; we will be announcing our next hackathon in a few weeks, so stay tuned Drishti is a company focused on improving manufacturing workflows using computer vision. A manufacturing environment consists of assembly lines. A line is composed of sequential stations along that manufacturing line. At each station on the assembly line, a worker performs an operation on the item that is being manufactured. This type of workflow is used for the manufacturing of cars, laptops, stereo equipment, and many other technology products. With Drishti, the manufacturing process is augmented by adding a camera at each station. Camera footage is used to train a machine learning model for each station on the assembly line. That machine learning model is used to ensure the accuracy and performance of each task that is being conducted on the assembly line. Krish Chaudhury is the CTO at Drishti. From 2005 to 2015 he led image processing and computer vision projects at Google before joining Flipkart, where he worked on image science and deep learning for another four years. Krish had spent more than twenty years working on image and vision related problems when he co-founded Drishti. In today’s episode, we discuss the science and application of computer vision, as well as the future of manufacturing technology and the business strategy of Drishti.
#1118
Apr 16, 2019 · 54 min
Lyft Data Discovery with Tao Feng and Mark Grover
RECENT UPDATES: Podsheets is our open source set of tools for managing podcasts and podcast businesses New version of Software Daily, our app and ad-free subscription service Software Daily is looking for help with Android engineering, QA, machine learning, and more FindCollabs Hackathon has ended–winners will probably be announced by the time this episode airs; we will be announcing our next hackathon in a few weeks, so stay tuned Lyft is a ridesharing company with petabytes of data. Within Lyft, many different employees can use those data sets to build useful applications. A business analyst creates a dashboard to see how driver satisfaction is changing over time. An economist studies the pricing data to ensure that Lyft’s prices are competitive. A data scientist creates a report of how the speed of a ride correlates with 5 star ratings. A machine learning engineer trains a model to detect fraud on the platform. All of these use cases make sense–and in each of them, the employee at Lyft needs to find the necessary data sets within the company to build their application. Amundsen is a tool for finding and discovering data sets within the company. Tao Feng and Mark Grover are engineers at Lyft and join the show to talk about the problem of data discovery and the tools they have built at Lyft.
#1117
Apr 15, 2019 · 56 min
Protein Structure Deep Learning with Mohammed Al Quraishi
RECENT UPDATES: Podsheets is our open source set of tools for managing podcasts and podcast businesses New version of Software Daily, our app and ad-free subscription service Software Daily is looking for help with Android engineering, QA, machine learning, and more FindCollabs Hackathon has ended–winners will probably be announced by the time this episode airs; we will be announcing our next hackathon in a few weeks, so stay tuned Until Google DeepMind came into the field, protein structure prediction was dominated by academics. Protein structure prediction is the process of predicting how a protein will fold by looking at genetic code. Protein structure prediction is a perfect field to approach through the application of deep learning, because the inputs are highly dimensional and there is a plentiful array of different sets of labeled data. Protein structure deep learning is a field in which many different approaches are taken, often involving supervised learning and reinforcement learning. Mohammed Al Quraishi is a systems biologist at Harvard. His background spans computer engineering, statistics, and genetics. In his work, Mohammed explores the interplay between biology and computer systems. One area of Mohammed’s focus is protein structure prediction. In a blog post last year, Mohammed gave a brief history of protein structure prediction and described the significance of DeepMind entering the field. DeepMind’s AlphaFold technology surpassed all other competitors in the most recent CASP protein structure competition. Mohammed joins the show to discuss biology, academia, deep learning, and DeepMind.
#1116
Apr 14, 2019 · 56 min
Podsheets: Open Source Podcasting
Podsheets is a set of open source tools for podcast hosting, publishing, ad management, community engagement, and more. Podsheets is influenced by our experience managing Software Engineering Daily, a full-time podcast business. Software Engineering Daily is a podcast that airs 5 times per week. With 4 ads per show and 50 business weeks per year, we
#1115
Apr 12, 2019 · 1 hr 34 min
Bubbles with Haseeb Qureshi
Haseeb Qureshi is an entrepreneur and investor. As a teenager, Haseeb played poker professionally through the online poker bubble. His path from poker to software entrepreneurship has been explored in previous episodes. In 2007, Haseeb and I met at an online poker table. As we battled each other for thousands of dollars, Haseeb and I realized we shared an affinity for obnoxious screen names, obnoxious online avatars, and the city of Austin, Texas. We were both living in the city, and met each other in the real world. In our earliest days, Haseeb and I were not friends. It was a strange time–we were disembodied minds, drifting on the Internet, attached mostly to the fluctuating balances of our Full Tilt Poker and Pokerstars accounts. This was not a time for friendship–it was a time for ruthless, modern competition. Haseeb grew tired of poker. He wrote a book about the game to memorialize his thoughts, then abandoned it. He studied philosophy and literature, searching for something new in the historical musings of humanity. He traveled Europe, working as a farmer to reconnect with the physical world. He discovered the Effective Altruism movement. Finding no solace in his poker spoils, Haseeb gave away most of his money and started from scratch. As he rebuilt himself, he found software engineering and charted a path to San Francisco, where we reconnected. In this episode, Haseeb joins me for a discussion of software, philosophy, poker, and the nature of bubbles. Indeed, Haseeb and I have now lived through four major bubbles: dot coms, poker, the 2008 financial crisis, and the crypto bubble. Throughout these bubbles, the mediums change but never does the message: human beings are deeply irrational, tribalistic, and emotional.
#1112
Apr 11, 2019 · 58 min
Consul Service Mesh with Paul Banks
RECENT UPDATES: FindCollabs $5000 Hackathon Ends Saturday April 15th, 2019 New version of Software Daily, our app and ad-free subscription service Software Daily is looking for help with Android engineering, QA, machine learning, and more Consul is a tool from HashiCorp that allows users to store and retrieve information from a highly available key/value data store. Consul is used for storage of critical cluster information, such as service IP locations and configuration data. A service interacts with Consul via a daemon process on the node of that service. The daemon process periodically shares information with the Consul server over a gossip UDP protocol and can share data on a more immediate basis using TCP. Consul’s functionality has increased recently to add secure service connectivity. Consul Connect allows services to establish mutual TLS encryption with each other. The addition of mutual TLS to the Consul feature set is closely incidental with Consul gaining a title of “service mesh.” Service mesh is an increasingly popular pattern that can encompass a variety of features: load balancing, security policy management, service discovery, and routing. Tools which offer self-described “service mesh” functionality include Linkerd, Kong, AWS App Mesh, Solo.io Gloo, and Google’s Istio open source project. Paul Banks is the engineering lead of Consul at HashiCorp. He joins the show to talk about the service mesh category and the past, present, and future of Consul.
#1111
Apr 10, 2019 · 1 hr 2 min
Machine Learning Joins with Arun Kumar
RECENT UPDATES: FindCollabs $5000 Hackathon Ends Saturday April 15th, 2019 New version of Software Daily, our app and ad-free subscription service Software Daily is looking for help with Android engineering, QA, machine learning, and more Data sets can be modeled in a row-wise, relational format. When two data sets share a common field, those data sets can be combined in a procedure called a join. A join combines the data of two data sets into one data set that is often bigger than the initial two data sets independently occupied. In fact, this new data set is often so much bigger that it creates problems for the machine learning engineers. Arun Kumar is an assistant professor at UC San Diego. He joins the show to discuss the modern lifecycle of machine learning models, and the gaps in the tooling. Arun’s research into improving processing of joined data sets has been adopted by companies such as Google. Some of that research has been adapted into open source machine learning tools that improve the performance of machine learning jobs with minimal code required.
#1108
Apr 9, 2019 · 48 min
Streaming with Holden Karau
RECENT UPDATES: FindCollabs $5000 Hackathon Ends Saturday April 15th, 2019 New version of Software Daily, our app and ad-free subscription service Software Daily is looking for help with Android engineering, QA, machine learning, and more Distributed stream processing allows developers to build applications on top of large sets of data that are being rapidly created. Stream processing is often described as an alternative to batch processing. In batch processing, a single large computation is performed over a large, static data set. In stream processing, a computation is performed repeatedly and continuously over a data set that is being appended to. A stream is often stored in a distributed queue such as Kafka, Kinesis, Pulsar, or Google PubSub. A stream is often processed with a stream processing tool such as Spark, Flink, Storm, or Google Cloud Dataflow. Holden Karau is an engineer who works on open source projects at Google. She returns to the show to describe the state of stream processing and discuss modern best practices.
#1106
Apr 8, 2019 · 51 min
AWS Storage with Kevin Miller
RECENT UPDATES: FindCollabs $5000 Hackathon Ends Saturday April 15th, 2019 New version of Software Daily, our app and ad-free subscription service Software Daily is looking for help with Android engineering, QA, machine learning, and more A software application requires compute and storage. Both compute and storage have been abstracted into cloud tools that can be used by developers to build highly available distributed systems. In our previous episode, we explored the compute side. In today’s episode we discuss storage. Application developers store data in a variety of abstractions. In-memory caches allow for fast lookups. Relational databases allow for efficient retrieval of well-structured tables. NoSQL databases allow for retrieval of documents that may have a less defined schema. File storage systems allow the access pattern of nested file systems, like on your laptop. Distributed object storage systems allow for highly durable storage of any data type. Amazon S3 is a distributed object storage system with a wide spectrum of use cases. S3 is used for media file storage, archiving of log files, and data lake applications. S3 functionality has increased over the years, developing different tiers of data retrieval latency and cost structure. AWS S3 Glacier allows for long-term storage of data at a large cost reduction, in exchange for increased latency of data access. Kevin Miller is the general manager of Amazon Glacier at Amazon Web Services. He joins the show to talk about the history of storage, the different options for storage in the cloud, and the design of S3 Glacier.

Showing 1,221–1,240 of 2,221 episodes