NewsTech News

Software Engineering Daily

softwareengineeringdaily.com

Software Daily: The world through the lens of software.

Play

Open in

2,221 episodes
Avg 53 min
English

Episodes 2,221

Newest Oldest Longest

All20 Unplayed20 In progress0 Played0

Counted on this page — what you have heard stays on this device, so it is not something the list can be paged by.

#674
Nov 20, 2017 · 56 min
Run Less Software with Rich Archbold
There is a quote from Jeff Bezos: “70% of the work of building a business today is undifferentiated heavy lifting. Only 30% is creative work. Things will be more exciting when those numbers are inverted.” That quote is from 2006, before Amazon Web Services had built most of their managed services. In 2006, you had no choice but to manage your own database, data warehouse, and search cluster. If your server crashed in the middle of the night, you had to wake up and fix it. And you had to deal with these engineering problems in addition to building your business. Technology today evolves much faster than in 2006. That is partly because managed cloud services make operating a software company so much smoother. You can build faster, iterate faster, and there are fewer outages. If you are an insurance company or a t-shirt manufacturing company or an online education platform, software engineering is undifferentiated heavy lifting. Your customers are not paying you for your expertise in databases or your ability to configure load balancers. As a business, you should be focused on what the customers are paying you for, and spending the minimal amount of time on rebuilding software that is available as a commodity cloud service. Rich Archbold is the director of engineering at Intercom, a rapidly growing software company that allows for communication between customers and businesses. At Intercom, the engineering teams have adopted a philosophy called Run Less Software. Running less software means reducing choices among engineering teams, and standardizing on technologies wherever possible. When Intercom was in its early days, the systems were more heterogeneous. Different teams could choose whatever relational database they wanted–MySQL or Postgres. They could choose whatever key/value store they were most comfortable with. The downside of all this choice was that engineers who moved from one team to another team might not know how to use the tools at the new team they were moving to. After switching teams, you would have to figure out how to onboard with those new tools, and that onboarding process was time that was not spent on effort that impacted the business. By reducing the number of different choices that engineering teams have, and opting for managed services wherever possible, Intercom ships code at an extremely fast pace with very few outages. In our conversation, Rich contrasts his experience at Intercom with his experiences working at Amazon Web Services and Facebook. Amazon and Facebook were built in a time where there was not a wealth of managed services to choose from, and this discussion was a reminder of how much software engineering has changed because of cloud computing. To learn more about Intercom, you can check out the Inside Intercom podcast. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#673
Nov 17, 2017 · 1 hr 3 min
Training the Machines with Russell Smith
If I am building a mobile app to play podcast episodes, and I make a change to the user interface, I want to have manual quality assurance (QA) testers run through tests that I describe to them, to make sure my change did not break anything. QA tests describe high level application functionality. Can the user register and log in? Can the user press the play button and listen to a podcast episode on my app? Unit tests are not good enough, because unit tests only verify the logic and the application state from the point of view of the computer itself. Manual QA tests ensure that the quality of the user experience was not impacted. With so many different device types, operating systems, and browsers, I need my QA test to be executed in all of the different target QA environments. This requires lots of manual testers. If I want manual testing for every deployment I push, that manual testing can get expensive. RainforestQA is a platform for QA testing that turns manual testing into automated testing. The manual test procedures are recorded, processed by computer vision, and turned into automated tests. RainforestQA hires human workers from Amazon Mechanical Turk to execute the well-defined manual tests, and the recorded manual procedure is used to train the machines that can execute the same task in the future. Russell Smith is the CTO and co-founder of RainforestQA, and he joins the show to explain how RainforestQA works: the engineering infrastructure, the process of recruiting workers from mechanical turk, and the machine learning system for taking manual tasks and automating them. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#672
Nov 16, 2017 · 1 hr
High Volume Event Processing with John-Daniel Trask
A popular software application serves billions of user requests. These requests could be for many different things. These requests need to be routed to the correct destination, load balanced across different instances of a service, and queued for processing. Processing a request might require generating a detailed response to the user, or making a write to a database, or the creation of a new file on a file system. As a software product grows in popularity, it will need to scale these different parts of infrastructure at different rates. You many not need to grow your database cluster at the same pace that you grow the number of load balancers at the front of your infrastructure. Your users might start making 70% of their requests to one specific part of your application, and you might need to scale up the services that power that portion of the infrastructure. Today’s episode is a case study of a high-volume application: a monitoring platform called Raygun. Raygun’s software runs on client applications and delivers monitoring data and crash reports back to Raygun’s servers. If I have a podcast player application on my iPhone that runs the Raygun software, and that application crashes, Raygun takes a snapshot of the system state and reports that information along with the exception, so that the developer of that podcast player application can see the full picture of what was going on in the user’s device, along with the exception that triggered the application crash. Throughout the day, applications all around the world are crashing and sending requests to Rayguns servers. Even when crashes are not occurring, Raygun is receiving monitoring and health data from those applications. Raygun’s infrastructure routes those different types of requests to different services, queues them up, and writes the data to multiple storage layers–ElasticSearch, a relational SQL database, and a custom file server built on top of S3. John-Daniel Trask is the CEO of Raygun and he joins the show to describe the end-to-end architecture of Raygun’s request processing and storage system. We also explore specific refactoring changes that were made to save costs at the worker layer of the architecture. This is useful memory management strategy for anyone working in a garbage collected language. If you would like to see diagrams that explain the architecture and other technical decisions, the show notes have a video that explains what we talk about in this show. Full disclosure: Raygun is a sponsor of Software Engineering Daily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#671
Nov 15, 2017 · 56 min
Fiverr Engineering with Gil Scheinfeld
As the gig economy grows, that growth necessitates innovations in the online infrastructure powering these new labor markets. In our previous episodes about Uber, we explored the systems that balance server load and gather geospacial data. In our coverage of Lyft, we studied Envoy, the service proxy that standardizes communications and load balancing among services. In shows about Airbnb, we talked about the data engineering pipeline that powers economic calculations, user studies, and everything else that requires a MapReduce. In today’s episode, we explore the business and engineering behind another online labor platform: Fiverr. Fiverr is a marketplace for digital services. On Fiverr, I have purchased podcast editing, logo creation, music lyrics, videos, and sales leads. I have found people who will work for cheap, and quickly finish a job to my exact specification. I have discovered visual artists who worked with me to craft a music video for a song I wrote. Workers on Fiverr post “gigs”–jobs that they can perform. Most of the workers on Fiverr specialize in knowledge work, like proofreading or gathering sales leads. The workers are all over the world. I have worked with people from Germany, the Philippines, and Africa through Fiverr. Fiverr has become the leader in digital freelancing. The staggering growth of Fiverr’s marketplace has put the company in a position similar to an early Amazon. There is room for strategic expansion, but there is also an urgency to improve the infrastructure and secure the market lead. Gil Scheinfeld is the CTO at Fiverr, and he joins the show to explain how the teams at Fiverr are organized to fulfill the two goals of strategic, creative growth and continuous improvement to the platform. One engineering topic we discussed at length was event sourcing. Event sourcing is a pattern for modeling each change to your application as an event. Each event is placed on a pub/sub messaging queue, and made available to the different systems within your company. Event sourcing creates a centralized place to listen to all of the changes that are occurring within your company. For example, you might be working on a service that allows a customer to make a payment to a worker. The payment becomes an event. Several different systems might want to listen for that event. Fiverr needs to call out to a credit card processing system. Fiverr also needs to send an email to the worker, to let them know they have been paid. Fiverr ALSO needs to update internal accounting records. Event sourcing is useful because the creator of the event is decoupled from all of the downstream consumers. As the platform engineering team works to build out event sourcing, communications between different service owners will become more efficient. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#670
Nov 14, 2017 · 55 min
Serverless Event-Driven Architecture with Danilo Poccia
In an event driven application, each component of application logic emits events, which other parts of the application respond to. We have examined this pattern in previous shows that focus on pub/sub messaging, event sourcing, and CQRS. In today’s show, we examine the intersection of event driven architecture and serverless architecture. Serverless applications can be built by combining functions-as-a-service (like AWS Lambda) together with backend as a service tools like DynamoDB and Auth0. Functions-as-a-service give you cheap, flexible, scalable compute. Backend as a service tools give you robust, fault-tolerant tools for managing state. By combining these sets of tools, we can build applications without thinking about specific servers that are managing large portions of our application logic. This is great–because managing servers and doing load balancing and scaling is painful. With this shift in architecture, we also have to change how data flows through our applications. Danilo Poccia is the author of AWS Lambda In Action, a book about building event-driven serverless applications. In today’s episode, Danilo and I discuss the connection between serverless architecture and event driven architecture. We start by reviewing the evolution of the runtime unit–from physical machines to virtual machines to containers to functions as a service. Then, we dive into what it means for an application to be “event driven.” We explore how to architect and scale a serverless architecture, and we finish by discussing the future of serverless–how IoT and edge computing and on-premise architectures will take advantage of this new technology. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#669
Nov 13, 2017 · 39 min
BigQuery with Jordan Tigani
Large-scale data analysis was pioneered by Google, with the MapReduce paper. Since then, Google’s approach to analytics has evolved rapidly, marked by papers such as Dataflow and Dremel. Dremel combined a column-oriented, distributed file system with a novel way of processing queries. A single Dremel query is distributed into a tree of servers, starting with the root server, splitting into the intermediate servers, and ending with the leaf servers talking to the file system. Once the data is pulled from the file system into the leaves, the data propagates back to the root server, and is shuffled along the way so that the root server receives a sorted response. When Google started turning its internal services into customer-facing cloud products, the effort to productize Dremel began, and BigQuery was born. Jordan Tigani is an engineering lead who works on BigQuery, and he joins the show to discuss the evolution of the data warehouse. Large scale distributed queries still can take a long time–but queries get faster every year. Queries that required a nightly Hadoop job 10 years ago can be viewed in a frequently updated user-facing dashboard. Power users of BigQuery talk about the speed and the query interface as being two of its most valuable differentiating features. As the job of a large scale data analyst becomes less technically intensive, tools like BigQuery will continue to rise in popularity. We have done some great shows about Google papers like Spanner, Dremel, and Dataflow. To find these old episodes, you can download the Software Engineering Daily app for iOS and for Android. In other podcast players, you can only access the most recent 100 episodes. With these apps, we are building a new way to consume content about software engineering. They are open-sourced at github.com/softwareengineeringdaily. If you are looking for an open source project to get involved with, we would love to get your help. Shout out to today’s featured contributor Shreyans Sheth. Shreyans has worked on the Software Engineering Daily search API, and has also helped us understand open source best practices, which we are still learning. Thanks again Shreyans for your work. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#668
Nov 10, 2017 · 58 min
Legal Technology with Justin Kan
Justin Kan has been building startups for a decade, and in that time he has interacted with lots of lawyers. From incorporation to fundraising to selling his company Twitch, the interactions with lawyers consistently seemed less transparent and less efficient than would be optimal. For an engineer like Justin, the natural inclination here was to build software and sell it to lawyers. But there would be so much resistance–you would have to convince the lawyers to change their pricing model to fixed-pricing, which would give them the incentive to buy software and work more efficiently. Instead, Justin teamed up with a few entrepreneurial lawyers who were willing to start a new law firm from scratch, and use software on day 1. The software company is called Atrium Legal Technology Services (or Atrium LTS for short), and the law firm that uses the software is Atrium LLP. Both of these companies are very new, and were publicly announced a few months ago. The two companies work side-by-side in undecorated office in downtown San Francisco. When I took the elevator up to see the company, the elevator doors opened and revealed two paper signs pointing to opposite ends of the office. On the Atrium LTS side of the office, engineers were writing software to extract the meaning from documents. Today, lawyers at old law firms are paid hundreds of dollars an hour to fill in document templates by editing a text document. As the Atrium LTS software gets better, document preparation will be done through web applications, with the variable names disambiguated from the parts of the document that never change from client to client. On the other side of the office sat Atrium LLP. The legal team was dressed a little more formally than their engineer counterparts, but there was nothing close to the formality of a traditional Silicon Valley law firm. Far from the decor of a Menlo Park law firm, the office space was actually more spartan than most well-funded startups, signaling to the employees that this is an unproven business strategy, and there is a ton of work to be done to validate it. This sentiment was echoed in my conversation with Justin. It’s possible (even plausible) that Atrium LLP could become the biggest law firm in the world, but the road to getting there will take patience and steady execution. I enjoyed hearing Justin explain the motivation for starting Atrium LTS, and look forward to covering the company in the future. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#667
Nov 9, 2017 · 49 min
Early Investments with Semil Shah
An engineer who wants to start a business using investment capital needs to understand the expectations of investors. The market for the business needs to be huge. The team needs to have a differentiated understanding of the market, or a differentiated product. The CEO needs to have the determination to continue operating the company even when it gets very difficult. And the price needs to be right for the investor. Even if you are just working at a startup, or considering joining a startup, you must understand how the investment market works. From a raw financial standpoint, it only makes sense to spend your time at a startup that has equity with a high expected value. Your equity will only have high expected value if the company continues to exist long enough to have an exit–the company must either go public or get acquired. In order to make it down the long and winding road to an exit, a technology company often needs to raise money on multiple occasions. That money is used to pay employees like you! If the company can’t earn enough revenues or raise money, you are going to get fired. Then, you may not have the spare cash to execute your stock options, and you might lose the rights to the equity that you worked so hard for. The best way to avoid this is to learn to think like an investor–because as an engineer working for equity, you are an investor. Semil Shah is an early stage seed investor with Haystack, a fund that he started. He also works with GGV Capital, a venture firm investing out of the United States and China. Semil has been blogging about technology for many years, and eventually evolved from a commentator to an investor. In this episode, we explore the dynamics between investors and founders of early-stage technology companies. We also explore the strange market of podcasting. Semil worked at a company called Concept.io, which was acquired by Apple for $30M. We have done some great shows with other engineering investors like Chris Dixon and Adrian Colyer. To find these old episodes, you can download the Software Engineering Daily app for iOS and for Android. In other podcast players, you can only access the most recent 100 episodes. With these apps, we are building a new way to consume content about software engineering. They are open-sourced at github.com/softwareengineeringdaily. If you are looking for an open source project to get involved with, we would love to get your help. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#666
Nov 8, 2017 · 1 hr 2 min
OpenBazaar with Brian Hoffman
Cryptocurrencies give us a decentralized financial system. OpenBazaar is a decentralized commerce system. A merchant can log onto OpenBazaar and post a listing for an item–for example, a t-shirt that I want to sell for $15. My item listing will spread throughout the OpenBazaar P2P network. A shopper can download the OpenBazaar desktop application and see my listing for a t-shirt. The shopper can pay me $15 in bitcoin, and I will send the t-shirt to their address. If I were selling that shirt on Amazon, the corporation would take a cut of that transaction. OpenBazaar has no transaction costs–so users get to save some money. However, users also miss out on the benefits of a corporate marketplace. Amazon makes sure that the seller will send the item to the buyer, and makes sure that the buyer pays the seller. On OpenBazaar, an escrow system is needed to place money in the hands of a neutral third party until the goods are delivered. Amazon ensures that the distributor sends the item to the customer. On OpenBazaar, users need to figure out how to send the goods to each other. Brian Hoffman was the first developer to start working on OpenBazaar. The project has grown significantly since his initial commit, and OpenBazaar now has buyers, sellers, and open source committers. There is a clear desire for an open system of commerce. Brian is also the CEO of OB1, a company that provides services on top of OpenBazaar. OpenBazaar is a protocol–and other companies will undoubtedly emerge to build on top of it as well. In our conversation, Brian discussed how OpenBazaar works–the peer-to-peer protocol, the escrow system, the dispute resolution, and the open source community management. It is a fascinating, unique project, and I hope you learn something about it from this episode. To find all of our old episodes about decentralized technology and blockchains, you can download the Software Engineering Daily app for iOS and for Android. In other podcast players, you can only access the most recent 100 episodes. With these apps, we are building a new way to consume content about software engineering. They are open-sourced at github.com/softwareengineeringdaily. If you are looking for an open source project to get involved with, we would love to get your help. Shout out to today’s featured open source contributor Justin Lam. He has been working on improving the iOS codebase, and I know all the SE Daily mobile users appreciate his effort. Thanks Justin! Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#665
Nov 7, 2017 · 55 min
Netflix Serverless-like Platform with Vasanth Asokan
The Netflix API is accessed by developers who build for over 1000 device types: TVs, smartphontes, VR headsets, laptops. If it has a screen, it can probably run Netflix. On each of these different devices, the Netflix experience is different. Different screen sizes mean there is variable space to display the content. When you open up Netflix, you want to efficiently browse through movies. The frontend engineers who are building different experiences for different device types need to make different requests to the backend to fetch the right amount of data. This was the engineering problem that Vasanth Asokan and his team at Netflix was tasked with solving: how do you enable lots of different frontend engineers to get whatever they need from the backend? This problem led to the development of a “serverless-like platform” within Netflix, which Vasanth wrote about in a few popular articles on Medium. This platform enables frontend developers to write and deploy backend scripts to fetch data, decoupling the responsibilities of frontend engineers and backend engineers. The tight coupling of frontend and backend engineering was problematic to the development velocity of Netflix. We have done many shows about Netflix engineering, covering topics like data engineering, user interface design, and performance monitoring. To find these old episodes, you can download the Software Engineering Daily app for iOS and for Android. With these apps, we are building a new way to consume content about software engineering. They are open-sourced at github.com/softwareengineeringdaily. If you are looking for an open source project to get involved with, we would love to get your help. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#664
Nov 6, 2017 · 56 min
Serverless Authentication with Bobby Johnson
Serverless architecture is software that runs without an addressable server. Serverless is made possible by two types of technology: platform as a service providers like Auth0, and functions as a service like AWS Lambda. With both of these technologies, we can program logic that runs without being deployed to a server. Functions as a service are cheap and scalable. Write your code for a Serverless function, and the cloud provider will cheaply deploy and execute that function on some server somewhere. The difficult part is maintaining state. Since Serverless compute instances are ephemeral, you aren’t dealing with a system that will keep track of your state—it is going to disappear eventually. The ephemeral nature of Serverless code requires us to shift our thinking—but the dramatic cost and simplified scalability make it well worth the effort. Serverless functions can add complexity in exchange for lower price. Serverless “platform as a service” often lowers complexity at a slightly higher price. A Serverless database like Firebase handles database scaling and gives you a nice web interface. A Serverless machine learning platform like Google CloudML gives your models scalability and controlled deployment. A Serverless authentication service like Auth0 manages your authentication. In addition to authentication, Auth0 has built a set of tools to allow SaaS companies to extend their platforms into a sandboxed code execution environment. Bobby Johnson is an engineer at Auth0, and he joins the show to describe the toolbox that Auth0 has developed: authentication, webtasks, and extensibility–and how the world of “serverless” architecture is evolving. Full disclosure, Auth0 is a sponsor of Software Engineering Daily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#663
Nov 3, 2017 · 56 min
Parlaying Failure to Fortune with Paul Martino
In 2003, Paul Martino co-founded Tribe.net, one of the earliest social networking sites. Tribe had significant traction, with hundreds of thousands of users. In the early 2000s, hundreds of thousands of users was enough traffic to pose a company with engineering challenges. Paul had studied computer science, and was able to use his knowledge of high-performance computing to write an efficient graph database, and solve the other technical puzzles that the company faced–but the business did not ultimately work out. The failure of Tribe made the founders even hungrier for success–and it taught them lessons that they carried into subsequent businesses. Paul went on to start Aggregate Knowledge, a marketing technology company that sold for $119 million. His Tribe co-founder Mark Pincus went on to start Zynga, the multi-billion dollar gaming company. Another Tribe employee co-founded Yammer, which sold to Microsoft for a billion dollars. Since his exit from Aggregate Knowledge, Paul Martino started Bullpen Capital, which makes post-seed investments. The Bullpen Capital portfolio is appealing to me–partly because of the number of Internet gambling companies. Paul and I talked about gambling and other taboo business sectors–as well as what makes a good investment in the “post-seed” category. I enjoyed speaking to Paul because he has a straightforward, no-nonsense way of talking about things–it’s very charismatic and uncommon. We have done some great shows with other engineering investors like Chris Dixon and Adrian Colyer. To find these old episodes, you can download the Software Engineering Daily app for iOS and for Android. In other podcast players, you can only access the most recent 100 episodes. With these apps, we are building a new way to consume content about software engineering. They are open-sourced at github.com/softwareengineeringdaily. If you are looking for an open source project to get involved with, we would love to get your help. Shout out to today’s featured contributor Kurian Vithayathil. He has made significant contributions to the Software Engineering Daily Android app. Thanks again Kurian for your work. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#662
Nov 2, 2017 · 1 hr 6 min
Bad Men with Bob Hoffman
Bob Hoffman’s long career in advertising included CEO positions at three different agencies. He helped huge brands craft their messaging, and grab consumer attention. In Bob’s world of advertising, lots of money was spent on creativity. Were the campaigns successful? That depends who you ask. In the old world of advertising, everyone acknowledged that success was subjective. If you could imagine the opposite of what Bob Hoffman built his career doing, it might look like search advertising. Bob’s campaigns were about creating a brand’s voice, with colorful art and subtlety and ambient messaging. Advertising was about turning a brand into an entity you recognize, teaching the consumer to associate Nike with fitness, or Dove soap with clean hands, or Cheetos with cheesy, salty attitude. Search advertising, on the other hand, is just text. You enter a search query, you are looking for black socks, and the top link that comes back is a line of text that says “cheap black socks.” Search advertising catches people who have an intent to do something. They have stated their intent by typing into a box. With search advertising, a brand might not even need a sexy, flashy creative. As money poured into adtech, user tracking, and Google, brands started to care more about metrics. When Bob met with a brand, the brand wouldn’t be asking about the cool new advertising campaign featuring a young actress drinking a Coca Cola. The brand would be asking about the click-through rate of a display advertising campaign. Brands moved their focus to statistics, and away from creativity. And technology companies were happy to provide them with statistics. Whether those statistics were true or not is another story altogether. The industry was moving from creative BS to outright lying, and Bob decided to leave. In today’s episode, Bob explains how the state of advertising became so problematic, and the ways in which it harms us Internet users. We have done lots of reporting about advertising fraud for the last year, and it is a popular topic because people are often shocked to find that online advertising is inextricably linked to organized crime, surveillance, and Twitter botnets. That’s not to say that online advertising doesn’t work–it certainly does! But understanding the dark underbelly of the Internet’s cash cow is a necessary precondition to finding a solution. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#661
Nov 1, 2017 · 56 min
Augmented Reality with Scott Montgomerie
Augmented reality applications are slowly making their way into the world of the consumer. Pokemon Go created the magical experience of seeing Pokemon superimposed upon the real world. IKEA’s mobile app lets you see how a couch would fit into your living room, which has a significant improvement on the furniture buying process. Augmented reality applications can have even more dramatic impact on industrial enterprises. Have you ever set up a factory? You might need to build a conveyor belt. You might need to put together the parts of a giant machine that extrudes steel. You might need to fix a silicon wafer fabrication machine. It takes an expert to set up these heavy, complicated machines. ScopeAR is a company that builds augmented reality tools. One of the ScopeAR products allows users to telepresence with each other to collaborate on the construction and maintenance of heavy machinery. Imagine I am setting up my factory, and I have a complicated piece of machinery (let’s say a conveyor belt) in front of me. I have never constructed a conveyor belt before. I put on a HoloLens, and set up a VoIP call with an expert who has experience with that piece of machinery, and they point out what I need to do by superimposing 3-D arrows, text, and other instructions on my field of vision. They can share my experience and help guide me through the process. This is such a flexible tool–you can imagine applications for augmented reality assistance being useful in medicine, construction, education and other fields. Scott Montgomerie is the CEO of ScopeAR and in today’s episode, we talk about the state of AR, how the AR tools from Apple and Google compare, and how the similarity between tools used for mapping the world in AR relate to the tools used to map the world by autonomous cars. Scott was a great guest, and I hope to have him back on in the future. We have done some great shows about how to build augmented reality and virtual reality applications. To find these old episodes, you can download the Software Engineering Daily app for iOS and for Android. In other podcast players, you can only access the most recent 100 episodes. With these apps, we are building a new way to consume content about software engineering. They are open-sourced at github.com/softwareengineeringdaily. If you are looking for an open source project to get involved with, we would love to get your help. Shout out to today’s featured contributor Edgar Pino. He is working on a real-time chat application for Software Engineering Daily, so that we can have chat rooms for people to discuss the episodes easily. Innovative work! Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#660
Oct 31, 2017 · 59 min
Elastic Load Balancing with Ranga Rajagopalan
Computational load is the amount of demand that is being placed on a computer system. “Load” can take the form of memory, CPU, network bandwidth, disk space, and other finite resources. When we design systems, we need to prepare for high-load events. On a social network, people are much more active in the mornings. On an e-commerce site, Black Friday causes many more users to come online for discount shopping. Our distributed application must be able to scale in response to these spikes in traffic. Cloud computing has changed the popular software architecture patterns, and load balancing has changed along with it. With on-demand, infinite infrastructure, we don’t need to worry about ordering servers and provisioning. With infrastructure as code, it becomes simpler to manage lots of deployable units–so we can break up our monolith into microservices, and have hundreds or thousands of virtual machines or containers running. Enterprises that were started before cloud computing have large on-premise server deployments–but today, many of them also use the cloud. The cloud can be used to augment their classic on-prem deployments with cloud platform-as-a-service features. The cloud can also be used as a reliable way to scale during high load events. Today, a common architectural pattern is to have your application broken up into services. Each of those services has multiple instances. When the load on a particular service is under lots of demand, you create more instances to handle the increased load. How do you monitor the load on each service? How do you know when to spin up new instances of the service? Load analysis and load balancing across different services can be implemented by placing “agents” throughout your infrastructure. These agents gather data about services and service instances, and route that data to a centralized place. The centralized “control plane” can be used to make decisions about load-balancing and traffic routing. Ranga Rajagopalan worked on networking at Cisco for a decade before co-founding Avi Networks as CTO. Avi Networks builds modern load balancing software, and in today’s episode, Ranga describes the requirements of load balancing. We talked about the evolution of network infrastructure, the impact of the cloud, and the technical decisions that his team has made when architecting Avi Networks. Full disclosure: Avi Networks is a sponsor of Software Engineering Daily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#659
Oct 30, 2017 · 58 min
Kafka at NY Times with Boerge Svingen
The New York Times is a newspaper that evolved into a digital publication. Across its 166 year history, The Times has been known for longform journalistic quality, in addition to its ability to quickly churn out news stories. Some content on the New York Times is old but timeless “evergreen” content. Readers of the New York Times website are not only looking for the most recent news–they want to know what the headlines were the day after Pearl Harbor. They want to read editorials about Martin Luther King. Over the last 30 years, New York Times has moved itself online, bringing old material with it. Since the 90s, several different content management systems (CMS) have been used by journalists within The Times. These different sources of content store data in different formats. This is a data management problem. Users want to search over the entire history of articles published by The Times, which means that The Times needs to unify those articles in a single index. These are articles from the 1920s that were digitized using OCR, articles from 1998 that were written on a legacy CMS, and articles from 2017 that use the latest CMS. Boerge Svingen is the director of engineering at NYT, and he wrote about this problem and its solution on Medium. This story describes the flexibility of Kafka; in contrast to the applications of Kafka as a place to buffer high volumes of data, the New York Times uses Kafka as a place to unify data and allow for other specific materialized views to be built on top of it. We have covered Kafka in the past with interviews of some of its creators–including Jay Kreps and Neha Narkhede. To find these old episodes, you can download the Software Engineering Daily app for iOS and for Android. With these apps, we are building a new way to consume content about software engineering. They are open-sourced at github.com/softwareengineeringdaily. If you are looking for an open source project to get involved with, we would love to get your help. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#658
Oct 28, 2017 · 1 hr 3 min
Cryptoeconomics with Vlad Zamfir
A cryptocurrency has a distributed ledger called a blockchain. The blockchain keeps track of every transaction that occurs across the cryptocurrency. This blockchain must stay up-to-date and verified–which requires someone in the network to do that validation. Bitcoin and Ethereum use the proof-of-work algorithm. Miners do computational work to validate the legitimacy of transactions across the network, and in return they are given cryptocurrency as a reward for that computational work. In the future, cryptocurrencies could move towards a proof-of-stake model. If you own a significant amount of cryptocurrency, you have incentive to keep the validity of the blockchain up to date. Proof-of-stake algorithms can be significantly less energy intensive. Vlad Zamfir is a researcher for the Ethereum Foundation, and he joins Haseeb Qureshi for a conversation about cryptoeconomics. This is an in-depth conversation between two active blockchain developers. We hope you enjoy it. You can send us feedback on the show by emailing me jeff@softwareengineeringdaily.com or joining us on the Slack channel at softwareengineeringdaily.com/slack. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#657
Oct 27, 2017 · 1 hr 8 min
Analyse Asia with Bernard Leong
In America, the tech companies we focus on are commonly known as FAANG: Facebook, Amazon, Apple, Netflix, Google. We all know what these companies do because they impact our daily lives. In Asia, there are three giant tech companies that have similar scale: Baidu, Alibaba, and Tencent, otherwise known as BAT. Technology within a location is shaped by the pressures of that location. You might think we live in a global society, but tech in Asia is dramatically different than it is in America. Differences in culture lead to differences in product development. In China, a different political system contributed to more rapid adoption of online payments. Because there is more payment data, people can be given loans more efficiently. Less of the population is “unbanked.” Online payments are mostly handled by WeChat, a social networking product from Tencent, and Alibaba, an ecommerce giant. If you live in the West, imagine that Facebook and Amazon handled most of your payments for everything. You would have a different relationship with those companies. Bernard Leong is the host of Analyse Asia, a podcast about Asian developments in technology in business. After studying materials science in Singapore and theoretical physics at Cambridge, he made his way into business and journalism, and developed an interest in the Singularity–a subject that few people took seriously until recently (one topic we explored in this show is Masayoshi Son, the Japanese tycoon who wants to invest nearly a trillion dollars into technology companies; Masayoshi believes firmly that the Singularity is coming). Shenzhen: The Silicon Valley of Hardware (Full Documentary) | Future Cities | WIRED In the Plex by Steven Levy The Hidden Forces Behind Toutiao: China’s Content King Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#656
Oct 26, 2017 · 57 min
IFTTT Architecture with Nicky Leach
It’s 9pm at night, and you are hungry. You order a pizza from Domino’s. You live on a street that’s dark, and so you have installed a smart lightbulb in front of your mailbox that lights up the address. When the pizza at Domino’s is ready, you want the lightbulb on your mailbox to light up so that the delivery person can read your address when they arrive in front of your house with the pizza. The Internet should make it possible to have this kind of event-driven, connected world. Anything that is connected to the Internet should be able to send signals to anything else on the Internet, so that our lives gradually become more automated. This is what IFTTT does. Users of IFTTT can easily create applets to wire different services together. You can use IFTTT to trigger an email whenever three of your friends retweet something on Twitter. You can use IFTTT to flash the lights in your house when Bitcoin hits new market highs. You can use IFTTT to order a pizza whenever Bitcoin crashes. IFTTT makes it easy to connect different services together, and a lot of work goes into the infrastructure that enables these billions of events to process correctly. Nicky Leach from IFTTT’s engineering team joins the show to describe how IFTTT allows for integrations between services that were not built to integrate–and he talks about the scheduling, data engineering, and monitoring of the company’s software stack. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
#655
Oct 25, 2017 · 1 hr 9 min
Dremio with Tomer Shiran
In 2015, eleven years had passed since MapReduce was first published, and companies were still having data problems. Tomer started working on Dremio, a company that was in stealth for another two years. I interviewed Tomer two years ago, when he still could not say much about what Dremio was doing. We talked about Apache Drill, an open-source project related to what Dremio eventually built. Earlier this year, two of Tomer’s colleagues Jacques Nadeau and Julien Le Dem came on to discuss columnar data storage and interoperability. What I took away from that conversation was that today, data within an average enterprise is accessible, but the different formats are a problem. Some data is in MySQL, some is in Amazon S3, some is in ElasticSearch, some is on HDFS stored in Parquet files. Different teams will set up different BI tools and charts that read from a specific silo of data. At the lowest level, the different data formats are incompatible–you have to transform MySQL data in order to merge it with S3 data. On top of that, engineers doing data science work are using Spark, Pandas, and other tools that pull lots of data into memory–if the in-memory formats are not compatible, the data teams can’t get the most out of their work. On top of THAT, at the highest level, data analysts are working with different data analysis tools, so there is even more siloing. Now I understand why Dremio took two years to bring to market. They are trying to solve data interoperability by making it easy to transform data sets between different formats. They are trying to solve data access speed by creating a sophisticated caching system. And they are trying to improve the effectiveness of the data analysts by providing the right abstractions for someone who is not a software engineer to study the different data sets across an organization. Dremio is an exciting project because it is rare to see a pure software company put so many years into up-front stealth product development. After talking to Tomer in this conversation, I’m looking forward to seeing Dremio come to market. It was fascinating to hear him talk about how data engineering has evolved to today.

Showing 1,601–1,620 of 2,221 episodes