Chess.com started in 2007 and grew steadily in the years following. The platform exploded in popularity during the pandemic, to the point that their servers struggled with the traffic. It was a great problem to have. Chess.com was instrumental in helping to elevate chess to its current height of mainstream popularity. But how did Chess.com The post Building Chess.com with Jay Severson appeared first on Software Engineering Daily.

Mastodon is an open source, decentralized social network. Eugen Rochko started building Mastodon in response to his dissatisfaction with centralized social networks like Facebook and Twitter. In the Mastodon model, users can run their own nodes, and other users can connect to them. You can follow users whose accounts reside in other nodes. Eugen joins The post Mastodon with Eugen Rochko appeared first on Software Engineering Daily.

This episode of Software Engineering Daily is part of our on-site coverage of AWS re:Invent 2023, which took place from November 27th through December 1st in Las Vegas. In today’s interview, host Jordi Mon Companys speaks with Mike Miller who is the Director of AWS AI Devices. Jordi Mon Companys is a product manager and The post AWS re:Invent Special: PartyRock Generative AI Apps with Mike Miller appeared first on Software Engineering Daily.

This episode of Software Engineering Daily is part of our on-site coverage of AWS re:Invent 2023, which took place from November 27th through December 1st in Las Vegas. In today’s interview, host Jordi Mon Companys speaks with Kevin Kelly who is the Director of the AWS Cloud Institute. Jordi Mon Companys is a product manager The post AWS re:Invent Special: The AWS Cloud Institute with Kevin Kelly appeared first on Software Engineering Daily.

This episode of Software Engineering Daily is part of our on-site coverage of AWS re:Invent 2023, which took place from November 27th through December 1st in Las Vegas. In today’s interview, host Jordi Mon Companys speaks with Rob Zuber who is the CTO at CircleCI. Jordi Mon Companys is a product manager and marketer that The post AWS re:Invent Special: CircleCI with Rob Zuber appeared first on Software Engineering Daily.

SQL databases were built for data consistency and vertical scalability. They did this very well for the long era of monolithic applications running in dedicated, single-server environments. However, their design presented a problem when the paradigm changed to distributed applications in the cloud. This shift eventually ushered in the rise of distributed SQL databases. One The post CockroachDB with Jordan Lewis appeared first on Software Engineering Daily.

Frédéric Harper is the Principal Developer Advocate at Kubefirst, which is an open source platform that integrates some of the most popular tools in the Kubernetes space. Frédéric has deep experience at major software companies having worked at npm, Mozilla, Microsoft, DigitalOcean, Fitbit, and others. He joins the show to talk about the challenges and The post Kubefirst with Frédéric Harper appeared first on Software Engineering Daily.

Supabase is an open source backend-as-a-service platform and competes directly with Google’s Firebase. A key distinction between them is that Firebase is a document store, while Supabase uses Postgres, which is a SQL-based database management system. Software Engineering Daily last covered Supabase in 2020 when its Founder Paul Copplestone came on the show, and a The post Supabase Security with Inian Parameshwaran appeared first on Software Engineering Daily.

Containers make it possible to standardize the deployment of software to any compute environment. However, managing and orchestrating containers at scale is a major challenge. Kubernetes was originally created by Google and solves the problem of scaling container deployment. Ben Elder is a Senior Software Engineer at Google, and an Elected Member of the Kubernetes The post Kubernetes at Google with Ben Elder appeared first on Software Engineering Daily.

This episode of Software Engineering Daily is part of our on-site coverage of KubeCon 2023, which took place from November 6th through 9th in Chicago. In today’s interview, host Jordi Mon Companys speaks with Justin Cormack who is the CTO at Docker. Jordi Mon Companys is a product manager and marketer that specializes in software The post KubeCon Special: Docker with Justin Cormack appeared first on Software Engineering Daily.

This episode of Software Engineering Daily is part of our on-site coverage of KubeCon 2023, which took place from November 6th through 9th in Chicago.   In today’s interview, host Jordi Mon Companys speaks with Darren Sheppard who is the Chief Architect and Co-Founder at Acorn Labs. Jordi Mon Companys is a product manager and The post KubeCon Special: Acorn with Darren Sheppard appeared first on Software Engineering Daily.

This episode of Software Engineering Daily is part of our on-site coverage of KubeCon 2023, which took place from November 6th through 9th in Chicago. In today’s interview, host Jordi Mon Companys speaks with David DeSanto who is the Chief Product Officer at GitLab. Jordi Mon Companys is a product manager and marketer that specializes The post KubeCon Special: GitLab’s AI Vision with David DeSanto appeared first on Software Engineering Daily.

Shopify is an e-commerce platform focused on enabling small businesses to sell online. The company was founded in 2006 and since then has become a core technology of online business infrastructure. Mike Shaver is a Distinguished Engineer at Shopify and previously worked at Facebook, Mozilla, Oracle and others. At Shopify he works on the core The post Shopify with Mike Shaver appeared first on Software Engineering Daily.

Observability is becoming an increasingly competitive space in the software world. Many developers have heard of Datadog and New Relic, but there are a seemingly countless number of observability products out there. Costa Tsaousis (he/him) is the Founder and CEO of Netdata. His goal was to build an open-source platform that was high-resolution, real-time, and The post A Different Monitoring Philosophy with Costa Tsaousis appeared first on Software Engineering Daily.

If you’re a developer, you’ve probably worked with an API, or application programming interface. An API is a set of rules for how to communicate with an applications or device. For example, when you build an app and want to use Stripe to handle payments, or use Slack to deliver notifications, it’s APIs that make The post Postman and the Growth of APIs with Joyce Lin appeared first on Software Engineering Daily.

There are hundreds of observability companies out there, and many ways to think about observability, such as application performance monitoring, server monitoring, and tracing. In a production application, multiple tools are often needed to get proper visibility on the application. This creates some challenges. Applications can produce lots of different observatory observability data, but how The post Observability with Eduardo Silva appeared first on Software Engineering Daily.

The Hypertext Transfer Protocol, or HTTP, is used to load webpages using hypertext links, and it’s the foundation of the web. Tim Berners-Lee famously created HTTP version 0.9 in 1989, and defined the essential behavior of a client and a server. Version 1.0 was eventually finalized in 1996, and its secure variant called HTTPS is The post The Future of HTTP with Nick Shadrin and Roman Arutyunyan appeared first on Software Engineering Daily.

A platform as a service, or PaaS, is the concept of a complete development and deployment environment in the cloud. One of the best examples is Heroku, which was created in 2007 and later acquired by Salesforce. Although these services are great for helping startups get off the ground quickly, they can ultimately become a The post Flightcontrol and Going Beyond Heroku with Brandon Bayer appeared first on Software Engineering Daily.

Picture a user interacting with a web app on their phone. When they tap the screen the app triggers communication with a server, which in turn communicates with a database. This process then happens in reverse to eventually update what the user sees on-screen. The latency for this round trip depends a lot on the The post Edge Databases with Glauber Costa appeared first on Software Engineering Daily.

Developer experience, or DevX, is a critical aspect of modern software development that focuses on creating a seamless and productive environment for developers. It encompasses everything from the tools and technologies used in the development process to the documentation, libraries, and frameworks available to streamline coding tasks. An emphasis on DevX can enhance individual developer The post AI-Powered DevX at AWS with Deepak Singh appeared first on Software Engineering Daily.

Infrastructure as code refers to the use of software and configuration files to convey infrastructure specifications. This is in contrast to the traditional approach of manually provisioning servers, operating systems, storage, and other infrastructure components. With the growth of cloud computing, the infrastructure as code paradigm is becoming more integral to managing compute resources. AutoCloud The post AutoCloud and Infrastructure as Code with Tyson Kunovsky appeared first on Software Engineering Daily.

Incident management is the process of responding to unplanned events or service interruptions, and then restoring service to an operational state. Having robust incident management is vital to many software teams. Blameless is a framework designed to help software companies manage their production incidents effectively. It provides a workflow for managing the incident response as The post Blameless with Ken Gavranovic appeared first on Software Engineering Daily.

Cloud applications continue to grow in popularity, but ensuring the security of these applications often presents a formidable engineering challenge. This challenge motivated the creation of Jit. Jit is a continuous security platform for developers, and seeks to enable every cloud app to start with minimum viable security, or MVS, without slowing development velocity. David The post Minimum Viable Security for Cloud Apps with David Melamed appeared first on Software Engineering Daily.

ScyllaDB is a fast and highly scalable NoSQL database designed to provide predictable performance at a massive cloud scale. It can handle millions of operations per second at a scale of gigabytes or petabytes. It’s also designed to be compatible with Cassandra and DynamoDB APIs. Scylla is used by Zillow, Comcast, and for Discord’s 350M+ The post Highly Scalable NoSQL with Dor Laor appeared first on Software Engineering Daily.

Exafunction is a leader in deploying deep learning models at scale. One of their products is Codeium, a coding assistant for software developers based on Exafunction’s deep learning technology. Codeium provides AI-assisted autocompletion in your IDE, making it easier for you to incorporate deep learning technology in your software development workflow. Varun Mohan is the The post Modern Coding Superpowers with Varun Mohan appeared first on Software Engineering Daily.

The CAP theorem, also known as Brewer’s theorem, is a fundamental principle in distributed systems that states that it is impossible to simultaneously achieve three desirable properties in a distributed data system: Consistency, Availability, and Partition tolerance. Eric Brewer is the VP of Infrastructure & Google Fellow at Google and he joins us today. This The post CAP Theorem 23 Years Later with Eric Brewer appeared first on Software Engineering Daily.

Elasticsearch is the most established solution today to search and analyze large amounts of logs. However, it can be costly and complex to manage. Quickwit searches large amounts of append only cloud data like logs or ledgers in a fraction time with significantly less cost than Elasticsearch. In this episode, we interview Paul Masurel, one The post Cloud-native Search with Paul Masurel appeared first on Software Engineering Daily.

Hyperscalers refer to expansive cloud service providers capable of delivering enterprise-scale computing and storage services. These Hyperscalers like Google, Amazon, Facebook that have huge data centers and are either running their own software or renting out this infrastructure realized a long time back that the traditional Network, Storage & Compute server racks were not enough The post Shipping Oxide with Bryan Cantrill appeared first on Software Engineering Daily.

CRaC (Coordinated Restore at Checkpoint) is a new technology that can improve startup and warmup times by orders of magnitude. It is a project of OpenJDK that was proposed and led by Azul. The CRaC Project defines public Java APIs that allow for the coordination of resources during checkpoint and restore operations. With CRaC, a The post Superfast JVM Startup with Gerrit Grunwald appeared first on Software Engineering Daily.

WebAssembly is a low-level binary format for the web that is compiled from other languages to offer maximized performance and is meant to augment the places where JavaScript isn’t sufficient. High Performance applications like AutoCAD, Figma and Photoshop are now leveraging the capabilities of WebAssembly to provide native experiences on the web. Matt Butcher is The post WebAssembly with Matt Butcher appeared first on Software Engineering Daily.

In this episode, we are talking to Robert Cooke, founder and CTO of 3forge. He has spent the last decade creating a full stack software platform that revolutionized enterprise real-time data management, visualization, and workflows through its inventive “high impact code” concept. With offices in New York, London, and Singapore, 3forge has been serving a The post Building Enterprise Applications with Robert Cooke appeared first on Software Engineering Daily.

Usage-Based Pricing is becoming more and more popular. Led by the wild popularity of cloud service providers such as AWS, customers are demanding more visibility into usage information and more accountability for the dollars they spend on a service. But historically service oriented products have shied away from usage based pricing because of the complexity The post Solving Usage-Based Pricing with Puneet Gupta appeared first on Software Engineering Daily.

Mach5 Search is a slide-in, cloud-native replacement for Elasticsearch and OpenSearch that immediately saves up to 90% in operating cost. Mach5 Search can run on top of Google BigQuery, Snowflake, and Databricks, or natively on Object Stores in all the major clouds. Vinayak Borkar is the CEO and Co-Founder of Mach5 Software and he joins The post Cloud Native Search with Vinayak Borkar appeared first on Software Engineering Daily.

Metrist is an observability platform designed to commoditize application observability. Jeff Martens is the CEO of Metrist, and he joins us today. This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential The post Observability for Your Cloud Dependencies with Jeff Martens appeared first on Software Engineering Daily.

Crossplane is an innovative open source control plane framework that helps companies provide managed access to cloud native control planes. Upbound provides a single global platform to build, deploy, and operate these internally managed control planes that are powered by cross plane. Bassam Tabbara is the CEO of Upbound, and he joins us today. Free The post Cloud-native Control Planes with Bassam Tabbara appeared first on Software Engineering Daily.

When your application fails, finding the reason quickly is essential for limiting downtime. Often, most of the time it takes to repair a problem is taken by figuring out what exactly went wrong. PromptOps is a service designed to make tracing the answer to why something happened back to the original cause. Dev Nag is The post PromptOps with Dev Nag appeared first on Software Engineering Daily.

Cloud native technologies empower organizations to build and run scalable applications in modern, dynamic environments such as public, private, and hybrid clouds. Containers, service meshes, microservices, immutable infrastructure, and declarative APIs exemplify this approach. These techniques enable loosely coupled systems that are resilient, manageable, and observable. Combined with robust automation, they allow engineers to make The post Cloud Native in 2023 with Chris Aniszczyk appeared first on Software Engineering Daily.

Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications. Although it improves the management and scaling of infrastructure and applications, Kubernetes frequently has challenges managing the complexity of releasing applications. Git is the most widely used version-control system in the software industry today. GitOps is a set of procedures that uses The post GitOps for Kubernetes with Priyanka Ravi and Stefan Prodan appeared first on Software Engineering Daily.

The cloud has become an all-encompassing platform for running diverse applications and enabling individuals and teams to add value by utilizing services and infrastructure that streamline the process of software building and operation. Nonetheless, the cloud has presented new hurdles for developers as it is intricate, and application development demands comprehension of cloud service intricacies. The post Cloud-Oriented Programming (Part 2) with Elad Ben-Israel appeared first on Software Engineering Daily.

The goal of having a single pane of glass that allows us to see what is happening with our organization’s IT operations has been a long-standing goal for many organizations. The goal makes a lot of sense. Without a clear end-to-end picture, it is hard to determine where your problems are if you can’t determine The post Challenges of Multi-Cloud and Hybrid Monitoring appeared first on Software Engineering Daily.

The cloud has become an all-encompassing platform for running diverse applications and enabling individuals and teams to add value by utilizing services and infrastructure that streamline the process of software building and operation. Nonetheless, the cloud has presented new hurdles for developers as it is intricate, and application development demands comprehension of cloud service intricacies. The post Cloud-Oriented Programming (Part 1) with Elad Ben-Israel appeared first on Software Engineering Daily.

Finout helps FinOps, DevOps, and Finance to manage & reduce cloud spend and improve the company’s profitability without adding code or changing existing tags. Roi Ravhon is the Co-founder and CEO at Finout and he joins us today. Full disclosure: Finout is a sponsor of Software Engineering Daily. Alex is an AWS Data Hero, an The post Cloud Cost Management with Roi Ravhon appeared first on Software Engineering Daily.

Distributed databases are necessary for storing and managing data across multiple nodes in a network. They provide scalability, fault tolerance, improved performance, and cost savings. By distributing data across nodes, they allow for efficient processing of large amounts of data and redundancy against failures. They can also be used to store data across multiple locations The post Turso: Globally Replicated SQLite with Glauber Costa appeared first on Software Engineering Daily.

The increasing complexity of modern cloud-native architectures has led to the emergence Platform Engineering. This practice involves the development and upkeep of an integrated product, known as an “Internal Developer Platform,” which serves as a flexible and supported abstraction layer between application developers and the underlying technologies. Luca Galante leads Product at Humanitec and he The post Platform Engineering with Luca Galante appeared first on Software Engineering Daily.

What are the reasons why we need to improve performance? The JVM is a powerful piece of software – this is the reason why Java has maintained its popularity over the past 26+ years. But… there are some ways that the JVM works that can impact the performance of your applications. One of the biggest The post Improved Java Performance Sans Code Changes with Simon Ritter appeared first on Software Engineering Daily.

Netlify is a cloud-based platform that provides web developers with an all-in-one workflow to build, deploy, and manage modern web projects. Matt Biilmann is the CEO of Netlify and he joins us today. This episode is hosted by Mike Bifulco. To learn more about Mike visit mikebifulco.com Sponsorship inquiries: sponsor@softwareengineeringdaily.com     The post Netlify with Mathias Biilmann Christensen appeared first on Software Engineering Daily.

Maintaining availability in a modern digital application is critical to keeping your application operating and available and to keep meeting your customers growing demands. There are many observability platforms out there and certainly Prometheus is a popular open source solution for cloud native companies yet operating an observability platform, costs money, and all of the The post Cloud Native Observability with Martin Mao appeared first on Software Engineering Daily.

Originally published on February 12, 2022. Lee Atchison spent seven years at Amazon working in retail, software distribution, and Amazon Web Services. He then moved to New Relic, where he has spent four years scaling the company’s internal architecture. From his decade of experience at fast-growing web technology companies, Lee has written the book Architecting The post Architecting for Scale with Lee Atchison appeared first on Software Engineering Daily.

Cloud native applications utilizing microservice architectures has grown into one of the most popular application architectural patterns in recent years. The value of leveraging dynamic cloud resources, along with the flexibility and scalability of microservice architectures, creates a strong paradigm that’s hard to miss. The strong adoption of Kubernetes has strengthened the pattern enormously. The The post K8s Troubleshooting with Itiel Shwartz appeared first on Software Engineering Daily.

Enterprise-grade authentication is often an essential ingredient to virtually all applications in today’s world. However, companies often have a hard time understanding the value of that authentication especially during the early stages of product development. And hardening of an application is often left as an afterthought. Add enterprise-level requirements such as single sign-on and two-factor The post WorkOS with Michael Grinich appeared first on Software Engineering Daily.

Mailchimp is an all-in-one marketing platform for growing businesses, empowering millions of customers around the world to launch, build, and grow their businesses with world-class marketing technology, award-winning customer support, and inspiring content. Eric Muntz is Mailchimp’s CTO, responsible for the engineering teams that design, implement, and maintain Mailchimp’s products and infrastructure. He joins the The post Mailchimp Engineering with Eric Muntz appeared first on Software Engineering Daily.

Venture capital investment has continued to flow into technology startups.  No one builds technology from scratch.  There are cloud services, software libraries, 3rd party services, and software platforms that modern entrepreneurs must adopt to build their products efficiently and quickly.  These layers of infrastructure are a key area for many investors. Tim is a partner The post Technical Venture Capital with Tim Tully appeared first on Software Engineering Daily.

This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His most recent book, Architecting for Scale (O’Reilly Media) is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments. Lee is the host The post Kubernetes Spend with Webb Brown appeared first on Software Engineering Daily.

The Kubernetes ecosystem has drastically changed how development teams ship software. While Kubernetes has provided many advancements in cloud infrastructure, it has also left organizations with massive security blindspots. KSOC was created to give developers and security teams a single control plane to harden multi-cluster Kubernetes environments through event-driven analysis, least privilege enforcement, and remediation-as-code. The post Kubernetes Security Compliance with Jimmy Mesta appeared first on Software Engineering Daily.

In software engineering, telemetry is the data that is collected about your applications. Unlike logging, which is used in the development of apps to pinpoint errors and code flows, telemetry data includes all operational data including logs, metrics, events, traces, usage, and other analytical data. Companies usually visualize this information to troubleshoot problems and understand The post New Relic Architecture with Nic Benders appeared first on Software Engineering Daily.

One of the challenges with Microservices architecture is how you manage dependencies between your services when implementing workflows. Conductor is an open-source microservices and workflow orchestration platform. Boney Sekh co-founded Orkes Inc – a company focused on offering Conductor as a service. Boney joins the show to discuss how engineers leverage Conductor to build highly The post Conductor Orchestration with Boney Sekh appeared first on Software Engineering Daily.

The advent of the cloud introduced a new form of technical debt in which organizations can lose track of what infrastructure they have and how it relates to the business.  While the cloud’s native APIs offer some transparency into your infrastructure, these offerings are often described as necessary but not sufficient.  When companies have a The post CloudGraph with Tyson Kunovsky appeared first on Software Engineering Daily.

WP Engine is a domain specific cloud provider that hosts high performance WordPress infrastructure. This website, Software Engineering Daily, runs on WP Engine. Scaling a domain specific cloud provider for WordPress includes complexities at the level of the database, application, load balancer, and other areas. Brandon DuRutte from WP Engine joins the show to talk The post Scaling WordPress with Brandon DuRette appeared first on Software Engineering Daily.

Loft is a platform for Kubernetes self-service and multi-tenancy. Loft allows you to control Kubernetes clusters with added multi-tenancy and self-service capabilities to get more value out of Kubernetes beyond simply cluster management. It allows for cost optimization, more efficient provisioning, and other features. Lukas Gentele joins the show to talk about Kubernetes multi-tenancy and The post Loft Kubernetes Namespaces with Lukas Gentele appeared first on Software Engineering Daily.

Ben Sigelman   Alex Kehlenbeck Observability consists of metrics, logs, and traces. Lightstep is a company that builds distributed tracing infrastructure, which requires them to store and serve high volumes of trace data. There are numerous architecture challenges that come with managing this data. Ben Sigelman and Alex Kehlenbeck join the show to discuss the implementation The post Distributed Tracing Infrastructure with Ben Sigelman and Alex Kehlenbeck appeared first on Software Engineering Daily.

SingleStore is a multi-use, multi-model database designed for transactional and analytic workloads, as well as search and other domain specific applications. SingleStore is the evolution of the database company MemSQL, which sought to bring fast, in-memory SQL database technology to market. Jordan Tigani is Chief Product Officer of SingleStore and joins the show to talk The post SingleStore with Jordan Tigani appeared first on Software Engineering Daily.

Vantage is a system for optimizing cloud costs. It provides tools and interfaces for developers to analyze how they are spending on AWS resources, and has recently expanded into GCP as well. Vantage users gain an easy interface into their costs that would otherwise be hard to analyze via the raw AWS console. Ben Schaechter The post Vantage Engineering with Ben Schaechter appeared first on Software Engineering Daily.

Managing Kubernetes nodes leads to operational complexity, security issues, and nodes that are perhaps more expensive to run than necessary. Deferring the node management to an underlying platform abstracts away these problems and can improve operations. Madhuri Yechuri runs Elotl, a nodeless Kubernetes platform. She joins the show to talk about the architecture and purpose The post Nodeless Kubernetes with Madhuri Yechuri appeared first on Software Engineering Daily.

Lee Atchison spent seven years at Amazon working in retail, software distribution, and Amazon Web Services. He then moved to New Relic, where he has spent four years scaling the company’s internal architecture. From his decade of experience at fast-growing web technology companies, Lee has written the book Architecting for Scale, from O’Reilly. As an The post Architecting for Scale with Lee Atchison appeared first on Software Engineering Daily.

Linkerd is a service mesh that runs efficiently with a low memory footprint. We have covered the details of Linkerd in previous episodes. Buoyant is the company that sells Linkerd as a service, and today’s show focuses on the engineering details of the company, and how Linkerd is architected in 2022. William Morgan is the The post Buoyant Cloud with William Morgan appeared first on Software Engineering Daily.

Database product companies typically have a few phases. First, the company will develop a technology with some kind of innovation such as speed, scalability, or durability. The company will offer support contracts around that technology for a period of time, before eventually building a managed, hosted offering. PlanetScale is a database company built around the The post Scaling PlanetScale with Sugu Sougoumarane appeared first on Software Engineering Daily.

Rackspace is a multi cloud solutions provider that has evolved beyond its cloud computing origins into a diverse set of services and support offerings. Customers work with Rackspace to adopt cloud application deployments, modern data analytics, and all the other opportunities offered by cloud computing. Much of this occurs through partnerships where Rackspace provides teams The post Rackspace with Jeff DeVerter appeared first on Software Engineering Daily.

Infrastructure as code is a concept that has delighted software engineers, dev ops, and engineering management across the board. It’s neither fun nor efficient to configure the infrastructure and environments software teams require. Operating software at scale on a cloud, on-prem, or hybrid model is a problem of modernity that many enterprises find surprisingly challenging. The post Infrastructure as Code with Rob Hirschfeld appeared first on Software Engineering Daily.

As cloud providers enable greater levels of specificity and control, they empower compliance-driven enterprise companies.  This level of parameterization is downright inhospitable to a new software engineer and can be a cognitive barrier to entry for a senior professional with a great idea but limited time.  Developers want to focus on their code, algorithms, front The post Render with Anurag Goel appeared first on Software Engineering Daily.

Thanks to the amazing books, blogs, videos, quickstarts, frameworks, and other software-related resources, getting started as a software engineer is easier than ever.  Although you can get started in a day, it can take years to become a master of the craft and most practitioners describe it as a profession of lifelong learning. Titus Winters The post Software Engineering at Google with Titus Winters appeared first on Software Engineering Daily.

Modern businesses run on the cloud and increasingly so they run on multi-cloud infrastructure.  As any growing company can tell you, cloud costs can easily run far out of control.  Today’s enterprises are trying to deliver new products and services at a fast pace.  That needs to be done in a cost-effective, ideally cloud-agnostic way. The post Yotascale with Jake Reichert appeared first on Software Engineering Daily.

One of the most painful parts of getting started on a new development team is getting one’s environment set up.  Whether it’s undocumented steps, overly complex setups, or simply the challenges of understanding how the pieces fit together, getting a dev environment up often feels like a chore to be suffered through in order to The post Remote Development in the Cloud with Gitpod & OpenVSCode Server with Sven Efftinge appeared first on Software Engineering Daily.

The last 15 years have seen the emergence of cloud-based developer APIs and services as dominant components of the developer toolchain. As a result, there has never been more power at developers’ fingertips. But making that power usable and accessible is a challenge that is shared between the providers and the consumers of these services. The post Developer Relations at Google with Luke Mahe appeared first on Software Engineering Daily.

The way we write, compile, and run software has continued to evolve since computer programming began.  The cloud, serverless, no-code, and CI/CD are all contemporary ideas introduced to help software engineers spend more time on their application and less time on the chores of running it.  Darklang is a new way of building serverless backends.  The post Darklang Deployless Applications with Paul Biggar appeared first on Software Engineering Daily.

An application network is a way to connect applications, data and devices through APIs that expose some or all of their assets and data on the network. That network allows other consumers from other parts of the business to come in and discover and use those assets (mulesoft.com). The company Tetrate provides the tools necessary The post Tetrate: Application Aware Networking with Varun Talwar appeared first on Software Engineering Daily.

Whether organizing projects, working from home, or conducting business, you need to use many necessary apps and cloud services to do the job. Despite these apps being necessary, switching between them and keeping them interconnected and updated is a challenge. They very easily become disorganized which makes using them less efficient. The company ClickUp solves The post ClickUp: Workflow Vision with Zeb Evans appeared first on Software Engineering Daily.

Kubernetes is an open source container orchestration service released by Google in 2014. It has quickly grown into a platform with a huge community of enthusiasts and professionals. Besides becoming the de facto standard for container orchestration, it has fostered an ecosystem of related tools and services with increasing power and sophistication (opensource.com). Argo, a The post Argo: Kubernetes-Native Tools with Alex Collins appeared first on Software Engineering Daily.

Pay-as-you-go pricing has become a strong selling point for modern SaaS companies as well as cloud-based companies. Public cloud providers, for example, typically only charge you for only what you use. But implementing this option is challenging because it requires advanced platform analytics. The company Octane is a drop-in metered billing system that gives you The post getoctane.io: Pay-As-You-Go Pricing with Akash Khanolkar appeared first on Software Engineering Daily.

Serverless computing is a cloud computing solution that lets developers deploy applications to containers without managing the servers themselves. Servers and resources are provisioned automatically, pay only for what you use, and experience little to no errors or downtime (ionos). Google Cloud Run is a managed compute platform that enables you to run containers that The post Cloud Run: Serverless Applications with Steren Giannini appeared first on Software Engineering Daily.

Latency is the time it takes to get from point A to point B. In programming, this might be the time from a user selecting their photos library to the pictures reaching their computer screen from the database.  Fly.io is a simple platform for running full-stack apps and databases close to your users. Some available The post Fly.io: Geographic App Deployment with Kurt Mackey appeared first on Software Engineering Daily.

Kubernetes is an open-source container orchestration system. It makes managing container clusters possible as well as deploying code changes to these containers. Microservice architecture is widely used today in large part because of Kubernetes. However, using it can require a large time commitment due to its learning curve. The company Okteto empowers developers to innovate The post Okteto: Cloud-Native Applications on Kubernetes with Ramiro Berreleza appeared first on Software Engineering Daily.

In today’s containerized world, it’s common to encounter similar issues with known solutions across multiple pods. For most people there are 2 solutions: go pod-by-pod finding and fixing the problem, or do that while also spending months trying to automate that process. This is significant time and manual labor. The company Shoreline orchestrates real-time debugging The post Shoreline: Fleet Automation with Anurag Gupta appeared first on Software Engineering Daily.

Apache Pulsar is a cloud-native, distributed messaging and streaming platform originally created at Yahoo! and now a top-level Apache Software Foundation project (pulsar.apache.org). Pulsar is used by many large companies like Yahoo!, Verizon media, Tencent, and Splunk. The company DataStax, an open, multi-cloud stack for modern data apps, has added to their product stack Astra The post Pulsar Revisited with Jonathan Ellis appeared first on Software Engineering Daily.

The company Dynatrace provides intelligent observability, continuous automation, and causation-based AI to help Cloud Ops, DevOps, and SRE teams transform faster, innovate more, and deliver better business outcomes. They offer application performance monitoring, infrastructure monitoring, cloud automation, application security, and much more. While being an industry leader in simplifying complex cloud applications, Dynatrace is continuously The post Dynatrace for DevOps and SRE with Aloïs Reitbauer appeared first on Software Engineering Daily.

Corey Quinn is guest hosting on Software Engineering Daily this week, presenting a Tour of the Cloud. Corey Quinn is the Chief Cloud Economist at The Duckbill Group, where he helps companies fix their AWS bill by making it smaller and less horrifying. If you’re looking to lower your AWS bill or negotiate a new The post Oracle Cloud with Salman Paracha appeared first on Software Engineering Daily.

Corey Quinn is guest hosting on Software Engineering Daily this week, presenting a Tour of the Cloud. Corey Quinn is the Chief Cloud Economist at The Duckbill Group, where he helps companies fix their AWS bill by making it smaller and less horrifying. If you’re looking to lower your AWS bill or negotiate a new The post Digital Ocean with John Allspaw appeared first on Software Engineering Daily.

Corey Quinn is guest hosting on Software Engineering Daily this week, presenting a Tour of the Cloud. Corey Quinn is the Chief Cloud Economist at The Duckbill Group, where he helps companies fix their AWS bill by making it smaller and less horrifying. If you’re looking to lower your AWS bill or negotiate a new The post GCP with Liz Fong-Jones appeared first on Software Engineering Daily.

Corey Quinn is guest hosting on Software Engineering Daily this week, presenting a Tour of the Cloud. Corey Quinn is the Chief Cloud Economist at The Duckbill Group, where he helps companies fix their AWS bill by making it smaller and less horrifying. If you’re looking to lower your AWS bill or negotiate a new The post Azure with Troy Hunt appeared first on Software Engineering Daily.

Corey Quinn is guest hosting on Software Engineering Daily this week, presenting a Tour of the Cloud. Corey Quinn is the Chief Cloud Economist at The Duckbill Group, where he helps companies fix their AWS bill by making it smaller and less horrifying. If you’re looking to lower your AWS bill or negotiate a new The post AWS with Pete Cheslock appeared first on Software Engineering Daily.

Next week Corey Quinn will be guest hosting on Software Engineering Daily, presenting a Tour of the Cloud. Corey Quinn is the Chief Cloud Economist at The Duckbill Group, where he helps companies fix their AWS bill by making it smaller and less horrifying. If you’re looking to lower your AWS bill or negotiate a The post Special Edition Repeat: AWS Analysis with Corey Quinn appeared first on Software Engineering Daily.

AWS Outposts is a fully managed service that offers the same AWS infrastructure, AWS services, APIs, and tools to virtually any datacenter, co-location space, or on-premises facility for a truly consistent hybrid experience. AWS Outposts is ideal for workloads that require low latency access to on-premises systems, local data processing, data residency, and migration of The post AWS Outpost Engineering with Joshua Burgin appeared first on Software Engineering Daily.

Running applications in containerized environments involves regularly organizing, adding and replacing containers. This complex job may involve managing clusters of containers in different geographic locations with different configuration requirements. Platforms like Kubernetes are great for managing this complexity, but include steep learning curves to efficiently get anything off the ground. The company Portainer provides a The post Portainer: Container Management with Neil Cresswell appeared first on Software Engineering Daily.

Apache Spark is a popular open source analytics engine for large-scale data processing. Applications can be written in Java, Scala, Python, R, and SQL. These applications have flexible options to run on like Kubernetes or in the cloud.  The company Data Mechanics is a cloud-native Spark platform for data engineers. It runs continuously optimized Apache The post Data Mechanics: Data Engineering with Jean-Yves Stephan appeared first on Software Engineering Daily.

Microservice architecture has become very common over the past few years because of the availability of containers and container orchestrators like Kubernetes. While containers are overall positive for scaling apps and making them more available, they’ve also introduced hurdles like persisting data and state, and container restarts or pod failures. Development teams put significant work The post Temporal Product: Managing State with Ryland Goldstein appeared first on Software Engineering Daily.

Cloud computing provides tools, storage, servers, and software products through the internet. Securing these resources is a constant process for companies deploying new code to their cloud environments. It’s easy to overlook security flaws because company applications are very complex and many people work together to develop them. Wyze Labs, for example, had millions of The post Bridgecrew: Cloud Security with Barak Schoster appeared first on Software Engineering Daily.

Large portions of software development budgets are dedicated for testing code. A new component may take weeks to thoroughly test, and even then mistakes happen. If you consider software defects as security issues then the concern goes well beyond an application temporarily crashing. Although even minor bugs can cost companies a lot of time to The post Speedscale: Automated Testing with Ken Ahrens and Matt LeRay appeared first on Software Engineering Daily.

IT infrastructure are the components required to operate IT environments, like networks, virtual machines or containers, an operating system, hardware, data storage, etc…. As companies build out different deployment environments with infrastructure configurations, they must maintain the different environments, replicate them, and update them. The management of infrastructure, often automated to some extent, is referred The post Gold Fig Labs: Cloud Infrastructure Security with Vikrum Nijjar and Greg Soltis appeared first on Software Engineering Daily.

Software-Defined Networking describes a category of technologies that separate the networking control plane from the forwarding plane. This enables more automated provisioning and policy-based management of network resources. Implementing software-defined networking is often the task of Site Reliability Engineers, or SREs. Site reliability engineers work at the intersection of development and operations by bringing software The post Equinix Infrastructure with Tim Banks appeared first on Software Engineering Daily.

Google Cloud, AWS, and Azure are the dominant cloud providers on the market today. But the market is still highly competitive, and there is significant overlap in the services offered by all three large providers. Since all three offer a broad range of services, developers looking to choose a platform for their application must focus The post Google Cloud Databases with Andi Gutmans appeared first on Software Engineering Daily.

AWS offers over 200 services as part of its IaaS platform, and that number continues to grow. Organizing all of these services, and tracking the costs they incur, can be a significant challenge, often requiring teams of AWS-certified sysadmins working together to get a handle on an enterprise-scale system.  Vantage provides an alternative, streamlined AWS The post Vantage: AWS Console Alternative with Ben Schaechter appeared first on Software Engineering Daily.

Build automation tools automate the process of building code, including steps such as compiling, packaging binary code, and running automated tests. Because of this, build automation tools are considered a key part of a continuous delivery pipeline. Build automation tools read build scripts to define how they should perform a build. Common build scripts include The post Earthly: Build Automation with Vlad Ionescu appeared first on Software Engineering Daily.

Modern SaaS products are increasingly delivered via the cloud, rather than as downloadable, executable programs. However, many potential users of those SaaS products may need that software deployed on-prem, in a private network. Organizations have a variety of reasons for preferring on-prem software, such as security, integration with private tools, and compliance with regulations. The The post Multi-Prem Software Delivery and Management with Grant Miller appeared first on Software Engineering Daily.

Cloud platforms are often categorized as providing either Infrastructure-as-a-Service or Platform-as-a-Service. On one side of the spectrum are IaaS giants such as AWS, which provide a broad range of services for building infrastructure. On the other are PaaS providers such as Heroku and Netlify which abstract away the lower-level choices and focus on developer experience.  The post Digital Ocean Platform with Cody Baker and Apurva Joshi appeared first on Software Engineering Daily.

In the past several years, Kubernetes has become the de-facto standard for orchestrating containerized, stateless applications. Tools such as StatefulSets and Persistent Volumes have helped developers build stateful applications on Kubernetes, but this can quickly become difficult to manage as an application scales. Tasks such as machine learning, distributed AI, and big data analytics often The post KubeDirector with HPE’s Kartik Mathur appeared first on Software Engineering Daily.

Serverless computing refers to an architectural pattern where server-side code is run on-demand by cloud providers, who also handle server resource allocation and operations. Of course, there is a server involved on the provider’s side, but administrative functions to manage that server such as capacity planning, configuration, or management of containers are handled behind-the-scenes, allowing The post Serverless Properties with Johann Schleier-Smith appeared first on Software Engineering Daily.

Cilium is open-source software built to provide improved networking and security controls for Linux systems operating in containerized environments along with technologies like Kubernetes. In a containerized environment, traditional Layer 3 and Layer 4 networking and security controls based on IP addresses and ports, like firewalls, can be difficult to operate at scale because of The post Cilium: Programmable Linux Networking with Dan Wendlant and Thomas Graf appeared first on Software Engineering Daily.

Microservices are built to scale. But as a microservices-based system grows, so does the operational overhead to manage it. Even the most senior engineers can’t be familiar with every detail of dozens- perhaps hundreds- of services. While smaller teams may track information about their microservices via spreadsheets, wikis, or other more traditional documentation, these methods The post OpsLevel: Service Ownership Platform with John Laban and Kenneth Rose appeared first on Software Engineering Daily.

Cost management is growing in importance for companies that want to manage their significant cloud bill. Kubernetes plays an increasing role in modern infrastructure, so managing cost of Kubernetes clusters becomes important as well. Kubecost is a company focused on giving visibility into Kubernetes resources and reducing spend. Webb Brown is a founder of Kubecost The post Kubecost with Webb Brown appeared first on Software Engineering Daily.

Serverless has grown in popularity over the last five years, and the space of applications that can be built entirely with serverless has increased dramatically. This is due to two factors: the growing array of serverless tools (such as edge-located key value stores) and the rising number of companies with serverless offerings. One of those The post Serverless Revolution with Tyler McMullen appeared first on Software Engineering Daily.

Originally published September 13, 2019 Amazon Web Services first came out in 2006. It took several years before the software industry realized that cloud computing was a transformative piece of technology. Initially, the common perspective around cloud computing was that it was a useful tool for startups, but would not be a smart option for The post Cloud-Native Applications with Cornelia Davis (Repeat) appeared first on Software Engineering Daily.

Originally published May 29, 2020 Kubernetes has become a highly usable platform for deploying and managing distributed systems.  The user experience for Kubernetes is great, but is still not as simple as a full-on serverless implementation–at least, that has been a long-held assumption. Why would you manage your own infrastructure, even if it is Kubernetes? The post Kubernetes vs. Serverless with Matt Ward (Repeat) appeared first on Software Engineering Daily.

Microservices route requests between each other. As the underlying infrastructure changes, this routing becomes more complex and dynamic. The interaction patterns across this infrastructure requires operators to create rules around traffic management. Tobias Kunze Briseno is the founder of Glasnostic, a system for ensuring resilience of microservice applications. Tobias joins the show to talk about The post Microservice Routing with Tobias Kunze Briseño appeared first on Software Engineering Daily.

Netlify is a cloud provider for JAMStack applications. To make those applications more performant, Netlify has built out capacity for edge computing–specifically “edge handlers”. Edge handlers can be used for a variety of use cases that need lower latency or other edge computing functionality. Matt Biilmann Christensen is the CEO of Netlify and joins the The post Edge Handlers with Mathias Biilmann Christensen appeared first on Software Engineering Daily.

Cloud resources can get out of control if proper management constraints are not put in place. Cloud Custodian enables users to be well managed in the cloud. It is a YAML DSL that allows you to easily define rules to enable a well-managed cloud infrastructure giving security and cost optimization. Kapil Thangavelu works on Cloud The post Cloud Custodian with Kapil Thangavelu appeared first on Software Engineering Daily.

Containers and virtual machines are two ways of running virtualized infrastructure. Containers use less resources than VMs, and typically use the runc open source container runtime. Sysbox is a containerization runtime that offers an alternative to runc, and allows for the deployment of Docker or Kubernetes within a container. Cesar Talledo is the founder of The post Sysbox: Containerization Runtime with Cesar Talledo appeared first on Software Engineering Daily.

Development environments are brittle and hard to manage. They lack the kind of fungibility afforded by infrastructure-as-code. Gitpod is a company that allows developers to describe development environments as code to make them easier to work with, and enabling a more streamlined GitOps workflow. Johannes Landgraf and Sven Efftinge are creators of Gitpod and they The post Gitpod: Cloud Development Environments with Johannes Landgraf and Sven Efftinge appeared first on Software Engineering Daily.

Ray is a general purpose distributed computing framework. Ray is used for reinforcement learning and other compute intensive tasks. It was developed at the Berkeley RISELab, a research and development lab with an emphasis on practical applications. Ion Stoica is a professor at Berkeley, and he joins the show to talk about the present and The post Ray Ecosystem with Ion Stoica appeared first on Software Engineering Daily.

A private network connects servers, computers, and cloud instances. These networked objects are often separated by firewalls and subnets that create latency and complication. David Crawshaw is the CTO of Tailscale, a company that works to make private networks easier to build and simpler to configure and maintain. David joins the show to talk about The post Tailscale: Private Networks with David Crawshaw appeared first on Software Engineering Daily.

The Java ecosystem is maturing. The GraalVM high performance runtime provides a virtual machine for running applications in a variety of languages. TornadoVM extends the Graal compiler with a new backend for OpenCL. TornadoVM allows the offloading of JVM applications onto heterogeneous hardware. Juan Fumero works on TornadoVM. He joins the show to talk about The post TornadoVM: Accelerating Java with GPUs with Juan Fumero appeared first on Software Engineering Daily.

Every software company works off of several different development environments–at the very least there is staging, testing, and production. Every push to staging can be spun up as an application to be explored, tinkered with, and tested. These ad hoc spin-ups are known as release apps. A release app is an environment for engineers to The post Release Apps with Tommy McClung appeared first on Software Engineering Daily.

Across a company, there is a wide range of resources that employees need access to. Documents, S3 buckets, git repositories, and many others. As access to resources changes across the organization, a history of the changes to permissions can be useful for compliance and monitoring. Indent is a system for simplifying access management across infrastructure. The post Access Control Management with Fouad Matin and Dan Gillespie appeared first on Software Engineering Daily.

Managing microservices becomes a challenge as the number of services within the organization grows. With that many services comes more interdependencies–downstream and upstream services that may be impacted by an update to your service.  One solution to this problem: a dashboard and newsfeed system that lets you see into the health and changes across your The post Cortex: Microservices Management with Anish Dhar and Ganesh Datta appeared first on Software Engineering Daily.

GitHub has been a social network for developers for many years. Most social networks are centered around mobile applications, but GitHub sits squarely in a developer’s browser-based desktop workflow. As a result, the design of a mobile app for GitHub is less straightforward. GitHub did acquire a popular mobile client called GitHawk, which was developed The post GitHub Mobile with Brian Lovin and Ryan Nystrom appeared first on Software Engineering Daily.

A service mesh provides routing, load balancing, policy management, and other features to a set of services that need to communicate with each other. The mesh can simplify operations across these different services by providing an interface to configure them.  There are lots of different vendors who offer service mesh technology: AWS has AppMesh, Google The post Multimesh with Luke Kysow appeared first on Software Engineering Daily.

AWS has over 150 different services. Databases, log management, edge computing, and lots of others. Instead of being overwhelmed by all of these products, an engineering team can simplify their workflow by focusing on a small subset of AWS services–the defaults. Daniel Vassalo is the author of The Good Parts of AWS. An excerpt from The post The Good Parts of AWS with Daniel Vassallo appeared first on Software Engineering Daily.

Kubernetes continues to mature as a platform for infrastructure management. At this point, many companies have well-developed workflows and deployment patterns for working with applications built on Kubernetes. The complexity of some of these deployments may be daunting, and when a new employee joins a company, that employee needs to get quickly onboarded with the The post Tilt: Kubernetes Tooling with Dan Bentley appeared first on Software Engineering Daily.

Kubernetes has become a highly usable platform for deploying and managing distributed systems.  The user experience for Kubernetes is great, but is still not as simple as a full-on serverless implementation–at least, that has been a long-held assumption. Why would you manage your own infrastructure, even if it is Kubernetes? Why not use autoscaling Lambda The post Kubernetes vs. Serverless with Matt Ward appeared first on Software Engineering Daily.

Every software company is a distributed system, and distributed systems fail in unexpected ways.  This ever-present tendency for systems to fail has led to the rise of failure testing, otherwise known as chaos engineering. Chaos engineering involves the deliberate failure of subsystems within an overall system to ensure that the system itself can be resilient The post Distributed Systems Research with Peter Alvaro appeared first on Software Engineering Daily.

Amazon’s virtual server instances have come a long way since the early days of EC2. There are now a wide variety of available configuration options for spinning up an EC2 instance, which can be chosen from based on the workload that will be scheduled onto a virtual machine. There are also Fargate containers and AWS The post AWS Virtualization with Anthony Liguori appeared first on Software Engineering Daily.

Serverless computing is a way of designing applications that do not directly address or deploy application code to servers. Serverless applications are composed of stateless functions-as-a-service and stateful data storage systems such as Redis or DynamoDB.  Serverless applications allow for scaling up and down the entire architecture, because each component is naturally scalable. And this The post Cloudburst: Stateful Functions-as-a-Service with Vikram Sreekanti appeared first on Software Engineering Daily.

When a developer spins up a virtual machine on AWS, that virtual machine could be purchased using one of several types of cost structures. These cost structures include on-demand instances, spot instances, and reserved instances. On-demand instances are often the most expensive, because the developer gets reliable VM infrastructure without committing to long-term pricing. Spot The post Reserved Instances with Aran Khanna appeared first on Software Engineering Daily.

A large cloud provider has high volumes of network traffic moving through data centers throughout the world. These providers manage the infrastructure for thousands of companies, across racks and racks of multitenant servers, and cables that stretch underseas, connecting network packets with their destination. Google Cloud Platform has grown steadily into a wide range of The post Google Cloud Networking with Lakshmi Sharma appeared first on Software Engineering Daily.

A cloud provider gives developers access to virtualized server infrastructure. When a developer rents this infrastructure via an API call, a virtual server is instantiated on physical machines. That virtual server needs to be made addressable through the allocation of an IP address to make it reachable from the open Internet. When the virtual server The post Go Networking with Sneha Inguva appeared first on Software Engineering Daily.

Distributed systems are required to run most modern enterprise software. Application services need multiple instances for scalability and failover. Large databases are sharded onto multiple nodes. Logging services, streaming frameworks, and continuous integration tools all require the orchestration of more than one server. Deploying a distributed system has historically been difficult because the nodes of The post Replicated Software Delivery with Grant Miller and Marc Campbell appeared first on Software Engineering Daily.

The ridesharing infrastructure of Lyft has a high volume of traffic that is mostly handled by servers on AWS. When Vicki Cheung joined Lyft in 2018, the company was managing containers with an internally built container scheduler. One of her primary goals at the company was to move Lyft to Kubernetes. In today’s episode, Vicki The post Lyft Kubernetes with Vicki Cheung appeared first on Software Engineering Daily.

Cloud infrastructure is usually consumed in the form of virtual machines or containers. These VMs or containers are running on a physical host machine that is also running other VMs and containers. This is called multitenancy. Servers across cloud providers such as AWS have a high utilization because there are multiple virtual instances running on The post Packet: Baremetal Infrastructure with Zachary Smith and Nathan Goulding appeared first on Software Engineering Daily.

Edge computing is the usage of servers that are geographically close to the client device. The first common use case for edge computing was CDNs: content-delivery networks. A content delivery network placed media files such as images and videos on multiple servers throughout the world. These are big files, and they take lots of bandwidth The post Edge Computing Platform with Jaromir Coufal appeared first on Software Engineering Daily.

Amazon EC2 (Elastic Compute Cloud) is a virtualized server product that provides the user with scalable compute infrastructure. EC2 was created in 2006 as one of the first three AWS services along with S3 and Simple Queueing Service. Since then, EC2 has provided the core server infrastructure for many of the companies that have been The post Amazon EC2 with Dave Brown appeared first on Software Engineering Daily.

Amazon’s container offerings include ECS (Elastic Container Service), EKS (Elastic Kubernetes Service), and Fargate. Through these different offerings, Amazon provides a variety of ways that a user can manage Kubernetes clusters and standalone container instances. The choice of which containerization system to choose depends on the needs of the user, and the tradeoffs they want The post Amazon Kubernetes with Abby Fuller appeared first on Software Engineering Daily.

When the Kubernetes project was started, Amazon Web Services was the dominant cloud provider. Most of the code that runs AWS is closed source, which prevents an open ecosystem from developing around AWS. Developers who deploy their application onto AWS are opting into a closed, controlled ecosystem, which has both costs and benefits. The software The post Kubernetes Progress with Kelsey Hightower appeared first on Software Engineering Daily.

Cruise is a company that is building a fully automated self-driving car service. The infrastructure of a self-driving car platform presents a large number of new engineering problems. Self-driving cars collect vast quantities of data as they are driving around the city. This data needs to be transferred from the cars onto cloud servers. The The post Kubernetes at Cruise with Karl Isenberg appeared first on Software Engineering Daily.

The container orchestration wars ended in 2016 with Kubernetes being the most popular open source tool for deploying and managing infrastructure. Since that time, most large enterprises have been implementing a “platform strategy” based around Kubernetes. A platform strategy is a plan for creating a consistent experience for software engineers working throughout an enterprise. At The post Linkerd Market Strategy with William Morgan appeared first on Software Engineering Daily.

Kubernetes has created a widespread system for deploying and managing infrastructure. As Kubernetes has been increasingly adopted, companies are thinking about how to leverage that common layer of infrastructure. With the common infrastructure abstraction of Kubernetes, it becomes easier to adopt other abstractions that are uniform across the entire company.  This has created a market The post Istio Market Strategy with Zack Butcher appeared first on Software Engineering Daily.

A cloud provider gives a developer low-cost compute infrastructure on-demand.  Cloud providers can be divided up into two categories: Layer 1 cloud providers and Layer 2 cloud providers. A Layer 1 cloud provider such as Amazon Web Services owns server hardware and sells compute infrastructure as a commodity. A Layer 2 cloud provider purchases compute The post Heroku Infrastructure with Mark Turner appeared first on Software Engineering Daily.

New software abstractions always take advantage of the abstractions that have been built before. Software libraries allow us to import code that sits on the same host as a new program. Open source software let us copy and paste existing code, or clone entire repositories. Cloud providers offer hosted tools and APIs that we can The post Cloud Dependencies with Mya Pitzeruse appeared first on Software Engineering Daily.

“Serverless” is an execution model where applications are scheduled and deployed to servers that are not directly managed by the application developer.  In serverless execution, an application only loads and operates when a user actually needs to get a response from that application. This saves on resources, because many applications do not need to run The post Cloudflare Serverless with Zack Bloom appeared first on Software Engineering Daily.

Java programs run in a different environment than they did ten years ago. Modern infrastructure runs on containers sitting in a Kubernetes cluster. The optimal configuration for a Java program in that context is different than it was for an environment dominated by virtual machines and bare metal. When you are co-scheduling your services with The post GraalVM Quarkus: Java Acceleration with Guillaume Smet and Emmanuel Bernard appeared first on Software Engineering Daily.

Dark Lang is a programming language that is tightly integrated with the cloud.  Dark takes an opinionated approach that most developers are going to want to run their applications in the cloud, and this perspective influences how Dark looks at deployments, IDEs, exception handling, and other aspects of software development. Paul Biggar is the founder The post Dark Lang with Ellen Chisa and Paul Biggar appeared first on Software Engineering Daily.

Modern applications are distributed systems. These applications require an installation mechanism that can run and update the software across multiple nodes.  When a SaaS company starts to work with large enterprise customers, that company needs to figure out a way to deliver their product to the enterprise. This requires the SaaS company to deploy the The post Gravity: Distributed Application Delivery with Ev Kontsevoy appeared first on Software Engineering Daily.

Render is a cloud provider built on top of Amazon Web Services and Google Cloud. Render uses the compute abstractions provided by the major cloud providers to build a second layer cloud provider with the goal of providing a better user experience. Anurag Goel is the founder of Render, and he returns to the show The post How To Build A Cloud Provider with Anurag Goel appeared first on Software Engineering Daily.

Cloud Foundry is a system for managing distributed applications. Cloud Foundry was released in 2011, and has been widely adopted by enterprises that need a platform for deploying and scaling the applications that run within their company. The ecosystem around Cloud Foundry includes systems for continuous delivery, pubsub messaging, and containerization. Abby Kearns is the The post Cloud Foundry with Abby Kearns appeared first on Software Engineering Daily.

A new employee at a software company needs access to a variety of tools. In order to get started working, the employee might need Slack, email, Google Docs, and Amazon Web Services, and all of these require an account with a username and password. Setting up all of these accounts can be time consuming, because The post Okta Engineering with Hector Aguilar appeared first on Software Engineering Daily.

Amazon Web Services first came out in 2006. It took several years before the software industry realized that cloud computing was a transformative piece of technology. Initially, the common perspective around cloud computing was that it was a useful tool for startups, but would not be a smart option for large, established businesses. Cloud computing The post Cloud-Native Applications with Cornelia Davis appeared first on Software Engineering Daily.

The service mesh abstraction allows for a consistent model for managing and monitoring the different components of a microservices architecture.  In the service mesh pattern, each service is deployed with a sidecar container that contains a service proxy. These sidecars are collectively referred to as the “data plane.” Each sidecar provides the service that it The post Service Mesh Deployment with Varun Talwar appeared first on Software Engineering Daily.

Software applications running within a host operating system need to be isolated. Isolation prevents security vulnerabilities, such as one application accessing the memory of another. In modern cloud environments, a single physical host might be running multiple virtual machines on top of a hypervisor. Those virtual machines might be divided up into containers. The different The post gVisor Container Isolation with Michael Pratt and Yoshi Tamura appeared first on Software Engineering Daily.

Sheng Liang was the lead developer on the original Java Virtual Machine. Today he works as the CEO of Rancher Labs, a company building a platform on top of Kubernetes. Sheng joins the show to discuss his experiences in the technology industry. The container orchestration wars had many victims. The competing standards for how an The post Infrastructure Wars with Sheng Liang appeared first on Software Engineering Daily.

Kubernetes has made distributed systems easier to deploy and manage. As Kubernetes has become reliable, engineers have started to look for higher level abstractions we can define on top of Kubernetes. An operator is a method of packaging, deploying, and managing a Kubernetes application. Operators are useful for spinning up distributed systems such as Kafka, The post Kubernetes Operators with Rob Szumski appeared first on Software Engineering Daily.

Cloud computing was popularized in 2006 with the launch of Amazon Web Services. AWS allowed developers to use remote server infrastructure with a simple set of APIs. But even with AWS, it was still not simple to deploy and manage a web application. In 2007, Heroku launched a platform built on top of AWS. Heroku The post Render: High Level Cloud with Anurag Goel appeared first on Software Engineering Daily.

Google Cloud was started with a vision of providing Google infrastructure to the masses. In 2008, it was not obvious that Google should become a cloud provider. Amazon Web Services was finding success among startups who needed on-demand infrastructure, but the traditional enterprise market was not yet ready to buy cloud resources. Googlers liked the The post Kubernetes Vision with Joe Beda appeared first on Software Engineering Daily.

Containers are made to fail gracefully. When your container shuts down due to a hardware or software failure, your distributed application should be able to tolerate that failure. One simple way to be able to tolerate such a failure is to make all of your application logic “stateless.” If your application does not maintain state, The post Kubernetes Storage with Saad Ali appeared first on Software Engineering Daily.

Amazon Web Services is the leading cloud provider by a large margin. Amazon established its lead by being first to market in 2006, with Google and Microsoft taking several years to catch up to the huge business opportunity of the cloud. Since 2008, Google Cloud has been working on cloud products for developers. It started The post Kubernetes Market with Adam Glick appeared first on Software Engineering Daily.

Containers offer a lightweight abstraction for running a server. Cloud providers are able to manage billions of containers from different users, allowing for economies of scale so that each user can pay less. Today, there is a variety of ways that users can deploy containers on a cloud provider. These containers can run in managed The post Service Mesh Interface with Lachlan Evenson appeared first on Software Engineering Daily.

Each cloud provider offers a different set of services which are not always compatible with each other. What are the challenges of building an application that interoperates with multiple different clouds? The first issue is API compatibility. Most cloud providers have a managed SQL offering, a bucket storage system, and server abstractions like virtual machines The post Multicloud Future with Bassam Tabbara appeared first on Software Engineering Daily.

Kubernetes has evolved from a nascent project within Google to a thriving ecosystem of cloud providers, open source projects, and engineers. Tim Hockin is a principal software engineer who has been with Google for 15 years. Tim joins the show to talk about the early days of the Kubernetes projects, and the engineering efforts that The post Kubernetes Development with Tim Hockin appeared first on Software Engineering Daily.

Google’s cloud business was long regarded as a place where startups could build a business, but not established enterprises. For serious workloads, enterprises chose Amazon almost unanimously. This phenomenon of Amazon as the default was described by a phrase that harkened back to the days of IBM’s dominance: “nobody ever got fired for choosing AWS.” The post Google Anthos with Aparna Sinha appeared first on Software Engineering Daily.

A service mesh is an abstraction that provides traffic routing, policy management, and telemetry for a distributed application. A service mesh consists of a data plane and a control plane. In the data plane, a proxy runs alongside each service, with every request from a service being routed through the proxy. In the control plane, The post Service Mesh Wars with William Morgan appeared first on Software Engineering Daily.

Netflix started with a DVD-by-mail product. The software infrastructure and operations practices needed for the DVD business were very different from those needed by a streaming video company. Since the early days of Netflix, CEO Reed Hastings knew that the company would evolve to becoming a streaming video platform. But he did not know when The post Netflix Early Days with Greg Burrell appeared first on Software Engineering Daily.

Alex Balazs is the Intuit Chief Architect and has been working at the company for almost twenty years. Intuit’s products include QuickBooks, TurboTax, and Mint. These applications are used to file taxes, manage business invoices, conduct personal accounting, and other critical aspects of a user’s financial life. Because the applications are managing money for users, The post Scaling Intuit with Alex Balazs appeared first on Software Engineering Daily.

Modern server infrastructure usually runs in a virtualized environment. Virtual servers can exist inside of a container or inside of a virtual machine. Containers can also run on virtual machines. Kubernetes has allowed developers to manage their multiple containers, whether those containers are running in VMs or on bare metal (servers without VMs). As organizations The post Kubernetes Virtualization with Paul Czarkowski appeared first on Software Engineering Daily.

RECENT UPDATES: FindCollabs is a company I started recently The FindCollabs Podcast is out! FindCollabs is hiring a React developer FindCollabs Hackathon #1 has ended! Congrats to ARhythm, Kitspace, and Rivaly for winning 1st, 2nd, and 3rd place ($4,000, $1000, and a set of SE Daily hoodies, respectively). The most valuable feedback award and the The post Cloud with Eric Brewer appeared first on Software Engineering Daily.

RECENT UPDATES: FindCollabs is a company I started recently The FindCollabs Podcast is out! FindCollabs is hiring a React developer FindCollabs Hackathon #1 has ended! Congrats to ARhythm, Kitspace, and Rivaly for winning 1st, 2nd, and 3rd place ($4,000, $1000, and a set of SE Daily hoodies, respectively). The most valuable feedback award and the The post Intricately: Mapping the Internet with Fima Leshinsky appeared first on Software Engineering Daily.

RECENT UPDATES: Podsheets is our open source set of tools for managing podcasts and podcast businesses New version of Software Daily, our app and ad-free subscription service FindCollabs is hiring a React developer FindCollabs Hackathon #1 has ended! Congrats to ARhythm, Kitspace, and Rivaly for winning 1st, 2nd, and 3rd place ($4,000, $1000, and a The post gVisor: Secure Container Sandbox with Yoshi Tamura appeared first on Software Engineering Daily.

RECENT UPDATES: Podsheets is our open source set of tools for managing podcasts and podcast businesses New version of Software Daily, our app and ad-free subscription service FindCollabs is hiring a React developer FindCollabs Hackathon #1 has ended! Congrats to ARhythm, Kitspace, and Rivaly for winning 1st, 2nd, and 3rd place ($4,000, $1000, and a The post Observability Engineering with James Burns appeared first on Software Engineering Daily.

RECENT UPDATES: Podsheets is our open source set of tools for managing podcasts and podcast businesses New version of Software Daily, our app and ad-free subscription service FindCollabs is hiring a React developer FindCollabs Hackathon #1 has ended! Congrats to ARhythm, Kitspace, and Rivaly for winning 1st, 2nd, and 3rd place ($4,000, $1000, and a The post Serverless Runtimes with Steren Giannini appeared first on Software Engineering Daily.

RECENT UPDATES: FindCollabs $5000 Hackathon Ends Saturday April 15th, 2019 New version of Software Daily, our app and ad-free subscription service Software Daily is looking for help with Android engineering, QA, machine learning, and more   A software application requires compute and storage. Both compute and storage have been abstracted into cloud tools that can The post AWS Storage with Kevin Miller appeared first on Software Engineering Daily.

Upcoming event: FindCollabs Hackathon at App Academy on April 6, 2019 On Amazon Web Services, there are many ways to run an application on a single node. The first compute option on AWS was the EC2 virtual server instance. But EC2 is a large abstraction compared to what many people need for their nodes–which is The post AWS Compute with Deepak Singh appeared first on Software Engineering Daily.

Upcoming events: A Conversation with Haseeb Qureshi at Cloudflare on April 3, 2019 FindCollabs Hackathon at App Academy on April 6, 2019 Uber’s infrastructure supports millions of riders and billions of dollars in transactions. Uber has high throughput and high availability requirements, because users depend on the service for their day-to-day transportation. When Uber was The post Uber Infrastructure with Prashant Varanasi and Akshay Shah appeared first on Software Engineering Daily.

Upcoming events: A Conversation with Haseeb Qureshi at Cloudflare on April 3, 2019 FindCollabs Hackathon at App Academy on April 6, 2019 Google has been building large-scale scheduling systems for more than fifteen years. Google Borg was started around 2003, giving engineers at Google a unified platform to issue long-lived service workloads as well as The post Workload Scheduling with Brian Grant appeared first on Software Engineering Daily.

Upcoming events: A Conversation with Haseeb Qureshi at Cloudflare on April 3, 2019 FindCollabs Hackathon at App Academy on April 6, 2019 Google’s Borg system is a cluster manager that powers the applications running across Google’s massive infrastructure. Borg provided inspiration for open source tools like Apache Mesos and Kubernetes. Over the last decade, some The post Peloton: Uber’s Cluster Scheduler with Min Cai and Mayank Bansal appeared first on Software Engineering Daily.

Cloud computing started to become popular in 2006 with the release of Amazon EC2, a system for deploying applications to virtual machines sitting on remote data center infrastructure . With cloud computing, application developers no longer needed to purchase expensive server hardware. Creating an application for the Internet became easier, cheaper, and simpler. As the The post Netlify with Mathias Biilmann Christensen appeared first on Software Engineering Daily.

A Kubernetes cluster presents multiple potential attack surfaces: the cluster itself, a node running on the cluster, a pod running in the node, a container running in a pod. If you are managing your own Kubernetes cluster, you need to be aware of the security settings on your etcd, your API server, and your container The post Kubernetes Security with Liz Rice appeared first on Software Engineering Daily.

Cloud computing has been popular for less than twenty years. Large software companies have existed for much longer. If your company was started before the cloud became popular, you probably have a large, data center on your companies premises. The shorthand term for this software environment is “on-prem”. Deploying software to your own on-prem servers The post Replicated: On-Prem Deployments with Grant Miller appeared first on Software Engineering Daily.

Infrastructure software is having a renaissance. Cloud providers offer a wide range of deployment tools, including virtual machines, managed Kubernetes clusters, standalone container instances, and serverless functions. Kubernetes has standardized the container orchestration layer and created a thriving community. The Kubernetes community gives the cloud providers a neutral ground to collaborate on projects that benefit The post Knative: Serverless Workloads with Ville Aikas appeared first on Software Engineering Daily.

Virtualization software allows companies to get better utilization from their physical servers. A single physical host can manage multiple virtual machines using a hypervisor. VMware brought virtualization software to market, creating popular tools for allowing enterprises to deploy virtual machines throughout their organization. Containers provide another improvement to server utilization. A virtual machine can be The post VMware Kubernetes Strategy with Brad Meiseles appeared first on Software Engineering Daily.

HashiCorp was founded seven years ago with the goal of building infrastructure tools for automating cloud workflows such as provisioning, secret management, and service discovery. Hashicorp’s thesis was that operating cloud infrastructure was too hard: there was a need for new tools to serve application developers. Hashicorp founders Mitchell Hashimoto and Armon Dadgar began releasing The post Scaling HashiCorp with Armon Dadgar and Mitchell Hashimoto appeared first on Software Engineering Daily.

Prometheus is an open source monitoring system and time series database. Prometheus includes a multi-dimensional data model, a query language called PromQL, and a pull model for gathering metrics from your different services. Prometheus is widely used by large distributed systems deployments such as Kubernetes and Cloud Foundry. Prometheus gathers metrics from your services by The post Prometheus Scalability with Bryan Boreham appeared first on Software Engineering Daily.

When a developer provisions a cloud server, that server is called an “instance”. These instances can be used for running whatever workload a developer has, whether it is a web application, a database, or a set of containers. The cloud is cheap to get started on. New applications with few users can often be hosted The post Spot Instances with Amiram Shachar appeared first on Software Engineering Daily.

Chinese Internet companies operate at a massive scale. WeChat has over a billion users and is widely used as the primary means of payment by urban Chinese consumers. Alibaba ships 12 million packages per day, which is four times the amount of Amazon. JD.com, a Chinese ecommerce company, has perhaps the largest production Kubernetes installation The post Kubernetes in China with Dan Kohn appeared first on Software Engineering Daily.

Amazon Web Services changed how software engineers work. Before AWS, it was common for startups to purchase their own physical servers. AWS made server resources as accessible as an API request, and has gone on to create higher-level abstractions for building applications. For the first few years of AWS, the abstractions were familiar. S3 provided The post AWS Analysis with Corey Quinn appeared first on Software Engineering Daily.

Serverless computing is a technique for deploying applications without an addressable server. A serverless application is running on servers, but the developer does not have access to the server in the traditional sense. The developer is not dealing with IP addresses and configuring instances of their different services to be able to scale. Just as The post Zeit: Serverless Cloud with Guillermo Rauch appeared first on Software Engineering Daily.

Functions-as-a-service allow developers to run their code in a “serverless” environment. A developer can provide a function to a cloud provider and the code for that function will be scheduled onto a container and executed whenever an event triggers that function. An “event” can mean many different things. It is a signal that something has The post Cloud Events with Doug Davis appeared first on Software Engineering Daily.

Most applications today are either deployed to on-premise environments or deployed to a single cloud provider. Developers who are deploying on-prem struggle to set up complicated open source tools like Kafka and Hadoop. Developers who are deploying to a cloud provider tend to stay within that specific cloud provider, because moving between different clouds and The post Multicloud with Ben Hindman appeared first on Software Engineering Daily.

In a cloud infrastructure environment, failures happen regularly. The servers can fail, the network can fail, and software bugs can crash your software unexpectedly. The amount of failures that can occur in cloud infrastructure is one reason why storage is often separated from application logic. A developer can launch multiple instances of their application, with The post Stateful Kubernetes with Saad Ali appeared first on Software Engineering Daily.

Cloud providers created the ability for developers to easily deploy their applications to servers on data centers. In the early days of the cloud, most of the code that a developer wrote for their application could run on any cloud provider, whether it was Amazon, Google, or Microsoft. These cloud providers were giving developers the The post Crossplane: Multicloud Control Plane with Bassam Tabbara appeared first on Software Engineering Daily.

Originally posted on 16 June 2017. John Looney spent more than 10 years at Google. He started with infrastructure, and was part of the team that migrated Google File System to Colossus, the successor to GFS. Imagine migrating every piece of data on Google from one distributed file system to another. In this episode, John The post Google Early Days with John Looney Holiday Repeat appeared first on Software Engineering Daily.

Originally posted on 14 February 2017. Most tech companies are moving toward a highly distributed microservices architecture. In this architecture, services are decoupled from each other and communicate with a common service language, often JSON over HTTP. This provides some standardization, but these companies are finding that more standardization would come in handy. At the The post Service Proxying with Matt Klein Holiday Repeat appeared first on Software Engineering Daily.

Software products are distributed across more and more servers as they grow. With the proliferation of cloud providers like AWS, these large infrastructure deployments have become much easier to create. With the maturity of Kubernetes, these distributed applications are more reliable. Developers and operators can use a service mesh to manage the interactions between services The post Linkerd Service Mesh with William Morgan appeared first on Software Engineering Daily.

Not every company wants to move to the public cloud. Some companies have already built data centers, and can continue to operate their business with their own servers. Some companies have compliance issues with the public cloud, and want to operate their own servers to avoid legal risk. Operating a data center is not easy. The post On-Prem Cloud with Bob Fraser appeared first on Software Engineering Daily.

Cloud computing changed the economics of running a software company. Before the cloud, a software company had to purchase physical machines which often required thousands of dollars paid up front. The cloud allowed developers to deploy their applications for free, to operate a business for cheap, and to scale without hiring a dedicated team to The post Cloud Costs with Ran Rothschild appeared first on Software Engineering Daily.

Originally published on July 6, 2016. Scheduling is the method by which work is assigned to resources to complete that work. At the operating system level, this can mean scheduling of threads and processes. At the data center level, this can mean scheduling Hadoop jobs or other workflows that require the orchestration of a network The post Schedulers with Adrian Cockcroft Holiday Repeat appeared first on Software Engineering Daily.

The software release process is a barrier between written code and a live production environment that affects users. A software release process can involve a variety of different practices. Code might be tested for bugs using automation and manual testing. Static analysis tools can look at the code for potential memory leaks. A software release The post Liquid Software with Baruch Sadogursky appeared first on Software Engineering Daily.

Serverless computing abstracts away the idea of a server node. Serverless lets programmers treat compute resources as high-level, reliable APIs, rather than unreliable, low-level compute nodes that might fail. Serverless dramatically improves the efficiency of programmers. Instead of thinking of a database as a set of servers that need to be sharded and replicated, the The post Fission: Serverless on Kubernetes with Soam Vasani appeared first on Software Engineering Daily.

Matt Klein has worked for three rapidly growing Internet companies. At AWS, he worked on EC2, the compute-as-a-service product that powers a large percentage of the Internet. At Twitter, he helped scale the infrastructure in the chaotic days before Twitter’s IPO. Today he works at Lyft, building systems to allow for ride sharing infrastructure to The post Scaling Lyft with Matt Klein appeared first on Software Engineering Daily.

A smart security camera takes in a high volume of video images and processes those images using a set of machine learning models. Those models can be used to identify interesting snippets of movement throughout the day, and decide which of those snippets to keep. Some of the video snippets might contain movement of birds–but The post Flogo: Event-Driven Ecosystem with Leon Stigter and Matt Ellis appeared first on Software Engineering Daily.

DevSecOps emphasizes moving security out of a siloed audit process and distributing security practices throughout the software supply chain. In the past, software development usually followed a waterfall development process. Each step in building software was serialized, one after another. First, software was planned. Then it was built. Then it was tested. Finally, the software The post DevSecOps with Edward Thomson appeared first on Software Engineering Daily.

Deepak Singh is the director of compute services at AWS, where he works on cloud products relating to containers, Linux, and High Performance Computing. In today’s show, Deepak describes how the market for containers and serverless has evolved, and how Amazon thinks about product strategy. Back in 2014, Docker containers were becoming a popular way The post AWS Containers with Deepak Singh appeared first on Software Engineering Daily.

The Windows operating system is one of the most widely used pieces of software in history. Windows was started before there was any alternative to a monolithic codebase, because Microsoft was building software before the Internet was widely used by consumers. Networked computers gave rise to web applications, and software engineers began to rethink how The post DevOps at Microsoft with Martin Woodward appeared first on Software Engineering Daily.

Internet of Things is a concept that describes lots of devices that you interact with regularly being connected to the Internet and networked together. Technologists have been dreaming of the world of IoT for many years, where our connected refrigerator can detect that we are out of food, and automatically order more food. Or our The post Android Things with Wayne Piekarski appeared first on Software Engineering Daily.

Unity is a game engine for building 2-D and 3-D experiences, augmented reality, movies, and other applications. Unity is cross-platform, so that applications can be written once and deployed to iOS, Android, web, and other surfaces. Unity has been around for 13 years, and has grown in popularity with the rise in gaming and game The post Unity and WebAssembly with Brett Bibby appeared first on Software Engineering Daily.

Front is a shared inbox application that has seen rapid adoption within companies. Front allows multiple members of a company to collaborate together on a conversation–whether that conversation is in email, Twitter, or Facebook Messenger. This is useful when a customer email needs to be shared between the sales and engineering teams, or when a The post Front Engineering with Laurent Perrin appeared first on Software Engineering Daily.

Google has two consumer operating systems: Android and Chrome. The Android operating system has been widely deployed on mobile devices. Chrome is an operating system for laptops and tablets, originally based around the Chrome browser. For several years, these two ecosystems were mostly separate–you could not run Android apps on a Chrome operating system. Shahid The post Android on Chrome with Shahid Hussain and Stefan Kuhne appeared first on Software Engineering Daily.

Kubernetes is an open source container management system. Kubernetes is sometimes described as “the Linux of distributed systems” and this description makes sense: the large numbers of users and contributors in the Kubernetes community is comparable to the volume of Linux adopters in its early days. There are many different distributions of Linux: Ubuntu, Red The post Kubernetes Distributions with Brian Gracely and Michael Hausenblas appeared first on Software Engineering Daily.

Continuous integration and delivery allows teams to move faster by allowing developers to ship code independently of each other. A multi-stage CD pipeline might consist of development, staging, testing, and production. At each of these stages, a new piece of code undergoes additional tests, so that when the code finally makes it to production, the The post Continuous Delivery Pipelines with Abel Wang appeared first on Software Engineering Daily.

A company runs a variety of distributed systems applications such as Hadoop for batch processing jobs, Spark for data science, and Kubernetes for container management. These distributed systems tools can run on-prem, in a cloud provider, or in a hybrid system that uses on-prem and cloud infrastructure. Some enterprises use VMs, some use bare metal, The post Orchestrating Kubernetes with Chris Gaun appeared first on Software Engineering Daily.

Engineering organizations can operate more efficiently by working with a continuous integration and continuous deployment workflow. Continuous integration is the process of automatically building and deploying code that gets pushed to a remote repository. Continuous deployment is the process of moving that code through a pipeline of environments, from dev to test to production. At The post Kubernetes Continuous Deployment with Sheroy Marker appeared first on Software Engineering Daily.

Kubernetes is in production clusters around the world with hundreds of thousands of containers. Kubernetes provides a distributed systems management environment for small startups and giant enterprises with applications ranging from microservices to machine learning pipelines. Because the use cases are already so wide-ranging, and the project has had so much adoption, the focus of The post Kubernetes Impact with Clayton Coleman appeared first on Software Engineering Daily.

The main user interfaces today are the smartphone, the laptop, and the desktop computer. Some people today interact with voice interfaces, augmented reality, virtual reality, and automotive computer screens like the Tesla. In the future, these other interfaces will become more common. Developers will want to be able to expose their applications to these new The post Android Slices with Jason Monk appeared first on Software Engineering Daily.

Back in 2014, platform-as-a-service was becoming an increasingly popular idea. The idea of PaaS was to sit on top of infrastructure-as-a-service providers like Azure, AWS, or Google Cloud, and simplify some of the complexity of these infrastructure providers. Heroku had built a successful businesses from the idea of platform-as-a-service, and there was a widely held The post Helm with Michelle Noorali appeared first on Software Engineering Daily.

Building software today is much faster than it was just a few years ago. The tools are higher level, and abstract away tasks that would have required months of development. Much of a developer’s time used to be spent optimizing databases, load balancers, and queueing systems in order to be able to handle the load The post Build Faster with Nader Dabit appeared first on Software Engineering Daily.

Food gets thrown away from restaurants, homes, catering companies, and any other place with a kitchen. Most of this food gets thrown away when it is still edible, and could provide nutrition to someone who is hungry. Just like Airbnb makes use of excess living capacity, OLIO was started to connect excess food with people The post OLIO: Food Sharing with Lloyd Watkin appeared first on Software Engineering Daily.

At Google, the job of a site reliability engineer involves building tools to automate infrastructure operations. If a server crashes, there is automation in place to create a new server. If a service starts to receive a high load of traffic, there is automation in place to scale up the instances of that service. In The post Infrastructure Monitoring with Mark Carter appeared first on Software Engineering Daily.

Continuous delivery is a way of releasing software without requiring software engineers to synchronize during a release.  Over the last decade, continuous delivery workflows have evolved as the tools have changed. Jenkins was one of the first continuous delivery tools and is still in heavy use today. Netflix’s open sourced Spinnaker has also been widely The post GitOps: Kubernetes Continuous Delivery with Alexis Richardson appeared first on Software Engineering Daily.

Klarna is a payments company headquartered in Sweden. Since being established in 2005 it has grown to handling $21 billion in online sales in 2017. Roughly 40% of all e-commerce sales in Sweden go through Klarna. Klarna’s original differentiator was that it allowed users to checkout of e-commerce stores without entering in credit card information. The post Klarna Engineering with Marcus Granström appeared first on Software Engineering Daily.

Stripe is a payments API that allows merchants to transact online. Since the creation of the payments API, Stripe has expanded into adjacent services such as fraud detection, business management, and billing. These other verticals leverage the existing customer base and infrastructure that Stripe has developed from the success of their payments business. Raylene Yung The post Stripe Engineering with Raylene Yung appeared first on Software Engineering Daily.

Java programs compile into Java bytecode. Java bytecode executes in the Java Virtual Machine, a runtime environment that compiles that bytecode further into machine code, and optimizes the runtime by identifying “hot” code paths and keeping those hot code paths executing quickly. The Java Virtual Machine is a popular platform for building languages on top The post GraalVM with Thomas Wuerthinger appeared first on Software Engineering Daily.

“Edge computing” is a term used to define computation that takes place in an environment outside of a data center. Edge computing is a broad term. Your smartphone is an edge device. A self-driving car is an edge device. A security camera with a computer chip is an edge device. These “edge devices” have existed The post Edge Kubernetes with Venkat Yalla appeared first on Software Engineering Daily.

Enterprises want to update their technology faster. One way an enterprise can accelerate the adoption of new tools is to move more aggressively towards the cloud. By giving internal developers access to the cloud, it becomes easier to provision new servers–allowing for rapid experimentation, test environments, and scalability. In previous shows we have explored how The post Kubernetes in the Enterprise with Aparna Sinha appeared first on Software Engineering Daily.

Git is a distributed file system for version control. Git is extremely reliable, fast, and secure, owing to the fact that it is one of the oldest pieces of open source software. But even battle-tested software can have vulnerabilities. In this episode, we explore a subtle git vulnerability that could have potentially led to git The post Git Vulnerability with Edward Thomson appeared first on Software Engineering Daily.

Shopify runs more than 600,000 small business websites. When Shopify was figuring out how to scale, the engineering teams did not have a standard workflow for how to deploy and manage services. Some teams used AWS, some teams used Heroku, some teams used other infrastructure providers. To manage all those stores effectively, Shopify has built The post Shopify Infrastructure with Niko Kurtti appeared first on Software Engineering Daily.

“Serverless” is a word used to describe functions that get deployed and run without the developer having to manage the infrastructure explicitly. Instead of creating a server, installing the dependencies, and executing your code, the developer just provides the code to the serverless API, and the serverless system takes care of the server creation, the The post Function Platforms with Chad Arimura and Matt Stephenson appeared first on Software Engineering Daily.

When you interact with your bank, it probably feels different than when you interact with a software technology company. That’s because the biggest banks in the world were started before software became such a universally important tool. Their core competency is banking–not consumer software. Today, most banks make consumer-facing software. But these banks were not The post Build a Bank: Monzo with Richard Dingwall appeared first on Software Engineering Daily.

Rust is a systems programming language with a distinct set of features for safety and concurrency. In previous shows about Rust, we explored how Rust can prevent crashes and eliminate data races through its approach to type safety and memory management. Rust’s focus on efficiency and safety makes it a promising language for networking code. The post Rust Networking with Carl Lerche appeared first on Software Engineering Daily.

A database stores data to an underlying section of storage. If you are an application developer, you might think of your persistent storage system as being the database itself–but at a lower level, that database is writing to block storage, file storage, or object storage. A container orchestration system manages application containers. If you want The post Container Storage with Jie Yu appeared first on Software Engineering Daily.

Containers have improved deployments and resource utilization. Kubernetes created a platform to manage those containers and orchestrate them into distributed applications. In today’s episode, we explore tools that improve the workflow of the application developer who is working with Kubernetes, including Helm, Draft, and Brigade. Helm is a package manager for Kubernetes, which allows users The post Container Native Development with Ralph Squillace appeared first on Software Engineering Daily.

Deploying software to a container presents a different security model than deploying an application to a VM. There is a smaller attack surface per container, but the container is colocated on a node with other containers. Containers are meant to have a shorter lifetime than VMs, so there are generally fewer consequences if a container The post Container Security with Maya Kaczorowski appeared first on Software Engineering Daily.

Voice interfaces are a newer form of communicating with computers. Alexa is a voice interface platform from Amazon. Alexa powers the Amazon Echo, as well as Alexa-enabled cars, refrigerators, and dishwashers. Any developer can build a device with a voice interface using a Raspberry Pi. Paul Cutsinger works on Echo and Alexa at Amazon. He’s The post Alexa Voice Design with Paul Cutsinger appeared first on Software Engineering Daily.

Gloo is a function gateway built on top of the popular open source project Envoy. The goal of Gloo is to decouple client-facing APIs from upstream APIs. Gloo is similar to an API gateway, which is a tool that software companies can use to collect all their APIs and one place and impose security, monitoring, The post Gloo: Function Gateway with Idit Levine appeared first on Software Engineering Daily.

Mesos is a system for managing distributed systems. The goal of Mesos is to help engineers orchestrate resources among multi-node applications like Spark. Mesos can also manage lower level schedulers like Kubernetes. A common misconception is that Mesos aims to solve the same problem as Kubernetes, but Mesos is a higher level abstraction. Ben Hindman The post Cluster Schedulers with Ben Hindman appeared first on Software Engineering Daily.

When I buy a mattress online, I pay for it with my credit card. Behind the scenes, a complex series of transactions occur between a payment gateway, the credit card company, and a few banks. There are problems with this process–it is slow, complex, and involves the synchronization of several different parties. Some consumers will The post Affirm Engineering with Libor Michalek appeared first on Software Engineering Daily.

Superpedestrian is a robotic bicycle wheel that learns how you pedal and personalizes your bicycle ride. The engineering challenges of Superpedestrian are at the intersection of robotics, software, and real-time analytics. The first half of today’s show is about Superpedestrian. Goss Nuzzo Jones and Matt Cole are engineers at Superpedestrian. The slides for their presentation The post Superpedestrian Robotic Wheel / Infrastructure at HubSpot Meetup Talks appeared first on Software Engineering Daily.

Alexis Le-Quoc started Datadog in 2010, after living through the Internet boom and bust cycle of the late 90s and early 2000s. In 2010, cloud was just starting to become popular. There was a gap in the market for infrastructure monitoring tools, which Alexis helped fill with the first version of Datadog. Since 2010, the The post Building Datadog with Alexis Le-Quoc appeared first on Software Engineering Daily.

Google’s central system for managing to compute resources is called Borg. On Borg, millions of Linux containers process a wide variety of workloads. When a new application is spun up, Borg provides that application with the resources it needs. Workloads at Google usually fall into one of two distinct categories: long-running application workloads (such as The post Google Cluster Evolution with Brian Grant appeared first on Software Engineering Daily.

A message broker is an architectural component that sends messages between different nodes in a distributed system. Message brokers are useful because the sender of a message does not always know who might want to receive that message. Message brokers can be used to implement the “publish/subscribe” pattern, and by centralizing the message workloads within The post NATS Messaging with Derek Collison appeared first on Software Engineering Daily.

Stripe processes payments for thousands of businesses. A single payment could involve 10 different networked services. If a payment fails, engineers need to be able to diagnose what happened. The root cause could lie in any of those services. Distributed tracing is used to find the causes of failures and latency within networked services. In The post Stripe Observability Pipeline with Cory Watson appeared first on Software Engineering Daily.

Monitoring a Kubernetes cluster allows operators to track the resource utilization of the containers within that cluster. In today’s episode, Ilan Rabinovitch joins the show to explore the different options for setting up monitoring, and some common design patterns around Kubernetes logging and metrics gathering. Ilan is the VP of product and community at Datadog. The post Monitoring Kubernetes with Ilan Rabinovitch appeared first on Software Engineering Daily.

Go is a language designed to improve systems programming. Go includes abstractions that simplify aspects of low level engineering that are historically difficult—concurrency, resource allocation, and dependency management. In that light, it makes sense that the Kubernetes container orchestration system was written in Go. Erik St. Martin is a cloud developer advocate at Microsoft, where The post Go Systems with Erik St. Martin appeared first on Software Engineering Daily.

Tammy Butow has worked at Digital Ocean and Dropbox, where she built out infrastructure and managed engineering teams. At both of these companies, the customer base was at a massive scale. At Dropbox, Tammy worked on the database that holds metadata used by Dropbox users to access their files. To call this metadata system simply The post Database Chaos with Tammy Butow appeared first on Software Engineering Daily.

Software engineers have interacted with operations teams since the software was being written. In the 1990s, most operations teams worked with physical infrastructure. They made sure that servers were provisioned correctly and installed with the proper software. When software engineers shipped bad code that took down a software company, the operations teams had to help The post Site Reliability Management with Mike Hiraga appeared first on Software Engineering Daily.

Steve Herrod led engineering at VMWare as the company scaled from 30 engineers to 3,000 engineers. After 11 years, he left to become a managing director for General Catalyst, a venture capital firm. Since he has both operating experience and a wide view of the technology landscape as an investor, he is well-equipped to discuss The post Cloud and Edge with Steve Herrod appeared first on Software Engineering Daily.

On Software Engineering Daily, we have been covering the “serverless” movement in detail. For people who don’t use serverless functions, it seems like a niche. Serverless functions are stateless, auto-scaling, event-driven blobs of code. You might say “serverless sounds kind of cool, but why don’t I just use a server? It’s a paradigm I’m used The post Serverless Systems with Eduardo Laureano appeared first on Software Engineering Daily.

Earlier this year we did several shows about Cloud Foundry, followed by several shows about Kubernetes. Both of these projects allow you to build scalable, multi-node applications–but they serve different types of users. Cloud Foundry encompasses a larger scope of the application experience than Kubernetes. Kubernetes is lower level and is actually being used within The post Cloud Foundry Overview with Mike Dalessio appeared first on Software Engineering Daily.

Over 12 years of engineering, Box has developed a complex architecture of services. Whenever a user uploads a file to Box, that upload might cause 5 or 6 different services to react to the event. Each of these services is managed by a set of servers, and managing all of these different servers is a The post Box Kubernetes Migration with Sam Ghods appeared first on Software Engineering Daily.

When Box started in 2006, the small engineering team had a lot to learn. Box was one of the earliest cloud storage companies, with a product that allowed companies to securely upload files to remote storage. This was two years before Amazon Web Services introduced on-demand infrastructure, so the Box team managed their own servers, The post Scaling Box with Jeff Quiesser appeared first on Software Engineering Daily.

Applications need to be ready to scale in response to high-load events. With mobile applications, this can be even more important. People rely on mobile applications such as banking, ride sharing, and GPS. During Black Friday, a popular ecommerce application could be bombarded by user requests–you might not be able to complete a request to The post Load Testing Mobile Applications with Paulo Costa and Rodrigo Coutinho appeared first on Software Engineering Daily.

Over the last decade, computation and storage have moved from on-premise hardware into the cloud data center. Instead of having large servers “on-premise,” companies started to outsource their server workloads to cloud service providers. At the same time, there has been a proliferation of devices at the “edge.” The most common edge device is your The post Serverless at the Edge with Kenton Varda appeared first on Software Engineering Daily.

How do you build resilient, failure tested systems? Redundancy, backups, and testing are all important. But there is also an increasing trend towards chaos engineering–the technique of inducing controlled failures in order to prove that a system is fault tolerant in the way that you expect. In last week’s episode with Kolton Andrus, we discussed The post Linkedin Resilience with Bhaskaran Devaraj and Xiao Li appeared first on Software Engineering Daily.

The number of ways that applications can fail is numerous. Disks fail all the time. Servers overheat. Network connections get flaky. You assume that you are prepared for such a scenario because you have replicated your servers. You have the database backed up. Your core application is spread across multiple availability zones. But are you The post Chaos Engineering with Kolton Andrus appeared first on Software Engineering Daily.

On this show, we spend a lot of time talking about CI/CD, data engineering, and microservices. These technologies have only been widely talked about for the last 5-10 years. That means that they are easy to adopt for startups that get founded in the last 5-10 years, but not necessarily for older enterprises. Within a The post How to Change an Enterprise’s Software and Culture with Zhamak Dehghani appeared first on Software Engineering Daily.

After two weeks of episodes about Kubernetes, our in-depth coverage of container orchestration is drawing to a close. We have a few more shows on the topic before we move on to cover other aspects of the software. If you have feedback on this thematic format (whether you like it or not), send me an The post Serverless Containers with Sean McKenna appeared first on Software Engineering Daily.

In 2011, platform-as-a-service was in its early days. It was around that time that Gabe Monroy started a container platform called Deis, with the goal of making an open-source platform-as-a-service that anyone could deploy to whatever infrastructure they wanted. Over the last six years, Gabe had a front-row seat to the rise of containers, the The post Container Instances with Gabe Monroy appeared first on Software Engineering Daily.

Oliver Gould worked at Twitter from 2010 to 2014. Twitter’s popularity was taking off, and the engineering team was learning how to scale the product. During that time, Twitter adopted Apache Mesos and began breaking up its monolithic architecture into different services. As more and more services were deployed, engineers at Twitter decided to standardize The post Service Mesh Design with Oliver Gould appeared first on Software Engineering Daily.

Modern applications store most of their data on hosted storage solutions. We use hosted block storage to back databases, hosted object storage for objects such as videos, and hosted file storage for file systems. Using a cloud provider for these storage systems can simplify scalability, durability, and availability–it can be less painful than taking care The post Kubernetes Storage with Bassam Tabbara appeared first on Software Engineering Daily.

A common problem in a distributed system: how do you take a snapshot of the global state of that system? Snapshot is difficult because you need to tell every node in the system to simultaneously record its state. There are several reasons to take a snapshot. You might want to take a picture of the The post Kubernetes State Management with Niraj Tolia appeared first on Software Engineering Daily.

In the last four years, CoreOS has been at the center of enterprise adoption of containers. During that time, Brian Harrington (or “Redbeard”) has seen a lot of deployments. In this episode, Brian discusses the patterns he has seen among successful Kubernetes deployments–and the pitfalls of the less successful. How should you manage configuration? How The post Kubernetes Operations with Brian Redbeard appeared first on Software Engineering Daily.

A backend application can have hundreds of services written in different programming frameworks and languages. Across these different languages, log messages are produced in different formats. Some logging is produced in XML, some are produced in JSON, some is in other formats. These logs need to be unified into a common format and centralized for The post FluentD with Eduardo Silva appeared first on Software Engineering Daily.

Kubernetes has become the standard way of deploying new distributed applications. Most new internet businesses started in the foreseeable future will leverage Kubernetes (whether they realize it or not). Many old applications are migrating to Kubernetes too. Before Kubernetes, there was no standardization around a specific distributed systems platform. Just like Linux became the standard The post The Gravity of Kubernetes appeared first on Software Engineering Daily.

Kubernetes has become the standard system for deploying and managing clusters of containers. But the vision of the project goes beyond managing containers. The long-term goal is to democratize the ability to build distributed systems. Brendan Burns is a co-founder of the Kubernetes project. He recently announced an open-source project called Metaparticle, a standard library The post Kubernetes Vision with Brendan Burns appeared first on Software Engineering Daily.

You are requesting a car from a ridesharing service such as Lyft. Your request hits the Lyft servers and begins trying to get you a car. It takes your geolocation, and passes the geolocation to a service that finds cars that are nearby, and puts all those cars into a list. The list of nearby The post High Volume Distributed Tracing with Ben Sigelman appeared first on Software Engineering Daily.

Since Kubernetes came out, engineers have been deploying clusters to Amazon. In the early years of Kubernetes, deploying to AWS meant that you had to manage the availability of the cluster yourself. You needed to configure etcd and your master nodes in a way that avoided having a single point of failure. Deploying Kubernetes on The post Kubernetes on AWS with Arun Gupta appeared first on Software Engineering Daily.

A single user request hits Google’s servers. A user is looking for search results. In order to deliver those search results, that request will have to hit several different internal services on the way to getting a response. These different services work together to satisfy the user’s request. All of these services need to communicate The post Istio Motivations with Louis Ryan appeared first on Software Engineering Daily.

Docker was released in 2013 and popularized the use of containers. A container is an abstraction for isolating a well-defined portion of an operating system. Developers quickly latched onto containers as a way to cut down on the cost of virtual machines–as well as isolate code and simplify deployments. Developers began deploying so many containers The post Kubernetes Usability with Joe Beda appeared first on Software Engineering Daily.

In the first 10 years of cloud computing, a set of technologies emerge that every software enterprise needs; continuous delivery, version control, logging, monitoring, routing, data warehousing. These tools were built into the Cloud Foundry project, a platform for application deployment and management. As we enter the second decade of cloud computing, another new set The post Cloud R&D with Onsi Fakhouri appeared first on Software Engineering Daily.

Cloud Foundry is an open-source platform as a service for deploying and managing web applications. Cloud Foundry is widely used by enterprises who are running applications that are built using Spring, a popular web framework for Java applications, but developers also use Cloud Foundry to manage apps built in Ruby, Node and any other programming The post Cloud Foundry with Rupa Nandi appeared first on Software Engineering Daily.

Google Docs is used by millions of people to collaborate on documents together. With today’s technology, you could spend a weekend coding and build a basic version of a collaborative text editor. But in 2004 it was not so easy. In 2004 Steve Newman built a product called Writely, which allowed users to collaborate on The post High Volume Logging with Steve Newman appeared first on Software Engineering Daily.

Duolingo is a language learning platform with over 200 million users. On a daily basis millions of users receive customized language lessons targeted specifically to them. These lessons are generated by a system called the session generator.   Andre Kenji Horie is senior engineer at Duolingo. He wrote about the process of rewriting the session The post Scala at Duolingo with Andre Kenji Horie appeared first on Software Engineering Daily.

Ten years ago, if you wanted to build software, you probably needed to know how to write code. Today, the line between “technical” and “non-technical” people is blurring. Website designers can make a living building sites for people on WordPress or Squarespace–without knowing how to write code. Salesforce integration experts can help a sales team The post Cloud Marketplace with Zack Bloom appeared first on Software Engineering Daily.

Remember when the best game you could play on your phone was Snake? In 1998, Snake was preloaded on Nokia phones, and it was massively popular. That same year Half-Life won game of the year on PC. Metal Gear Solid came out for Playstation. The first version of Starcraft also came out in 1998. In The post Scalable Multiplayer Games with Yan Cui appeared first on Software Engineering Daily.

The Internet was designed as a decentralized system. Theoretically, if Alice wants to send an email to Bob, she can set up an email client on her computer and send that email to Bob’s email server on his computer. In reality, very few people run their own email servers. We all send our emails to The post Decentralized Objects with Martin Kleppmann appeared first on Software Engineering Daily.

Developers can build networked applications today without having to deploy their code to a server. These “serverless” applications are constructed from managed services and functions-as-a-service. Managed services are cloud offerings like database-as-a-service, queueing-as-a-service, or search-as-a-service. These managed services are easy to use. They take care of operational burdens like scalability and outages. But managed services The post Serverless Applications with Randall Hunt appeared first on Software Engineering Daily.

Functions as a service are deployable functions that run without an addressable server. Functions as a service scale without any work by the developer. When you deploy a function as a service to a cloud provider, the cloud provider will take care of running that function whenever it is called. You don’t have to worry The post Serverless Scheduling with Rodric Rabbah appeared first on Software Engineering Daily.

Are we a media company or a technology company? Facebook and the New York Times are both asking themselves this question. Facebook originally intended to focus only on building technology–to be a neutral arbiter of information. This has turned out to be impossible. The Facebook newsfeed is defined by algorithms that are only as neutral The post React and GraphQL at New York Times appeared first on Software Engineering Daily.

Cloud computing changed the economics of running a software company. A cloud is a network of data centers that offers compute resources to developers. In the 1990s, software companies purchased servers–an upfront capital expense that required tens of thousands of dollars. In the early 2000s, cloud computing started, and turned that capital expense into an The post How IBM Runs Its Cloud with Jason McGee appeared first on Software Engineering Daily.

Thumbtack is a marketplace for real-world services. On Thumbtack, people get their house painted, their dog walked, and their furniture assembled. With 40,000 daily marketplace transactions, the company handles significant traffic. On yesterday’s episode, we explored how one aspect of Thumbtack’s marketplace recently changed, going from asynchronous matching to synchronous “instant” matching. In this episode, The post Thumbtack Infrastructure with Nate Kupp appeared first on Software Engineering Daily.

The labor market is moving online. Taxi drivers are joining Uber and Lyft. Digital freelancers are selling their services through Fiverr. Experienced software contractors are leaving contract agencies to join Gigster. Online labor marketplaces create market efficiency by improving the communications between buyers and sellers. Workers make their own hours, and their performance is judged The post Marketplace Matching with Xing Chen appeared first on Software Engineering Daily.

Facebook serves interactive content to billions of users. Google serves query requests on the world’s biggest search engine. Uber handles a significant percentage of the transportation within the United States. These services are handling radically different types of traffic, but many of the techniques they use to balance loads are similar. Vivek Panyam is an The post Load Balancing at Scale with Vivek Panyam appeared first on Software Engineering Daily.

As a system becomes more complex, the chance of failure increases. At a large enough scale, failures are inevitable. Incident response is the practice of preparing for and effectively recovering from these failures. An engineering team can use checklists and runbooks to minimize failures. They can put a plan in place for responding to failures. The post Incident Response with Emil Stolarsky appeared first on Software Engineering Daily.

There is a quote from Jeff Bezos: “70% of the work of building a business today is undifferentiated heavy lifting. Only 30% is creative work. Things will be more exciting when those numbers are inverted.” That quote is from 2006, before Amazon Web Services had built most of their managed services. In 2006, you had The post Run Less Software with Rich Archbold appeared first on Software Engineering Daily.

A popular software application serves billions of user requests. These requests could be for many different things. These requests need to be routed to the correct destination, load balanced across different instances of a service, and queued for processing. Processing a request might require generating a detailed response to the user, or making a write The post High Volume Event Processing with John-Daniel Trask appeared first on Software Engineering Daily.

As the gig economy grows, that growth necessitates innovations in the online infrastructure powering these new labor markets. In our previous episodes about Uber, we explored the systems that balance server load and gather geospacial data. In our coverage of Lyft, we studied Envoy, the service proxy that standardizes communications and load balancing among services. The post Fiverr Engineering with Gil Sheinfeld appeared first on Software Engineering Daily.

In an event driven application, each component of application logic emits events, which other parts of the application respond to. We have examined this pattern in previous shows that focus on pub/sub messaging, event sourcing, and CQRS. In today’s show, we examine the intersection of event driven architecture and serverless architecture. Serverless applications can be The post Serverless Event-Driven Architecture with Danilo Poccia appeared first on Software Engineering Daily.

The Netflix API is accessed by developers who build for over 1000 device types: TVs, smartphontes, VR headsets, laptops. If it has a screen, it can probably run Netflix. On each of these different devices, the Netflix experience is different. Different screen sizes mean there is variable space to display the content. When you open The post Netflix Serverless-like Platform with Vasanth Asokan appeared first on Software Engineering Daily.

Serverless architecture is software that runs without an addressable server. Serverless is made possible by two types of technology: platform as a service providers like Auth0, and functions as a service like AWS Lambda. With both of these technologies, we can program logic that runs without being deployed to a server. Functions as a service The post Serverless Authentication with Bobby Johnson appeared first on Software Engineering Daily.

Augmented reality applications are slowly making their way into the world of the consumer. Pokemon Go created the magical experience of seeing Pokemon superimposed upon the real world. IKEA’s mobile app lets you see how a couch would fit into your living room, which has a significant improvement on the furniture buying process. Augmented reality The post Augmented Reality with Scott Montgomerie appeared first on Software Engineering Daily.

Computational load is the amount of demand that is being placed on a computer system. “Load” can take the form of memory, CPU, network bandwidth, disk space, and other finite resources. When we design systems, we need to prepare for high-load events. On a social network, people are much more active in the mornings. On The post Elastic Load Balancing with Ranga Rajagopalan appeared first on Software Engineering Daily.

It’s 9pm at night, and you are hungry. You order a pizza from Domino’s. You live on a street that’s dark, and so you have installed a smart lightbulb in front of your mailbox that lights up the address. When the pizza at Domino’s is ready, you want the lightbulb on your mailbox to light The post IFTTT Architecture with Nicky Leach appeared first on Software Engineering Daily.

Computer chips have physical limitations. When transistors get too small, electrons start to behave in ways that make the hardware modules less reliable. Our reliable technological progress has been enabled by Moore’s Law: the idea that the number of components we can fit on a chip doubles roughly every 12-18 months. We can’t keep shrinking The post Quantum Computing Introduction with Zlatko Minev appeared first on Software Engineering Daily.

How would you build a system for indexing and monitoring the entire Internet? Start by breaking the Internet up into IP address ranges. Give each of those address ranges to servers distributed around the world. On each of those servers, iterate through your list of IP addresses, sending packets to them. Depending on what sorts The post Internet Monitoring with Matt Kraning appeared first on Software Engineering Daily.

Scala is a functional and object oriented programming language built on the JVM. Scala Native takes this language, loved by many, and brings it to bare metal. Scala Native is an optimizing ahead-of-time compiler and lightweight managed runtime designed specifically for Scala. Denys Shabalin is a Research Assistant at the EPFL and the primary creator The post Scala Native with Denys Shabalin appeared first on Software Engineering Daily.

Tinder is a rapidly growing social network for meeting people and dating. In the past few years, Tinder’s userbase has grown rapidly, and the engineering team has scaled to meet the demands of increased popularity. On Tinder, you are presented with a queue of suggested people that you might match with, and you swipe left The post Tinder Engineering Management with Bryan Li appeared first on Software Engineering Daily.

Playing a video on the Internet seems simple. You press play, the video gets delivered, and boom–you are watching Game of Thrones, right? It’s a bit more complicated. Unless you have built an application that involves video, you probably have not dealt with the world of codecs, bitrates, and streaming. Depending on the bandwidth between The post Video Infrastructure with Matt McClure and Jon Dahl appeared first on Software Engineering Daily.

Tinder is a popular dating app where each user swipes through a sequence of other users in order to find a match. Swiping left means you are not interested. Swiping right means you would like to connect with the person. The simple premise of Tinder has led to massive growth, and the app is now The post Tinder Growth Engineering with Alex Ross appeared first on Software Engineering Daily.

Spotify is a streaming music company with more than 50 million users. Whenever a user listens to a song, Spotify records that event and uses it as input to learn more about the user’s preferences. Listening to a song is one type of event–there are hundreds of others. Opening the Spotify app, skipping a song, The post Spotify Event Delivery with Igor Maravic appeared first on Software Engineering Daily.

Applications built in the cloud are often serving requests from all around the world. A user in Hong Kong could have written to a database entry at the moment just before a user in San Francisco and a user in Germany simultaneously try to read from that database. If the user in San Francisco is The post Cloud-Native SQL with Alex Robinson appeared first on Software Engineering Daily.

When a user experiences an error in an application, the engineers who are building that application need to find out why that error occurred. The root cause of that error may be on the user’s device, or within a piece of server-side logic, or hidden behind a black box API. To fix a complex error, The post Error Diagnosis with James Smith appeared first on Software Engineering Daily.

Facebook was rapidly outgrowing its infrastructure in 2009. Classic data center design was not up to the task of the rapid influx of new users and data, photos, and streaming video hitting Facebook’s servers. A small team of engineers spent the next two years designing a data center from the ground up to be cheaper, The post Open Compute Project with Steve Helvie appeared first on Software Engineering Daily.

Serverless computing reduces the cost of using the cloud. Serverless also makes it easy to scale applications. The downside: building serverless apps requires some mindset shift. Serverless functions are deployed to transient units of computation that are spun up on demand. This is in contrast to the typical model of application delivery–the deployment of an The post Serverless Continuous Delivery with Robin Weston appeared first on Software Engineering Daily.

After raising $18 million, social networking startup Yubl made a series of costly mistakes. Yubl hired an army of expensive contractors to build out its iOS and Android apps. Drama at the executive level hurt morale for the full-time employees. Most problematic, the company was bleeding cash due to a massive over-investment in cloud services. The post Serverless Startup with Yan Cui appeared first on Software Engineering Daily.

Continuous delivery is a model for deploying small, frequent changes to an application. In a continuous delivery workflow, code changes that are pushed to a repository set off a build process that spins up a new version of the application. Testing is performed against that new build before advancing it to production, merging it with The post Platform Continuous Delivery with Andy Appleton appeared first on Software Engineering Daily.

Reinforcement learning is a type of machine learning where a program learns how to take actions in an environment based on how that program has been rewarded for actions it took in the past. When program takes an action, and it receives a reward for that action, it is likely to take that action again The post Reinforcement Learning with Michal Kempa appeared first on Software Engineering Daily.

In its most basic definition, machine learning is a tool that makes takes a data set, finds a correlation in that data set, and uses that correlation to improve a system. Any complex system with well-defined behavior and clean data can be improved with machine learning. Several precipitating forces have caused machine learning to become The post Apparel Machine Learning with Colan Connon and Thomas Bell appeared first on Software Engineering Daily.

Every software company backs up critical data sources. Backing up databases is a common procedure, whether a company is in the cloud or on-prem. Backing up virtual machine instances is less common. Rubrik is a company that is known for building backup infrastructure for enterprises. Their main product is an appliance that sits on prem The post Backups with Kenny To appeared first on Software Engineering Daily.

Software deployment evolves over time. In the 90s, a “deployment” might have meant issuing a new edition of your software via CD-ROM. Today, a deployment is often a multi-stage process. A new software build will undergo automated unit tests and integration tests, before being deployed to users.  The deployment might only go out to a The post Deployment with Avi Cavale appeared first on Software Engineering Daily.

Apache Kafka is an open-source distributed streaming platform. Kafka was originally developed at LinkedIn, and the creators of the project eventually left LinkedIn and started Confluent, a company that is building a streaming platform based on Kafka. Kafka is very popular, but is not easy to deploy and operationalize. That is why Confluent has built The post Kafka in the Cloud with Neha Narkhede appeared first on Software Engineering Daily.

Modern software applications are often built out of loosely coupled microservices. These services can be written in different languages, by different people, but communication between services needs to be standardized. For this reason, a service proxy is useful. A service proxy is a sidecar container that sits next to a service and facilitates communications with The post Istio Service Mesh with Varun Talwar and Louis Ryan appeared first on Software Engineering Daily.

Containers make it easier for engineers to deploy software. Orchestration systems like Kubernetes make it easier to manage and scale the different containers that contain services. The popular container infrastructure powered by Kubernetes is often called “cloud native.” On Software Engineering Daily, we have been exploring cloud native software to get a complete picture of The post Service Mesh with William Morgan appeared first on Software Engineering Daily.

Software architecture address the challenge of communicating and navigating large, complex systems to stakeholders, both technical and non-technical.  Over the years software architecture has gone in and out of fashion.  Today we discuss why software architecture is important, what it means to have software architecture, and how to properly structure teams and incorporate architecture. Today’s The post Software Architecture with Simon Brown appeared first on Software Engineering Daily.

A self-driving car needs to be able to quickly respond to changes in driving conditions. A factory needs to be able to quickly respond to changes in workplace safety. For these kinds of applications, we need processing power closer to the user of the application. If we put all of our application logic in the The post IoT Edge with Olivier Bloch appeared first on Software Engineering Daily.

John Looney spent more than 10 years at Google. He started with infrastructure, and was part of the team that migrated Google File System to Colossus, the successor to GFS. Imagine migrating every piece of data on Google from one distributed file system to another. In this episode, John sheds light on the engineering culture The post Google Early Days with John Looney appeared first on Software Engineering Daily.

Kubernetes makes it easier for engineering teams to manage their distributed systems architecture. But it’s still not simple to deploy and operate a Kubernetes cluster. Google Container Engine (GKE) is a managed control plane for Kubernetes. Just as developers can use Google App Engine to easily deploy monolithic apps against a platform as a service, The post Container Engines with David Aronchick and Chen Goldberg appeared first on Software Engineering Daily.

DNS stands for domain name system. This is the naming system that maps the entire internet. It associates information with domain names. More specifically, DNS specifies mappings between numerical IP addresses and domain names. Most engineers know these basic facts about DNS, but they may not know how much engineering a complex company like Etsy The post DNS with Phil Stanhope appeared first on Software Engineering Daily.

On January 31st 2017, GitLab experienced a major outage of their online repository hosting service. The primary database server experienced data loss due to a combination of malicious spam attacks and engineering mistakes that occurred while trying to respond to those spam attacks. GitLab responded to the event transparently. The company put up a postmortem The post GitLab with Pablo Carranza appeared first on Software Engineering Daily.

Healthcare is a complex business. Oscar is a company that wanted to build a new insurance provider–but realized that healthcare is so interconnected that in order to build a new insurance provider, realized it actually needed to build an entire healthcare business too, complete with patient management and facilities. Since Oscar is a modern technology The post Healthcare Engineering with Isaac Councill appeared first on Software Engineering Daily.

Many companies are transitioning from a monolith to microservices architecture. Tools for cloud computing, containerization, and continuous delivery are making this easier. But there are still technological and organizational challenges that a company will encounter while making this transition. Cassandra Shum is an engineer with ThoughtWorks. She has worked with major financial institutions and other The post Microservices Transition with Cassandra Shum appeared first on Software Engineering Daily.

Firebase is a backend-as-a-service. The key efficiency of a backend-as-a-service is that it enables developers to go from having a 3-tier architecture (client, server, database) to a 2-tier architecture (client, backend-as-a-service). The team who started Firebase built it as a pivot. They had started a social network, and then they realized there wasn’t a good The post Firebase with Doug Stevenson appeared first on Software Engineering Daily.

Spring Framework is an application framework for Java and JVM languages. Spring was originally built around dependency injection, but grew to become an entire ecosystem of tools and plugins for Java developers. Spring was originally released 15 years ago, and since then a lot has changed around application development. For example, many engineers deploy applications The post Spring Boot with Josh Long appeared first on Software Engineering Daily.

The word “microservices” started getting used after a series of events–companies were moving to cloud virtual machines. Those VMs got broken up into containers, and the containers can fit to the size of the service. Services that are more narrowly defined take up smaller containers, and can be packed more densely into the virtual machines–hence The post Microservices Practitioners with Austin Gunter and Richard Li appeared first on Software Engineering Daily.

Search is a common building block for applications. Whether we are searching Wikipedia or our log files, the behavior is similar: a query is entered and the most relevant documents are returned. The core data structure for search is an inverted index. Elasticsearch is a scalable, resilient search tool that shards and replicates a search The post Elasticsearch with Philipp Krenn appeared first on Software Engineering Daily.

There are various standards at play when creating and consuming Application Program Interfaces (APIs).  These standards, though, are mostly technical and mostly lower-level than the content of the API. Andy Beier has experienced the broad range of API quality in his role with Domo in creating integrations with other businesses.  He has made standardization of The post API Design Standards with Andy Beier appeared first on Software Engineering Daily.

Servers in a data center fail. Sometimes entire data centers have a power outage. Bugs in an application make it into production. Human operators make mistakes and cause data to be deleted. Failure is unavoidable. We make backups and replicate our servers so that when a failure occurs, we can quickly respond to it without The post Failure Injection with Kolton Andrus appeared first on Software Engineering Daily.

Designers and software engineers need to communicate with each other. From Apple to Slack to Uber, the emphasis on visual design within a product is rising in importance. Much like development and operations siloes have been bridged with the DevOps movement, design and engineering teams are working more closely together to align the vision of The post Software Psychology with Bjorn Freeman Benson appeared first on Software Engineering Daily.

If you are building a service that processes payments, your software architecture has a lot of requirements. Not only do you need to be highly available, consistent, and fast–you need to be PCI compliant. In this episode, we explore the infrastructure of Stripe with Evan Broder, who has been with the company for five years. The post Stripe Infrastructure with Evan Broder appeared first on Software Engineering Daily.

Observability allows engineers to understand what is going on inside their systems. In its most raw form, observability comes from log data. Modern systems have many layers of logs–virtualized cloud infrastructure, container orchestration, the container runtime itself, and the application logic running within the container. With all of these layers, it is not practical for The post Stripe Observability with Cory Watson appeared first on Software Engineering Daily.

Command Query Responsibility Segregation (CQRS) is a powerful concept that has the potential to make for reliable and maintainable systems.  It is also broadly misunderstood and means different things to different people. Derek Comartin learned about the idea after viewing some talks by Greg Young and has since successfully applied the approach with great success The post Using CQRS to Make Controllers Lean with Derek Comartin appeared first on Software Engineering Daily.

Load testing measures performance of a system undergoing a large volume of requests. Before an application is pushed to production, engineers will often load test their software to ensure it is resilient in the face of high traffic. As web applications have changed, the requirements around load testing have changed as well. External APIs, internal The post Load Testing with Mark Gilbert appeared first on Software Engineering Daily.

Parse was a backend as a service company built in 2011 before being acquired by Facebook in 2013. Building a backend as a service for developers requires walking a thin line between giving engineers lots of control and preventing those engineers from shooting themselves in the foot. While she was at Parse, Charity Majors learned The post Parse and Operations with Charity Majors appeared first on Software Engineering Daily.

When an application is using all of its available resources, that application needs to be scaled. Scaling an application means giving it more resources–typically servers. Autoscaling is an engineering practice where an application is automatically given more or less resources based on how healthy the application performance is at a given time. Applications on Heroku The post Heroku Autoscaling with Andrew Gwozdziewycz appeared first on Software Engineering Daily.

In the mid 90s, data warehousing might have meant “using an Oracle database.” Today, it means a wide variety of things. You could be stitching together a big data pipeline using Kafka, Hadoop, and Spark. You could be using managed tools like BigQuery from Google. How did we get from the simple days of Oracle The post Data Warehousing with Mark Rittman appeared first on Software Engineering Daily.

Most tech companies are moving toward a highly distributed microservices architecture. In this architecture, services are decoupled from each other and communicate with a common service language, often JSON over HTTP. This provides some standardization, but these companies are finding that more standardization would come in handy. At the ridesharing company Lyft, every internal service The post Service Proxying with Matt Klein appeared first on Software Engineering Daily.

Infrastructure is a term that can mean many different things: your physical computer, the data center of your Amazon EC2 cluster, the virtualization layer, the container layer–on and on. In today’s episode, podcasters Chris Wahl and Ethan Banks discuss the past, present, and future of infrastructure with me. Ethan and Chris host Datanauts, a podcast The post Infrastructure with Datanauts’ Chris Wahl and Ethan Banks appeared first on Software Engineering Daily.

Giphy is a search engine for gifs, the short animated graphics that we see around the Internet. Giphy is also a creative platform where people create new gifs. Every search engine requires the construction of a search index, which is a data structure that responds to search queries efficiently. Since Giphy is a search engine The post Giphy Engineering with Anthony Johnson appeared first on Software Engineering Daily.

Back in 2008, the range of tools that engineers could use to connect computer systems together were getting quite good. Cloud computing was democratizing access to servers. But the telephony ecosystem was still inaccessible to the average developer. If you needed your program to make a phone call and connect a user to a customer The post Twilio Engineering with Pat Malatack appeared first on Software Engineering Daily.

A company like Pinterest has millions of transactional emails to send to people. The scalability challenges of sending high volumes of email mean that it makes more sense for most companies to use an email as a service product rather than building their own. Chris McFadden is the VP of engineering and cloud operations at SparkPost The post Email Infrastructure with Chris McFadden appeared first on Software Engineering Daily.

Meetup is an online service that allows people to gather into groups and meet in person. Since 2002, the company has been growing and its technology stack has been changing. Today, they are in the process of migrating to the cloud, using both Amazon Web Services and Google Compute Platform. Yvette Pasqua is the CTO The post Meetup Architecture with Yvette Pasqua appeared first on Software Engineering Daily.

When a useful new technology comes out, companies that are in a position to adopt that new technology can gain an edge over competitors. As our industry grows and moves faster, these kinds of changes are coming faster–some recent examples are Docker, ReactJS, and Kubernetes. Evolutionary architecture supports incremental, guided change as a first principle The post Evolutionary Architecture with Neal Ford appeared first on Software Engineering Daily.

Self-contained systems is an architectural approach that separates the functionality of a system into many independent systems. Each self-contained system is an autonomous web application, and is owned by one team. Communication with other self-contained systems or 3rd party systems is asynchronous where possible. As Eberhard Wolff explains in this episode, self-contained systems is not The post Self-Contained Systems with Eberhard Wolff appeared first on Software Engineering Daily.

Application performance monitoring helps an engineer understand what is going on with an application. An application on a single machine is often monitored by inserting bytecode instructions into the application after it has been interpreted. Distributed cloud applications with functionality broken up across multiple servers often use distributed tracing. Andi Grabner from Dynatrace joins today’s The post Performance Monitoring with Andi Grabner appeared first on Software Engineering Daily.

Online marketplaces and social networks often have a trust and safety team. The trust and safety team helps protect the platform from scams, fraud, and malicious actors. To detect these bad actors at scale requires building a system that classifies every transaction on the platform as safe or potentially malicious. Since every social platform has The post Antifraud Architecture with Josh Yudaken appeared first on Software Engineering Daily.

For many years, software companies have been breaking up their applications into individual services for the purpose of isolation and maintainability. In the early 2000s, we called this pattern “service-oriented architecture”. Today we call it “microservices”. Why did we change that terminology? Did the services get smaller? Not exactly. Jonas Boner suggests that the movement The post Reactive Microservices with Jonas Boner appeared first on Software Engineering Daily.

Some tasks are simple, but cannot be performed by a computer. Audio transcription, image recognition, survey completion–these are simple procedures that almost any human could execute, but the machine learning models have not gotten consistent enough to do them accurately. Scale is an API for human labor, created by Lucy Guo and Alexandr Wang. Similar The post Scale API with Lucy Guo and Alexandr Wang appeared first on Software Engineering Daily.

Caching is a fundamental concept of computer science. When data is accessed frequently, we put that data in a place where it can be accessed more quickly–we put the data in a cache. When data is accessed less often, we leave it in a place where the access time is slow or expensive. Netflix has The post Netflix Caching with Scott Mansfield appeared first on Software Engineering Daily.

When you are working on a program, a lot of things are going through your head. In some sense, you become part machine when you are programming. Learnable Programming is a concept that facilitates this, by showing developers what the computer is doing in real time, before compiling. In this episode, Josh Varty, co-founder of The post Developer Tools with Josh Varty appeared first on Software Engineering Daily.

Microservices are a widely adopted pattern for breaking an application up into pieces that can be well-understood by the individual teams within the company. Microservices also allow these individual pieces to be scaled independently and updated in isolation. Past Software Engineering Daily episodes have covered the microservice architectures of Twitter, Netflix, Google, Uber, and other The post Microservices with Rafi Schloming appeared first on Software Engineering Daily.

Slack is a chat client that has reached wide adoption. The rise of Slack has coincided with the rise of chatbots. A chatbot is a simple, conversational interface into a computer program that may have simple functionality, like telling you some simple statistics, or more complex functionality, like helping you manage your continuous integration pipeline. The post Slack Bots with Amir Shevat appeared first on Software Engineering Daily.

Amazon Web Services changed the economics of building an internet application. Instead of having to invest tens of thousands of dollars up front for hardware, developers can pay for services over time as their application scales. As AWS has grown to be a gigantic platform, the documentation about how to use cloud infrastructure has become The post AWS Open Guide with Joshua Levy appeared first on Software Engineering Daily.

The blueprint for a typical startup involves investing heavily in cloud services–either from Amazon, Google, or Microsoft. The high costs can quickly eat away at all of the money that startup has raised. In today’s episode, Avi Freedman outlines some of the infrastructure mistakes that can set back a company severely–cloud jail, hipster tools, and The post Infrastructure Mistakes with Avi Freedman appeared first on Software Engineering Daily.

Chat bots are your newest co-worker. Slack, HipChat, and other chat clients allow developers and other team members to communicate more dynamically than the limits of email. Companies have started to add bots to their chat rooms. These bots can give you technical information, restart a server, or notify you that a build has finished. The post ChatOps with Jason Hand appeared first on Software Engineering Daily.

Kafka is a distributed log for producers and consumers to publish messages to each other. We’ve done many shows about Kafka as a key building block for distributed systems, but we often leave out the discussion of the complexities of setting up Kafka and monitoring it. Kafka deployments can be a complex piece of software The post Managed Kafka with Tom Crayford appeared first on Software Engineering Daily.

Google Compute Engine is the public cloud built by Google. It provides infrastructure- and platform-as-a-service capabilities that rival Amazon Web Services. Today’s guest Joe Beda was there from the beginning of GCE, and he was also one of the early engineers on the Kubernetes project. Google’s internal systems have made it easy for employees to The post Google Cloudbuilding with Joe Beda appeared first on Software Engineering Daily.

Cloud computing was something much different in 2011, when Brian Gracely and Aaron Delp started The Cloudcast, a podcast I listen to on a regular basis. The Cloudcast features technical discussions about cloud infrastructure technology, and one of the most recent shows was a monologue by Brian Gracely where he explained his perspective on the The post Docker Cloudcasting with Brian Gracely appeared first on Software Engineering Daily.

When a user of a social network updates her profile, that profile update needs to propagate to several databases that want to know about such an update–search indexes, user databases, caches, and other services. When Neha Narkhede was at LinkedIn, she helped develop Kafka, which was deployed at LinkedIn to help solve this very problem. The post Kafka Event Sourcing with Neha Narkhede appeared first on Software Engineering Daily.

The intent of the DevOps movement is to get organizations moving faster and more effectively by breaking down siloes, and improving communication. Gene Kim’s book The Phoenix Project illustrated this by telling the fictional story of a company adopting a DevOps mentality. Although that book was fiction, Gene is an experienced engineer, having worked as The post DevOps Handbook with Gene Kim appeared first on Software Engineering Daily.

  At Netflix, developers write applications with a variety of requirements–from simple requests for a list of movies to more resource-intensive requests like a complex machine learning workflow. Netflix wants developers to be able to request the resources they need from a compute cluster and receive those resources on-demand, without thinking too much about the The post Netflix Scheduling with Sharma Podila appeared first on Software Engineering Daily.

Building a monitoring system is a complex distributed systems problem. Events are produced from different points in an application and must be aggregated in order to form metrics. These events are often ingested by a time series database, which forms the backbone of our monitoring system. Theo Schlossnagle is the CEO of Circonus, where he The post Monitoring Architecture with Theo Schlossnagle appeared first on Software Engineering Daily.

In order to move software updates from the development team to production, companies do a variety of things. Some teams might email files to each other or use FTP or even floppy disks. Most companies today at least use version control systems like Git together with separate servers for development and production. When code is The post Continuous Delivery with David Rice appeared first on Software Engineering Daily.

Kafka Streams is a library for building streaming applications that transform input Kafka topics into output Kafka topics. In a time when there are numerous streaming frameworks already out there, why do we need yet another? To quote today’s guest Jay Kreps “the gap we see Kafka Streams filling is less the analytics-focused domain these The post Kafka Streams with Jay Kreps appeared first on Software Engineering Daily.

Platform as a service can mean different things to different people. The most prominent feature of a PaaS is the ability to abstract away issues that every developer within an organization has to deal with. As an example, developers today don’t need to fear scalability and load balancing issues as much as engineers of the The post Platform as a Service with Sinclair Schuller appeared first on Software Engineering Daily.

Google builds cloud services for developers, such as PubSub, Cloud Storage, BigQuery, and Cloud DataStore. On Software Engineering Daily, we’ve done lots of shows about how these types of services are built. In this episode, we are zooming in on the interaction between the developer using a cloud service and the design and engineering of The post Cloud Clients with Jon Skeet appeared first on Software Engineering Daily.

Batch and stream processing systems have been evolving for the past decade. From MapReduce to Apache Storm to Dataflow, the best practices for large volume data processing have become more sophisticated as the industry and open source communities have iterated on them.   Dataflow and Apache Beam are projects that present a unified batch and The post Cloud Dataflow with Eric Anderson appeared first on Software Engineering Daily.

Slack is a chat application that is rapidly growing in popularity. The focus of Slack is to create a polished, responsive tool for productivity that cuts down on the emailing, context switching, and useless meetings that take place at a typical enterprise.   Keith Adams, the chief architect at Slack, joins the show to explain The post Slack’s Architecture with Keith Adams appeared first on Software Engineering Daily.

Uber has a software architecture with unique requirements. Uber does not have the firehose of user engagement data that Twitter or Facebook has, but each transaction on Uber is both high value and time-sensitive. Users are paying for transportation that they expect to be available and reasonably close by. When Uber’s system is trying to The post Uber’s Ringpop with Jeff Wolski appeared first on Software Engineering Daily.

Kubernetes is a cluster management tool open sourced by Google. On Software Engineering Daily, we’ve done numerous shows on how Kubernetes works in theory. Today’s episode is a case study in how to deploy Kubernetes to production at a company with existing infrastructure.   GolfNow is a fifteen year-old application written in C# .NET. It The post Kubernetes Migration with Sheriff Mohamed appeared first on Software Engineering Daily.

In a microservices architecture, a user request will often make its way through several different services before it returns a result to the end user. If a user experiences a failed request, the root cause could be in any of the services along that request path. Even more problematic is the challenge of debugging latency The post Distributed Tracing with Reshmi Krishna appeared first on Software Engineering Daily.

“Serverless” usually refers to an architectural pattern where the server side logic is run in stateless compute containers that are event-triggered and ephemeral. Mike Roberts has written a series of articles about serverless computing, in which he discusses theories and patterns around serverless architecture. In this episode, Mike and I discuss how to reimagine our The post Serverless Architecture with Mike Roberts appeared first on Software Engineering Daily.

Unbounded data streams create difficult challenges for our application architectures. The data never stops coming, and we are forced to assume that we will never know if or when we have seen all of our data. Some streaming systems give us the tools to deal partially with unbounded data streams, but we have to complement The post Apache Beam with Frances Perry appeared first on Software Engineering Daily.

Prometheus is a tool for monitoring our distributed applications. It allows us to focus on the services we are deploying rather than the individual machines that make up instances of that service.   The monitoring service itself is a portion of a distributed system that is treated differently than the services we are monitoring. We The post Prometheus Monitoring with Brian Brazil appeared first on Software Engineering Daily.

Monitoring translates machine data into actionable business metrics, and is a key component of a modern software company. James Turnbull’s new book “The Art of Monitoring” describes how organizations can build their monitoring infrastructure.     James joins the show today to outline the strategies that a company can use to proactively monitor their systems. The post The Art of Monitoring with James Turnbull appeared first on Software Engineering Daily.

Lee Atchison spent seven years at Amazon working in retail, software distribution, and Amazon Web Services. He then moved to New Relic, where he has spent four years scaling the company’s internal architecture. From his decade of experience at fast growing web technology companies, Lee has written the book Architecting for Scale, from O’Reilly. As The post Scalable Architecture with Lee Atchison appeared first on Software Engineering Daily.

Scheduling is the method by which work is assigned to resources to complete that work. At the operating system level, this can mean scheduling of threads and processes. At the data center level, this can mean scheduling Hadoop jobs or other workflows that require the orchestration of a network of computers. Adrian Cockcroft worked on The post Schedulers with Adrian Cockcroft appeared first on Software Engineering Daily.

In 1999, it took $50,000 to buy a server. Once you bought that server, you had to know how to operate and maintain it. Today, cloud service providers have changed how we build software. Servers, load balancers, networking, storage–these hardware concerns have been turned into software. Don Pezet joins the show today to discuss the The post Cloud Providers with Don Pezet appeared first on Software Engineering Daily.

Six years ago, Twitter was experiencing outages due to high traffic. Back in 2010 Twitter was built as a monolithic Ruby on Rails application. Twitter migrated to a microservices architecture to fix these problems. During this migration, the engineers at Twitter learned how to build and scale highly distributed microservice architectures. William Morgan was an The post Scaling Twitter with Buoyant.io’s William Morgan appeared first on Software Engineering Daily.

Mass customization is the process of making customized, personalized products that are accessible to individuals and small businesses. The process involves manufacturing, assembly lines, supply chains, and software at every step along the way. Today’s guests are Jim Sokoloff and Maarten Wensveen, who work on infrastructure and technology at Cimpress, a mass customization platform. Cimpress The post Manufacturing and Microservices with Cimpress’ Jim Sokoloff and Maarten Wensveen appeared first on Software Engineering Daily.

The unit of computation has evolved from on premise servers to virtual machines in the cloud to containers running in those virtual machines. Serverless computation is another stage in the evolution of computational unit management. With a serverless architecture, a function call to the cloud spins up a transient container, calls the function on that The post Serverless Code with Ryan Scott Brown appeared first on Software Engineering Daily.

Google’s site reliability engineers are responsible for maintaining the highly available services that power the Google software that we all use on a regular basis. O’Reilly recently published the book “Site Reliability Engineering: How Google Runs Production Systems”, and the book provides a comprehensive window into how the site reliability engineering role works. Todd Underwood The post Google’s Site Reliability Engineering with Todd Underwood appeared first on Software Engineering Daily.

Dropbox has been storing files on Amazon Web Services for 8 years, and Dropbox’s core business is storing files. For the past three years, Dropbox has been working on a project to migrate its file storage from Amazon Web Services to its own custom-built infrastructure. Magic Pocket is the name of Dropbox’s new infrastructure layer, The post Dropbox’s Magic Pocket with James Cowling appeared first on Software Engineering Daily.

Distributed systems products are often marketed with terms like “real-time data” and “hassle-free scaling”, but what do those terms actually mean? Is data in a distributed system ever reliably “real time”? Do we ever have strong enough plans about our scalability strategy to say that scaling will be “hassle free”? Camille Fournier joins us today The post Distributed Systems Tradeoffs with Camille Fournier appeared first on Software Engineering Daily.

Exception monitoring services and log management services are two sides of a gradient. Exception monitoring services capture and aggregate the problems that occur on your application. Log management services aggregate all of your logs, so that you can decide for yourself what constitutes a problem. Brian Rue from Rollbar joins the show today to talk The post Distributed Systems and Exception Monitoring with Brian Rue appeared first on Software Engineering Daily.

Kubernetes is an open source system for automating deployment, operations, and scaling of containerized applications. Google developed Kubernetes after fifteen years of running containers in production. Brendan Burns is a founder of the Kubernetes project, and he joins us to talk about the lessons learned as Google has built containerized applications to distribute across its The post Google’s Container Management with Brendan Burns appeared first on Software Engineering Daily.

“You need to build more things yourself to be highly available, but one of the very good consequences of being bare metal is that the prices are very low compared to what you could get on the cloud provider.” Engineers who want to add search to their application usually deploy Elasticsearch, or write their own The post Search as a Service with Julien Lemoine appeared first on Software Engineering Daily.

“We’re not always in control of other people’s networks.” CDN stands for content delivery network. A content delivery network is a system of distributed servers that delivers web pages and other web content. Without CDNs, the internet would be much slower, because CDNs function as a caching layer for most web resources. Carl Gustas is The post Managing a CDN with Carl Gustas appeared first on Software Engineering Daily.

“You write the code, but you don’t run it? That’s just preposterous.” Software applications are constantly generating logs. These logs are necessary to understand how an application is functioning, and logs are key to debugging. As applications have gotten more complex, logging infrastructure has become complex as well. Storing and managing all of our log The post Logging and NoOps with Christian Beedgen appeared first on Software Engineering Daily.

“As the scale continues to increase, certain effects of architecture become less and less efficient.” When you spend money online, you expect a receipt to come in your email. When you register for a new web site, you need to verify your sign up in your email. These types of emails are called “transactional email” The post Scaling Email with J.R. Jasperson appeared first on Software Engineering Daily.

“SaaS, whether we want it or not, in enterprise technology or in our data centers, is coming.” Application delivery has become more complex as software architectures have moved into the cloud. Data center infrastructure has turned into code to be manipulated, and software engineering teams are adjusting their strategies. HashiCorp is a company that builds The post Automating Infrastructure at HashiCorp with Mitchell Hashimoto appeared first on Software Engineering Daily.

“It’s an amazing era for software developers – we have all this amazing infrastructure behind the scenes that we can build upon.” Ten years ago, building a highly scalable image delivery service would require millions of dollars in upfront costs, and hours of work configuring hardware server infrastructure. Today, it is possible to bootstrap this The post Bootstrapping a SaaS for Developers with Itai Lahan appeared first on Software Engineering Daily.

“Its sort of like the old joke in computer science – what do you do when you have a problem? Well, add a layer of abstraction.” Today’s guest is Calvin French-Owen, the CTO of Segment, a tool that companies use to aggregate their analytics into once place. As Segment has scaled, the company has had The post Developer Analytics with Calvin French-Owen appeared first on Software Engineering Daily.

“It’s Friday night and you’re basically out of the office on your way to meet with friends. And you just merge this thing and put it into production because you have that trust – that the system will capture any kind of problem.” Continuous integration and deployment are important tools for modern software development. With The post Continuous Delivery and Test Automation with Flo Motlik appeared first on Software Engineering Daily.

This episode is a republication from my interview with Leslie Lamport on Software Engineering Radio. Leslie Lamport won a Turing Award in 2013 for his work in distributed and concurrent systems. He also designed the document preparation tool LaTex. Leslie is employed by Microsoft Research, and has recently been working with TLA+, a language that is The post Distributed Systems with Leslie Lamport appeared first on Software Engineering Daily.

“A lot of our customers are kind of AWS refugees who really don’t want all the stuff amazon has, and they just want a really easy to use system that is reliable.” Continue reading… The post Engineering Cloud Services with Sam Kottler appeared first on Software Engineering Daily.

“You can have a monolith, and it can be a perfectly good thing.” Continue reading… The post Moving to Microservices at SoundCloud with Lukasz Plotnicki appeared first on Software Engineering Daily.

“If an engineer is doing something repeatedly over and over again, their mind is immediately going to jump to that place where it’s like ‘Okay how can I make this faster?’ or ‘How can I save myself time?’” Quora is a Q&A website where questions are asked, answered, edited and organized by its community of The post Engineering at Quora with Shreyes Seshasai appeared first on Software Engineering Daily.

“If you can make a system that can survive this random failure testing, then you will more or likely survive whatever other chaotic conditions exist.” Continue reading… The post Scaling Uber with Matt Ranney appeared first on Software Engineering Daily.

“Sometimes there’s a misconception that Genie is a job scheduling platform... Genie really represents our extraction layer, from what our computational resources are, to our end user jobs.” Genie is an open-source tool that provides job and resource management for the Hadoop ecosystem in the cloud. Continue reading… The post Netflix Genie with Tom Gianos appeared first on Software Engineering Daily.

Distributed systems programming will always be a world of tradeoffs -- there is no silver bullet in the future. But life can be made easier with tactics such as the actor pattern and the use of conflict-free replicated data types (CRDTs). Caitie McCaffrey is a distributed systems engineer who currently works at Twitter. She previously worked on Halo 4 at Microsoft and 343 Industries. At QCon San Francisco, she will be hosting the track Taming Distributed Architecture. Continue reading… The post Taming Distributed Architecture with Caitie McCaffrey appeared first on Software Engineering Daily.

“DevOps is not a thing. It is a set of problem statements and solution possibilities that are always growing.” The hosts of DevOps Cafe joined Software Engineering Daily for a conversation about DevOps culture and misconceptions. Questions What do software engineers need to know about DevOps? What are the biggest misconceptions around DevOps? Is DevOps The post Origin of DevOps with John and Damon from DevOps Cafe appeared first on Software Engineering Daily.

Jenkins is an extensible open source continuous integration server. Kohsuke Kawaguchi is the primary developer of Jenkins CI and the CTO of CloudBees, a provider of enterprise Jenkins. Questions: How does continuous integration affect DevOps? What has changed in the five years since Jenkins was created? In what ways is Jenkins opinionated? What are the The post Continuous Delivery with Jenkins Creator Kohsuke Kawaguchi appeared first on Software Engineering Daily.

Container infrastructure has benefits of security, scalability and efficiency. Containers are a central component of the DevOps movement. Joyent provides simple, secure deployment of containers with bare metal speed on container-native infrastructure Bryan Cantrill is the CTO of Joyent, the father of DTrace and an OS kernel developer for 20 years. Questions: Why are containers The post Containers with Bryan Cantrill from Joyent appeared first on Software Engineering Daily.

Rocana applies big data, advanced analytics, and visualizations to dev ops in order to guide users to the root causes of problems. Eric Sammer is the co-founder and CTO of Rocana. At Cloudera, he served as an Engineering Manager responsible for tools and partner integrations. Within that role, he developed many of Cloudera’s best practices for The post Hadoop Ops: Rocana CTO Eric Sammer Interview appeared first on Software Engineering Daily.