Greatest Hits Archives - Software Engineering Daily

C++ is a powerful programming language that has been in use for several decades. Its importance lies in its versatility and efficiency, making it a popular choice for developing software and systems across different domains. The impact of C++ is significant, as it has been used to create numerous high-performance applications, including operating systems, browsers,

Comma is a startup aimed at solving self-driving cars. A lot of the new cars in the market have built-in stock Advanced driver assistance systems. Comma takes this system to the next level with Openpilot. Openpilot is an open-source driver assistance system. Currently, with features like Adaptive Cruise Control (ACC), Automated Lane Centering (ALC), Forward

The a16z Podcast is a show that is produced by Andreessen Horowitz, an investment fund based in Silicon Valley. The a16z Podcast covers topics including software engineering, biology, media, cryptocurrencies and entrepreneurship. A16z is one of the most popular podcasts about technology. Sonal Chokshi is the editor in chief at Andreessen Horowitz and the showrunner

Large software companies have become a target for criticism. Google, Facebook, Amazon and other prominent technology giants find themselves under a kind of scrutiny that is reminiscent of banks in 2008 and oil companies in the early 1900s. Across the planet, there is a growing sentiment that “big tech” has too much power, and that

Software companies such as Slack, Zoom, and Uber have recently gone public. When a company goes public, they issue a document called an S-1. Within the S-1, there is a wealth of information about the company, providing a detailed story about the company’s business model, economics, and future prospects. The S-1 describes the operating model

Envoy is an open source edge and service proxy that was originally developed at Lyft.  Envoy is often deployed as a sidecar application that runs alongside a service and helps that service by providing features such as routing, rate limiting, telemetry, and security policy. Envoy has gained significant traction in the open source community, and

Facebook has released open source software projects that have changed the industry. The most impactful projects to date are the React frontend user interface tools: ReactJS and React Native. Before React became popular, there were multiple competing solutions for the dominant frontend JavaScript framework. React became the most prominent because of its invention of JSX,

Facebook was built using PHP, a programming language that was used widely in the late 90s and early 2000s. PHP allows developers to get web applications built quickly and easily, although PHP has a reputation for being difficult to scale.  In the early days of Facebook, the company was scaling rapidly on every dimension.  New

Today’s episode is a keynote I gave at Full Stack Tech Radar Day in Tel Aviv. The talk is called “You Are Not a Commodity”. This talk is also available as a YouTube video. The slides can be accessed here. The world of commodity engineering is coming to an end. Developers are becoming more productive,

Sheng Liang was the lead developer on the original Java Virtual Machine. Today he works as the CEO of Rancher Labs, a company building a platform on top of Kubernetes. Sheng joins the show to discuss his experiences in the technology industry. The container orchestration wars had many victims. The competing standards for how an

Cloud computing was popularized in 2006 with the launch of Amazon Web Services. AWS allowed developers to use remote server infrastructure with a simple set of APIs. But even with AWS, it was still not simple to deploy and manage a web application. In 2007, Heroku launched a platform built on top of AWS. Heroku

Software engineering is an art and a science. To manage engineers is to manage artists and scientists. Software companies build practical tools like payment systems, messaging products, and search engines. Software tools are the underpinnings of our modern lives. You might expect this core infrastructure which modern humans rely on to have been constructed with

A service mesh is an abstraction that provides traffic routing, policy management, and telemetry for a distributed application. A service mesh consists of a data plane and a control plane. In the data plane, a proxy runs alongside each service, with every request from a service being routed through the proxy. In the control plane,

Google’s codebase is managed in a single monolithic repository. An engineer at Google can explore almost any area of the codebase within the entire company. In order to enable this, Google has built tooling to support the monolithic repo, including a virtual file system and a set of build tools. A monolithic repository is not

Facebook’s strategy is shaped by long term goals, short term requirements, and the available resources of the company. Long term goals are necessary for thinking through big decisions such as acquisitions, hardware product investments, and open source software ecosystems. To implement long term goals, Facebook needs to communicate the vision of the company and foster

Software engineering is harder than it should be. There are many people who have an app idea that they are not sure how to build. Some of these people are highly technical professionals like real estate agents, scientists, and accountants. These professionals learn to use spreadsheets in their day-to-day work. Spreadsheets are also used widely

RECENT UPDATES: FindCollabs is a company I started recently The FindCollabs Podcast is out! FindCollabs is hiring a React developer FindCollabs Hackathon #1 has ended! Congrats to ARhythm, Kitspace, and Rivaly for winning 1st, 2nd, and 3rd place ($4,000, $1000, and a set of SE Daily hoodies, respectively). The most valuable feedback award and the

RECENT UPDATES: Podsheets is our open source set of tools for managing podcasts and podcast businesses New version of Software Daily, our app and ad-free subscription service Software Daily is looking for help with Android engineering, QA, machine learning, and more FindCollabs Hackathon has ended–winners will probably be announced by the time this episode airs;

RECENT UPDATES: FindCollabs $5000 Hackathon Ends Saturday April 15th, 2019 New version of Software Daily, our app and ad-free subscription service Software Daily is looking for help with Android engineering, QA, machine learning, and more Haseeb Qureshi is an entrepreneur and investor. As a teenager, Haseeb played poker professionally through the online poker bubble. His

Upcoming events: A Conversation with Haseeb Qureshi at Cloudflare on April 3, 2019 FindCollabs Hackathon at App Academy on April 6, 2019 Chris Yeh is an entrepreneur, investor, and author. He co-wrote Blitzscaling with LinkedIn founder Reid Hoffman.   Blitzscaling is a strategy for growing a company that has found product market fit. Blitzscaling prioritizes speed

GitLab is an open source platform for software development. GitLab started with the ability to manage git repositories and now has functionality for collaboration, issue tracking, continuous integration, logging, and tracing. GitLab’s core business is selling to enterprises who want a self-hosted git installation, such as banks or other companies who prefer not to use

Cloud computing started to become popular in 2006 with the release of Amazon EC2, a system for deploying applications to virtual machines sitting on remote data center infrastructure . With cloud computing, application developers no longer needed to purchase expensive server hardware. Creating an application for the Internet became easier, cheaper, and simpler. As the

Uber manages the car rides for millions of people. The Uber system must remain operational 24/7, and the app involves financial transactions and the safety of passengers. Uber infrastructure runs across thousands of server instances and produce terabytes of monitoring data. The monitoring data is used to understand the health of the software systems as

Tyler Cowen’s book Stubborn Attachments outlines a framework that individuals can use to make decisions grounded in economic philosophy. In his previous books, Tyler examined recent economic history. Stubborn Attachments gives his perspective for navigating the future. Tyler is a professor of economics at George Mason University. He is also the host of Conversations with

Artificial intelligence is reshaping every aspect of our lives, from transportation to agriculture to dating. Someday, we may even create a superintelligence–a computer system that is demonstrably smarter than humans. But there is widespread disagreement on how soon we could build a superintelligence. There is not even a broad consensus on how we can define

Mars is a cold, inhospitable planet far from earth. It presents one of the most complex challenges faced by engineers: how can we create a new world? To create a new world, first we have to get there. We can build new rockets with improved propulsion systems. We can build ships that allow us to

Social media has transformed our lives. It has also transformed how wars are fought. P.W. Singer’s new book “Likewar: The Weaponization of Social Media” describes the far-reaching impact of social media on the tactics and strategies used by military, business, and everyday citizens. We have all read about stories such as Russian bots and Cambridge

Infrastructure software can be a great business. An infrastructure software company sells core technology to a large enterprise such as a bank or insurance company. This software has near zero marginal cost and generates a large annuity for the infrastructure software company. Once a bank has purchased your infrastructure software, the bank is likely to

Netflix has petabytes of data and thousands of workloads running across that data every day. These workloads generate movie recommendations for users, create dashboards for data analysts to study, and reshape data in ETL jobs, to make it more accessible across the organization. Over the last ten years, data engineering has become a key component

Serverless computing is a technique for deploying applications without an addressable server. A serverless application is running on servers, but the developer does not have access to the server in the traditional sense. The developer is not dealing with IP addresses and configuring instances of their different services to be able to scale. Just as

Most applications today are either deployed to on-premise environments or deployed to a single cloud provider. Developers who are deploying on-prem struggle to set up complicated open source tools like Kafka and Hadoop. Developers who are deploying to a cloud provider tend to stay within that specific cloud provider, because moving between different clouds and

In a cloud infrastructure environment, failures happen regularly. The servers can fail, the network can fail, and software bugs can crash your software unexpectedly. The amount of failures that can occur in cloud infrastructure is one reason why storage is often separated from application logic. A developer can launch multiple instances of their application, with

When a user makes a request to product like The New York Times, that request hits an API gateway. An API gateway is the entry point for an external request. An API gateway serves several purposes: authentication, security, routing, load balancing, and logging. API gateways have grown in popularity as applications have become more distributed,

An instruction set defines a low level programming language for moving information throughout a computer. In the early 1970’s, the prevalent instruction set language used a large vocabulary of different instructions. One justification for a large instruction set was that it would give a programmer more freedom to express the logic of their programs. Many

Open source software powers everything we do on the Internet. Google runs on Linux servers. Content sites are served by WordPress. Our data is queued in Kafka clusters and stored in MongoDB instances. The success of an open source project often leads to the creator of that open source software becoming wealthy. An open source

Matt Klein has worked for three rapidly growing Internet companies. At AWS, he worked on EC2, the compute-as-a-service product that powers a large percentage of the Internet. At Twitter, he helped scale the infrastructure in the chaotic days before Twitter’s IPO. Today he works at Lyft, building systems to allow for ride sharing infrastructure to

Google Search allows humans to find and access information across the web. A human enters an unstructured query into the search box, the search engine provides several links as a result, and the human clicks on one of those links. That link brings up a web page, which is a set of unstructured data. Humans

Google Search is a highly interactive JavaScript application. As you enter a query, results are being automatically suggested to you before you even finish typing. When you press enter, some of your search results may be widgets that represent the weather, the price of a stock, a recipe for green bean soup, or a language

Airbnb began in 2008 as a monolithic Rails application serving the simple purpose of listing homes for rental. Over time, the number of listings increased dramatically, as did the number of people who were renting. With that scale, the Rails app had to be broken into different services, and entire teams were built out to

Google Brain is an engineering team focused on deep learning research and applications. One growing area of interest within Google Brain is that of generative models. A generative model uses neural networks and a large data set to create new data similar to the ones that the network has seen before. One approach to making

GraphQL allows developers to communicate with all of their different data backends through a consistent query interface. A GraphQL query can be translated into queries to MySQL, MongoDB, ElasticSearch, or whatever kind of API or backend is needed to fulfill the GraphQL query. GraphQL users need to set up a GraphQL server to fulfill this

Stock traders have access to high volumes of information to help them make decisions on whether to buy an asset. A trader who is considering buying a share of Google stock can find charts, reports, and statistical tools to help with their decision. There are a variety of machine learning products to help a technical

Building software today is much faster than it was just a few years ago. The tools are higher level, and abstract away tasks that would have required months of development. Much of a developer’s time used to be spent optimizing databases, load balancers, and queueing systems in order to be able to handle the load

In the smartphone market there are two dominant operating systems: one closed source (iPhone) and one open source (Android). The market for self-driving cars could play out the same way, with a company like Tesla becoming the closed source iPhone of cars, and a company like Comma.ai developing the open source Android of self-driving cars.

Chad Fowler was the CTO of Wunderlist prior to its acquisition by Microsoft. Since the acquisition, Chad has become the general manager of developer advocacy at Microsoft. He also works as a venture capitalist at BlueYard Capital, an early stage investment firm. I’ve had a lot of fun talking to Chad, because he can move

React Native allows developers to reuse frontend code between mobile platforms. A user interface component written in React Native can be used in both iOS and Android codebases. Since React Native allows for code reuse, this can save time for developers, in contrast to a model where completely separate teams have to create frontend logic

Investing is an infinite game. In a game, a player can formulate a strategy based on the available resources, the apparent variance of the environment, and the metagame of the other actors involved. For an investor, the game board includes companies, currencies, and people. A successful game player can model their actions mathematically. They can

Moore’s Law states that the number of transistors in a dense integrated circuit double about every two years. Moore’s Law is less like a “law” and more like an observation or a prediction. Moore’s Law is ending. We can no longer fit an increasing amount of transistors in the same amount of space with a

Applications of artificial intelligence are permeating our everyday lives. We notice it in small ways–improvements to speech recognition; better quality products being recommended to us; cheaper goods and services that have dropped in price because of more intelligent production. But what can we quantitatively say about the rate at which artificial intelligence is improving? How

Vint Cerf is Chief Internet Evangelist at Google. He contributes to global policy development and continued spread of the Internet. This episode is republished from The Quoracast. Questions: What will the world look like in 5 years? What are the biggest problems associated with rapid spread and development of the Internet? Does blockchain technology present

“It’s a classic case where you have to be contrarian. It seems like the worst idea in the world to start a cloud hosting business. We didn’t know any better.” Moisey Uretsky is the cofounder of Digital Ocean, a leading cloud hosting provider based in New York. “It’s the usual immigrant story. My parents moved

“Humans are the most complicated thing out there – judging human skill is extremely hard, there’s all kinds of ways that people can be good.” Triplebyte is a technical hiring platform that vets engineers using a comprehensive evaluation platform and connects them to companies that are interesting in hiring them. Triplebyte was part of the

“Systems are giving up correctness for latency, and I’m arguing that stream processing systems have to be designed to allow the user to pick the tradeoffs that the application needs."

Continue reading…

“You’ve listened to podcasts where you gotta fast forward 8-9 minutes in before the actual meat happens.”

Continue reading…

“If you can make a system that can survive this random failure testing, then you will more or likely survive whatever other chaotic conditions exist.”

Continue reading…

"Software is this really unique field that is growing so rapidly that people are almost forced to specialize into one subdomain – and that kind of stratification is good for your job and for your employers, but it's not necessarily good for you as an individual trying to grow in the field of software.”

Continue reading…

“Changing anything changes everything.”

Technical debt, referring to the compounding cost of changes to software architecture, can be especially challenging in machine learning systems.

Continue reading…

“The cloud as an environment – I had thought it was a purely utilitarian kind of thing. What I realized is that it’s a fascinating centralized repository of computation.”

Wolfram Research makes computing software powered by the Wolfram language, a knowledge-based programming language that draws from symbolic and functional programming paradigms.

Continue reading…

“Mutable state is the root of all evil.”

Erlang is a functional, concurrent programming language that was originally designed within Ericsson in the 1980's. It was built to support distributed, fault-tolerant, non-stop applications suitable for telecommunications infrastructure.

Continue reading…

“Free Code Camp is my effort to correct the extremely inefficient and circuitous way I learned to code. I’m committing my career and the rest of my life towards making this process as efficient and painless as possible.”

Continue reading…

“If I was trying to learn coding on my own, to the level that App Academy was able to teach me, it definitely would have taken me significantly longer.”

Continue reading…

“The official motto that we have in our help manual is ‘Losing is fun!’ ”

Dwarf Fortress is a construction and management simulation computer game set in a procedurally generated fantasy world in which the player indirectly controls a group of dwarves, and attempts to construct a successful underground fortress.

Continue reading…

“Creativity never comes to you – she will only meet you halfway.”

Derek Sivers is a programmer, musician, and writer. He has created several companies and products, including CD Baby, which became the largest seller of independent music online.

Continue reading…

“There are a lot more people who have the problem that Hadoop solves than there are people using Hadoop.”

Pachyderm is a containerized data analytics platform that seeks to replace Hadoop.

Continue reading…

"If you focus on improving the developer experience, it will naturally translate into good documentation."

ReadMe is simplifying the process of writing documentation. The platform provides a readymade developer hub with the ability to integrate API endpoints into documentation.

Continue reading…

What is it like to be a young software engineer in Nigeria?

Osine and Anesi Ikhianosime have a deep understanding of the startup tactics that have led to so many successful companies in the Web 2.0 boom.

Their favorite podcast is a16z. Their role models include Elon Musk, Steve Jobs, Mark Zuckerberg, and Bill Gates. They recently worked on a machine learning compression project, inspired by the show Silicon Valley.

Continue reading…

Functional languages encourage practices and patterns that can simplify concurrent programming. Scala, Clojure, and Akka are functional tools built on the Java Virtual Machine.

Jessica Kerr is a functional developer on the JVM. She currently works at Monsanto. At QCon San Francisco, she will be giving a talk called Contracts in Clojure: Settling Types vs. Tests.

Continue reading…

"What we learn again and again is that security is less about what you think of, and more about what you didn't think of."

Bruce Schneier is a security researcher and author of Data and Goliath.

Continue reading…

Container infrastructure has benefits of security, scalability and efficiency. Containers are a central component of the DevOps movement. Joyent provides simple, secure deployment of containers with bare metal speed on container-native infrastructure Bryan Cantrill is the CTO of Joyent, the father of DTrace and an OS kernel developer for 20 years. Questions: Why are containers

Bitcoin’s cultural implications inform the engineering opportunities and constraints. Andreas Antonopoulos is a bitcoin researcher, journalist, and evangelist. Questions What are the taboo topics within the bitcoin community? What do you think of when people say “we know bitcoin is the first real cryptocurrency, but the big question is whether it will be the last”?

Tim O’Reilly’s book What’s the Future? is an overview of business, technology, and society. As the founder of O’Reilly Media, Tim has been steeped in technology trends for the last 40 years. From his vantage point running conferences and publishing technical content, Tim has been able to make informed predictions about what is coming next.

When software is performing suboptimally, the programmer can use a variety of tools to diagnose problems and improve the quality of the code. A profiler is a tool for examining where a program is spending time. Every program consists of a set of different functions. These functions call each other. The total amount of time

Self-driving, electric cars will someday outnumber traditional automobiles on the road. As transportation becomes autonomous, it is hard to imagine an industry that will not be affected by the downstream effects of this change. These cars will likely be managed by fleet operators like Lyft and Uber. We will need fewer cars, and the amount

When a user takes a ride on Uber, the app on the user’s phone is communicating with Uber’s backend infrastructure, which is writing to a database that maintains the state of that user’s activity. This database is known as a transactional database or “OLTP” (online transaction processing). Every active user and driver and UberEATS restaurant

A sample of the human voice is a rich piece of unstructured data. Voice recordings can be turned into visualizations called spectrograms. Machine learning models can be trained to identify features of these spectrograms. Using this kind of analytic strategy, breakthroughs in voice analysis are happening at an amazing pace. Rita Singh researches voice at

Mesos is a system for managing distributed systems. The goal of Mesos is to help engineers orchestrate resources among multi-node applications like Spark. Mesos can also manage lower level schedulers like Kubernetes. A common misconception is that Mesos aims to solve the same problem as Kubernetes, but Mesos is a higher level abstraction. Ben Hindman

Technology is pushing us rapidly toward a future that is impossible to forecast. We try to imagine what that future might look like, and we can’t help having our predictions shaped by the media we have consumed. 1984, Terminator, Gattaca, Ex Machina, Black Mirror–all of these stories present a dystopian future. But if you look

Google’s central system for managing to compute resources is called Borg. On Borg, millions of Linux containers process a wide variety of workloads. When a new application is spun up, Borg provides that application with the resources it needs. Workloads at Google usually fall into one of two distinct categories: long-running application workloads (such as

Machine learning tools are rapidly maturing. TensorFlow gave developers an open source version of Google’s internal machine learning framework. Cloud computing provides a cost effective, accessible way of training models. Edge computing allows for low latency deployments of models. But even if you are a kid with a laptop who has learned all the machine

“The Federal Reserve System is fraudulent. Whatever its stated purpose, its effective purpose is to create a mechanism of deficit spending by politicians, through the insidious invisible taxation of monetary debasement (aka inflation).” These are the words of Erik Voorhees, the CEO of crypto financial exchange ShapeShift. Long before he started ShapeShift, Erik was opposed

Cryptocurrency speculation has pulled in a large population of people who do not know what they are investing in. If you hear about an investment of $1000 turning into $1M, it’s tempting to get sucked in yourself. For most of these everyday people, the game is completely rigged. A large percentage of market activity is

Joseph Bonneau is co-author of Bitcoin and Cryptocurrency Technologies, a popular textbook. At NYU, he works as an assistant professor exploring cryptography and security. His YouTube lessons teaching Bitcoin have hundreds of thousands of views. His material offers clear explanations of how Bitcoin works. Since Joseph has a clear understanding of the objective facts around

Dogecoin was started in 2013 as a joke. Jackson Palmer forked Bitcoin and created his cryptocurrency as a play-off of the “doge” meme. The currency became popular as a means of Reddit users “tipping” each other. If I made a comment on Reddit that you liked, you might send me some Dogecoin. This use case

Apache Spark is a system for processing large data sets in parallel. The core abstraction of Spark is the resilient distributed dataset (RDD), a working set of data that sits in memory for fast, iterative processing. Matei Zaharia created Spark with two goals: to provide a composable, high-level set of APIs for performing distributed processing;

When Box started in 2006, the small engineering team had a lot to learn. Box was one of the earliest cloud storage companies, with a product that allowed companies to securely upload files to remote storage. This was two years before Amazon Web Services introduced on-demand infrastructure, so the Box team managed their own servers,

Your friends from college are asking you how to buy Bitcoin. Your mom is emailing you articles about the benefits of decentralized peer-to-peer networks. Your shoe shiner is telling you to buy XRP. It is 2018, and cryptocurrencies have become a daily part of news headlines. The general public may not understand how this technology

Kubernetes has become the standard way of deploying new distributed applications. Most new internet businesses started in the foreseeable future will leverage Kubernetes (whether they realize it or not). Many old applications are migrating to Kubernetes too. Before Kubernetes, there was no standardization around a specific distributed systems platform. Just like Linux became the standard

Kubernetes has become the standard system for deploying and managing clusters of containers. But the vision of the project goes beyond managing containers. The long-term goal is to democratize the ability to build distributed systems. Brendan Burns is a co-founder of the Kubernetes project. He recently announced an open-source project called Metaparticle, a standard library

You are requesting a car from a ridesharing service such as Lyft. Your request hits the Lyft servers and begins trying to get you a car. It takes your geolocation, and passes the geolocation to a service that finds cars that are nearby, and puts all those cars into a list. The list of nearby

Originally published January 6, 2016 “The best computer science is the kind where the theory is inspired by some practical problem, you develop a better theoretical understanding of what you want to do, and that feeds back into better practice.” Brian Kernighan is a professor of computer science at Princeton University and the author of

Originally published November 18, 2015 “The playing field has never ever been more leveled – that means everything you don’t build is your choice not to build it.” Seth Godin is a writer, speaker, and entrepreneur. He is the author of many books, including most recently, What To Do When It’s Your Turn. Questions How

Originally published November 10, 2015 “The cloud as an environment – I had thought it was a purely utilitarian kind of thing. What I realized is that it’s a fascinating centralized repository of computation.” Wolfram Research makes computing software powered by the Wolfram language, a knowledge-based programming language that draws from symbolic and functional programming

Originally published November 17, 2015 “Changing anything changes everything.” Technical debt, referring to the compounding cost of changes to software architecture, can be especially challenging in machine learning systems. D. Sculley is a software engineer at Google, focusing on machine learning, data mining, and information retrieval. He recently co-authored the paper Machine Learning: The High

There is a quote from Jeff Bezos: “70% of the work of building a business today is undifferentiated heavy lifting. Only 30% is creative work. Things will be more exciting when those numbers are inverted.” That quote is from 2006, before Amazon Web Services had built most of their managed services. In 2006, you had

Automation is changing the labor market. To automate a task, someone needs to put in the work to describe the task correctly to a computer. For some tasks, the reward for automating a task is tremendous–for example, putting together mobile phones. In China, companies like FOXCONN are investing time and money into programming the instructions

A popular software application serves billions of user requests. These requests could be for many different things. These requests need to be routed to the correct destination, load balanced across different instances of a service, and queued for processing. Processing a request might require generating a detailed response to the user, or making a write

As the gig economy grows, that growth necessitates innovations in the online infrastructure powering these new labor markets. In our previous episodes about Uber, we explored the systems that balance server load and gather geospacial data. In our coverage of Lyft, we studied Envoy, the service proxy that standardizes communications and load balancing among services.

Imagine that you are a lawyer. Your work involves managing files with dense, technical text. Your co-workers collaborate with you to accomplish a complex goal that can be broken down into smaller pieces. Your work has formal specifications, but there are degrees of freedom in how you express an idea. In all of these ways,

An engineer who wants to start a business using investment capital needs to understand the expectations of investors. The market for the business needs to be huge. The team needs to have a differentiated understanding of the market, or a differentiated product. The CEO needs to have the determination to continue operating the company even