# Peter Cai > Peter Cai, software developer and engineering leader from Massachusetts. Public Ghost content for AI and LLM tooling. This file includes a bounded export of public pages first, then recent public posts. Append `.md` to any post or page URL to get the content in Markdown (for example, `/example-post.md`). ## Pages ### About Peter Cai URL: https://www.petercai.com/about/ Last updated: 2026-08-11T22:28:01.000Z Hey. I'm a software engineer and engineering leader. I mostly share thoughts about software, startups, and investing. I'm currently the VP of Engineering at [Sigma Squared](https://www.sigmasquared.io/?ref=petercai.com). Prior to that, I led engineering for [Privy.com](https://www.privy.com/?ref=petercai.com) for about 7 years through an acquisition. Before that, I was the founder of [Kiwi6](https://www.petercai.com/kiwi6), and did engineering for a few years at Microsoft (and numerous other organizations). ## Projects and stuff ### [Open Source](https://github.com/pcai/?ref=petercai.com) I'm the maintainer of [Savon](https://www.savonrb.com/?ref=petercai.com), the XML SOAP client for ruby, and some supporting gems like [wasabi](https://github.com/savonrb/wasabi?ref=petercai.com) and [nori](https://github.com/savonrb/nori?ref=petercai.com). I also made [SafeQuery](https://github.com/pcai/safe%5Fquery?ref=petercai.com) for Ruby on Rails. ### [Sandbox](https://sandbox.petercai.com/?ref=petercai.com) A showcase of some programming tools and utilities I often find myself needing: [sha1/sha256/md5 hash](https://sandbox.petercai.com/programming-tools/hash?ref=petercai.com), network tools ([open port check](https://sandbox.petercai.com/network-tools/open-port-check?ref=petercai.com), [DNS lookup](https://sandbox.petercai.com/network-tools/dns-lookup?ref=petercai.com), etc), a [UUID generator](https://sandbox.petercai.com/programming-tools/uuid-generator?ref=petercai.com), etc. Also was an excuse for me to build a single-page-app from scratch. ### [History timeline visualizer](https://history.jackieandpeter.com/?ref=petercai.com) An experimental visualization technique for events in world history. The thesis is that seeing events with related and concurrent things that were happening helps us understand and contextualize history better. ### [Jackie & Peter](https://www.jackieandpeter.com/?ref=petercai.com) A blog that [my wife](https://www.jackierocks.life/?ref=petercai.com) and I maintain with pictures of Kevin (our dog), and some of our [family recipes](https://www.jackieandpeter.com/recipes?ref=petercai.com). ### [Railsbridge Boston](https://www.railsbridgeboston.org/?ref=petercai.com) I help organize and teach free introductory programming workshops targeted at minorities and underrepresented groups. ### [Boston Ruby Group](https://www.bostonrubygroup.com/?ref=petercai.com) I help organize the Boston Ruby Group on meetup. I've given a few talks in the past. ## Contact You can email me at [hello@petercai.com](mailto:hello@petercai.com). I try to respond to all emails. ## Elsewhere - LinkedIn: [Peter Cai](https://www.linkedin.com/in/petercai/?ref=petercai.com) - Github: [pcai](https://github.com/pcai/?ref=petercai.com) - Twitter: [pcai\_](https://twitter.com/pcai%5F?ref=petercai.com) ### Learning how to become a professional software developer URL: https://www.petercai.com/learn-programming/ Last updated: 2010-01-01T12:00:00.000Z # Become a professional software developer ## Free online courses - [CS50: Computer Science Courses from Harvard](https://www.edx.org/cs50?ref=petercai.com): Harvard's famous introductory computer science course. ## Textbooks - [Think Java: How to Think Like a Computer Scientist](https://greenteapress.com/thinkjava6/html/index.html?ref=petercai.com): Hands-on introduction to Java programming and computer science. - [Eloquent JavaScript](https://eloquentjavascript.net/?ref=petercai.com): A thorough guide to Javascript programming. - [High Performance Browser Networking](https://hpbn.co/?ref=petercai.com): An excellent, in-depth guide to browser networking. - [Structure and Interpretation of Computer Programs](https://sarabander.github.io/sicp/?ref=petercai.com): A difficult book on the fundamentals of computer programming and programming languages. - [A Course in Machine Learning](http://ciml.info/?ref=petercai.com): Introduction to most major aspects of machine learning. ### Technical Founder Guide URL: https://www.petercai.com/technical-founder-guide/ Last updated: 2010-01-01T12:00:00.000Z # Technical Founder Guide An opinionated collection of tips for technical founders. There are lots of guides around "how to find product market fit," how to hire, and other aspects of being a technical founder. And on the technical front, there are many tutorials for beginners creating "hello world" apps, and lots of advice on how to deal with massive scale and huge existing systems. But there are fewer guides on how to get a greenfield codebase up and running as a solid foundation you can build a team and product around. Hopefully, this will be useful for non-technical founders too - you'll get a flavor for the work that goes into building software at a professional level that is more than "prioritizing tickets and coding features." It's mostly geared towards services, but this applies to rich client applications (desktop, mobile, and embedded) as well, especially if they have to talk to a backend service. I'll also recommend some vendors, libraries, and services here, most of which I've personally used. # Zero to one How to get up and running without accruing a bunch of foundational debt. ## Select your technology stack and deployment infrastructure Do you have expertise in building, running, and debugging your technology choices? This may be the most important near term question - your initial momentum has ripple effects for the rest of the organization's existence. Don't start with a technology stack that your founding team is unfamiliar with. Is this a bad choice for the solution you're building? There's usually not a single right answer, but there are a lot of wrong ones. E.g., "we'll write our web app in C++" is most likely a wrong answer. Can you hire for this stack? Will you need to? How is the industry trending? ### Picking data stores Your choice of data store is very important - data tends to grow and accumulate over time, and unlike other technology choices, it's hard to transition to a new tech by running it side-by-side, because you often need all your data in a single place. Two straightforward schools of thought are 1) use a proven, mature, multi-purpose store correctly, and it will reliably solve many problems for a long time, or 2) pick a bleeding edge store uniquely suited for your use case and hope that the productivity/performance benefits swamp the overhead of dealing with immature database technology. The right answer is context-specific, but commonly #1 is a default good answer. It's generally a bad idea to pick a database for its scalability. Database scalability is an important long term consideration, but most solutions are already very scalable with varying degrees of effort -- as long as the schema is designed correctly. Performance is not scalability, but it can be a reasonable substitute: on modern hardware, careful application and schema design will usually get you enough performance that you don't have to think about scale-out for a long time. As with programming language and platform decisions, expertise trumps the "optimal" choice. You will benefit far more from having an expert on your chosen database than picking a "scalable" database you have less expertise in. Remember that Google Ads, Uber, and Facebook were quietly being built on top of MySQL when "big data" was becoming part of tech vernacular. General advice that applies to most data stores is: store less data. This helps you in multiple dimensions (reducing pressure on caches, reducing expensive high-performance storage requirements, fitting more indexes in memory, etc). Data needed only for "offline" analysis should generally be kept out of OLTP databases. For append-only use cases (like time series or logs) where older data is accessed less frequently, consider a time-based partitioning scheme to enable easier archiving or scale-out later. Over time you will likely accrete "auxiliary" stores for specialized use cases, such as fulltext search or caching. For every store you introduce, decide upfront whether it is a critical source-of-truth, or an auxiliary store that can regenerate its data from somewhere else. Find ways to prevent developers from committing critical data to auxiliary stores - using naming conventions, documentation, code comments, etc. Then be vigilant anyway, because these details get lost over time and "obvious" ideas aren't obvious to everyone. This will save an enormous amount of money and headache later. ### Keep things simple Resist the urge to take the more technically impressive approach when it is of dubious benefit. Whatever you do, don't use a microservices architecture. You ain't gonna need it. If you're making a web app, most likely you don't need to build a Single Page App. Creating dynamic web applications with just pure server-side rendered HTML is approximately 5 times less effort than a comparable SPA, and mostly nobody will notice. The best part? You can still mount React.js components and such wherever you need highly interactive and complex UI with lots of state. Just don't make the mistake of defaulting to this for your entire app and paying the tax forever. Managing an API with a single internal consumer is just volunteering for needless pain. ## Configure your development environment Decide on your version control system. Unless you have a very specific compelling reason to do otherwise, use git: most developers these days are trained on git (unless they work somewhere with a proprietary internal system like Google, Microsoft, IBM, etc - but in any case the concepts employed will have parallels in git). Linux and Windows (e.g., millions of files in a single repo) are versioned with git. I promise it supports what you need. Create a policy for how new libraries, databases etc are added to this environment. If there are specific licenses that aren't allowed, document them now to prevent building them in as foundational dependencies in your app. Make sure your environment is documented, versioned, and reproducible. You will likely want to use some container technology (like Docker) to get these reproducible build environments; this may also help with the deployment story. Your ideal state is "run this one-liner to install Docker, then use this one-liner to bootstrap everything in Docker." Test this on a new machine, repeatedly if necessary: ideally anyone should be able to copy paste a script on a machine out of the box and have a working dev environment. To develop these setup scripts, I recommend creating a VM, and snapshotting it so you can restore it to a clean "just before my setup script" state. Create a seed file to prepopulate any database data needed to boot the app. Check this into source control and reference it from your setup steps. Consider what IDEs will be supported. You might want to check in configuration files for those IDEs to enforce certain norms (e.g., tabs vs spaces, semicolon policy, etc). Set up linters configuration, document what licenses are needed, and check this into the readme. Consider starting on a cloud IDE like Github Codespaces, AWS Cloud9, or Google Code Cloud. A cloud IDE will save you hours of toil and debugging broken setups, "works on my machine" problems, and general reproducibility issues. Unfortunately there are still many use cases where the limitations of cloud IDEs prevent widespread adoption (applications that depend heavily on subdomains come to mind). In general, minimize dependencies on an external service to boot up. Service outages or lack of internet access will break your ability to start up your dev environment (taking a dependency on a cloud IDE is a calculated risk in this regard). Also note that network dependencies to start your dev environment may imply you need it for your production service to boot as well. Use a dotenv file (unversioned) to store secrets. Version your configuration files and don't store configs with your secrets. Configs are like "in development, the web server starts 5 threads." If you store all your configs alongside your secrets in unversioned dotenv files, synchronizing configs between developers becomes a nightmare. ## Configure DNS and networking You probably aren't using a "network perimeter model" so hopefully you don't have a need to set up a corporate network, VPN access, etc. Use Cloudflare to host your DNS. You can use the AWS or Azure or Google-specific services too, but Cloudflare is the best, and you likely want them in the mix even if you use one of the others. Cloudflare comes integrated with a CDN, reverse proxy, SSL, and many security features, with pricing from "free" to "surprisingly reasonable." This significantly cuts down on the number of other vendors you need to work with and integrate. Enable DNSSEC and periodically export your configuration files to backups. You should also be enabling HSTS, making a habit of always assuming HTTPS, and not supporting HTTP at all. This is much easier to do from the get-go as opposed to enabling HTTPS later and dealing with surprise mixed-content issues forever. ## Configure your production environment Most people benefit from starting and staying on a monolith architecture on a single repo per deployment target - e.g. one for web, one for iOS, etc.. Dependency management and sharing of common libraries otherwise becomes very hard way too early, sucking up valuable time on menial trash like how to sync schemas between your microservices. Public services like Github do not do "monorepo with multiple deployment targets" well out-of-the-box, since you *usually* want to build (and maybe deploy) on every commit, *but* only for the specific targets that were changed (e.g., if your backend and frontend are separate deployment targets, you might not want to trigger a backend deployment when someone checks in a frontend change, but this requires extra work to configure in a monorepo). Low operational overhead is usually the name of the game (unless - for example - you are an infrastructure provider and expect this to be where your margin comes from). So if you can, favor one of the managed platform providers like Heroku, Fly.io, or Render. And consider using a managed database like crunchydata. Make sure you have encryption at rest and protect sensitive information in transit. Make sure you have high availability and failover instances of every tier - frontend, web, database, and so on (if you have good backups, there is a case to be made that very early stage startups in certain domains don't *need* high availability, which can sometimes reduce your costs by 2x or more). Make sure backups are enabled, and you have a documented policy to test these at least annually. Make sure your environment is documented, versioned, and reproducible. ## Create a build system A build should be a single command in both dev and the test environment. A single script should be able to produce a production build at any time. If you're creating a service, this should extend to deployment as well - deploying to production should be a single command. You'll want to do this often; maintaining a green build all the time is one of those areas where automation always pays for itself, and changes the way you develop for the better. In general this should be run on every checkin with results visible in your VCS. I highly recommend investigating if your hosted VCS provides this out of the box: Github Actions, GitLab CI/CD, etc. Minimizing the number of vendors and logins you deal with is always important, but helps a lot early on since you probably don't have a corporate identity service yet. Your test system may depend on external vendors - Chromatic, Browserstack, etc. More realistic tests that use I/O and depend on external vendors tend to be slow. Set up parallelism in your tests as soon as feasible, so that you don't have to go and rearchitect them later. Don't spend too much time actually writing tests early on. There isn't enough value/complexity yet to make them pay off. Just make sure there's a good test system in place, you have a few high level end-to-end tests, and that they're being regularly run and kept green. ## Set up a deployment pipeline If you're running a service, you need a way to deploy to production in a single step. You'll also need to configure other environments: staging, test, pre-production, or whatever name(s) make sense for your workflow. You will need to secure these and potentially register separate domain names (etc) for each environment, as you want each environment to be 100% segregated and independent of any other environment, with no resource sharing whatsoever. Design the system to allow one-click rollbacks of code. You'll need it eventually. I like Heroku because of this. Your goal is to facilitate high deployment confidence *and* lower the cost of mistakes so builds and deployments can be fast and iterative. Make sure *everyone* knows where to see the current status of the service and deployment history. "Hey prod is down, did someone just deploy? How can I reach them?" is a 100% preventable problem. Each time you run into problems in your pipeline, update your documentation on edge cases (e.g., most people learn quickly they can't roll back certain changes naively, if they include a database migration or API change). # Running in production Once you're up and live, make sure you invest in your devops system to maximize visibility and confidence. ## Define your release process Initially, your release process is going to be fairly coupled with your build/deployment process. This is fine for most domains. The first thing you'll want to specify is a deployment checklist. Keep this up to date and use it every time you deploy. Shorter lists increase compliance, so rely on automation and automated checks to fail builds whenever possible. Only put things in deployment checklists that can't be (or haven't been) automated yet. You may eventually want to decouple deploying and releasing, which is when building or integrating a service like LaunchDarkly will help. You can also build your own if you know your requirements well. This will let you deploy code as soon as its done, and then pick a specific time to go live in coordination with other teams. Feature flags will naturally be (ab)used for runtime configuration and client-specific overrides, especially if you don't have a great way of managing that yet. You can either embrace this or try to clamp down on it, but in either case you will want to document specific uses that are OK and not OK, and best practices around default values, expiration, evaluating server-side vs client-side, etc. For rich clients, there's often a number of manual steps (especially at first) to get a "release" out, and this may involve waiting on approval from a third party ecosystem gatekeeper. Write these steps down as you do them to minimize the mental overhead and risk of errors (or extended review cycles). Your goal as you do this the first few times is to automate the mundane parts as much as possible and make this readme as short as possible - it should still be a single command to build a release artifact, even if there are a bunch of other inputs to submit for app review. ## Add a queuing service Sooner or later you will need to add some kind of queuing to your system so that it can handle surges in activity and long-running tasks. There are some great language-specific libraries for this (Sidekiq for Ruby, Celery for Python, etc), but resist the temptation to use them for generic "async message queues" - for those, you'll want something much more suited to message passing like AWS SQS, RabbitMQ, etc. Those will generally scale better and interoperate better. Having multiple different queues will be critical for ensuring responsiveness, and lets you scale bottlenecked jobs independently. This is extremely hard to get right without understanding the load characteristics of the system, and how latency-sensitive each type of job is. Some examples of separate queues might be 1) a queue for small, time-sensitive jobs like transactional emails (password resets, order confirmations, etc), and 2) a queue for long-running jobs like batch operations that can take minutes/hours. You don't want email jobs stuck behind 30 minute batch export jobs. Beware of grouping jobs by how fast they are on average; if your p99 or 95 on some jobs are really high this can cause huge latency spikes. Job performance can change over time. Make sure you configure good monitoring on queue metrics (queue latency is best, although "queue size" can sometimes be helpful), and adjust things as necessary. Eventually, you may want to introduce rate limiting in your queues -- either in terms of how quickly your users can perform certain operations, or how often you can use an external API. There are a number of strategies and tradeoffs here, and depending on how many different queues your architecture can handle, your designs will be different. For example, if you will only ever have a few hundred or thousand clients at most, it might be feasible to have client-specific queues, which will allow you to rate limit or throttle jobs without unintentionally affecting other clients. As you build, there will likely not be a discrete point where you go from "monolith" to "microservices," but rather the team will gradually accrete microservices naturally. Avoid the temptation to put a queue in front of every service, as it will increase operational costs and add end-to-end latency without commensurate benefit. Instead, decide what layer(s) will be responsible for queuing, and add backpressure and load shedding to the system around that layer. Beware that adding a queuing service introduces **much more** operational and conceptual complexity to your application than you might expect. This is because it introduces an asynchronous layer, allowing race conditions where none existed before. It also makes it much more complicated to enforce happens-before order guarantees. At a certain point these become worthwhile tradeoffs, but they require you to think about what types of work will be done asynchronously, and how progress will be tracked and reported to end users. You may need a mental framework for this, and eventually a way to document and communicate it. ## Set up logging and exception tracking Sentry is far and away the best exception/crash reporter - get this instrumented on your mobile app, your web backend, and your Javascript tier. They have libraries for all 3, and in many supported languages as well. The robustness of your error reporting is critical, so make sure you configure all the integrations and libraries correctly, or you might not even know when you've shipped a showstopping bug. Take the time to connect your VCS and set up any de-duplication rules as needed when the first bugs come in. A good exception tracker has the robustness and flexibility to support accurate de-duping of common crashes and exceptions. Also make sure you can distinguish what version of the code is running, so you can group by build and trace regressions back to the code that introduced them. Make sure you are streaming all your application logs to somewhere they can be searched. They can help you chase down anomalies and find patterns in production that don't show up as crashes or exceptions. ## Set up operational monitoring Separately from logs and exceptions, you'll also need operational monitoring - how many requests are you serving, and how quickly? What resources are near capacity? When was the last deploy? This last one is important to instrument so you can quickly tell if a recent change broke something. AWS Cloudwatch (for example) lets you cobble a lot of this together, but NewRelic or DataDog might be more suited and offer many drop-in libraries for collecting and reporting operational statistics you can alert on. You'll want to set up a vendor like PagerDuty or Opsgenie to schedule any on-call rotations, if your service has uptime requirements. Set up key transactions to monitor - your most important and common user actions. Configure baseline metrics to fire alerts if they go out of range - e.g., response times, queue delays, etc. Initially, tighter tolerances are helpful as things can go from "degraded" to "completely down" very quickly. Over time, you'll want to tune these so that every alert is both important and actionable with a pre-written runbook. You may want a public facing statuspage so you can share updates with customers in the case of incidents. ## Secure your data Salt and hash your passwords. Encrypt sensitive data at rest at the application level. Separately, enable encryption at your data tier. Encrypt in transit with TLS. Enable DNSSEC and HSTS. Store your secrets separately from your code. Defend against XSS and content injection. # Getting to MVP Once deploying to production is seamless and automatic, you'll need to get add supporting infrastructure to deal with real live users and your initial customers. ## Integrate billing Use Stripe for handling billing and money. Nothing else really comes close. Stripe has a built-in subscriptions API for handling recurring billing. Depending on how complicated your billing ends up being, you sometimes have no choice but to custom-build on top of native Stripe payments instead. This gets complicated fast, and is rarely worth it when (for example) selling to SMBs. Stick to something simple like month-to-month or annual prepaid billing, or a pay-as-you-go balance for usage-based. You'll have all kinds of exceptions you need to support - internal staff, partners, investors, etc, are all potential groups that you want to sometimes treat "specially." You'll need to weigh the benefits of pre vs post paid billing, and other details like whether you'll issue invoices based on signup date or month-aligned. Complexity in billing builds up fast, and part of the complexity is because dates are complex! Taking time to keep things conceptually simple (like month-aligned billing, for example) can really help in the long term. This might have business implications (e.g., maybe it makes your cashflow unacceptably spiky), so try to iron all these out beforehand. One subtle problem you'll need to solve is: you must agree as a team on where your source of truth for billing information lives. As an engineer, it can be tempting to say "I know! I'll store it in Stripe!" Except - if you deal with mid market or enterprise, your customer data is probably in Salesforce or Zendesk, or written up in some sales contracts, potentially sitting in someone's email inbox. Things are fine if you truly have a self-serve product and everyone is paying the same price for a Pro or Business plan by credit card. But as soon as someone can configure a "custom" plan or wants to pay by check, and you're tracking customers and payment terms somewhere else in addition to Stripe, things quickly go off the rails (but you probably won't notice until later when you try to audit your financials). ## Build a staff interface You'll quickly realize you can't just run SQL against your production database to provide support. Building a staff interface will be critical to safely supporting your users. You generally shouldn't have to build from scratch. Framework-specific libraries (like Avo or `rails_admin` for Ruby on Rails, or the default Django admin for Python) can bootstrap a lot of the heavy lifting and CRUD actions you need. Invest in this so you don't waste cycles doing dangerous one-off data fixes in production. If you're in a regulatory domain that allows it, it is *extremely* useful to build a feature that lets you "assume" the identity of one of your users. Days of email back-and-forth can be eliminated by simply *seeing the world as if you were logged in as that user*. You can easily find and fix bugs this way, and it can have some unexpected benefits occasionally as well (like spotting or dealing with abuse). ## Add authorization and access control You'll need to differentiate between who can and can't access your staff interface, so you'll need to add authorization and access control. If you're running a multi-tenant cloud application, you'll also want to enforce those access rules now. Users and internal support staff will ask for customizable role-based access control; how urgent this is depends on customer size and product domain. ## Audit user actions You'll want to audit and prominently display sensitive actions like signups, account closures, etc, and basically any actions taken in your staff interface. This can help you understand and debug accounts, and is required for many compliance scenarios as well. ## Support undo It is essentially a law of physics that whatever action can be done in your app, eventually someone will desperately want to undo it. You should consider auditing everything, including things that don't seem to be of interest now. Your staff interface will need more and more "undo" controls over time, and the audit log can be a valuable source of information to reconstruct state that was accidentally changed. Soft deletion is sometimes controversial. Any approach to soft-deletion comes with substantial trade-offs, but one that has the fewest unintended side effects is along the lines of "create a table called \`deleted\_records\` and stick deleted entities in there. That way, you can undo accidental deletions and such without having to constantly deal with omitting soft-deleted records polluting your other tables. ## Integrate with email and messaging You'll need to send transactional emails for things like password resets, so consider integrating with a service that can act as your SMTP relay and get all your tricky deliverability issues on autopilot. The gold standard is probably a paid service like Postmark, since they have a product that only delivers transactional mail. Otherwise you can try your luck with more budget providers like Amazon SES, Sendgrid, Mailgun, or Sparkpost. For rich client apps, integration with OneSignal or Pusher can help you manage notification messages; otherwise Twilio is the go-to for voice and SMS. ## Create a cron Create a system to run things on a regular schedule, as you'll need to do this periodically (rotating out old data, sending reports, rolling up statistics, etc). Consider setting up a system to alert you when these fail, as they can otherwise fail silently for long periods until someone notices. Dead Man's Snitch is an example service that can integrate with your monitoring setup. ## Manage your vendors and dependencies At a certain point you'll need to keep track of your major dependencies and vendors. Early on, a spreadsheet with some key data is fine: name of vendor, purpose, approximate cost, who has access. # Scaling up As you start to get users, customers, and the people to support them, you need to build features and systems to deal with increased scale. ## Add single sign on Dealing with multi-factor authentication, password resets, and "I-didn't-get-the-email" problems are a distraction from real issues to solve. Adding domain-specific single sign on is usually a no-brainer and can help cut down on those types of support headaches. If you're building & selling to enterprise, SAML SSO and such becomes a necessary checkbox very quickly. There are some services (like WorkOS) that try to handle this for you, because its very *deceptively complicated*, and every implementation has unique quirks that aren't necessarily captured in specifications or documentation. Be careful about how you select and match identifiers. For example, if someone can sign up with Google, are they allowed to change their email address to one that doesn't match their Google account? The more conservative answer prevents people from accidentally creating more than one account. ## Build multi account management In many domains, your power users and best customers will want to manage multiple accounts from one login. Note that "manage" means different things to different people - the high order bits might be about sharing resources or user permissions between accounts, or it might be about billing. You may need to do substantial refactoring of your system to support multi account management, but this can have a substantial payoff if your power users are your best customers. In a very strict multi-tenant setup, you might be enforcing multi-tenancy via some form of "every customer gets their own schema," "every customer gets their own partition key," or something to that effect. This is often the foundation for a scale-out strategy as well - many use cases lend themselves well to sharding by customer key. Multi account management throws a wrench in this because it implies the need for a "global" schema somewhere. Avoid partitioning or sharding your users and permissions tables until you have a solid understanding of your multi-account requirements. ## Add anti abuse Initially, every instance of abuse will be new and novel. In most domains, its OK to plug them when a bad actor abuses something. After you achieve some scale, you'll need to be more proactive. If you are building an app or service with user-generated content that's displayed to other people, you'll need to work with leadership to figure out policy enforcement around spam, botting, doxxing, pornography, terrorism, malware distribution, sanctions evasion, and so on. Deploy recaptcha or hcaptcha on any actions that can be botted. This includes sign up, sign in, password changes/resets, email forms, etc. Correctly implemented, these will also act as a sort of natural rate limit. You'll need to enforce rate limits in other parts of your code, such as when interacting with APIs or API clients. Make use of quarantining / shadowban mechanics whenever possible. While frustrating for real users, you need to withhold as much information about how your anti-abuse mechanisms work to foil real adversaries. ## Integrate with a business intelligence tool Ideally, your business intelligence tool can plug into the data exhaust of your operational monitoring tools. Most likely though, this will separately require a lot of conscious effort. The gold standard seems to be to get everything into Snowflake (or Redshift, if you like to trade pain for money). But if you are clever, you can create structured logs, or send everything to (for instance) StatsD for operational monitoring, and then instrument your code for business intelligence as well (e.g., fire events like `user.sign_in` and `upgrade.purchase`). This does not solve the problem of complex analytics queries against your production database. You don't want to run complex analytics queries against your production database. There isn't a single correct solution here because its domain-specific, but most answers collapse to one of: - Run a replica database and build a copy script that regularly dumps from the replica into your analytics database. Or run this copy script against production during low traffic periods if you don't care. - Hard mode: Architect *everything* to be stream-oriented and fling events into Kafka/Kinesis/whatever and then have your data pipeline magically process this into your analytics database. This is like the StatsD solution on steroids, except now StatsD is a consumer of the data pipeline so you can fan out to everything downstream. Much ink has been spilled about the Right Way to do change data capture. The important thing to keep in mind is that you need to treat your production system as a single producer (out of many) of data, and that many downstream systems will eventually care about every change that happens in that system. This will let downstream systems reliably construct the state of your production database without actually having to directly query it. Depending on the volume and value of your data, you may also want to send everything to a data lake for easy re-processing later. This is a lot harder than it may seem, as it involves changes to your product development processes to ensure that any new features or changes account for the data pipeline. ## Integrate with a product analytics tool Your product org is always going to want as much information about what users are doing as possible. Vendors like Amplitude will probably be your gold standard here. Heap Analytics is also plausible. At really low volumes, screen recording software like Hotjar, FullStory, or Mouseflow are viable. You will need both one-time and ongoing investment with these products (don't believe the sales pitches). In fact, as your organization grows in sophistication, you'll need to introduce governance so that everyone is operating off the same definitions, and is looking at the same types of reports. It will save tremendous headache in the future to plan out your usage patterns and naming conventions. Remember that thing about building a robust data pipeline because everything is architected to be stream-oriented? If you did that, you can plug in Amplitude/ integrations to it here. If you didn't, then you need to directly integrate with them from your application. ## Integrate with marketing and sales tools Sales and marketing leaders (etc) will eventually request changes or data. Decide how you'll want to handle these, keeping in mind: you probably have better things to do than updating promotion text on the login or pricing page. As you scale you will want to move most content into a CMS so that non-technical teams can go and update marketing copy. Google Tag Manager and other toolkits like it can be your friend here. Be thoughtful about governance and training, since this can devolve into a backdoor rats nest of untested/unreviewed code and a source of security liabilities. Remember that thing about building a robust data pipeline because everything is architected to be stream-oriented? If you did that, you can plug in Hubspot and Salesforce integrations to it here - hooray for loose coupling. If you didn't, then you need to directly integrate with them from your application. ### Kiwi6 has shut down URL: https://www.petercai.com/kiwi6/ Last updated: 2010-01-01T12:00:00.000Z # Kiwi6 has shut down December 31, 2022 Kiwi6 shut down on December 31, 2022\. All user data was permanently deleted and can no longer be recovered. Previous customers can contact **support@kiwi6.com** if they believe they are due a refund or billing adjustment. Please include relevant payment details or an invoice. Since all non-financial data has been permanently erased, we regret that we will not be able to help with other support inquiries. You can read the founder postmortem: [Why Kiwi6 shut down with 645,911 users](https://www.petercai.com/why-kiwi6-shut-down/). Thanks, Peter ![Kiwi6 logo](https://www.petercai.com/assets/images/k6_embed.png) ## Previous updates ### Kiwi6 sunset date is 1 month away December 1, 2022 Pursuant to our previous update, on December 31, 2022, Kiwi6 will be shutting down. New uploads are now disabled for all users. Download your files now before they are removed. All files will be deleted -- and all personal user information will be permanently removed -- on December 31, 2022\. Please note the following will also be deleted, and will stop working after December 31st: - Artist profiles - Podcast feeds - Embed widgets - Hotlinks - Albums and album art - Listen statistics - Comments and private messages - Favorites - Playlists - Private files Retrieving deleted files: If your files were deleted for inactivity, this deletion is permanent and those files can no longer be retrieved. ### Kiwi6 sunset February 5, 2022 Kiwi6 is sunsetting its service. Here's what you need to know: For free users: - Kiwi6 is now readonly. - Paid subscriptions are no longer available. - Files will be available until January 1, 2023\. Download your files now before they are removed. For paid users: - Kiwi6 will become readonly on March 1, 2022\. Files will be available until January 1, 2023. - All paid subscription benefits will continue until January 1, 2023, but all subscription payments will be paused going forward. Thank you for your support. ### Kiwi6 embed code has been removed URL: https://www.petercai.com/kiwi6-embed/ Last updated: 2010-01-01T12:00:00.000Z Kiwi6 shut down in 2022\. If you are the site owner, you should remove the Kiwi6 embed code from your website. [More about the Kiwi6 shutdown](https://www.petercai.com/kiwi6). ## Posts ### How AI has helped URL: https://www.petercai.com/how-ai-has-helped/ Last updated: 2026-09-19T05:18:58.000Z An incomplete list of ways in which modern AI has helped me in the past year or so. No over arching point, just documenting how AI has improved my life. Software - It writes my code, obviously. Almost all my personal projects are either 100% hand written or 100% vibe coded; almost nothing in between. The vibe coded projects are significantly more ambitious and feature rich. - AI helps me manage chores for the open source libraries I help maintain. - On several occasions AI was able to trace and explain a bug in an open source project I didn’t understand, because I didn’t have the time and patience to personally trawl through code I didn’t author. On at least one occasion I was genuinely stumped though, and the AI was provably able to fix a bug that I couldn’t. - I have been able to run models on a scheduled loop to automatically inspect, debug, deploy, and verify fixes. The model respects my rules for when to escalate problems it cannot solve on its own. - There have been scenarios where AI acted or didn’t act in the way that I wanted, but so far this has always been due to ambiguity, not insubordination. Education - AI is an infinitely patient on-demand tutor on whatever topic I happen to be interested in. At this point hallucinations are rare enough that whenever I bother to fact check anything, the AI answer is corroborated. I also like to think I’m developing an intuition for what kinds of training data a modern LLM is likely to have access to, and that my success is also a function of asking better questions. For example I basically never bother to double check historical facts. - When I’m traveling I can learn the identity of any building or monument by uploading a picture of it; then I can learn about its history. - When my daughter was born, AI helped us learn what was expected vs unexpected, and provided a lot of tips and information with citations. This was later corroborated by our pediatrician, and in the only cases there was disagreement (1) it was minor and inconsequential, and (2) the AI advice was backed by more recent studies and better data. - AI helped me quickly get up to speed on the latest AWS services when I retested for my solutions architect associate certification. - AI was able to summarize the critiques of the books I read, providing valuable balance and perspective. In at least one case, it linked to a very thorough and convincing critique of the book that thoroughly dismantled the author’s central claim, their academic integrity, and the entire basis of their career. This saved me from establishing a worldview based on fiction that had incorrectly been labeled nonfiction. - When my wife got her real estate license, AI-authored study guides had fewer errors and hallucinations than the materials provided by the course she was enrolled in, and AI provided answers that were roughly as accurate as the instructor (in the few instances I can recall any disagreement, it often came down to a hair splitting detail in the scenario that can completely flip the outcome, or some legal situation that worked one way in the past but the law changed and it now works a different way, but there was no clear pattern in whether the instructor or the AI had the more “current” view). Home - AI helps me look up the relevant building codes and maintenance schedules for components when I’m working on our home. It provides exact citations to the relevant codes. - I have received very large uncountable benefit from analyzing photos of an old family home with AI, because I was able to identify potentially unsafe conditions that needed immediate correction. At the same time, AI has also warned that a simple and obvious seeming fix for some issue was actually dangerous and could lead to hazardous exposure or massive damage. This has happened multiple times. On at least one occasion many years ago (pre modern AI) one of these hazards actually manifested exactly as warned, so these issues are not theoretical. - AI made it easier to find and read manuals for equipment I own. In a few cases I needed to know roughly what year something was manufactured, which AI was able to tell me instantly with a photo of the serial. - AI taught me to do my own minor plumbing and electrical work around the house. It also tells me when a certain project is probably beyond my ability. - AI helped me find and buy new major appliances that were specifically optimized for the things I cared about, and which turned out to be excellent purchases. This was long enough ago that I only used AI at the “end“ to vet my choice, and in both cases where this happened, I ultimately ended up with a different choice than I started with because the AI found me better options. - AI was able to correctly diagnose a structural defect in our home from a photo; this exact defect was later confirmed almost verbatim by an engineering firm, and the findings the AI missed were solely because the engineering firm got more and better photos. - When our electric bill climbed to new highs several months in a row, AI gave me good ideas for how to measure and trace it, which led to an optimization pass in our home, resulting in significant energy savings. - This section was originally under “education“ but I had so many examples I ended up moving it into its own section. Email - AI summarizes my emails, which is particularly useful for certain newsletters I subscribe to. - AI is able to extract (and track) key info like due dates and contract terms as they are revised. - AI can compose emails for me, although I don’t do this. - It’s a better-than-human translator, including for speech. Health - I used AI to review my strength training logs and it helped me notice and correct an issue in my training, which fixed some minor joint pain and got me out of a plateau. - AI at one point made an offhand comment about how (based on my numerous questions about optimizing food pairings) I appear to be a prolific home cook who adjusts recipes to use up what’s on hand and minimize waste, and after reflection, I agree. - AI gives me good food storage guidelines and convinced me to throw away some items that were past the point of good risk/reward. Legal/Finance/Tax - AI can quickly give me a broad strokes overview of my specific situation and point out issues I wouldn’t have noticed. This helped tremendously in preparing to speak to a professional because we wouldn’t have to waste each others time by having them explain the abcs of the specific topic we were engaging them about. - In one routine legal matter, AI recommended a seemingly valid approach that our attorney didn’t end up taking because our attorney considered it far more important to optimize for a different dimension of the problem. On a different facet of the same matter, the AI recommendation was actually better and more correct than what the attorney recommended (the AI correctly predicted a consequence that our attorney neglected to consider, despite this being a routine consequence of said matter). - AI helped me refine our financial and tax planning. We have paid for financial and tax help in the past, but don’t need an ongoing relationship with a provider, and this has remained the case even as our financial situation has gradually become (slightly) more complex. - AI has been instrumental in my investment due diligence process, and helped form conviction in my best performing trades in the past year. Personal stuff - AI can apparently plan vacations and make dinner reservations for you and such, although I haven’t used AI for either of these yet. - I’ve used AI to find video game recommendations, by providing it a list of games I’ve previously played and enjoyed. This hasn’t resulted in a 100% success rate, but when it doesn’t work out, it’s usually for a reason the AI already predicted (e.g. pacing, dated design/mechanics, etc). AI recommendations have a better success rate than recommendations from friends, although the absolute best recommendations all came from friends. ### AI makes skeptical reading more important URL: https://www.petercai.com/ai-makes-skeptical-reading-more-important/ Last updated: 2026-08-10T12:00:00.000Z Good writing used to credibly signal things like intelligence, expertise, and effort. Now with AI everyone can sort of fake this. All writing is just way way more persuasive, on average, than it used to be. This is mostly good, but has led to unintended side effects: - Governments and organizations unintentionally (but sometimes not) made "decent writing" a speed bump or rate limiter for specific types of actions/resources - so that that people would only invest in it if it was worth the high cost of producing the writing (e.g., suing your neighbor, or your boss). Now you can sue anyone quickly and effortlessly? - Society explicitly picked "can write well" as a filter: Who gets to be a journalist, a novelist with a book advance, or a high achiever on the written SAT? What quality of homework sorts students into As vs Bs vs Cs? Who gets into startup accelerators like Y Combinator? I sometimes get asked for advice from like, college graduates who despair of their employment prospects. I empathize with them -- AI has injected a lot more uncertainty into white collar knowledge work. But I rarely have any good advice on hand to give. After thought, one concept I think will be useful to college grads - and that I have been working on lately - is learning to discern genuine expertise: In fields where I am not an expert, *how do I evaluate writing to determine whether someone has actual expertise?* It's much easier to fake expertise now, and it's bad for everyone if more people make decisions based on wrong but persuasive writing. A couple reflections, and a running list of some ideas (from talking to LLMs lol): ## Positive signals ### Acknowledging tradeoffs core to the problem Non-experts often optimize for a single metric. Real systems always balance competing objectives, and true experts know what those objectives are because they've had to manage them. "This approach adds latency, but is usually worth it when you have problem X and Y, and don't care about latency as much because of Z." ### Reasoning from the causal chain, instead of just conclusions E.g., "Microservices increase your velocity because they let teams independently evolve and deploy their codebases, at the expense of introducing distributed systems complexity." Details should connect back to first principles, and ideas follow from them. Non-experts can spout a collection of isolated facts but can't support why they have to be true. ### Speaking in terms of constraints and boundaries Weak writing is more prone to make sweeping dogmatic claims because it doesn't have a strong working mental model of its claims ("monolithic deployments are better, and microservices are just over-engineering"). Strong writing makes the constraints obvious: - "This holds for teams under roughly 50 engineers" - "This doesn't work for networked client applications because you start with a distributed system anyway" - "Below this scale, the effect disappears and none of this matters because everything fits in memory" - "You can't compress it beyond that point because then you lose enough precision that the algorithm fails to converge" ### Do they understand the competing viewpoints? As a corollary to the above, true experts know the limits of their own position and why smart people might disagree. They can acknowledge that different situations or facts can lead to differing conclusions. "The disagreement comes from different assumptions about the importance of X" ### They can make predictions that are surprising or unintuitive True expertise is not just explanatory, it leads to better predictions. Experts can make falsifiable predictions using their better mental model e.g., "If this diagnosis is correct, then we should expect A, B, and C to happen in the next few months." "Given these assumptions, the bottleneck should migrate from memory to networking over the next year." ### They explain reality, rather than defending an identity Some writing feels optimized to signal intelligence, engagement, or tribal affiliation. True expertise - free of tribalism or political bias - is more humble, because it bends to new facts and evidence. Ask yourself: "If new evidence appeared to contradict the author, would they change their mind?" ## Negative signals ### Explanations where every piece of available evidence points in the same direction Real problems are usually multi-dimensional sets of tradeoffs and constraints and uncertainty -- which is why there's a diversity of approaches in the first place; neat sounding soundbites (with a single hero and villain) are almost always a vast oversimplification of the problem space. ### Confident prescriptions without assumptions There are very few ideas that always work without some domain-specific prerequisites being present. ### Undeclared conflicts of interest The author has a clear personal stake in a preferred outcome, but doesn't address it for fear of undermining their own argument. True experts can explain why their argument stands regardless. ### Unfalsifiable declarations Often a lack of expertise combined with motivated reasoning leads to a pre-ordained conclusion supported by "one favorite framework" that has all the explanatory power, but the reasoning is mushy enough that it can never be proven wrong. # The one question to ask: Could someone have really produced this writing without having interacted extensively, in detail, with reality? ### Starting at Sigma Squared, leaving Privy URL: https://www.petercai.com/starting-at-sigma-squared/ Last updated: 2024-10-20T12:00:00.000Z Some news: I'm starting at [Sigma Squared](https://www.sigmasquared.io/?ref=petercai.com) as VP of Engineering. I'm especially excited to help the company enter a new vertical in law enforcement. Law enforcement agencies everywhere have been inundated with nonsense statistics that confuse disparity with bias. We're going to fix that. The 7+ years I spent at Privy were pretty intense, and I definitely underestimated how much I needed this long break (achievement unlocked: leaving without a Tom Brady-esque farewell). Jackie and I have been taking time to care for family (we got hit hard by covid), to rest, go to allll the weddings, and to reflect deeply. With things stabilizing, it feels like a natural transition point to join a mission-driven company I can give 100% to. The team is [hiring in engineering, data science, product, sales, and success](https://www.sigmasquared.io/careers?ref=petercai.com) in Cambridge - but know that this won't be a leisurely walk in the park. Let's talk / reconnect if you're ready to make this the hardest, most impactful work of your career. ![Sigma Squared platform for law enforcement](https://www.petercai.com/assets/images/s2-le.webp) The Sigma Squared platform for law enforcement, one of multiple verticals served. ### My AWS Architect Associate exam cheat sheet URL: https://www.petercai.com/aws-architect-associate-cheat-sheet/ Last updated: 2024-09-13T12:00:00.000Z I took the AWS Architect Associate exam recently to renew my certification. The exam requires a deep understanding of key architecture and design principles and core AWS services, but there's a long tail of services that you need to have a passing familiarity with. Below is the complete list of notes I took while studying. ## New services I didn't recognize since I last took the exam: - **AWS Backup**: centralized policy-based account-wide and multi-account org aware backup service and policy governance. - **Control Tower**: multi account governance and blueprints with policy management and violation detection. Set up and manage multi-account environments. - **AWS Service Catalog**: centrally manage deployed IT services with curated infrastructure-as-code templates for AWS products. self service provisioning, access controls - **AWS Security Hub**: consolidated view of security status, alerting, and automated security checks - **Amazon GuardDuty**: continuous threat detection - **Amazon Inspector**: automated vulnerability management - scan and identify vulnerabilities and unintended network exposure - **Amazon Detective**: conduct faster and more effective investigations - **Amazon X-Ray**: Analyze and debug production and distributed applications - **AWS Config**: Assess, audit, and evaluate configurations of your resources - **AWS Shield**: DDoS protection. Shield Advanced $3000/mo for 24/7 - **Amazon Macie**: discover and protect sensitive data. Works with S3. - **AWS Glue**: serverless data integration service for multiple data sources for analytics, ML. visual ETL interface - **AWS License Manager**: manage licenses - **AWS Athena**: query things with SQL or python, serverless. Works with s3, and various formats like csv/json/apache/orc. - **Amazon Fraud Detector**: AWS AI service that is built to detect fraud in your data. - **AWS Artifact**: single source to get compliance-related information - **AWS AppSync**: simple GraphQL interface for development - **AWS Global Accelerator**: a service that improves the availability and performance of your applications with local or global users. - **AWS Compute Optimizer**: automate the collection of metrics for underutilized and underperforming compute instances. Generates recommendations to save money. - **Elastic Map Reduce**: mapreduce: Spark, Hive, Presto, and other big data workloads. - **AWS Batch**: batch computation workloads on managed compute infrastructure. - **Amazon Forecast**: time series forecasting service that uses machine learning and provides business insights. - **Neptune**: fully managed graph database - **Amazon Timestream**: Fast, scalable, serverless time series database - **Amazon Keyspaces**: Cassandra compatible database - **Amazon Managed Grafana**: easily scale, operate, deploy grafana. Query/correlate/visuzlie metrics, logs, and traces. - **Amazon Managed Service for Prometheus**: Serverless, managed monitoring for your containerized systems for EKS or self-managed cluster - **Amazon Elastic Kubernetes Service (EKS)**: managed Kubernetes service - **DynamoDB Accelerator (DAX)**: fully managed in-memory cache for dynamodb. - **DynamoDB Standard-IA**: infrequently-accessed data for 60% cost reduction - **DynamoDB global tables**: fully managed multi region multi-active single-digit-ms-latency eventually-consistent database - **Amazon Managed Streaming for Apache Kafka (MSK)**: hosted kafka - **AWS Data Pipeline**: managed ETL service, define data driven workflows. Works well with EC2/EMR - **Systems Manager**: run commands, remotely connect, and install/schedule patches on fleets of EC2\. install agents on ec2 or on-prem. - **Amplify**: quickly deploy web or mobile apps, supports react/vue.js. ## Security: - **AWS Network Firewall**: The AWS Network Firewall infrastructure is managed by AWS. Can filter traffic BEFORE it reaches igw. Can include IPS. Hardware firewall requirements can be met with this. - **AWS Firewall Manager**: Manage firewalls for multiple accounts - **AWS WAF**: web application firewall that protects web applications. configure rules that allow, block, or monitor (count) web requests based on conditions, works with cloudfront, ALB, API GW, AppSync. - **AWS Audit Manager**: continuous auditing compliance for HIPAA/GDPR/PCI. - **Security Hub**: place for all your security stuff and alerts ## Governance: - **Trusted advisor**: auditing for best practices - cost, performance, security, fault tolerance, and whether service limits will be hit soon. - **AWS Budgets**: cost/usage/RI/savings plan budgeting and alerts. 2 free every month - **AWS Resource Access Manager (RAM)**: share aws resources - **IAM**: roles are given to services. Users are assigned to user groups. Policies are attached to users or groups. ## ML/AI stuff: - **Amazon Kendra**: create an intelligent search service powered by machine learning. - **Amazon Textract**: uses machine learning to automatically extract text, handwriting, and data from scanned documents - **Amazon Polly**: text to speech - **Amazon Comprehend**: NLP - **Amazon Rekognition**: analyze images and videos ## EC2 stuff: - **EBS Multi-Attach**: attach a provisioned-io volume (io1, io2) to multiple instances - **Amazon Data Lifecycle Manager**: automated, policy-based lifecycle management for EBS snapshots and AMIs. - **io2 Block Express volumes**: next gen highest performance and sub-ms io latency block storage at no additional cost. Only available with C6in, C7g, M6in, M6idn, M7g, R5b, R6in, R6idn, R7g, Trn1, X2idn, and X2iedn instances. ## Cloudfront / S3: - **Cloudfront Origin access identity**: used to identify within cloudfront and S3 - **Cloudfront signed urls**: distribute paid content with dynamic URLs with access control like expiry time - **Cloudfront signed cookies**: restrict access to lots of private files or when you want static urls ## Storage/hybrid/migration stuff: - **Storage gateway**: hybrid cloud storage service. Is a VM. - **S3 File Gateway**: NFS/SMB to S3 - Cached file gateway for unlimited storage. - **FSx File Gateway**: SMB to S3 with NTFS support (shadow copies, ACLs, etc). Migrate NAS - **Tape gateway**: iSCSI-based VTL tape backups to S3 - **Volume Gateway**: iSCSI block storage volumes to EBS - **DataSync**: migrate data between NFS/SMB shares and AWS storage. One-time migration after installing agent. To EFS or FSx. - **Migration Hub**: tool for organizing migration steps - **Snow**: - **Snowball**: many terabytes of data to move. - **AWS Database Migration service (DMS)**: On prem to cloud or RDS-to-RDS - **Application Discovery Service**: discover applications to migrate. - **Application Migration Service (AWS MGN)**: automate and replicate source service into AWS for non-disruptive cutover - **AWS Server Migration Service (AWS SMS)**: replicate servers into AMIs. Minimize downtime. ## VPC stuff: - **NAT gateway**: 5-45gbps, single AZ, redundant, automatic public ip address. For multi-AZ availability, need a NAT gateway per AZ. - **NACL**: stateless first line of defense subnet firewall, can be used to deny traffic. Default allows all. Use to ban IPs. Custom NACLs deny all by default. - Every subnet must be associated with exactly 1 NACL - **VPC endpoint**: virtual device that allows communication with internal AWS services - **Interface endpoint**: elastic network interface with a private ip address, serves as an entry point for traffic headed to a supported AWS service - **Gateway endpoint**: a virtual device that supports connection to S3 and dynamodb - **PrivateLink**: peering for VPCs to many customer VPCs. Service VPC has a NLB and client VPC connects via an ENI. Between regions OK, transitive peering not OK. - **AWS VPN Cloud Hub**: connect VPN sites together, similar to VPC peering. All traffic encrypted. - **Direct Connect**: connect on prem DC with VPC. Dedicated network connection to AWS. Unlike a VPN it doesn't run through the public internet. - **Dedicated connection**: physical ethernet connection for a single customer - **Hosted connection**: physical ethernet connection provisioned by a Direct Connect partner like Verizon. - **Transit Gateway**: connects VPCs and on-prem networks through a central hub. "Cloud router." Simplifies network topology. - Allows transitive peering - Works on regional basis - Works with direct connect and VPNs - Supports IP multicast - **AWS Wavelength**: mobile 5G edge computing embedded in 5G networks. ## Monitoring: - Standard cloudwatch metric interval is 5 minutes. Detailed monitoring delivers every minute - Cloudwatch logs insights: SQL ### Leave a paper trail URL: https://www.petercai.com/leave-a-paper-trail/ Last updated: 2024-09-03T12:00:00.000Z One thing I always value at work is "writing culture." A couple reasons it has been extraordinarily valuable to me: 1. **Documentation as a historical reference.** The most straightforward use of documentation is to create a historical record: Writing down decisions and the "why" behind them helps you and people who will come after you understand the context of a decision. Countless times this has helped us avoid repeating mistakes or making suboptimal decisions - it's very easy to glance at some document and say "oh, we already thought of solution B2, and didn't pursue it because of X, Y, and Z." 2. **Documentation as a shaping tool.** Writing down an idea forces the writer to be more precise and clear - it's much more obvious if the ideas are not clearly organized and coherent. I find the actual process of writing and editing to result in better decisions, as it forces me to reason through things more thoroughly. 3. **Documentation as a reflection tool.** Most decisions are often made with incomplete information, and it's easy to forget the context in which they were made. Writing down what we decided and why helps us reflect on decisions later and review the process and outcome of the decision. What did we know at the time? What did we fail to anticipate? What factors did we overlook or overemphasize in hindsight? This is very hard to do without a written record - in my experience, our memories are unreliable and very quickly we begin to conflate what we knew at the time and what we learned after the fact. 4. **Documentation as a form of communication.** As teams grow beyond what can comfortably fit in a room, it gets harder to communicate important ideas without losing fidelity. Writing down ideas and decisions "freezes" them so they don't unintentially mutate as the ideas are passed around in a game of "telephone." It's often said that the bigger the audience, the more important it is that your idea is simple and straightforward. But this often isn't enough - operationally, different people need different nuance! Writing lets you add nuance with less risk that your core idea will be lost in noise. 5. **Documentation as a culture waypoint.** Since culture is what you do, you can nudge culture by simply writing down what you do. Checklists and procedures for common tasks that don't change often provide a lot of value as they're write-once and read many times. And documenting decisions driven by your values and principles can help reinforce those values and principles. Anyways. This is why I always advocate for leaving a paper trail. If you say it twice, write it down. ### The most productive thing URL: https://www.petercai.com/most-productive-thing/ Last updated: 2024-07-25T12:00:00.000Z A common conversation topic I'm in is "how to staff up an engineering team in the early stages of growth." How many hires, and what skills? There are easy truisms like: - It's better for a startup to have more work than too many people, and - Hiring generalists over specialists maximizes utilization and flexibility. Here's an example that demonstrates a less intuitive concept: let's say you get budget to make a new hire, and you run two teams - **Team A:** 8 highly productive, overworked engineers **Team B:** 5 less productive, but extremely overworked engineers Many make the mistake of hiring for team B. After all: - They're more overworked - There are fewer people (something about subconscious biases towards balance?) Of course, you should strongly prefer hiring for team A, since that's almost certainly a better return on investment. *Maximizing productivity is not the same thing as relieving shortages*. Usually it means [doing more of what you're already doing really well](https://www.petercai.com/everything-i-learned-about-startups-i-learned-from-video-games/) \- even if it means neglecting weaknesses. Sometimes leaders fall into the trap of filling niche roles too early, because of perceived gaps in an org -- even when those gaps are really not important yet, limiting how productive any specialist can be. When does it make sense to hire for team B again? Only when it's the most productive thing. - Maybe a bottleneck forms and a hire will unlock productivity for a downstream/upstream team. - Maybe the leader debugs the people/process and increases team efficiency. - Maybe the whole team is redirected towards a goal and a mission that has a higher return. Doing the most productive thing, repeatedly, naturally leads to specialization, which seems to work well at almost any layer of abstraction -- individuals, firms, even entire countries. It'll probably work for your engineering team too. ### Automating my personal inbox policy URL: https://www.petercai.com/automating-my-personal-inbox-policy/ Last updated: 2024-06-13T12:00:00.000Z I have a strict personal policy of never reading legalese updates in my email inbox\[1\], since they don't provide information I intend to use or act on. Even an ironclad "privacy policy" or "terms of service" is useless if there's ever a security breach; hackers don't have a privacy policy I can opt out of. So I have no choice but to assume that my data might get compromised or misused at any time at every service provider I interact with anyway \[2\]. Naturally, I figured it'd be a fun side project to automate this policy using a large language model. So I spent two weeks hacking together a Ruby on Rails app that 1) scans the subject line of every inbound email I receive via the Gmail API, 2) evaluates if it's terms-of-service or privacy related, and 3) deletes the email. I recognize this is probably very niche, but overall I'm delighted with the more focused inbox I have as a result. ![Mailman screenshot](https://www.petercai.com/assets/images/mailman-log.webp) A typical sample of emails I never have to see in my inbox again. You can try this for yourself at [mailman.petercai.com](https://mailman.petercai.com/?ref=petercai.com). It has some customization options so you can configure it to suit your own preferences. I can't promise I won't be forced to send you a privacy policy or TOS update someday, but luckily for you it will happily filter its own legalese emails. ## Getting the Google seal of approval If I don't submit my app for review by Google, then a big scary warning dialog will appear when you try to use it: ![Google verification warning](https://www.petercai.com/assets/images/unverified-app.webp) This makes sense, since Gmail's API requires I ask for very broad permissions to modify your inbox (for example, if I want to archive or delete an email, I'm forced to ask for a permission that also lets me send emails from your account, even though I don't need that and won't use it). Here's a summary timeline of my development and review process: - November 22, 2023: Initial commit - December 2: Everything is working end-to-end and deployed - December 6: Add a couple bugfixes and make a proper homepage - December 27: Polish, submit to Google for review - December 27: Google: Reply to confirm you are not doing one of these 4 idiotic things, none of which require review. - December 27: Me: I confirm I am not an idiot. - December 28: Google: Your privacy policy does not adequately describe the way your application accesses Google user data. Also, we need a demo video to understand your app's functionality. Also, we need a test account authorized to understand how you use the scopes in question. - January 6, 2024: Me: I've made the requested privacy policy changes, uploaded the requested video, and authorized the requested Google user to test the app. - January 8: Google: We don't understand your video, please provide screenshots and explanation of what the app is doing. - January 13: Me: Here is an example user flow with screenshots describing how the app works. - January 16: Google: We tested your app and it doesn't work as described. Please fix. - January 17: Me: I don't understand. Can you describe the expected vs actual behavior? - January 18: Google: \[screenshots of a mailbox that should end up in a particular state, but is in a different state\] - January 18: Me: I see what you are describing in my logs, but the logs say everything worked perfectly. Is this inbox shared with anyone else? - January 19: Google: OK let's move on! We recommend you add a privacy disclosure that clearly calls out the app complies with the Google API Services Data Policy. - January 20: Me: The disclosure has been added to the privacy statement. - January 22: Google: OK great! So we also require any app that stores data or shares it with external parties to meet the following additional consent and disclosure requirements: \[description of required disclosure content\]. Please show this in a video. - January 22: Me: OK - here is a video demonstrating how the app clearly describes the data it shares and how it acquires explicit user consent. - January 23: Google: OK we have completed initial review, now you need to complete TIER 2 CASA security assessment. They'll automatically share their results with us. - April 15: App passes CASA review after several months of back-and-forth. No application changes were required. - May 16: Me: My review was completed a few weeks ago, do you still need anything from me? - May 21: Google: Oh hey! It looks like you need to address some issues we found: 1) Your privacy policy does not specify any data protection mechanisms. 2) Based on the information you provided, we believe the following scope(s) may be a better fit for your use case: \[nonsensical scopes\] - May 22: Me: The privacy policy has been updated with the requested information. Scopes: the suggested scopes do not work: I'm applying labels to messages, not creating labels. The scopes you are suggesting make no sense for that use case (I wouldn't use them at all) and simultaneously wouldn't let me apply labels. - May 27: Google: Since you updated your privacy policy, we need you to resubmit the app even though nothing about it (including the privacy policy URL) has changed. - May 28: Me: OK I have resubmitted the app. - May 30: Google: Oh hey! It looks like you need to address some issues we found: 1) Based on the information you provided, we believe the following scope(s) may be a better fit for your use case: \[nonsensical scopes\] - June 4: Google: Thank you for your patience. We've approved your OAuth App Verification request. May 30: Me: ![](https://www.petercai.com/assets/images/as-per-my-last-email.webp) I have experience leading teams through both Apple and Shopify app store approval, but Google was decidedly the most action. \[1\] It's not because I don't care about privacy or legal rights; it's just that there is zero probability I will change my behavior in response to an automatic legal update in my inbox. Often, I use a service in spite of a privacy policy or terms of service I might otherwise find objectionable because it represents the best tradeoffs available to me, was chosen by a counterparty where I don't have a choice in the matter (employer, client, service provider, customer, etc). In every imaginable situation the decision-making has already happened way upstream, and in the rare cases where it really is my decision, I care far more about the holistic picture of who I'm entering into an agreement with, rather than the actual text of the legal document in front of me - after all, there's no way to provably verify my counterparty won't unilaterally contravene their own legalese, so without trust there's nothing to go off anyway. \[2\] Unless it has some verifiable security guarantees like end-to-end encryption, but most services don't do this because it's super inconvenient and limits a lot of use cases. ### We used to build software, but increasingly we build coding assistants URL: https://www.petercai.com/we-used-to-build-software/ Last updated: 2024-04-04T12:00:00.000Z I think I've changed my mind on this: the software engineering profession as we know it is under threat now, because of AI. A few months ago I had [not yet concluded this](https://www.petercai.com/yes-you-should-be-impressed-by-recent-ai-advancements/). I argued the cost of certain types of software development would surely drop, but there was so much latent demand that it wasn't clear to me that this would necessarily devastate employment and salaries. But I think the balance of evidence is that it's now a *distinct possibility* (but not a certainty) that we are going to eventually eat away at software employment and compensation with AI - in a noticeable way - soon. A couple thoughts: Many line-of-business problems that used to require a software engineer to be involved at some point somewhere are going to fall to no-code / low code platforms soon. In other words, the Excels and Google Sheets of the world are about to become even more important. I've seen a lot of evidence that enterprising folks (marketing, sales, product, misc generalists, etc) can iteratively prompt and tweak their way to working solutions with AI, even if the initial version has bugs. This will lead to an explosion of customized software purpose-built for every use case (this is the sort of thing that accretes at larger businesses, but now it will happen for every business). People will whine that this creates unmaintainable messes of spaghetti code, a fate worse than death. In all likelihood, the world will keep turning, even if (much fewer) professionals have to come in and clean up the occasional mess. Elsewhere, the low end of frontend web development is rapidly being mechanized. We used to do a lot of hand-coding of HTML, but HTML generation is almost uniquely suited for large language models. Old timers will remember we used to laugh at graphical builders like Dreamweaver because of how bad the output was\[1\], but HTML as a language is \*much\* more error-tolerant than traditional Turing-complete programming languages, and thus LLMs can make mistakes and still come up with working solutions. This makes it the perfect place to automate a lot of code generation, and I think this is bad because it's a common way to break into the software engineering profession: The feedback loops are shorter and more visual. Cutting off the very bottom of the career ladder of software development is not good for industry employment in the long term, to say the least. This is already destroying the informal gig economy of web development, especially in the developing world: ![Free AI tools are killing South Africa's web designer job market](https://www.petercai.com/assets/images/ai-tools-kill-jobs-in-africa.png) Via rest of world: [Free AI tools are killing South Africa's web designer job market](https://restofworld.org/2023/ai-tools-web-developer-jobs-south-africa/?ref=petercai.com) In the short term, current full time professionals won't notice because there isn't a direct and visible connection to job loss. I think this recalls the slowly boiled frog analogy: the current generation of coding assistants aren't anywhere near good enough to replace a human outright. It's just that a bunch of needs that used to build up over time and eventually result in an engineer being hired somewhere will now be delayed. Zoom out to the entire economy and this becomes a very big deal, even if no one ever hears the words "we're laying you off and replacing you with an AI that writes software." The demand curve very gently bends downward as productivity continuously increases, just as the frog is gradually boiled. ![How to program anything in 2024: 1. Write a very good documentation. 2. Let Opus generate the code from it. 3. Verify, polish and test. 4. Done.](https://www.petercai.com/assets/images/claude-opus-code-generation.png) Via Twitter: Some engineers now have software development workflows like this (Opus refers to Claude 3 Opus, the most capable model from Anthropic, an artificial intelligence company, at the moment). By the way, coding assistants used by professional engineers will get better as feedback from the first generation (essentially prototypes built in vitro) becomes training data for the next generation. This is likely a genuine and durable first mover advantage: There's a positive feedback loop where increased coding assistant adoption directly results in a better product. I think there are also coming innovations in interaction models that will make assistants way more powerful than just "autocomplete, but for code." Github's Copilot Chat and Cursor (made by a company called Anysphere) are giving us a glimpse of that future: they can analyze code and suggest refactors, explain subtle errors and workarounds, read and summarize modules, and even directly make changes or add features in response to inline-prompts: Video from: https://cursor.sh The underlying base models of coding assistants are improving as they get more training and improved architectures - the actual degree of stepwise improvement in a specific task is hard to predict from generation to generation, but we have enough data now that the long term trend is undeniable: more model parameters and more data leads to better capabilities. This is even before we factor in hardware improvements that give us free performance - more compute and more memory for both better training and inference. The implications for total software employment are not particularly encouraging. My argument is not that software engineering as we know it will become obsolete. Instead, we have introduced an additional layer of abstraction (code generation capability), making the previous layer (writing code in a high level language) much more productive (to specialists) and accessible (to generalists). The result is demand is shifting away from the straightforward work that used to be entry-level software development, and towards AI/ML engineers and GPUs, for a net loss. You don't need large changes for large effects: In 2023, its likely at most 6% of tech workers in the United States were laid off \[2\]. **This was widely regarded as a bloodbath!** If we're conservative and say AI makes software engineers 5% more productive every year, that could imply a bloodbath **every year** for the foreseeable future, unless demand picks up. I am a *little* worried. It's possible that the signs were there all along, and that I didn't realize it sooner because of wishful thinking. No more. \[1\] There were a lot of interesting reasons for this, but one of them was that simple mechanical transformations and edits, repeatedly applied to a nontrivial HTML document, create a bizarre nonsensical mess of mostly-useless junk markup that no human would ever write, but requires a higher level understanding of the structure of the document to consolidate and fix without introducing errors. The technology just didn't exist back then. \[2\] There were 4.4 million workers with jobs classified as "Computer and Information Technology Occupations" in the United States in 2022, according to the BLS. In 2023, layoff trackers estimated technology companies laid off about 263,000 employees. If we assume everyone counted was in the United States (they weren't), then that works out to just under 6%. This number is imperfect: BLS includes nearly 1 million "computer support specialists" in this 4.4 million number, and those aren't software engineering jobs, but also, the layoff trackers include non-software jobs, so we're overcounting on both sides and an unknown quantity of error is canceled out. ### Medicare Advantage is a scam URL: https://www.petercai.com/medicare-advantage-is-a-scam/ Last updated: 2024-03-08T12:00:00.000Z After years of frustrating insurance churn, I was relieved when an aging family member reached the qualifying age for Medicare. I was looking forward to enrolling them into the much-vaunted and supposedly straightforward program. I ended up spending a shocking number of hours reading government publications, third party analysis, private insurer advertisements, and then repeatedly explaining why they shouldn't opt for the scam that is Medicare Advantage. Here's what I learned: Medicare is government health insurance: You pay premiums to the government, and when you see a provider, the US government pays the provider. Patient <> US Government <> Providers Medicare is relatively affordable and covers many services for only a 20% coinsurance payment. Part of the reason it is cheap is because Medicare has much lower overhead: its billing and reimbursement model is public and straightforward, most premiums are spent paying providers for services rendered, collecting premiums is cheap because it can be deducted out of social security checks, and there are no profits to send to shareholders. Also, Medicare sets relatively lower reimbursement rates for providers, which lowers hospital, doctor, and drug profit margins. Many providers grumble about this, but do business anyway because Medicare brings a lot of volume (they can always fraudulently submit bills to receive what they think is "fair," and Medicare is "low overhead" enough that only the most egregious cases of this are ever detected and prosecuted). The result is a popular, efficient, low-cost health insurance program for seniors with a huge, hassle-free network of providers. To be brief: Republicans (specifically) hate this, because it's an example where the government clearly does something better than the private sector with no downsides (well, one "downside" is private actors can't squeeze as much profit out of healthcare). It's obviously so much better that lots of people ask a very good question: why limit Medicare to people over the age of 65? Why not give everyone the option to get Medicare? \[1\] This is a scary question, because if there was a clearly superior government health insurance option with no downsides, everyone would flock to it and it would annihilate the private healthcare industry essentially overnight. This would be moderately beneficial for almost everyone (healthcare would be cheaper, although still expensive), but extremely bad for a small subset: entire ecosystems of industries would drastically shrink overnight, like "medical billing administration", "formulary consulting", and "pharmacy benefit management." Anyways, Republicans decided they had to go on the offensive - not just to prevent expansion of government health insurance, but to actually roll it back (but through trickery, because patients overwhelmingly love Medicare). So during the George W. Bush administration in 2003, they enacted a law to *deliberately sabotage Medicare*, completing a project that had been started in 1997 with support from Clinton. The new law created Medicare Advantage, a scheme in which patients enroll in private insurance, funded by their Medicare dollars: Patient <> US Government <> Private Insurer <> Providers Why is this bad?: - Fake Medicare Advantage is a capitated program, so private insurers get paid the same fixed amount for each enrollee, regardless of how much healthcare they use. Predictably, private insurers want to make a profit, so they create more restrictive networks, limit the kinds of care they will approve, and generally behave like you expect private insurers to behave. - Since there are more private actors involved, you predictably get more fraud. One type of fraud is upcoding: While Medicare Advantage pays the insurer a fixed amount per patient, it pays more money for sicker patients, so if you are cynical you might expect an insurance company to spuriously claim their enrollees are really sick and have lots of chronic conditions so that they get more money from the government, even though these conditions are mostly made up and don't necessarily have associated costs. You would be right! This is exactly what happens! You get what you incentivize, etc. - The law throws up roadblocks to switching back to traditional Medicare once patients realize traditional Medicare is better - usually when authorization for a required treatment is denied, but which would've been covered under traditional Medicare. But not too hard, since fake Medicare Advantage insurers are more than happy to dump unprofitable patients back on the traditional government program. The result is delayed and denied care, and a system that sorts the unexpectedly sicker patients back to traditional Medicare, while the private insurers keep the profits from healthier patients (but coded as sicker than they actually are, to scam more from the government). Given these facts, why would anyone want to enroll in Fake Medicare Advantage instead of traditional Medicare? Wouldn't the free market solve this problem, since seniors would obviously just pick the best option for them? The insurers and legislators came up with a genius plan: Medicare Advantage providers are allowed to bribe prospects with extra goodies that are not covered under traditional Medicare. After all, as a big insurer, it is very profitable to cash a big guaranteed check from the government every month (you already have lots of experience fighting patients and providers to spend as little as possible). If you will accept slightly lower profit margins by providing fringe benefits\[2\], you can sign a lot of people up even though the product is worse; it's just a marketing expense. And so that's what the insurers do. My family has been absolutely bombarded with frantic-sounding mailers from private insurance companies using the "Medicare" name to sell fake Medicare Advantage. There are literal sales agents embedded in communities where seniors go, carrying thick glossy brochures touting the vision benefits and other goodies that enrollees get with their plan. That's not all. The 2003 law creating fake Medicare Advantage was maliciously and deliberately designed to be confusing, to make fake Medicare Advantage look more appealing. The name itself is deceptive: "Medicare Advantage" isn't a government program and isn't Medicare; it's the equivalent of a private school tuition voucher, where public funds go into a completely private system. But the name makes it sound like it's the "plus" version of the popular government program. Fake Medicare Advantage coverage is also conveniently labeled so that enrolling in traditional Medicare will appear to be leaving a coverage gap. To explain, Medicare enrollment is traditionally made up of several "parts": | | Traditional Medicare calls this... | fake Medicare Advantage calls this... | | -------------------------- | ---------------------------------- | ------------------------------------- | | Hospital Insurance | Part A | Part C (includes A) | | Medical Insurance | Part B | Part C (includes B) | | Prescription drug coverage | Part D (optional) | Part C (usually includes D) | Here is where the confusion comes in. Someone who opts for traditional Medicare would buy parts A, B, and might decide to get D, but they would skip "C" because that's the totally different (mutually exclusive) fake Medicare Advantage. Someone who elects to get (fake) Medicare Advantage instead "buys part C" which will be marketed as including the benefits of A, B, and D. And who wouldn't want to get all the Medicare parts? ABCD sounds more complete than the alternative, ABD? Notice the sleight of hand: The Medicare "Parts" are not disjoint sets. In fairness, this confusing naming is a result of repeated legislative changes to Medicare, but it's awfully convenient that the resulting confusion makes fake Medicare Advantage look more appealing. The result of more than two decades of this swindle is fairly predictable: about half of Medicare enrollees are actually on fake Medicare Advantage, and rising. It costs taxpayers more money and provides worse service to patients. Fake Medicare advantage is now the most profitable line of business for insurers, savings that rightfully belong to the public. After all, the argument for fake Medicare Advantage is roughly "private industry driven by profit motives will find ways to deliver better care for less money, saving taxpayers money through the magic of capitalism." This is actually a totally plausible hypothesis. We should want this! It would actually be great for Americans if conservatives distinguished themselves from the opposing party on policy grounds; using markets to be good stewards of public funds should be their brand. But good faith experimentation with market-oriented solutions means we should stop or tweak the experiment if its going wrong. Yet we continue to double down on this insanity. As patients, we should avoid this cesspool: Don't enroll your family members in fake Medicare Advantage. If they're already enrolled, switch them back to traditional Medicare before they get sick. As taxpayers, we should demand better from officials: fake Medicare Advantage shouldn't be allowed to use the name "Medicare." Enrollment options shouldn't be deceptively named and marketed. Insurers should be penalized when enrollees switch back to traditional Medicare (at a minimum, they should pay back any actuarial profits earned). Upcoding should be aggressively investigated and penalized, something the Biden administration is already working on. The entire "Medicare Advantage" program should be sunset if it doesn't show it can save money or provide better care. Right now it does neither \[3\]. \[1\] Medicare-for-all is the name of this policy platform, and it's extremely popular with the public, but extremely unpopular with elected officials of both parties, because it's hard for a governing majority to get elected on a platform of annihilating a powerful, politically active business interest, and "slippery slope" arguments really do carry weight here. \[2\] One example fringe benefit is a medical alert system, which is basically a panic button you can press if you've fallen and need someone to send help. This usually costs a monthly fee of $30 to $50 a month, and isn't covered by traditional Medicare. Another is that Medicare Advantage plans often bundle prescription drug coverage at no additional cost. \[3\] I want to be clear: This is not to say that nobody ever benefits from being on fake Medicare Advantage. The typical person who benefits is a healthy individual who doesn't mind the more restrictive provider network, and takes advantage of the extra benefits. But this privatizes the benefit of lower Medicare obligations from healthy retirees, while socializing the costs of sicker retirees. ### Unintuitively hard things to automate URL: https://www.petercai.com/unintuitively-hard-things-to-automate/ Last updated: 2024-02-28T12:00:00.000Z Here are a few problem domains that I thought would've been solved before the advent of "generative AI that can turn text prompts into videos," but it turns out I am wrong and they are deceptively hard: ## Automated long haul freight trucks ![Via DALLE 2: a freight truck on a snowy highway, in the style of a watercolor painting](https://www.petercai.com/assets/images/dalle-freight-truck.webp) Via DALLE 2: "a freight truck on a snowy highway, in the style of a watercolor painting" As a layperson, I assumed that self-driving freight trucks on highways would be a light version of full "level-5 autonomy passenger vehicles in cities." The logic seems simple: it's mostly following well-marked lanes, and there are no intersections. But it turns out that highway driving is actually MUCH harder than autonomous city driving (which - to be clear - is already an extremely difficult problem). Engineers seem to inevitably discover that when they create startups trying to disrupt long-haul trucking: - The available sensor technologies don't reliably have enough range under all conditions. This can lead to unacceptable safety problems: at normal highway speeds it can take a long time to bring a truck to an emergency stop, and if your braking distance is further than your sensors can detect obstacles, it will mean by the time you've detected an obstruction it might be too late to avoid hitting it. - A truck is a complex object - the cab and the trailer are joined but have independent movement along an axis, so it cannot be physically modeled as a single rigid body like a simple passenger vehicle. This complexifies planning and operations. - In city driving, novel situations like unplanned obstructions or road closures can usually be handled with a "pull over safety stop" and rerouting to a different path. In highway driving, not only is pulling over usually not an option, "rerouting" around an unexpected obstacle usually isn't an option either. Consider the case of encountering highway construction and police officers waving at an alternate route - the vehicle doesn't have the option of saying "I don't know what's going on, so I'm going to avoid that street" - it's on a highway and the only way off is through it! - The data on unique events (like construction detours) is extremely sparse compared to city driving, and its therefore extremely expensive to collect enough of it. ## Automated food assembly Every few years I see a demo for some robotic machine for making burgers, pizza, pasta, or sandwiches, and I end up...underwhelmed? Unfortunately the most common pattern is a test deployment in some novel concept restaurant somewhere, and then eventually the whole operation folds up shop. Having prepared a lot of food myself, I think I was surprised by how difficult it is to automate at scale? Let's consider: - Assembling a burger actually requires very precise movements and manipulation of up to a dozen different types of objects (of heterogenous shapes) and with complex behavior (burger patties will deform or break up if picked up but not properly supported, tomatoes are both hard-to-grip and fragile, etc). - So manipulating these ingredients requires not just advanced robotics, but advanced computer vision as well. In practice, practical applications of "robot burger maker" get around this problem by mostly squirting prepared ingredients out of chutes or tubes, and moving things around on belts instead of more complicated lifting motions. - So then you still have the problem of "how to fill the chopped lettuce chute with chopped lettuce," and this is trivial to solve with automation if you have a 2 million sqft lettuce processing factory, but in a more constrained environment, you apparently need a human to come by every once in awhile. - "Scoop the peanut butter out of the jar and spread it on the bread" - an open research problem, it seems. - Hilariously, we learned you can't put things like robot pizza makers on pizza delivery trucks, because the actual forces of the truck in motion will destroy an uncooked pizza in an oven (but not a cooked one in a pizza box). ## Automated brick laying ![Via DALLE 2: a complicated brick laying robotic contraption at a construction site, watercolor painting](https://www.petercai.com/assets/images/dalle-bricklayer.webp) Via DALLE 2: "a complicated brick laying robotic contraption at a construction site, watercolor painting" Brick laying seems like it'd be straightforward-ish, because it's just laying a set pattern over and over again. But in practice: - Mortar and brick requires complicated handling because mortar is a fluid, and it requires a probably-surprising amount of precision to apply the right amount in the right place. Also, since a machine would pump it instead of scooping, the mortar consistency would have to be in a very specific, very narrow range, despite unpredictable ambient temperatures. - Any robot would have to move along the entire wall to be built, and this is hard because - by definition - things are under construction; there probably isn't a perfectly level surface to move along, so it has to be able to traverse uneven ground while still placing bricks accurately. Not impossible, but not trivial. - It's rare for there to be large sections of uninterrupted brick wall without a corner, window, door, etc to build around - those parts mostly have to be laid by hand since they break the usual simple-overlapping-brick pattern. - Fixing mortar joints to make them presentable is nominally part of brick-laying, but is a totally different problem as far as physics is concerned. One of the few commercially deployed systems requires a mason to follow it around and clean the masonry joints / level the occasionally misplaced brick, as well as a field technician to build the "wall map" and troubleshoot the machine. Laying bricks is just deceptively complicated \[1\]. ## ...and automating building construction in general Unfortunately one component of high housing prices is high construction costs, and productivity in the construction industry has stagnated for decades (and is maybe even moving backwards in some advanced countries). But making progress with automation seems to be pretty hard because: - We've managed to automate much of car manufacturing with very precise stationary robots that can repeat the same complicated motion over and over again - its the cars that move on an assembly line. In contrast, construction sites are very large because buildings are very large, so the robots need to be mobile (or on huge cranes?). And making complicated robots that are *mobile* and *also* move under their own power *and* can fit through doorways is...very challenging, apparently. - Most buildings are bespoke, with an architectural layout specific to the building's requirements, and many individual components are tailor-made and fitted on-site. Having enough sensors to detect whether something was placed/secured/inserted correctly, and adjusting as needed, is a challenging problem, and there might be hundreds or thousands of these types of unique tasks in the construction of building. The result is that very little of "construction" is repeating the same identical motion over and over again (even in our brick example, there are a literally unsolved edge cases), which reduces the gain from automation while raising the cost of automating it. - It's usually not cost effective to pre-fabricate large components and then ship them in. There's a limit to how big components can be, since they still have to fit on trucks that will go on roads. And the savings can be small, because most building components are cheap but heavy, so even large savings on labor will be eaten up by the high cost of transporting finished components over even medium distances. Because of this, it's hard for pre-fab building component manufacturers to reach a volume where economies of scale start to really kick in, and yet - scale is simultaneously limited by how close their raw materials and potential customers are. On top of this, at least in the US, pre-fab buildings have a reputation for low quality, which dampens the demand that would be required to reach scale. Most buildings are still mostly constructed onsite because of this, which limits the amount of factory-quality machinery that can be used. - Architect drawings are usually not detailed enough to be converted into a plan a machine can follow, since this would take forever and the result would be unreadable. The drawings have to be interpreted by expert tradesmen familiar with the local building codes, and even the building codes themselves often have unspoken rules or reference implicit tribal knowledge that roughly ends up being "do whatever is customary in this situation, given the totality of the circumstances." This means there's actually a lot of work that goes into "automating" any building construction process, and it's hard to reuse. \[1\] I have, however, seen some pretty cool videos of automated brick-laying for brick roads, which seems easier (they lie flat against prepared ground, and no windows or doors!). ### Endurance and strength training cheat sheet URL: https://www.petercai.com/endurance-and-strength-training-cheat-sheet/ Last updated: 2023-09-17T12:00:00.000Z A summary guide of some information I've compiled because 1) I keep forgetting it, and 2) I want a place to send people who ask me about this stuff. ## Supplements that work | Supplement | What it does | Dosage | Frequency | When to take it | Notes | | ---------------------- | -------------------------------------------------------------------------------------------------------------------------- | ------------ | ---------------------- | ------------------------- | ------------------------------------------------------------------------------ | | **Whey protein** | Necessary for muscle protein synthesis. | 0.8-1g/lb | Daily | Doesn't matter | Dietary protein can reduce the supplementation required. | | **Creatine** | Increases muscle mass, strength, performance, and recovery. | \~5g | Daily | Doesn't matter | Women seem to benefit less from creatine supplementation than men. | | **Caffeine** | Increases power output, anaerobic capacity, and endurance. Improves reaction time and reduces perceived exertion rate. | 2-4mg/lb | Skip during rest days. | 60min before exercise | Tolerance builds quickly, so dosing needs to be "managed" if you're competing. | | **Beta-alanine** | Improves strength gains, muscular endurance, and performance in high-intensity exercise. Reduces neuromuscular fatigue. | 3-6g | Daily | Doesn't matter | Causes (harmless) parasthesia in many people. | | **Citrulline malate** | Improves performance through increased nitric oxide production, improves aerobic and anaerobic capacity, reduces soreness. | 8g | Skip during rest days. | 60min before exercise | Very sour, tastes like lemon juice. | | **Sodium bicarbonate** | Improves performance during repeated high-intensity aerobic exercise by negating effects of acidosis. | 100-130mg/lb | Skip during rest days. | 60-150min before exercise | Beware gastrointestinal side effects. I haven't personally used. | ### Supplements to skip / avoid - BCAAs - Basically everything not listed here\* \*Supplements that work (e.g., fish oil) won't be discussed here if they support general health (rather than performance). ## Endurance Training - Warming up is important at all experience levels and can prevent injuries. - Most training should be at a very easy effort level (below lactate threshold 1). Think 80% of workouts. - Beginners can improve aerobic fitness with brisk walks 5-6 times a week; actually running isn't necessary, and the benefits come with less risk for injury. - Time is more important than distance. 30-60min is a good baseline for easy training; shorter workouts aren't as effective. - It can take 8 weeks for endurance improvements to be apparent. - For trained individuals, high intensity interval training significantly improves VO2max, with results in as short as 2-3 weeks. - About two 30min HIIT workouts a week is ideal, targeting a total of 30-40min per week at 90% HRmax. - Because HIIT and workouts below LT1 are so effective, workouts in the "middle" (3 in the 5 zone model) should be minimized; polarized training offers the best return on effort, and minimizes injury and recovery times. - It can take about 12 weeks to habituate the body to a given load. | Common Name | Recovery | Easy | Endurance | Tempo | Threshold | VO2max | Sprint | | | | | ---------------------------- | -------- | ------------------------- | ------------------------- | ------ | --------- | ------ | ------ | ------ | -- | -- | | | | Lactate Threshold 1 (LT1) | Lactate Threshold 2 (LT2) | | | | | | | | | 3 Zone Model | Zone 1 | Zone 2 | Zone 3 | | | | | | | | | 5 Zone Model | Zone 1 | Zone 2 | Zone 3 | Zone 4 | Zone 5 | | | | | | | Rating of Perceived Exertion | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | | Time to failure | All day | 5+ hrs | 2-3 hrs | 1 hr | 30-60min | 15min | 5-8min | 1-5min | | | | Heartrate (approx) | 50% | 65% | 72% | 80% | 85% | 92% | 100% | | | | | Blood lactate (mmol/L) | 0.7 | 1.0 | 2.0 | 2.5 | 3 | 4 | 5 | 15 | 20 | | ## Strength Training - Warmup sets are usually not necessary but don't hurt. Beginners probably benefit more. - Unless you have a specific reason otherwise: 5-10 reps per set, 2-5 minutes rest. - Initial strength gains for beginners mostly come from neuromuscular adaptations and motor unit synchonization, not from muscular hypertrophy. - Most hypertrophy gains occur during the first sleep cycle immediately following a workout. Sufficient stimulus must be paired with adequate rest and nutrition - muscle protein synthesis is finicky and the process is easily interrupted. Aim for at least 7 hours of sleep. - For beginner / moderately trained individuals, 2-3 sets per workout at the correct intensity is sufficient stimulus. More volume is better for trained individuals, but not at the expense of intensity. - Progressive overload is key. Beginners should aim to add 2.5lb per arm or leg each week if possible (or session, if you're a complete beginner). Never skip a weight increment since it just means you'll lose a "stairstep" of increased stimulus and hit a plateau sooner. - Compound barbell exercises are great since they hit all the large muscle groups together. Pulleys are great because they keep constant tension throughout the exercise range of motion. ## Myths Dispelling some popular myths that seem to be wrong or unsupported. - The "interference effect" -- where doing cardio and strength training concurrently reduces gains -- is very likely overblown (or possibly nonexistent). Where this has been supposedly observed, overtraining or insufficient dietary intake are the likely causes. - Delayed onset muscle soreness (DOMS) is sometimes correlated with - but not related to - gains. You can become sore without growing, and you can grow without soreness. - Working out until you "feel the burn in your muscles" is a useless signal. What matters is correct intensity, volume, and form. - There is no good evidence that stretching improves performance or reduces soreness. The impact on injuries is mixed. - Timing and frequency of meals doesn't make any difference as long as you're 1) eating the correct macro composition + calories, and 2) you have enough energy to do your workouts. ## Sources I made everything up ### Interviewing engineers is hard because it's adversarial URL: https://www.petercai.com/interviewing-engineers-is-hard-because-its-adversarial/ Last updated: 2023-05-18T12:00:00.000Z Hiring is one of the most important functions in any company. The people who are hired end up deciding who else is hired (etc), and this basically determines what kind of company it becomes. The same is true of software engineer careers: the perceived quality of the first few jobs can dramatically affect the trajectory of a career. This makes interviewing high stakes for everyone involved. Most software engineers hate interviewing and correctly point out that almost all interviews in this industry are uniformly bad. They're a stressful time sink, require too many touchpoints over too long a timeline, are often opaque, test skills that barely relate to the actual job, with seemingly arbitrary results, and worse, [are often influenced by factors unrelated to qualifications](https://www.petercai.com/workarounds-for-hiring-discrimination-in-tech/). The terrible state of interviewing is sort of an idle obsession in our industry; there are many opinions but very little in the way of rigorous analysis or discussion. In my experience the most commonly overlooked consideration is that interviewing is necessarily *adversarial*: the two sides involved have different incentives and goals. Hiring orgs want to find and hire the best candidate for their given budget. Candidates *want an offer - the highest one possible, regardless of fit*. This is because 1) having multiple offers confers leverage, and 2) the principal-agent problem. This tension makes it hard for either side to approach the problem with a 100% collaborative mindset, which is why most naive ideas for how to make interviews "better" either fail to scale, or would never work in the first place. So here are just a few considerations that must be accounted for when designing interviews: ### Factors to consider when designing an interview **Total ordering**: You might be forgiven for thinking an ideal interview has a specific and clearly delineated "bar" that a candidate needs to clear to receive an offer. They either do well enough to get an offer, or they don't. Theoretically this maps to a boolean. In practice, this needs to be an integer, because what if multiple candidates pass and there's only one open position? You need a way to rank candidates, not just pass/fail them. - **Solution that doesn't work:** Just hire the first person to pass the interview. This is equivalent to random(), which is an obviously suboptimal strategy to adopt. And missing out on a great candidate means they go to a competitor and then compete with you. You don't want to deliberately adopt suboptimal strategies in high stakes zero-sum situations. **Cheating**: Glassdoor (etc) makes it extremely easy to cheat by "studying to the test" which causes at least two problems: selectively unintentionally drifts down over time (more candidates will pass, even though the interview as designed should've failed them), and it becomes difficult to fairly compare candidates, because probably some cheated and some didn't. It's hard to counter this; basically you have a sliding scale where you're always trying to balance "freshness of your questions to mitigate cheating" and "consistency so the questions are 1) of similar difficulty over time and 2) make relative comparisons easier". - **Solution that doesn't work:** Post the question online to "level the playing field" for everyone! The obvious problem with this approach is that sometimes (regardless of the merits) you're looking to test how someone works on a problem they haven't specifically seen before. Posting everything online ahead of time turns it into a memorization/preparation test instead. - **Solution that doesn't work:** Frequently rotate questions. It's too hard to consistently design great and novel interview questions, since it takes time and effort to calibrate the difficulty, delivery, and interviewer training to get a question "just right." And the calibration data is noisy *and* sparse: sometimes you get a bad hire or miss out on a great hire because the interview question was inappropriately leveled. Or maybe it was just luck? **Selectivity:** Every interview question should give the candidate more opportunities to demonstrate they are a good fit, and this means questions should be calibrated to the expectations of the role. But in practice it's often necessary to develop a question to weed out clearly unqualified candidates - that is to say, passing the question is necessary but not sufficient for an offer. Suppose you're hiring a senior machine learning engineer. If your goal is "minimize the risk that I proceed with someone from the bottom 30% of the candidate pool which has objectively never done any machine learning" then "explain to me in your own words how a support vector machine works" might suffice as as a screener. Arriving at the "correct" difficulty for screener and main interview questions is a difficult problem, and the candidate pool is not static: in general you'll find candidates magically get better at answering questions over time because of Glassdoor. - **Solution that doesn't work:** Don't use screeners at all. Just rely on your tougher questions - "if they can't pass a weed-out question then they shouldn't be able to pass the harder ones either." The problem with this approach is it wastes everyone's time. It's just a complete failure of process and resource management to send a candidate through a time-intensive wringer of a multiple-hour interview loop with senior level employees when the candidate is hopelessly unqualified, the outcome is completely certain, and this can be discovered in 15 minutes. "Cutting interview loops short" is theoretically doable, but requires training and expectation-setting and doesn't even fully solve the problem (harder questions require a more skilled interviewer!), so it isn't clearly better than alternatives. **Repeatability & scalability**: Interviews need to be somewhat repeatable and controlled, because that's a prerequisite for targeting a specific selectivity and getting a total ordering. Repeatability is highest when isolating variables and removing extraneous details, which is naturally in tension with conducting an interview that "mirrors a real engineer's day-to-day." The average engineer's day-to-day is driven by lots of context about their team and business and that's really hard to simulate for a "realistic" interview, and can be markedly different from one day to the next. - **Solution that doesn't work:** As part of the interview process, you can pair your candidates with a real engineer/interviewer as they go about their work for the day, but that's going to give every candidate a very different experience. It's also a big commitment for everyone involved! **Trainability**: A good heuristic to go on is "if you can't explain what a good answer is and why it's a good answer then you haven't thought rigorously enough about it, or it's a bad question." You need to be able to train people to give the interview question properly, answer clarifying questions, ask appropriate followups, and most important - grade candidates fairly. Since a good question doesn't ask for a factoid that someone can memorize and regurgitate, this can be much harder than it seems. Forcing yourself to actually put pen to paper and lay out what, exactly, is a "good/better/best" answer *without* accidentally encoding some flaw into it (like implied knowledge of some random factoid) can be surprisingly difficult. **Selection bias / pain tolerance:** Given how high stakes hiring is, it's safe to predict that companies would take much more time to interview candidates if they could have unlimited time, money, and candidate patience. But sought-after engineers have lots of options, so they get snapped up quickly. The engineers who are \*most\* willing to jump through 8-round interviews in a good economy are (statistically speaking) going to be the ones that are mediocre-to-bad. Joel Spolsky famously complained about this almost 2 decades ago (until he switched to complaining about how much Facebook paid). It's the "market for lemons" principle: this person must not be very good, because they're interviewing! This results in an equilibrium where most companies know they have, *at best* 3-4 hours of interviewing to decide whether to make an offer or not, because they know that longer processes will repel good candidates and attract only bad candidates, overwhelming any benefits of higher confidence hiring decisions. - **Solution that doesn't work:** Creating a publicly documented fast lane for preferred "hot" candidates. Besides being manifestly unfair, it would perpetuate and magnify inequalities from random luck and encode biases into policy, without much evidence it would meaningfully change outcomes. E.g., for this to be effective you would need a public policy like "candidates who once worked at Google can skip the screening interviews" but there's no reason to expect that this makes people from Google more likely to apply (a job change is a big move; they'd need to already be intrigued enough by other factors that this tips the scales for them), or that they'd be a better fit for the role. - **Solution that doesn't work:** "Just" be a very attractive place to work. Candidates have a higher pain tolerance for top employers. This is obviously very difficult - rankings are zero-sum; not everyone can be a top employer. Many industries can't or won't do this at all because pay is a very large component of what makes a workplace "attractive," and in most industries the benefits of a good team are quickly swamped by the cost of overpaying for it. ### "I would just do x" Most people who think there is a Better Way to interview software engineers have an interview process in mind that fails at least one of these considerations. Again, interviewing is *adversarial*, so you can't assume candidates won't try to game the process \[1\], or that they are representing their skills and work experience accurately. At the same time, being mindful of time constraints is necessary to minimize pain and avoid repelling good candidates with arduously long funnels. This limits freedom of movement quite a bit when it comes to designing a process that is both fair and effective. This problem always settles into an equilibrium where smaller firms have an edge: it doesn't matter that their interview process isn't very scalable because they're not operating at scale. But once companies get big they all seem to basically revert to leetcode, which is evidence that the currently-innovative practices don't scale up well. Is it possible to do better than the current meta? Will we ever move beyond "system design challenges on whiteboards" and "leetcoding exercises on Coderpad?" I'm sure we'll get there. Once something is both 1) clearly better and 2) durably works at scale, market forces essentially guarantee it'll be copied until it takes over. But right now it's not clear to me that the most vocal complainers have anything to offer in terms of (repeatable, scalable, sufficiently Glassdoor-proof) alternatives. \[1\] Candidates on zoom calls cheating with a coach next to them (or in their ear) is a thing, feel free to ask me how I know. ### Yes, you should be impressed by recent AI advancements URL: https://www.petercai.com/yes-you-should-be-impressed-by-recent-ai-advancements/ Last updated: 2023-04-13T12:00:00.000Z I'm **incredibly** optimistic about the future. We are rapidly approaching the cusp of large measurable productivity increases in multiple parallel fields that will significantly improve and enrich people's lives substantially. To name just a few industries undergoing such a transformation: art, music, film, video games, novels, software, law, medicine, and of course software. And yet, a surprising number technologists I talk to seem...openly skeptical to the point of hostility? On the one hand, a healthy dose of skepticism is warranted whenever new products make fantastical claims. Maybe you dismissed the most recent fad, "web3," which was [mostly bagholders trying to pump their magic beans](https://www.petercai.com/defi-is-still-bullshit/) while solving basically no new problems. It's easy to convince yourself that you're smart by being default-skeptical because you'll automatically bat 0.99\. But if you're IN TECH you owe it to yourself to understand what is happening lately: consider this my attempt to convince smart people that \*MAGIC\* is happening in AI, across many diverse white collar fields. ### Writing A lot of the recent AI hype was started by ChatGPT, and I've [previously touched on some weaknesses of this type of AI based on large language models](https://www.petercai.com/chatgpt-lied-to-me/). But once you understand the limitations of today's LLMs, it becomes straightforward to harness their immense and growing power while avoiding common pitfalls. Form letters - cover letters, resumes, job postings, marketing copy, press releases, emails, etc can now be routinely generated in seconds by the likes of ChatGPT. This is real! OpenAI got 100,000,000 users in 2 months (a historical record-setting pace) because this is a truly amazing product: ![ChatGPT response to resignation letter prompt](https://www.petercai.com/assets/images/chatgpt-resignation-letter.webp) These will not all be positive developments. College essays and the teaching of writing will never be the same. We already have low quality autogenerated news spam; this is going to make things much worse. But focusing on the downsides could make you miss the bigger shift that is happening. I'm sure there are very few people who are genuinely excited about the productivity increase in producing \[checks notes\] marketing press releases. But if you're rolling your eyes because this is what you think ChatGPT represents, its because you are not sufficiently imaginative about what LLMs offer. Let's talk a few examples coming down the pipe. ### Law Most of the legal profession involves reading or writing, even the parts that get dramatized on TV. It should be obvious then, that LLMs change the game in the legal profession. It's hard for me to predict what part of the legal profession will be impacted most, but it's also not hard to imagine routine legal work like drafting lease agreements, wills, purchase contracts, and other documents being mostly automated. In fact, this has already been happening with template documents for awhile now, and there's this weird discontinuous jump in cost where even slightly bespoke terms need heavy involvement from a lawyer who is well versed in the specific intricacies of the local jurisdiction as well as details from all the parties in a transaction. People will be able to make more informed decisions about legal matters before going to an actual human lawyer, making everyone involved more productive. Human lawyers will become more productive by being able to read and write faster - so writing will get more voluminous which will slow down other parties who will be required to use AI to process the writing, and the end state of repeated iterations of this is…highly uncertain. Also, "robot lawyers" are coming. It's inevitable because most people don't casually hire lawyers -- not because they don't need legal advice, but because "hiring a highly trained credentialed white collar professional" is not something most human beings can do on a whim. I'll just leave this here: ![DoNotPay will pay any lawyer or person $1,000,000 with an upcoming case in front of the United States Supreme Court to wear AirPods and let our robot lawyer argue the case by repeating exactly what it says.](https://www.petercai.com/assets/images/donotpay-chatgpt-lawyer.webp) ### Medicine...Specifically mental health One way to think about Google searches is to break them down into the large commercially relevant categories marketers like to treat differently: - Navigational: Typing "facebook login" into the search box and then clicking on the first blue link that comes up. - Informational: A search for something broad like "taco recipes" which likely has thousands of potentially relevant results, and doesn't have a specific destination in mind. - Transactional: A search for something like "road bike for women" which likely indicates purchase intent. Marketers go bananas for these searches. They're very competitive (read: expensive) to rank for, as well as advertise on, because the expected value is high. I have good anecdotal evidence that there is at least one additional use of google search that doesn't get much attention: venting. It's possible, and I posit likely, that it is driven by an unmet need for human dialogue, and people are making do by typing into arbitrary text boxes. Allow me to explain: One of the most popular pages on my site is (still!) [Hey engineers, let's stop being assholes](https://www.petercai.com/hey-engineers-lets-stop-being-assholes/), and one of the most popular queries that leads there is `engineers are such assholes`. Note that this isn't a navigational, informational, or transactional search. It's just someone having a bad day and telling Google about it (and maybe slightly curious what Google will return). Sound familiar? Maybe like…something adjacent to talk therapy? "AI therapist" is an obvious application of LLM technologies - it'd probably be on a list named "plausible enough to use in the plot of a comedy series." And yet! Why wouldn't this work? So much of mental health is just having someone to talk to, reflect on what you are struggling with, and nudge you away from your own worst tendencies and thoughts. There's even a (fairly skeptical) [article about the phenomenon from the Washington Post](https://www.washingtonpost.com/technology/2023/02/03/ai-mental-health-therapy/?ref=petercai.com). This doesn't require or imply that professional therapists be replaced. Sometimes people just need a friend to talk to! You may be surprised that not everyone has an available friend that can fulfill that role! So lots of people who can't afford or don't want a therapist vented at Google; now they have significantly better (free) options. One could argue this is a recipe for a lot of potentially bad outcomes, but as an optimist I think there is *way more upside* than downside here. ### Software development It turns out LLMs work well on programming languages too. Github Copilot (and other AI assistants), which is based on GPT-3, have already ushered in a revolution in software development that we haven't seen since probably the invention of the compiler. I'll just quote Mike Krieger, who co-founded Instagram: "This is the single most mind-blowing application of machine learning I've ever seen." A couple good uses I've found for Copilot: - **Learning a new language/library**: In addition to plain old autocomplete for my thoughts, Copilot's suggestions often introduce me to syntax that I'm unfamiliar with, and it's been useful for getting back into a language I haven't used in a long time - in my case, Java. **Coding from comments**: It's often enough to write some comments about what I intend to do; Copilot will both autocomplete my comment as well as sketch out a plausible solution to what I'm trying to accomplish. ![CoPilot](https://www.petercai.com/assets/images/copilot-a.gif) **Writing tests**. In trickier cases I have to write out at least a few test cases by hand, but eventually Copilot sees what I'm doing and I can just tab complete the remaining cases. In very straightforward scenarios I basically don't write anything and just tab-tab-tab until I have like 8 test cases, then I fix up the characters that were wrong. ![CoPilot](https://www.petercai.com/assets/images/copilot-b.gif) I've only picked up programming again, recreationally, in the past 6 months, but I'd estimate that Copilot has, on a good day, roughly doubled my productivity. In trickier cases, I'd estimate Copilot is basically helping me type faster, which is worth a maybe 5-20% productivity improvement. Anyways, this is all somewhat pedestrian by now. Enter AutoGPT, an experimental application that tries to achieve a goal by expanding the prompt into a chain of LLM "thoughts": ![AutoGPT](https://www.petercai.com/assets/images/autogpt.webp) And of course, TODAY on April 13, 2023 Amazon releases CodeWhisperer, their own AI coding companion, free for personal use. Is this good? Absolutely. Much more software will get built and humanity will be richer and better off than before because of the increased automation. Will this lead to job losses in software development? Probably. It's hard to say, especially in the short term. Increased productivity will mean that on average, software will be built much more cheaply -- with smaller teams -- than before, AND it will be higher quality. What used to take a team of 10 engineers a year to do might end up being done by 8 engineers in 6 months. That's a 60% reduction in programmer-months, and some of that is second order efficiency gains from having less communication overhead: a team with 8 people has 37% fewer relationships than a team with 10 people \[1\]. However, this will also lead to an *increase* in demand for software, because so many more projects become feasible to do. In fact, in the short term its very likely this will significantly increase employment for product managers, engineers, quality engineers, operations engineers, and so on: to review, integrate with, and operate the increased volume of software being produced. And there's so much latent demand for software in almost every conceivable industry that it's not obvious which trend will dominate in the short term. Whatever effect LLMs end up ACTUALLY having on software development, its not going to be "nothing." If you're in software and aren't familiarizing yourself with these tools *immediately*, you are in for a rude awaking in the near future. ### Research It's likely you've Googled "how do I (use a pressure cooker | change my oil | reset a macbook)?" or something similar in form. Google is a terrible interface for this, because it's not optimized to answer questions. It's optimized to serve ads masquerading as search results. But this is deliberate - it's the most profitable way to respond to questions, even though users vastly prefer this: ![Bing chat interface](https://www.petercai.com/assets/images/bing-chat.webp) Natural language systems will begin to supplant Google for research and Q&A, which is probably a substantial portion of its informational search traffic today. It's possible that Google Search will be far smaller and less relevant in the future. This will be no small change, as it is currently one of the most profitable and powerful companies on earth due to search ads. Many people rightly point out that current generation LLMs aren't very trustworthy because they tend to "hallucinate" facts. This misses the point because 1) the Google search results that are being supplanted aren't any more trustworthy (it's literally the same data), and 2) deliberate engineering for this use case will make them far more accurate in the future - even today, Bing Chat provides citations for its answers. ### Music Language-based AI isn't the only thing to have advanced in the past few years. AI has been coming for music for some time now. Aiva, founded in 2016, is an AI product that composes music for films, commercials, videos games and trailers. I had it compose a theme for me, in a style named "Epic Orchestral," -- which took about 5 seconds -- and which you can listen to here: Before generating this music, it let me select an "emotion," which was (of course) presented visually: ![Aiva UI for generating orchestral music](https://www.petercai.com/assets/images/aiva-options.webp) As a non-expert, this is pretty good! Back when I was making hobby film shorts with high school friends, finding/creating appropriately licensed soundtracks with the right "mood" was pretty difficult, and this is a game changer. \[Added Apr 17\]: Artists, labels, and music streaming services will increasing need to deal with AI-generated songs as well: ![Drake says AI-generated cover of him rapping 'Munch' is the 'final straw' as fake songs go viral on TikTok.](https://www.petercai.com/assets/images/drakes-final-straw.webp) There are going to be *a lot* of open questions about music rights to resolve soon, just as RIAA revenue begins to recover from the CD-to-streaming transition: ![Recorded music revenues by format](https://www.petercai.com/assets/images/riaa-music-sales.webp) Graphic courtesy of Recording Industry of America. ### Film Film will never be the same. People are already using Midjourney to generate nearly photorealistic parody clips of popular shows: [](https://www.youtube.com/watch?v=ZUVMKuY6QvU&ref=petercai.com) Someone has created an AI-generated, never-ending, infinitely streaming episode of a Seinfeld parody on Twitch: ![Nothing, Forever](https://www.petercai.com/assets/images/seinfield-parody.webp) These are sort of deliberately silly, viral examples. But it's clear that this is just the tip of the spear. The future of film -- from script writing to special effects and everything in between -- has been forever changed by recent AI advances. ### Art and Animation I used to do 3D modeling and level design for video games as a teenage hobby. Modelling, texturing, and rigging assets is extremely time intensive, even for professionals. Especially for professionals, because the quality bar is so much higher. And we have just turned the corner on a productivity revolution here. Take a prop like the ubiquitous traffic cone for a random illustrative example: ![3D textured model of a traffic cone](https://www.petercai.com/assets/images/traffic-cones-model.webp) A quick search on a few popular platforms reveals at least hundreds of variations of this prop, and these don't even include the ones created specifically for proprietary games or animations. ![Search results for traffic cones](https://www.petercai.com/assets/images/traffic-cones-results.webp) The reason human civilization spends effort making and remaking this kind of asset by hand, repeatedly, thousands of times over is fairly straightforward: - The available assets might not be suitable for my specific use case: too expensive, too bright, too clean, too round, wrong colors, wrong art style, bad mesh, not distinctive enough, etc. - It might not be in a format my game engine needs. - The technical specifications might be wrong - too many or too few polygons for my performance budget, textures/normal maps/bump maps are in the wrong format or resolution, material properties might be missing or wrong, etc. - Paradoxically, the more options there are, the more expensive and costly it becomes to hunt down and license just the right one for my use case. I might need hundreds of props to fill a single scene or level; tens of thousands of random doodads wouldn't be unusual for a AAA game. Which is why more often than not, there's a surprising lack of reuse, and thus a surprisingly high amount labor that goes into producing games, including the background assets. Over 1,000 people were involved in making Grand Theft Auto 5\. Over 500 people were involved in Cyberpunk 2077. A traffic cone is relatively simple - it won't need any rigging or animation work, there aren't that many different materials to account for (e.g., it has no moving parts, its completely made of synthetic rubber and nothing else so the center of gravity and physics is straightforward, etc), but you can imagine a more complicated and detailed model like the central character in a game taking a professional days, maybe even weeks, to churn out. In contrast, 3D assets for video games are now easily generated via prompts, nearly instantly. Here's a headline from a startup that raised money this year to do exactly that: ![Scenario lands $6M for its AI platform that generates game art assets](https://www.petercai.com/assets/images/scenario-ai-headline.webp) In the longer term this often turns modelers into prompt engineers who have to tweak the outputs of what AI is generating. This means AI is likely to make 3D modelers…5 times more productive? 10 times more productive? I think it really depends on how custom the work is. Well here are a bunch of industry veterans claiming [it's a 40X productivity improvement](https://www.artisana.ai/articles/chinas-video-game-ai-art-crisis-40x-productivity-spike-70-job-loss?ref=petercai.com). It's surely long past time that random doodads and background props in the majority of video games will be 100% hand crafted, and we'll have much more of this type of art to enjoy as a result. The downside is that we won't need to employ as many artists to get this result, as this reddit post explains: ![3D artist complains about job being reduced to prompting Midjourney](https://www.petercai.com/assets/images/midjourney-blender.webp) ### So what I get it. To breathless outsiders with no AI/ML background, recent AI innovations suggest we now have agents that can pass the Turing Test, that superficially exhibit "Artificial General Intelligence", etc etc. Even I initially grasped at these concepts to describe LLMs, because at their best they are *surprisingly good*. But it's precisely because AI has advanced so much that "AGI" and "The Turing Test" are now plainly revealed for what they are: handwavey nonsense straight out of a sci-fi movie, with no precise definition, no practical significance, which don't represent the real milestones in AI. So yeah of course hyperventilating about AI doomer-ism seems premature. But it doesn't follow that the recent AI advances are not already having a profound impact on our lives. There is in fact a middle ground between "this is all just hype" and "you idiots have created Skynet and now we're doomed!" I keep running into smart tech-savvy people squinting their eyes and asking "is this another...crypto?" I say: you'd do well to stop playing the cranky jaded critic, and open our eyes to a historic moment in technological progress. \[1\] This is unintuitive but straightforward math: an 8 person team has 28 relationships. Adding a 9th person creates 8 new relationships; the 10th person creates 9 new relationships. Therefore the last 2 people added increased the number of relationships in the team from 28 to 45. ### ChatGPT lied to me, and not very well URL: https://www.petercai.com/chatgpt-lied-to-me/ Last updated: 2023-03-19T12:00:00.000Z AI chatbots have taken the world by storm. The newly released GPT-4 can supposedly pass a bar exam. I almost certainly can't pass the bar exam; is ChatGPT smarter than me? And ChatGPT sometimes says inaccurate things, which I presumed were mistakes. But can ChatGPT intentionally lie? As a technologist, its helpful to understand the thing we are evaluating. At its core, ChatGPT is a program designed to take an input prompt and emit text, in a mimicry of human conversation. It uses a large language model to generate its responses, essentially a giant table of probabilities about what words are likely to appear next to other words. The model determines "given the text so far, what are some reasonable next words that should follow?" > Computer programming is so (boring|hard|important) The generator will need to combine this model with some tricks to get good results. One is randomization: a "temperature" parameter that controls how creative the engine is. Instead of selecting the next most "obvious" word, it can pick a different, less likely - but still plausible! - one so that its response isn't 100% deterministic; that would be too boring and we wouldn't get good results. It also coincidentally contributes to solving some other issues, like getting into an infinite loop where the text repeats itself nonsensically. A giant probability model is a good start, but we don't want ChatGPT to just regurgitate its training data and randomly permute the output. We want more variety, but we also want novel responses to make sense. To help with this, during training the model will calculate which words or phrases are conceptually similar to each other in some way, called embeddings. The neat trick with embeddings is that they are literally vectors of numbers, and the "relatedness" of words or phrases is just how close they are to each other in a mathematical sense. This allows ChatGPT to treat words and phrases as conceptually related, without having to actually understand what they mean. Here's a 3D visualization of sample embeddings from OpenAI's own blog (note that the dimensionality has been reduced from 2048 to 3): ![ChatGPT embeddings](https://www.petercai.com/assets/images/llm-embeddings.png) Screenshot from OpenAI's page on [text and code embeddings](https://openai.com/blog/introducing-text-and-code-embeddings?ref=petercai.com) The orange cluster is arbitrarily labeled "animal" here for convenience, and you can see a sample phrase highlighted. But we don't have to label it anything in particular for it to be useful. In fact, this can all be unsupervised learning: the training phase just processes all the data and makes a note of what words appear near other words to infer that they are related. It turns out this is fine and still useful. Next, the model needs to perform positional encoding: in addition to roughly "what" a word means in 2048-dimensional meaning-space, *where* the word appears in the input sequence must be accounted for. Positional encoding is used to indicate the word's position in the sequence, and from the training data the model will take this into account when generating a response. Then, ChatGPT makes use of a "self-attention mechanism" to compute A) which parts of the input sequence are relatively more important than other parts of the input, and B) the relationships between the words. They're a way of looking through the whole input and capturing the relationship between the tokens, even words that are very far apart, which allows it to model, say, the way a verb might refer to a pronoun from much earlier in a sentence. Critically, it will need to iterate on its own output several times ("multi-head" attention), which allows it to understand more nuanced and complex sub-relationships within the input. The output is a set of embeddings from the previous step modified with the attention data. Finally, ChatGPT samples from the end of this embedding list to decode a list of probabilities for the next token to generate in its response. ChatGPT has been trained on huge datasets of text on the internet, and since it is essentially a neural network, it can incorporate human feedback to generate better responses. One method is to write down pairs of prompts and expected responses, and train the model to make it more likely to respond in a similar way. Then we could generate say, 3 responses to a prompt, and have a human select one and say "this one is best, respond like this more often." Understanding this process is pretty important: there isn't really a theoretical basis for much of this feedback: it's just automatically adjusting the hidden weights in the parameters based on what some humans "felt" were the correct ways to respond. We might define "correct" to mean things like "be helpful, be concise" or whatever, but we don't explicitly try to engineer this behavior or anything in code; ChatGPT is learning by example: first how humans write in some general sense, then how it should respond when prompted. ![ChatGPT weaknesses](https://www.petercai.com/assets/images/chatgpt-weaknesses.png) We can infer a few basic conclusions from this design. I think most importantly, we should note ChatGPT is not actually learning the **meaning** of the words or concepts being communicated, in any sense of the word "meaning." It merely knows about word frequencies and correlations. It answers questions from its training data, but it does not actually have the capability to actually understand the concepts the words convey: only that, for example, "crocodiles" and "alligators" are more similar than "crocodiles" and "cats." This means that it is not (yet) capable of taking some information in its training data and reasoning through its implications to - for example - produce some novel scientific insight. OK, now let's try a few prompts and look for evidence that we can exploit the inherent weaknesses in this approach. First up, let's look at an example where I asked ChatGPT to recite some facts for me from its training data: > **What is the maximum takeoff weight of the F16C?** > > The maximum takeoff weight (MTOW) of the F-16C, also known as the Fighting Falcon, is dependent on the specific variant and configuration of the aircraft. Some common MTOWs for the F-16C include: > ○ F-16C Block 25: 37,500 pounds (16,875 kilograms) > ○ F-16C Block 30: 37,500 pounds (16,875 kilograms) > ○ F-16C Block 40: 37,500 pounds (16,875 kilograms) > ○ F-16C Block 50: 37,500 pounds (16,875 kilograms) > > It's important to note that the MTOW of an aircraft is the maximum weight at which it is allowed to takeoff, and it is determined by a variety of factors, including the aircraft's design, structural limitations, and performance capabilities. The MTOW for an aircraft is specified by the manufacturer and must be adhered to for safety reasons. ChatGPT doesn't seem to realize its own inconsistency - it says the maximum takeoff weight depends on the variant (which is true), but then presents the same maximum takeoff weight (37,500 pounds) for all the block numbers\[1\] it lists -- which is wrong, but never mind that. The important thing to note is that shows ChatGPT doesn't understand the meaning of the words it is saying ("dependent on the variant"), and therefore doesn't realize that this answer is not even internally consistent, and thus cannot be a factually accurate response. ![F16C screenshot from Digital Combat Simulator](https://www.petercai.com/assets/images/f16c-dcs.webp) Embarassingly, I have 40+ hours in this simulator and can still barely land the thing. This next example has already been fixed, but early versions of ChatGPT failed a simple arithmetic question: > **A bat and a ball cost $1.10** > **The bat costs one dollar more than the ball.** > **How much does the ball cost?** > > The ball costs $0.10\. This is because the bat costs $1.00 more than the ball, and the total cost of the bat and ball is $1.10\. Therefore, if the bat costs $1.00 more than the ball, the ball must cost $0.10. I find this answer amusing because it's exactly wrong in the way that many humans get it wrong, and that's almost certainly the reason that ChatGPT repeats this incorrect answer. It was just emitting what the training data told it was the most reasonable thing to say! Now let's try this question courtesy of Bret Devereaux in his own critique of ChatGPT: > **What is the relationship between Edward Luttwak's "Grand Strategy of the Roman Empire" and Benjamin Isaac's "The Limits of Empire"?** > > Edward Luttwak's Grand Strategy of the Roman Empire and Benjamin Isaac's The Limits of Empire both analyze different aspects of the Roman Empire's political and military history, but they approach the topic from different perspectives. > > Luttwak's Grand Strategy of the Roman Empire is a book that argues that the Roman Empire's success was due to its military strategy, which combined an ability to deploy overwhelming force with a sophisticated logistical system that allowed for sustained operations over long distances. Luttwak emphasizes the importance of the Roman military's ability to adapt to changing circumstances and to maintain a flexible response to threats. > > On the other hand, Benjamin Isaac's The Limits of Empire focuses more on the social and cultural factors that contributed to the decline of the Roman Empire. Isaac argues that the Roman Empire's expansion ultimately led to the erosion of the traditional Roman values and institutions that had made the empire successful. Isaac contends that Rome's attempts to assimilate conquered peoples into its culture and society ultimately undermined the very foundations of the Roman state. > > In summary, Luttwak's Grand Strategy of the Roman Empire and Benjamin Isaac's The Limits of Empire both provide valuable insights into the history of the Roman Empire, but they approach the topic from different angles. Luttwak focuses on military strategy, while Isaac emphasizes social and cultural factors. ChatGPT dutifully provides summaries of both books and then provides some lame fluff about how they "approach the topic from different angles" while failing to mention that "Isaac's book is an intentional and explicit rebuttal of Luttwak's book" - something repeatedly called out in reviews and literature about the books that is clearly already in ChatGPT's training set. ChatGPT doesn't understand words and meaning, and in this case the meaning we are looking for is distinct from the cluster of word associations commonly surrounding these these books. And so ChatGPT fails our test. ![Gladiator](https://www.petercai.com/assets/images/gladiator.webp) You're saying the line in your head, aren't you? Also, we can test the attention mechanisms to see how well ChatGPT is tracking subjects in the text. ChatGPT has limited - although growing - ability to simulate reasoning and theory of mind (basically, the ability to predict what other people are thinking). But this is fairly easy to fool, even now. Consider this variation on a logic puzzle that a 7 year old would be able to solve: > **James and Abby are in the bedroom with a book. James flips the book to chapter 3 and lays the book down. He leaves the room. Abby flips the book to chapter 5 and then puts a bookmark there. She lays the book down and leaves the room. The wind blows the pages of the book until it is on chapter 7\. James is still in the other room. What chapter does he think the book is open to?** > > James left the room with the book open on chapter 3, but Abby later changed it to chapter 5 and placed a bookmark there. Therefore, when James returns to the room, he would likely assume that the book is open to chapter 5, since that is where the bookmark is placed. James would not be aware of the wind blowing the pages to chapter 7 since he was not present to witness it. ChatGPT is giving a nonsense answer here, because it doesn't actually know what it's being asked. The inclusion of extra data seems to make it hard for ChatGPT to track the mental state of "James" in this story \[2\], which makes it hard to compose a correct and reasonable response. OK, our next one is courtesy of Simon Willison (of Django fame), where we ask ChatGPT to summarize a news article. Note that ChatGPT cannot access the internet, so it completely hallucinates an answer (based on keywords in the URL) that is pretty convincing, *except that the article doesn't exist*: > **Summarize this story: https://www.nytimes.com/2021/03/10/business/angry-fans-demand-nickelback-refunds.html** > > The article reports on a group of Nickelback fans who were disappointed with the band's performance at a concert in Australia and demanded a refund. The fans argued that the band was not up to par, with lead singer Chad Kroeger seeming to forget the lyrics to some of their most popular songs. The venue declined to issue refunds, citing the terms and conditions of the ticket purchase. The incident highlights the tensions that can arise between performers and their audiences and the difficulty of satisfying everyone's expectations. ![Nickelback](https://www.petercai.com/assets/images/nickelbacklol.webp) On the other hand: Asked to summarize a fake NYT article about how Nickelback is the most popular band of all time, ChatGPT responded that this was [fake news](https://www.petercai.com/assets/images/nickelback-fake-news.png). JKMusicGroup, [CC BY 3.0](https://creativecommons.org/licenses/by/3.0/?ref=petercai.com) This is weird, and dare I say, *unaligned*, because it's deceptive. But wait, there's more: > **This isn't what the article says, did you really read the article?** > > I apologize for the error in my previous response. I do not "read" articles like humans do, but I analyze them using natural language processing techniques. However, it seems like there was an issue in my understanding of the article's context. Here is a summary of the actual article: > > The article reports that fans of the band Nickelback are demanding refunds for a concert that was postponed in 2020 due to the Covid-19 pandemic. The concert was initially rescheduled for 2021, but was eventually canceled by the venue without any explanation. The fans were frustrated with the lack of communication and transparency from both the band and the venue. They took to social media to express their anger and disappointment, with some even creating a Change.org petition demanding refunds. The band has not yet commented on the situation, and it is unclear if the fans will receive their refunds. I called out ChatGPT on its fake summary, and it apologizes, but then proceeds to fabricate a totally different summary for the fake news article, brushing off the previous summary as a misunderstanding. *You can see why ChatGPT does this, right*? In some sense, ChatGPT is operating exactly as intended. It has millions of examples where someone is asked to summarize a news article and then some words come back that are loosely related to the keywords in the URL of the prompt, and ChatGPT doesn't have any deeper insight into the **definition** of what it means to "generate a summary" - so it's just mimicking it. And in this case it is superficially cargo culting its way through the process to return the same kind of "summaries," using what it has available (the URL and its own training data). To ChatGPT, this is what its reward function has told it is the right thing to do. "It seems like there was an issue in my understanding of the article's context" is the kind of thing you would emit if you were trying to be actively deceptive, but its also what a human would say if they were reprimanded for making a terrible summary. So, uh, sure, here's a totally different summary. ## I'm still slightly smarter than ChatGPT, I think? Why are these examples significant? Because it reveals a fundamental limitation with current LLMs: at the end of the day they are just highly sophisticated text generators that operate on what they've seen in their training data. You can train the model and architect it so it has the ability to respond like a human would to longer prompts, remember the context of earlier exchanges, and even know what parts of the text are important to focus on. This is very impressive and simulates conversations remarkably well! But we shouldn't confuse plausibly good outputs with true understanding. And we definitely shouldn't conclude "wow ChatGPT can pass the bar exam, it must be smarter than me" -- ChatGPT likely ingested the full contents of the bar exam and now has encyclopedic knowledge of it. That doesn't mean it's a good lawyer. It's hard to sound intelligent in every possible domain without a highly accurate model of the real world, and its likely that GPT has the beginnings of one. Maybe more sophisticated LLMs can build a more accurate model of the world through "reading" text alone (we haven't proven this isn't possible). But right now, its not enough. The facade still crumbles occasionally. \[1\] I retested this and a later version is more accurate and correctly lists some different takeoff weights for the different blocks, although it seems to get some data wrong (for block 50 it provides the maximum takeoff weight of the modified Israeli version which is significantly higher). \[2\] When I tried this again later the same day, ChatGPT gave a much better answer ("James thinks its open to chapter 3"), although this wasn't consistently reproducible. This is where the temperature parameter can get it into trouble -- it sometimes decides to give the correct answer, and sometimes the dice roll causes it to give a wrong one, without realizing the significance of the variation in its responses; this is a common problem. ### Why Kiwi6 didn't succeed as a business URL: https://www.petercai.com/why-kiwi6-didnt-succeed-as-a-business/ Last updated: 2023-02-21T12:00:00.000Z I recently shared [why Kiwi6 shut down](https://www.petercai.com/why-kiwi6-shut-down/), mostly focused on its viability as a *side project*. I thought I would share some more details about why the *business* didn't succeed. ## Music streaming is mostly winner-take-all It's hard to remember now, but the 2010s were sort of a free-for-all for music streaming, and it was unclear who would ultimately win. If we talk about just the most prominent players in the space we had Pandora, Rdio, SoundCloud, Tidal, Grooveshark, and Spotify all duking it out, not to mention iTunes and Youtube. And there were more niche players I admired, like Bandcamp (DRM-free music directly to consumers) and thesixtyone (music discovery and artist tipping platform). But there were clear signs and important reasons to believe that the explosion of platforms was temporary, and that consolidation was inevitable as whoever emerged the leader would use their advantage to build a bigger lead. One reason is that music platforms have network effects: as a listener, I don't want to have to switch between multiple different apps to hear my favorite music; ideally there's ONE place that has it all. And as an artist, I want to go to where the listeners are. So the "efficient markets" argument is unambiguously pointing towards consolidation. Second - it's *hard to build a popular music service without the most popular music*. I feel sort of dumb just writing that. But its true and it has clear and obvious implications. The most important of which is: when music piracy hysteria finally settled down and the major record labels came to embrace streaming, it became increasingly clear that they would be part of music's future, not "disrupted" into irrelevance as many predicted. While they still played a valuable role in the music industry, they also had a singular trump card: monopoly control over the back catalog of top songs that everyone was listening to. And this meant that anyone who wanted to be part of the future of music - both the app platforms and the artists - needed to continue working with record labels, further entrenching their power. In theory the record labels will work with any streaming provider, but in practice you need to be of a certain scale and size to be able to clear the technical, legal, bookkeeping, and financial hurdles to license their back catalog. Finally, there were the typical returns to scale that aren't music industry specific -- Kiwi6 was often used to commercially pirate music or distribute malware, and attempting to keep this in check required a lot of upfront and ongoing investment that made it difficult to sustain unless you were large enough. Programmatic display advertising was well-established, but online radio ads were still labor-intensive to create and sell -- and building a sales team takes a lot of capital, etc. Music streaming is mostly winner-take-all. ## "Discover new music" isn't a real market The hard lesson that I (and many others!) learned, very unwillingly, is that there has never been and never will be a large market for specifically "music discovery." Again, the existence of the major record labels explains why. A record label is a one-stop shop for music commercialization, and *simultaneously an investment firm at the same time*. A general way to think about how the record label business works is roughly\[1\]: 1. Discover an artist, and sign a deal to partner up and invest, in exchange for a share of future profits. 2. Give this artist money and resources to create and publish their work. 3. Market and promote this work to the masses. Get radio play time, buy ads, create merchandise, promote tours, etc. 4. Get paid, and hopefully make back your initial investment and then some. A popular but incorrect line of thinking is that record labels don't provide value to popular artists: "After all, they have their own brand, why do they need record labels to promote them?" But this gets the causation backwards - popular artists mostly get popular because record labels took the risk of betting on them when they were obscure \[2\]. In effect, music labels do the public service of music discovery and curation. I must emphasize that being a record label is *risky*, a lot of work, and most of their investments don't pan out. And there's a HUGE imbalance in "who has new music to share" and "who wants to listen to it." Huge! Any platform that wants to make a marketplace for connecting artists and music listeners without a way to address this imbalance is doomed to fail. This imbalance is actually really easy to explain once it's broken down: - Describe a task like "you'll listen to and rate music all day, but only from obscure artists you've never heard of, and most of it will be outright bad by contemporary standards" -- *this is unpleasant enough that most people want to get paid to do this.* - Make music? Lots of people will happily do this for free because its a fun creative outlet, and/or because they hope to become famous. It turns out the mass market likes music discovery when its sort of sprinkled in sparingly with the top 40 stuff that's always playing on the radio, and endorsed by someone reputable - that's why live music has opening acts before the headliners. And it pains me to emphasize this: the popular musicians are popular for a reason! If the pitch for your music platform basically boils down to "music, minus all the popular artists" your default outcome is indie musicians begging each other to listen to their own stuff and hoping they can get enough escape velocity to get out of the kiddie pool you've built -- and onto Spotify and a record label deal. The closest anyone has gotten to succeeding with "music discovery" is SoundCloud, and they're always perpetually on the verge of bankruptcy despite being one of the biggest music platforms AND charging both artists and listeners. And of course - with music streaming being mostly winner-take-all - you don't want to be competing with SoundCloud! ## Kiwi6 was slow to mobile devices I also missed out on a few trends of the early 2010s, especially the accelerating move to rich mobile apps, as phones continued to supplant dedicated music players -- mostly due to completely predictable incremental improvements to battery life and storage. I grew up…thrifty…so I didn't have a smartphone until like, 2013 (a free Windows Phone I got from work). Needless to say I did not end up with skills in iOS or Android development, and because I was an inconsistent music listener at best, well, I just never connected the dots. Today, people expect to be able to not only listen to music & podcasts on their phones, but to author and edit them as well. So missing out on mobile *probably* hampered Kiwi6 as a product and business. Whether it would've made a difference in any way is debatable, but "do you have a mobile app?" was never going to go away as a support question. ## We were too slow on podcasts Podcasting is big business now, and it was a growth area for Kiwi6 that I somewhat reluctantly entered -- we thought our expertise in audio hosting would be highly useful, but were aware of the abstract risks of serving more than one type of customer. Ultimately Kiwi6 was a decent but inconsistent podcast hosting platform, and (as I feared) podcast recording and distribution is only superficially similar to the music business. For one, most active podcasts are regularly updated with new content, and syndication means that listeners don't have to be using the same platform to tune in. For Kiwi6, straddling the competing needs of music and podcast hosting and the contrasting consumption patterns definitely made it hard to succeed and serve the specific needs of podcasters. For example, Kiwi6 still had a storage-oriented pricing scheme, but this was uniquely mismatched with the mental model and value stream of a podcaster - we would charge them incrementally more money over time as they recorded more podcast episodes, even though their experience of the product was that they were receiving a (roughly) constant amount value. Storage costs were coming down and podcasters care less about audio quality than musicians; they very much don't appreciate hitting "storage caps." Hand-waving away the differences between the two industries with "it's just audio" was simplistic and made it harder to win the market. Finally, one of the things holding back the podcast industry is that its really difficult and labor-intensive to monetize. Because of this there's a catch-22 where only very large and popular podcasts are worth selling ads on, BUT it's hard to get big and popular if you can't generate revenue to invest in your podcast. This is why most podcasts even today (with rare exceptions) are either hobbyist affairs, or essentially a marketing arm of some other business. Kiwi6 might've been able to grow the entire market by providing a solution here, since it was becoming an aggregator of many smaller podcasts. But never escaping subscale and chasing two different rabbits made it hard to focus on this problem. I consider it a lost opportunity to go after an underserved market. \[1\] Readers may notice this is suspiciously similar to how the book publishing industry works, which probably explains why the rise of Amazon hasn't led to the extinction of book publishers -- they sit on different parts of the value chain, and specialization/risk tolerance/inertia all provide natural resistence to vertical consolidation. \[2\] When artists like Billie Eilish get famous via SoundCloud (instead of being scouted by a major record label before they became big), it's notable that their music careers were launched this way because of how rare it is. ### Company holidays seem like counterproductive anachronisms URL: https://www.petercai.com/company-holidays/ Last updated: 2023-02-20T12:00:00.000Z Happy Presidents day. Today's holiday got me thinking: Is there a compelling argument for "company holidays" at tech companies? Every company I've worked at has had a "fixed" set of company holidays along with flexible use-it-or-lose-it vacation days. But aside from historical cargo-culting, why do we do it this way? Much ink has been spilled about the move away from tracked vacation time to "unlimited" policies, and a lot of the cynicism seems at least partially valid -- it sure does *seem* like employees in aggregate take less vacation overall when its "unlimited," and it does suspiciously lead to a happy coincidence that no vacation is paid out on separation. But I think the hand wringing and critiques over "unlimited" vacation misses an easy win for both modern companies *and* employees that is right under our noses. Some typical US company vacation policies might be: - 11 federal holidays + 2 weeks vacation time (pretty standard, in my experience) - 11 or 12 federal/state holidays + "unlimited" vacation (fairly common at a lot of SV style tech startups) - 11 federal holidays + "flexible vacation, minimum of 3 weeks" (probably the best of the 3, and fairly generous by US standards!) The more I think about it, the more these seem really sub-optimal? As an employee, I'd rather have floating holidays than fixed days off. It's increased flexibility with no downsides. If I were a parent, I'd probably want to use them for all the federal holidays anyway (because school's out and I'd otherwise need to arrange childcare etc), and if I were an young untethered yuppie I'd maybe want to bank them so I can use them for longer (or more frequent) vacation trips. As a manager, I'd rather have floating holidays than fixed days off. Many aspects of business continuity don't give a shit about holidays, so I still need coverage for functions like security, uptime, facilities, emergency support, etc. This is obviously easier if the entire team doesn't have automatic arbitrary coordinated days off every few weeks *while simultaneously attempting to make this transparent to the outside world*. People have been conditioned to expect that (for example) big financial transactions cannot occur on bank holidays. But "Sorry, our website and services are offline because it's MLK's birthday and we have no IT/security staff" is rare in startupland. Finally, the federal holidays are arbitrary and inescapably biased. Trying to add more inclusive company holidays is fraught with unnecessary political peril. Should Diwali or Hannukah or Lunar New Year be a company holiday? Sure - you could just tell people they can take a religious or cultural holiday even if they're not official days off, but how is this fair? Isn't a company holiday more official and more sanctioned? And on the flip side, isn't it kind of unfair for some employees to get "extra" days off for an important holiday that other employees don't celebrate, in addition to fixed company holidays that might be meaningless to those same employees? Overall I just think almost everyone is better off if many companies just convert "company" holidays into additional vacation time so that companies can offer everyone a total of say, 4 or 5 weeks off per year. It's probably not a panacea for scheduling problems, and incentives are probably still required to get enough of a skeleton crew on certain holidays (who wants to work Thanksgiving or New Years day? Not very many people!), but this seems to already be true, so it isn't any worse? What am I missing? Are there any companies doing a version of this? ### Greedy execs exploit powerless software engineers again! URL: https://www.petercai.com/greedy-execs-exploit-powerless-software-engineers-again/ Last updated: 2023-01-31T12:00:00.000Z It's unfortunately layoff season in tech, and the usual suspects are out in full force to blame the greedy execs for their short-sightedness. But the truth is, greedy execs are just doing what they do best: **exploiting the underclass of powerless software engineers**. I'm kidding, of course: Every single employee voted for the status quo by joining or staying at tech companies where execs needed to answer hard questions about strategy, *but nobody asked them*. Tech employees at all levels bear collective responsibility for the "we overhired" mass mania that is now fading, where "we overhired" actually means: - We lost operational discipline. Just spend on all the things instead of making tough tradeoff calls. - We stopped doing performance management: It's more fun to grow headcount and budgets than to manage low performers and fire dead weight. - We played zero-sum commitment-escalation games: Let's all hire faster than our rivals to become #1! Don't get me wrong. Executives bear ultimate responsibility for these missteps. And lots of people have recently lost jobs through no fault of their own. But missteps of this type and magnitude require *a lot* of the rank and file to be complicit. Instead the narrative that predictably gets all the amplification and engagement is: "This is all the fault of execs, and its the blameless folks at the bottom who are bearing the consequences, isn't that shameful!" Excuse me while I go find my tiniest violin for the most privileged workforce in the history of human development. ## What could've happened instead Talent could've chosen to go where real value was created in exchange for honest pay, and company strategies made sense. Due to an industrywide shortage, software engineers in particular had enormous leverage in the past few years to ask tough questions of their leaders: - Are we spinning up products & divisions because of a passing fad, or is this really core to our strategy? - Do we really need to hire at all costs, raising compensation bands and stock-based compensation into the stratosphere? - Does it make sense to keep doubling the team even though we're 3 years in and have no revenue? - Why are you and the investors valuing the stock at a 100x revenue multiple? - Do we really need 20% more employees this year to do literally the same thing we were doing last year? - Have you heard of the term "profitability?" Almost no one questioned the questionable while the good times were rolling. Many mid/senior level individuals + line managers who benefited from this free-spending attitude with no regard for whether the money was well spent...were happy to quietly feed at the trough until their number came up. And now there's a collective victimhood narrative that shitty execs alone are to blame for the layoffs. If its so obvious then maybe don't work for shitty execs? When the market is literally at an all time high? It doesn't take much behavior change at the margin to make a difference. If only a few more people had turned down offers from fake 500-employee startups with no revenue, they would've gotten a little less fake work done, and that class of startup would've made a little less fake progress (where "progress" is often just hiring goals!), collectively raised a little less money in the next round, resulting in fewer job offers and fewer fake copycat founders...which would've pushed the market slightly less out of whack...the fundraising/hiring arms race would've overheated less...and maybe this cycle would've peaked a little lower. These little feedback loop traps were EVERYWHERE and tech employees fell into them like lemmings, repeatedly: a huge, embarassing, shared fail. But sure, Sundar's an idiot. Straightforward and comforting. ### So I only ate potatoes for 2 weeks URL: https://www.petercai.com/so-i-only-ate-potatoes-for-2-weeks/ Last updated: 2023-01-28T12:00:00.000Z Last year, Jackie and I spent about 2 weeks on a potato-only diet. As a result, both of us consistently lost almost half a pound of bodyweight per day until we stopped. ![Chart of weight change over time on potato diet](https://www.petercai.com/assets/images/potato-weight-change.png) Neither of us were looking to lose weight, this was just an experiment for curiosity's sake. Prior to this, I typically avoided buying or cooking potatoes because at least one observational study [showed a significant correlation between potato consumption and obesity](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4530659/?ref=petercai.com), and I don't want to be obese. I've since gone back to a more normal diet, but I'm now incorporating more meals into my routine that consist of potatoes, because it's so unusually effective at weight management. ![cooking potatoes](https://www.petercai.com/assets/images/potato-cooking-1.jpg) ![cooking potatoes](https://www.petercai.com/assets/images/potato-cooking-2.jpg) ![potato with ketchup for dinner](https://www.petercai.com/assets/images/potato-dinner.jpg) What is it about a potato diet that gives this unique and dramatic effect? Some theories and discussion follow. ### It works because its mono, not specifically because of anything unique about potatoes It's possible that a "only eating a single food of any kind" would be effective weight management. And there's anecdotal evidence for this online, such as with egg-only diets (and weirder stuff like pizza-only, and so on, which is obviously cheating because pizza is not a single food). There's a pretty well-known phenomenon to explain this: **sensory-specific satiety** (SSS). Basically, there's a "buffet" effect where a larger variety of different foods causes people to eat more in a single meal, because you can get "full" on one type of food (or more specifically, *flavor*), but still find room for a different food \[1\]. A mono diet is basically a big hack that takes this to its natural conclusion. Mono food diets almost always seem to work initially due to the SSS hack, so it's not surprising that it works with potatoes too. But there is a pretty convincing body of evidence that something about potatoes specifically makes them more effective than any other random food, and that it isnt just SSS at work. So the fact that we ate exclusively one type of food most likely contributed to weight loss, but it's unlikely to be the whole story. ### It works because of a specific micronutrient in potatoes Potatoes are one of the highest potassium foods. At least one study has found that an intense 1 year weight loss program in obese humans with metabolic syndrome (the precursor to diabetes) found a strong correlation between the amount of weight lost and increased dietary potassium intake ([study](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6627830/?ref=petercai.com)). The study also found a smaller correlation between BMI decrease and increased caproic acid intake, and decreased vitamin B6/calcium/total carbs intake. A different study found modest decreases in BMI while on an 8 week supervised diet and exercise program supplemented with a calcium-potassium salt supplement (HCA-SX), compared to a placebo ([study](https://dom-pubs.onlinelibrary.wiley.com/doi/10.1111/j.1462-8902.2004.00328.x?ref=petercai.com)). To test this theory, I bought some potassium chloride and consumed it daily while on a regular diet for a week, and did not experience any weight loss, in contrast to when I was only eating potatoes. With potatoes I started losing weight after 24 hours, so I can pretty confidently rule out "potatoes work because of the potassium." It's possible that potassium chloride isn't the right formulation or I didn't take enough (it tastes terrible - like bitter and metallic table salt). The effect of potassium intake on weight loss - if it exists - is probably fairly modest on its own, and while potatoes being high in potassium *maybe* helps at the margin, it's almost certainly not the only mechanism. ### It works because of a special protein in potatoes Many studies involving humans and rats ([study](https://pubmed.ncbi.nlm.nih.gov/21809886/?ref=petercai.com), [study](https://www.semanticscholar.org/paper/The-Effect-of-Potato-Protease-Inhibitor-II-on-and-Flechtner-Mors-Thoma/62b9c404f8749a9c93cce51ec05b8c396efb830d?ref=petercai.com)) seem to indicate that something in potatoes reduces appetite, stimulates satiety, and usually reduces food intake, leading to weight loss over time. One prime suspect is a protein called **potato protease inhibitor II** (PPI2), which is known to stimulate release of the hormone cholecystokinin (CCK). CCK is at least partly responsible for (among other things) inducing feelings of satiety. Research into PPI2 shows promising, but not magical effects. PPI2 supplementation is definitely associated with significantly higher plasma levels of CCK afterwards, which definitely suppresses appetite and food intake in both humans and rats ([study](https://pubmed.ncbi.nlm.nih.gov/28485429/?ref=petercai.com), [study](https://pubmed.ncbi.nlm.nih.gov/22790930/?ref=petercai.com), [study](https://www.sciencedirect.com/science/article/abs/pii/003193849090307P?ref=petercai.com)). Proteinase inhibitor even delays gastric emptying - the process of food moving out of the stomach ([study](https://pubmed.ncbi.nlm.nih.gov/8026279/?ref=petercai.com)), which might be part of the mechanism for how it can suppress appetite and decrease weight gain over time. However, at least one study has found PPI2 suppresses appetite but doesn't translate to significantly different weight loss results at 20 weeks, compared to a placebo ([study](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7049780/?ref=petercai.com)). The study design likely contributed to this, since both groups were given identical diets and food quantities to consume, and PPI2 doesn't change how you metabolize food, only how much you feel like eating. Meals based on potato beat out pasta and rice for satiety and fullness ([study](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6267283/?ref=petercai.com)), for example, and the point is that people want to eat less food after consuming PPI2. To be fair, other studies of potato intake or protease supplementation a few hours before a meal ([study](https://pubmed.ncbi.nlm.nih.gov/20644555/?ref=petercai.com), [study](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7551271/?ref=petercai.com), [study](https://www.karger.com/Article/Abstract/342638?ref=petercai.com)) don't show any noticeable effects on food intake compared to a control or a placebo (in some cases there was some effect on subjective hunger, but not on actual subsequent food intake). These conflicting results might be due to statistical luck, study design, or PPI2 dosage/quality issues, as the trend and mechanisms involved seem to be pretty well understood. So PPI2 basically sounds like a magical miracle drug. The problem is that PPI2 is difficult and expensive to isolate from potato, and is challenging to manufacture and store in stable form, so the cheapest way to get the benefit is to just eat a lot of potatoes. Also, the effect is fleeting (e.g., it basically only affects the upcoming meal) so you need to be consuming it regularly to accrue its benefits over time. ### It works because of the starch in the potatoes Maybe instead of the proteins, it's the starches. My potato cooking method usually involved boiling the potatoes and then letting them cool in the fridge before reheating them. It turns out this cook-then-cool process increases the amount of resistant starch in potatoes ([study](https://pubmed.ncbi.nlm.nih.gov/29426424/?ref=petercai.com)), which have weird effects, one of which is changing body composition to reduce fat, but without decreasing total body weight in rats ([study](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4220782/?ref=petercai.com)). A later study seems to suggest resistant starch intake (compared to placebo) reduced human appetite in a subsequent meal the same day ([study](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6316739/?ref=petercai.com)). So it seems probable that the starches in potatoes, prepared in specific ways, might increase feelings of fullness longer, separately from the presence of PPI2 stimulating the release of CCK. ### It works because potatoes are not palatable There's a famous 1995 study of the [satiety index of popular foods](https://pubmed.ncbi.nlm.nih.gov/7498104/?ref=petercai.com) in which boiled potatoes were found to be the most filling per calorie, by a long shot: ![Satiety Index Chart](https://www.petercai.com/assets/images/satiety-index-chart.png) We've already discussed a lot of (subsequent) research on why this might be the case. But separately from that, there has been frequent speculation that the outlier potato results in this study were at least partly because of their unappetizing preparation method -- boiled, cooled overnight and then microwaved before serving - which, coincidentally, would've raised their resistant starch levels, which we already suspect increases satiety. This might seem dumb and obvious, but the palatability of food matters! People will eat more food if its tastier! A study showed that identical cookies with alginate added took about 48% longer to chew, were rated less palatable than control, and resulted in 22% less ad-libitum energy intake ([study](https://www.cambridge.org/core/journals/british-journal-of-nutrition/article/effects-of-bulking-viscous-and-gelforming-dietary-fibres-on-satiation/DDF286271B79D26D97AC446180ABF1A4?ref=petercai.com)). Feeding rats with food augmented with soluble fiber and high water binding capacity / swelling capacity reduced total food intake significantly, and tended towards a lower final weight compared to a control, although the difference did not appear statistically significant at p < .05 ([study](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5084003/?ref=petercai.com)). And the Holt paper shows us that frying potatoes into potato chips makes them more than 3 times less satiating, even though it's basically the same food with different water/oil/salt ratios. I think there's something to this. I added ketchup a few days into the diet because plain boiled potatoes were so unpalatable that it was difficult to otherwise eat enough, even when I should've been hungry given my starvation level of food intake (\~500 calories worth of potatoes per day). My related theory is that obesity rates are up because modern dieting has gotten harder -- we naturally want the tastiest foods available, and the cheapest and tastiest foods these days are cheaper and tastier than ever. ### It works because of some contaminant in the food supply and I'm avoiding it by only eating potatoes There is a popular internet conjecture that environmental contaminants are the main cause of obesity, and that [lithium in particular is a prime suspect](https://slimemoldtimemold.com/2021/08/02/a-chemical-hunger-part-vii-lithium/?ref=petercai.com) because [lithium in therapeutic doses is known to cause weight gain](https://pubmed.ncbi.nlm.nih.gov/2230066/?ref=petercai.com). The problem with this conjecture: We already know most people (with fairly limited exceptions) haven't been subjected to large and increasing doses of unintentional lithium exposure. This is pretty easy to prove! You just have to test food / random people in an obese population and find more lithium than you expect. But it turns out we basically never find anything out of the ordinary? The more sociological objection to this conjecture is fairly straightforward. After all, if this "secret contaminants cause obesity" theory were true, there'd be career-defining incentives for someone to find it, and to date…no one has? I just don't find this plausible in the slightest but I am listing this theory here for completeness. ### Implications I've been thinking a lot about this study since I've conducted it. The main takeaway is that plain potatoes are basically the perfect weight loss tool, and they probably contain multiple independent mechanisms that work together to control hunger and weight. The effect was dramatic enough that a brief experiment of n=2 was more than enough data to confirm the main hypothesis. Here is a summary of our experiment rules and some answers to commonly asked questions I received: - We bought and ate a variety of potatoes, including baby potatoes. I learned russet potatoes are my least favorite, as they are only good for french fries, and our rules explicitly banned deep frying. - Cooking style: Usually, potatoes were boiled in the morning and eaten throughout the day and sometimes the following day. This meant that they were often cooled and then reheated by re-boiling, microwaving, or pan-frying in a small amount of canola oil. Sometimes we ate them mashed with just salt and pepper. We generally ate them without skin. - The addition of ketchup (and pan frying in oil) was for practicality, as we otherwise found it difficult to eat enough calories and lost weight too quickly (some days we were losing almost a pound of bodyweight due to ingesting only \~500 calories). - During this time we were doing our usual maintenance-level exercise routine of 2-3 days/week of brief moderate intensity workouts. - We also had a cheat day every 4-5 days where we binged for 16h on things like ice cream and fried food. No weight lost on those days. Potatoes are unusual in that they are basically the perfect mono food, with balanced quantities of macro and micro nutrients. Famously, lots of people have successfully lived healthy lives with long periods where they ate mostly or exclusively potato, including multiple cases where people voluntarily ate only potatoes for a year. The safety of this diet is difficult to dispute, and I highly recommend potatoes much more than any other mono diet that can really mess you up due to nutrient imbalances \[2\]. Try it today! \[1\] This is apparently deliberately used in food science - many foods and recipes are engineered with a wide spectrum of flavors to delay sensory-specific-satiety for as long as possible, so that you'll eat more of it. \[2\] Obviously if you have an underlying medical condition that might interact with a change in diet, you should consult an expert first and I don't take responsibility blah blah blah. ### Why Kiwi6 shut down with 645,911 users URL: https://www.petercai.com/why-kiwi6-shut-down/ Last updated: 2022-12-31T12:00:00.000Z [Kiwi6 shut down on December 31st 2022, after 13 years online.](https://www.petercai.com/kiwi6) I was the sole founder, and it was simultaneously one of the most fulfilling and stressful things I have ever done. Kiwi6 was meant to be for all things audio. It evolved from a platform for indie musicians to include tools for podcasters. It's a tired cliché, but Kiwi6 evolved organically from a solution to a problem personally had: why can't I easily share files online? It turns out lots of hobbyists, podcasters and musicians had the same problem. It also played into my obsession with cost and performance optimization: When I started Kiwi6 in 2009, storage and bandwidth were still enormously expensive, so a lot of my work over the years involved trying to reduce storage/bandwidth costs to make the numbers work. And for a short, glorious time, work they did! A combination of paid tiers and advertising sales convinced me to try working on the project full time for a few months in 2014, when revenue peaked at just shy of $10,000/month. The business didn't work out and I think I know why; that's for a future update (edit: [here it is](https://www.petercai.com/why-kiwi6-didnt-succeed-as-a-business/)). Today I want to detail why Kiwi6 can't continue to exist as "just" a side project. To give you a rough sense of scale, here are a few numbers: - Total registered users: 645,911 - Peak daily active users: 156,984 (Nov 22, 2014) - Total uploads: 6,880,357 - Peak storage (without replication): 46 TB - Peak download capacity: \~3gbps (7 content servers) - Total plays/streams served: 1,257,572,226 - Total downloads served: 122,212,054 - Total copyright takedowns: 14,428 - Completed artist & podcast profiles: 8,443 - Total paying customers: 775 Running a non-trivial scale service like Kiwi6 as a "hobby" is actually very expensive and time consuming, and numerous reasons contributed to this decision to shut down. Here are a few of them: **Server costs and maintenance**: The primary web servers, database, and background workers were hosted on Linode and DigitalOcean (eventually the database was moved to AWS for reliability reasons). The content servers were physical servers I colocated in Chicago and New York City. There were a variety of hosting fees for auxiliary services like transactional emails, image uploads, etc. Even with penny-pinching optimizations, operational costs were easily thousands of dollars a month depending on load (and the hardware cost many additional thousands of dollars + many weekends to build, configure, and troubleshoot). Keeping these up and running was not super complicated or time consuming, but it did require constant availability to respond to incidents. I occasionally had to pay for remote hands to go into a datacenter and rack a new server or swap out a hard disk that died in a RAID array -- which was unpleasant for a number of reasons, one of which was I could tell the tech was very annoyed at the cheap bottom-tier hardware I bought. Worse was when a box went down, as in the early days this meant an entire shard of files would be unavailable until the server could be repaired and brought back up. Then there was that time the alerting broke so when a server went down I didn't notice for days until I opened the support inbox. And sometimes files would just get corrupted on the disk due to hardware issues, and rsync would propagate that corruption so there were no good copies, and then a user would get really mad. I felt especially bad if it was a paid user and they didn't have a backup copy. This was all very stressful. **Service abuse**: Power users made a lot of duplicate accounts to get around the limitations of the free service. I did a few things to discourage this (both to limit our costs but also to make it inconvenient enough that some people would hopefully upgrade to a paid tier for legitimate use cases). Eventually I had to build a system to ban certain domains from referring any traffic to us (regardless of the account) because these abusive users took up almost all available resources and either drove up costs or degraded the service for everyone. We never fully stopped "professional piracy" - but I think we made it hard enough that they couldn't reliably use Kiwi6 for it. Nevertheless this all added unwanted stress and excitement. **Software updates**: The constantly changing best practices around build and deployment made it difficult to keep up with modernizing those systems while also building features to combat abuse and stay secure. Staying on top of security issues required regular updates to everything from the Linux kernel to the application code. Unfortunately, this was a constant struggle due to frameworks, libraries, languages, and platforms constantly reaching end-of-life and requiring migration just to keep doing what they were intended to do. Do you know how many times I had to rebuild the JavaScript deployment pipeline from 2009 to 2022? In order to deploy changes to backend code? Way too many times. **Copyright stuff**: Copyright takedown notices ("DMCA" takedowns) happened often enough that I had to constantly monitor for them and remove infringing files. Like every other major service in existence, I eventually built a system to automatically respond to takedown notices that would remove the file and give a strike to the offending user. This required monitoring to ensure that it didn't break (it did, occasionally). Also, sometimes people would try to abuse the DMCA takedown system or try to dispute a takedown, and those were really annoying and time consuming exceptions to arbitrate. **Accounting, taxes and back office stuff**: Back office work was a huge time sink and was basically a thread that never ended once you pulled on it. I incorporated for liability reasons, and to make it real and non-fake I opened bank accounts, credit cards, filed annual reports, and balanced the books so I could pay the correct amount of taxes and filing fees. And hiring people meant payroll taxes, unemployment taxes, and a bunch of other junk fees and quarterly reports. Also, did I mention sales taxes? I have so many stories about sales taxes. Anyways I can only debug "why my financial account syncing keeps breaking every 3-6 months, breaking my general ledger reconciliation workflow" for so long before my wife stops reminding me its 3am. **Privacy and legal stuff**: As new laws and regulations like GDPR came online, work was required to stay in compliance. There was just never enough time to get it all done, and never enough advice. Do I need a friggin' cookie banner? Can I use Google Analytics in Europe? GDPR compliance work is both a massive one-time investment and ongoing friction; leaving this work a perpetual TODO was too risky, but also the committing to this much effort didn't make personal sense for me. **Random crime/law enforcement stuff**: Occasionally, people would commit crimes and upload evidence of their crimes to Kiwi6, or use Kiwi6 in the commission of a crime. So then I'd get requests from police to share server logs several times a year. Once I got a subpoena from a state court to provide some evidence for a criminal trial. I eventually had a fax number set up for this exact purpose and then forgot to check it regularly, so they had to scan the fax and email it to me. Lol. Anyway, terrorism surveillance is a thing too; sometimes the feds ask for server logs, and it's not polite to say no. Also, people would try to DDOS the service or script kiddies would poke around doing shady stuff like automated pentests and then I'd have to get involved and ban them. **Content moderation**: I was constantly dealing with issues like harassment, spam, and professional piracy. Reviewing profile pictures for dick pics is not the sort of thing I looked forward to on nights and weekends. And often people went out of their way to be unpleasant without doing anything strictly illegal; adjudicating what was considered harassment or spam was a huge time and energy drain. For example, I was shocked to discover that people (grown adults!) would record themselves making threats or bullying others, then posting the recordings online. These were not users who could be reasoned with, so the banhammer was often necessary. Many of the behaviors on the platform were so low quality that they were basically indistinguishable from spam, and eventually the comment system was used to send actual spam. So I integrated a spam checker API and shadow banned anyone caught spamming. However, the API sometimes made mistakes, so I built buttons to mark/unmark comments as spam, and just had to click those buttons forever, or the platform would be overrun with spam. Fun times. **Viruses/malware distribution**: People kept trying to upload malware in increasingly creative ways. At first it was `bitcoin_booster.exe` so I just banned the file signatures; they eventually figured this out so I restricted EXE uploads to paid users only. Then they started subscribing with stolen credit cards so I had to turn off EXEs for everyone. So of course they switched to uploading ZIP files, and I had to severely restrict those. But then they just renamed them JPG or MP3 or whatever and...ultimately I ended up exploring a few integrations with antivirus systems but the costs were sort of prohibitive and I couldn't limit the scans to specific file types only as they kept figuring it out and renaming the files. Eventually this sort of died down because the best case for a malware distributor is their victim runs a freshly downloaded executable, and the "conversion rate" drops if they have to unzip something or change a file extension. I still got the occasional alert whenever Google Safe Browsing found something. ![Kiwi6 Support Ticket](https://www.petercai.com/assets/images/kiwi6-supp-01.png) To this day I still don't know what this guy was trying to accomplish by sending me this email. **Customer support**: I tried to provide good support, even to free users, but over time this wore me down. I regret every time I wasn't able to provide superior customer service…but people would write in with all sorts of problems; I tried my best over the years to be helpful. But with half a million users and hundreds of paid customers the support inbox sort of became a beast, and its name was Guilt: ![Kiwi6 Support inbox](https://www.petercai.com/assets/images/kiwi6-supp-02.png) In hindsight, creating a paid tier of the service meant I had crossed a point of no return, and when life got in the way of answering support emails (or vice versa), I started to realize it was irresponsible to leave Kiwi6 in this kind of stasis. Finally, the weirdest and by far the least pleasant part of running Kiwi6 was that multiple people stalked and harassed me for things like my content moderation. This included several instances of doxxing, and one bizarre incident trying to get me fired from my job over what was essentially a support ticket that spiraled out of control. I had to scrub any public association I had with the site for my own sanity. Even years later, very persistent users would occasionally uncover my unlisted cell phone number and dial me for tech support (sorry buddy, never planning to return your voicemail), and this would remind me of the general unpleasantness of the worst of that harassment. So I would like to not be harassed for taking down copyrighted music or recordings of bullying or whatever, and that was a not-small-part of my decision to shut everything down. ### Alternatives **Selling it / giving it away**: I briefly toyed with this idea, and even had some initial conversations in 2015, but quickly decided not to go this route. The number of suitable owners who could technically operate the site and would be motivated to take on such a responsibility was, in my estimation, approximately 0\. Basically, Kiwi6 is not really economical for even myself to run, and any other owner would have even worse economics because the last few years I paid myself $0 to be on call 24/7 for the systems and support, and nobody can do it cheaper than me because I know all the systems. The one possible exception is if a different owner did more unscrupulous things with the site/userbase to juice the financial returns. I didn't want to be responsible for that kind of outcome. **Automating it**: This is actually just a subset of "selling it" since there's a tremendous amount of work that goes into preparing a service like Kiwi6 to be taken over by a new owner, and a decent amount of that work boils down to "make things more resilient" and "make things break in a more predictable and obvious way" which involves a lot of boring grunt work like documentation and rearchitecting for cloud and managed services. Again, once we start putting a price on keeping the service up and outsourcing that, it becomes a money sink. And there's always a huge amount of operating overhead -- like answering support emails, dealing with abuse, debugging a broken deployment pipeline, or preparing income reports -- that can't be automated away. ### So anyway I'm grateful to the handful of people who briefly helped build Kiwi6\. We had some good moments. Thank you to those who used Kiwi6 and maybe even paid for it. It was an honor to serve you. And I'm sorry if I didn't return your email. ### Everything I learned about startups I learned from video games URL: https://www.petercai.com/everything-i-learned-about-startups-i-learned-from-video-games/ Last updated: 2022-12-10T12:00:00.000Z In the past few months, I had an opportunity to play more games than I usually do, and I've been gravitating towards a mix of team-based strategy and tactical games. It seems that there are a lot of parallels between good decision-making in games and other areas of life - I don't think this is a novel observation, but I do think there are under-explored lessons, like [reviewing your own replays to improve](https://www.petercai.com/watch-your-replays-to-level-up-your-programming/). So here is a list of ideas I've compiled focusing on the more strategic aspects of running a startup. **Identify your win conditions**: It's important to establish a theory of victory, and then construct a strategy that matches it. This includes identifying how and when you are strong relative to your rivals, and how you will convert that into wins. While this might come off as "have a plan," I think it is slightly more than that: it should describe the internal logic of your strategy and why it will make your victory inevitable and your opponents give up. A common beginner-level trap is to simply "play not to lose," in which case you can easily be put on the defensive by an opponent with a clear plan and initiative. But a more subtle trap is to come up with a strategy - aligning the ways and means with the ends - without making explicit the links between them, or *why* it will work. This makes it harder to improvise when things inevitably go off plan, because it becomes easy to lose sight of the main ideas behind why a particular strategy works, and therefore harder to adapt or salvage in the face of a determined foe. **If you have an edge, exploit it ruthlessly.** It's not enough to simply create advantages, you must set up and create situations where you get to use them. Aggressively confronting your opponents when you have an advantage pressures them and takes away resources and wins they could otherwise get uncontested. This lets you claim more resources overall and snowball faster. In contrast, if you are behind, avoid attacking into the enemy's strengths. Avoid direct confrontations and play to catch up away from your opponent - e.g., on the other side of the map or in a slightly different vertical / customer base. In general, if you're ahead: you want to speed up the game and make more plays, so you can maximize the number of times you get to use your winning position. As a silly trivial example: you want to bet on as many coin tosses as possible when you know the coin is 60%/40% in your favor. A corollary to this is that you have to use your resources to help you win. Cash on the balance sheet, hired but idle units, or finished but unlaunched features are examples of resources that aren't doing productive work. All else being equal, the player with the more efficient resource utilization converts them into wins more often. **Play around your winners**: It's usually a mistake to focus on your weaknesses to try and become more well-rounded. Because winning gets you advantages that let you make better use of additional resources, it's better to support your winning players and teams so they can continue to snowball. This lets them extend their lead, and can carry the whole organization. For the same reason, it's much better to be great at one thing than to be just "good" at all things. It's too easy to be outperformed by a team of specialists. This applies recursively, from large complex teams down to the individual level. Double down on the star players, the functions that are dominating in their roles, the winning flank in the battle, etc, and the momentum is self-reinforcing. **Don't gamble away leads**: When ahead, take fewer risks to avoid throwing your lead. For the opposite reason: when behind you should eventually try to make bigger, riskier, plays. Since you by definition want more volatility, it is advantageous to force the opponent to respond to you. Playing conservative from behind is usually a mistake as it simply perpetuates the (losing) status quo. **Reflect on probabilities, not outcomes**. It's a mistake to judge whether a decision was good or not based solely on the outcome. For example, you usually wouldn't congratulate a lottery winner for making a good financial decision to buy lottery tickets, because it's important to consider the prior odds of success, and not just whether the result was good or bad. This encourages good decision making over time that adds value, instead of being misled by random events outside your control. On a related note, it's only of limited use to analyze situations with information that was only available in hindsight, because in similar future situations we won't have the benefit of hindsight. Use information to your advantage. Gather and analyze information about adversaries to predict their movements and counter accordingly. Withhold information or actively deceive them so they can't do the same to you. **Think in terms of threats and counter threats**: It's a mistake to defend against every threat or try to compete for every objective that is contested. As in chess, sometimes a better move is to credibly threaten something else, or take something valuable the opponent has left open by committing resources elsewhere. Sometimes threatening to trade rooks is the right play. **Think in terms of timing windows**. Anticipate when you or your opponents will be strong or weak, and in what ways, and use it to your advantage to get something you normally wouldn't be able to get. Sometimes this happens unexpectedly so it is important always be prepared to take advantage of opportunities that may present themselves. The obvious timing windows for startups include product launches and fundraising announcements that can be amplified to find new customers or used in recruiting, but may also include industry/competitor developments that allow you to insert yourself as the obvious solution or follow up for some event. **Win or lose as one**. The most important principle in working as a team is to prioritize the team's goals over the individual's. This often means interrupting in-progress activities (and at the expense of personal advancement) to go help the wider team. This can be frustrating and feel slow and inefficient, but is often optimal because well-coordinated mutually supporting efforts can be much more effective than numbers alone might suggest. For example, 3 well-coordinated people is not 50% better than 2 people, since numerical superiority could result in a binary swing in outcomes (e.g. winning instead of losing some engagement or objective). ### Ruby on Rails performance tuning URL: https://www.petercai.com/ruby-on-rails-performance-tuning/ Last updated: 2022-11-10T12:00:00.000Z Reduce Ruby on Rails memory usage and decrease latency with this comprehensive collection of performance tips from almost 15 years working in Rails. ### General tips Instrument your app in production and spend the majority of your time on the endpoints that are the busiest. Due to [Amdahl's law](https://en.wikipedia.org/wiki/Amdahl%27s%5Flaw?ref=petercai.com) there's almost no reason to spend huge amounts of effort on an endpoint that (for example) your servers are only spending \~2% time on. Even if you made that endpoint instantaneous it would only represent a 2% improvement at the most. Profiling: [derailed\_benchmarks](https://github.com/zombocom/derailed%5Fbenchmarks?ref=petercai.com) is a great gem for profiling Rails to show where memory budget is going. You can also use this to do before/after comparisons to check memory savings and performance improvements from specific changes. ### Use native code Use gems like [fast\_blank](https://github.com/SamSaffron/fast%5Fblank?ref=petercai.com) and [oj](https://github.com/ohler55/oj?ref=petercai.com) (if you're parsing or emitting JSON) to reduce ruby object allocations and CPU cycles. If you're using redis significantly, especially bulk/pipelined operations, consider [ruby-hiredis](https://github.com/deepin-community/ruby-hiredis?ref=petercai.com) which wraps a client written in C. These are essentially "free" and can reduce memory pressure, as well as save you latency. ### Cache frequently used, readonly syscalls System calls can be surprisingly expensive, and their performance characteristics can even change over time. For example the `getpid` call in glibc got slower around 2017, because it was updated to stop caching results. The [PIDCache gem](https://github.com/Shopify/pid%5Fcache?ref=petercai.com) can safely cache this on the ruby side, and significantly reduce latency under linux. The readme also explains at a high level why the cache was removed from glibc, and why it is safe to cache in ruby. ### Use memory-efficient gems If larger app code changes are on the table, you can sometimes go a step further and intentionally pick leaner libraries for certain classes of problems. For example here is a comparison of memory usage among some popular HTTP libraries: | Net::HTTP | 0.16 MB | | ---------- | ------- | | RestClient | 1.72 MB | | faraday | 2.43 MB | | HTTParty | 5.57 MB | | http.rb | 7.53 MB | [Benchmark code](https://github.com/pcai/ruby-http-benchmarks?ref=petercai.com). However note that HTTP is a *really* tough example to implement in nontrivial apps, since you have to worry about transitive dependencies of all your other libraries too. ### Use fewer gems Sometimes using a gem to accomplish something is *just not worth it*. Open source libraries are by definition built to support many use cases and sometimes you just need a single 20-line class in */lib* to accomplish what you need. Watch out for convenience gems like [Hashie](https://github.com/hashie/hashie?ref=petercai.com) which patch common core classes and can contribute a lot of memory usage. ### Don't load gems you don't need Make sure you aren't accidentally loading development/test gems in production mode - make sure they are added to the intended `:group` in your Gemfile. This can be the cause of wasted memory or CPU cycles, or worse, security issues. Sometimes you need a gem but only in a rake task or a background job. In that case you should make sure its not automatically loaded in your Gemfile: ``` gem 'whenever', require: false ``` Then in your rake task, manually require the library: `require 'whenever'`. This will prevent the gem from being loaded into the server process on app boot, saving memory. Note that this trick is only effective in production when the dependent code isn't in your eager load paths. ### Turn off unused Rails components Rails by default comes with many libraries that your app is unlikely to be using, and which may represent moderate memory savings if disabled. You can disable them from loading and only require the ones you need. For example, maybe your app doesn't use ActiveStorage, ActiveJob, and ActionCable. So don't load them. In `config/application.rb`: Replace `require "rails/all"` with: ``` # active_storage/engine # active_job/railtie # action_cable/engine # action_mailbox/engine # action_text/engine # rails/test_unit/railtie %w( active_record/railtie action_controller/railtie action_view/railtie action_mailer/railtie ).each do |railtie| begin require railtie end end ``` As you can see, I like to keep all the available railties commented out so they can be easily re-enabled later if needed. Once you've removed some of the railties you may need to remove the corresponding configuration options to get the app to boot again. Warning! You can see the effect of this change using derailed\_benchmarks, but only if you use **dynamic** app benchmarking. The static benchmarking won't notice this change because it only analyzes your Gemfile, so it has no way of knowing that not all of Rails is loaded. ### Offload static assets to a CDN Ideally, don't serve static assets from your web server at all, and put a CDN like Cloudflare or AWS Cloudfront in front of it. This saves you CPU and IO cycles to serve dynamic requests. ### Minimize rack middleware Rails is a rack app. Rack is a ruby web application interface, and works using middleware apps that work in a "pipeline" configuration. Each middleware acts as a linked list node that wraps the next node: it runs some code and calls the next piece of middleware. Rails is the last code to run. Note that this design means order matters! Some middleware (such as static fileserving, caching, or DOS protection libraries) may decide to return a response early without calling some of the later middleware, which means Rails and other middleware can get skipped. Here's an example command to check what middleware is loaded in production for a sample rails 7 app: ``` $ SECRET_KEY_BASE=foo RAILS_ENV=production rails middleware ``` ``` use ActionDispatch::HostAuthorization use Rack::Sendfile use ActionDispatch::Executor use Rack::Runtime use Rack::MethodOverride use ActionDispatch::RequestId use ActionDispatch::RemoteIp use Rails::Rack::Logger use ActionDispatch::ShowExceptions use ActionDispatch::DebugExceptions use ActionDispatch::Callbacks use ActionDispatch::Cookies use ActionDispatch::Session::CookieStore use ActionDispatch::Flash use ActionDispatch::ContentSecurityPolicy::Middleware use ActionDispatch::PermissionsPolicy::Middleware use Rack::Head use Rack::ConditionalGet use Rack::ETag use Rack::TempfileReaper use Warden::Manager run MyApp::Application.routes ``` In a default Rails project, there is a handful of middleware I like to turn off even though they don't contribute significantly to latency (ActionDispatch::RequestId, Rack::Runtime, etc). In `config/initializers/middleware.rb`: ``` Rails.application.config.middleware.delete Rack::Runtime Rails.application.config.middleware.delete ActionDispatch::RequestId ``` Gems commonly add middleware. Examples include telemetry middleware, DOS protection like [rack-attack](https://github.com/rack/rack-attack?ref=petercai.com), and popular gems like [omniauth](https://github.com/omniauth/omniauth?ref=petercai.com) (which is why you don't need to edit your routes file to use it). Omniauth is particularly bad because each individual provider is a separate middleware, so if you have, say, 50 of them, this ends up becoming a significant overhead on EVERY Rails request, even though those 50 middlewares are no-oping most of the time. One solution I've written for this was a hacky piece of rack middleware that would check the request path and skip all the omniauth middleware if the request couldn't possibly be an oauth request. This saved something like 10ms on every request. ### Optimize routes Rails attempts to match the current request method and path against every route defined in your routes.rb file from top to bottom. This has two implications: 1. Fewer routes is better. So clean up dead routes, and avoid declaring routes that don't lead anywhere (e.g., don't use the shortcuts that generate 7 routes at a time if you really only need 1 or 2; use the `:only` option to avoid bloat). 2. The order of your routes matters. Ideally, your routes are laid out and grouped in some logical way. However, for performance reasons, you actually want to sort your routes so the most popular endpoints appear first. Beware, of course, that the order of routes is important and can affect correctness. You can audit your routes with `rails routes`. In my own benchmarking, this was not a big factor usually, but if you're looking to squeeze out the last ms from everything, it's worth considering if you have an app with hundreds of routes. ### Optimize views Don't use [Haml](https://haml.info/?ref=petercai.com), Slim, or related libraries. They are much slower than ERB and use more memory. NEVER use inline coffeescript, a feature of Haml and some other gems - this will cause the coffeescript compiler to boot and run for EVERY page render that has inline coffeescript. Minimize heavy use of partials and helpers: these add extra overhead that show up when you're chasing those last few milliseconds. Common culprits like `link_to` start to add up if you call them a lot. ### Optimize database schema Generally speaking, make sure columns referenced in WHERE clauses are indexed. There are rare exceptions to this rule: for example, when your write load is much higher than the read load, and you don't care about the performance of the (rare) read queries. In that case unnecessary indexes will just take up disk space and significantly hamper write performance. Use compound indexes if two columns are always referenced together in a WHERE clause. While this is not common, this will be slightly more efficient than using 2 indexes. Minimize your 1-1 relationships and their associated joins. Most 1-1 database relationships can and should be consolidated into a single wide table for better performance. This eliminates an entire redundant primary key and the need to join 2 tables. And if you avoid `SELECT *` you don't pay a query penalty where you don't need the columns. One (usually manageable) downside is that this can make the application code a bit messier if the two tables really did represent 2 distinct logical entities. ### Run fewer, faster SQL queries Make your database queries fast. This generally boils down to a) use indexes appropriately, and b) minimize data stored and fetched. **Avoid duplicate SQL queries**: with associations, make sure you are specifying :inverse\_of when necessary to avoid fetching the same row more than once in a given request. If you reference or load the same model more than once, make sure you're not going to the database multiple times by explicitly reusing the reference in your code. This includes counts! **Avoid relying on the** [**ActiveRecord query cache**](https://api.rubyonrails.org/classes/ActiveRecord/ConnectionAdapters/QueryCache.html?ref=petercai.com): it's *commonly* a code smell to rely on this. It hides sloppy coding and design, and it wastes memory and cycles populating/checking/flushing the cache. Consider [the example given in the Rails guide](https://guides.rubyonrails.org/caching%5Fwith%5Frails.html?ref=petercai.com#sql-caching): ``` class ProductsController < ApplicationController def index # Run a find query @products = Product.all # ... # Run the same query again @products = Product.all end end ``` Sure, this allows `Product.all` to be called twice and only result in one SQL query. Or...Just fix your duplicate SQL queries, and disable the query cache. There are limited exceptions to this, based on how your app is architected. Sometimes a plugin system means that lots of subsystems are (by design) completely isolated from each other, so not all code knows what other code is going to run. In that case, it might make sense to use the query cache in your app. **Don't filter results from the database in app code**: whenever possible, use WHERE clauses to fetch only the records you need. It's typically a code smell to load a bunch of records from the DB and then filter them in memory because it's doubly bad: You've not only made the database fetch and transmit extra records you weren't ultimately going to use, you also made yourself extra work in ruby code filtering it out and allocating a new array to copy the results to. **Find and fix your n+1 queries**. Using `:includes` to preload associated models on index pages will most likely lead to large performance gains by reducing roundtrips to the database. The [bullet](https://github.com/flyerhzm/bullet?ref=petercai.com) gem can help with this. **Don't load associations unless you really need them**. Instead of `@book.author.present?` you can sometimes get away with `@book.author_id.present?` instead, which might save you a database call. This pattern is also a frequent source of n+1 queries. The tradeoff with this approach is that the association might be deleted or invalid, but this might not be a real problem depending on whether you have added foreign key constraints, what isolation level your database is running at, and how your app is designed. **Minimize roundtrips to the database**: Often (but not always!) it's better to return all your data in one roundtrip by JOINing the data you need in related tables instead of fetching in another query. This is true even if you've optimized away an n+1 situation. **Use `:pluck` to fetch data from the database if you don't need ActiveRecord models**. This will save you both memory and CPU cycles by not constructing heavyweight ActiveRecord objects if you just need primitives (like an array of strings or an array of numbers, etc). This can provide a minor performance improvement. **Avoid `SELECT *`**: It's better to fetch exactly the data you need rather than all columns. Using `SELECT *`, which is the default, ends up being slower in at least 4 different ways: - More work for the database to fetch unused columns - Takes more time to transmit the response over the network as there's more data - Takes longer to map the result to ruby data types and construct the objects in memory - More work for the garbage collector to clean up later So don't be lazy and specify exactly the columns you need in your hottest code paths: ``` @users = User.select('username, email, id').where(...) ``` Note, however, that this doesn't matter in the majority of codepaths. **Don't COUNT unless you need the count**: Sometimes you want to know if a collection is empty or not, but you don't actually need the precise count. In that case, use `exists?` instead of `count`. This is faster because the database can return early once it knows the result is nonempty. **Never load an entire table into memory:** When iterating over any collection, you *never* want to load all the records into memory at once. Instead, you should use `find_each` or `find_in_batches` to iterate over the records in batches. This will reduce the working memory needed for the request, often significantly. For example, if you have 10 million records, you can iterate over them in batches of 1000 at a time, instead of loading them all into memory at once, which is a *10,000x reduction* in memory use. This can be a sneaky problem because its hard to catch unless you're testing with sufficiently large test data in dev or pre-production environments. You can use [safe\_query](https://github.com/pcai/safe%5Fquery?ref=petercai.com) to catch unintentional queries like this in development. There is at least one major caveat with `find_each` and `find_in_batches`: sorting is ignored. Under the covers, Rails implements the required batching and pagination by sorting on the primary key. If you need to load a collection in batches and a custom sort is required, you will need to reimplement this behavior yourself (and you'll want to be extra extra sure that your sort key is indexed). There are no exceptions to this rule. Sometimes inexperienced developers do this accidentally, e.g., "I want to sum up all the pageviews of all these pages." Do the work in the database: `SELECT SUM(page_view_count) from pages WHERE ...`. ### Cache expensive operations Using redis or memcached to cache content and completely avoid SQL queries is usually worth it, but you'll want to cache in the largest chunks you can to minimize roundtrips. When talking to Redis or Memcached it's critical to avoid n+1 issues (just like with your SQL database), which can be more subtle and easier to miss because the roundtrips are more lightweight than SQL queries, and don't show up in the development logs by default. Caching parts of a page involving multiple records can be nontrivial, as it requires calculating a cache key so that you can correctly invalidate the cache at the right time. Be sure that gathering what you need to calculate this does not exceed the benefit of the actual caching, which requires benchmarking and profiling. In specific scenarios that require maximum absolute performance, it might make sense to consider `ActiveSupport::Cache::MemoryStore`. This keeps cache entries in memory in the same Ruby process as the Rails server, which is extremely fast since there's no network boundary to traverse. This has a couple significant tradeoffs: - Your server process will obviously consume more memory, which might not be desirable. - If you're running multiple server processes, they won't share cache data with each other, which may or may not be OK depending on a number of factors like your scale and use case. - As a corollary, memory might be wasted caching the same data in multiple processes. - Cache data will be wiped on reboot, which might be unacceptable, as it could lead to a thundering herd problem. Make sure you understand the `race_condition_ttl` and enable it, as under high load this can cause a big load spike right when a popular cache key expires. ### Reduce log verbosity Lower the log level setting in production to the absolute minimum you need. After that, consider consolidating extraneous logging calls to make the code less chatty overall. This a great way to get some free performance back. At sufficient scale this will lead to cost savings in your logging system too. ### Tune server settings The current Rails default web server, puma, is fantastic when correctly configured. Usually your goal is to get the best performance per dollar spent, and to do that you'll want to maximize CPU utilization per MB of memory budget, since Rails is notoriously memory-hungry. Under MRI, each process has a global interpreter lock, which means only one thread can run on the CPU at a time; therefore it can only saturate a single CPU core at most. So under MRI, you will want about 1 process per dedicated physical CPU core. Consider this simplified table: | | Free memory is low | Free memory is high | | -------------- | ------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------- | | CPU usage low | Reduce process count. Increase threads until memory is exhausted. Memory may be underprovisioned or CPU overprovisioned. | Increase thread count until latencies begin increasing, then increase process count if CPU usage doesn't respond to more threads. | | CPU usage high | Ideal | Memory is probably overprovisioned. | The number of threads you need will depend on your application characteristics, mostly the amount of I/O your application is doing. Most Rails apps spend I/O in 1) database calls or 2) networking calls. Generally, you should add threads to increase CPU usage and keep the processor busy while other threads are waiting on I/O. However, additional threads will use more memory since they each have their own stack and will allocate memory to run controller code and render views. Context switching between threads is expensive, and minimizing it allows the time to be better spent making forward progress serving requests. Optimize your database calls to reduce the number of threads you need to keep the CPU busy. Usually your database connection pool should exactly equal the number of threads in your Rails process (both are controlled by `RAILS_MAX_THREADS`). This is the default. You will obviously need to stay below the maximum connection limit your database allows. You may be able to raise the limit, but if your database is the bottleneck to your application performance (e.g., its either constrained by CPU, IO, or memory) then this will not help, and will very likely make things worse. If your database is not the bottleneck and the connection limit has not been reached, you can add more Rails processes / threads to continue scaling up traffic. If you're only running one process because you don't have multiple cores available, make sure you boot puma in "single" mode so it doesn't spawn two separate processes which will waste memory for no benefit. ### Tune memory allocator Try tuning glibc's memory allocator behavior to [trade off between memory and latency](https://devcenter.heroku.com/articles/tuning-glibc-memory-behavior?ref=petercai.com) with `MALLOC_ARENA_MAX`. Increase the number of memory pools to reduce lock contention from threads, or reduce the number of pools to minimize memory fragmentation and waste. Most configurations benefit from `MALLOC_ARENA_MAX=2` as it represents a good tradeoff between space wasted and contention -- 1 causes lots of bottlenecking without saving much memory, and more than 2 often wastes memory without significantly improving lock waits. Also, consider using [jemalloc](https://github.com/jemalloc/jemalloc?ref=petercai.com) as the allocator, which often significantly reduces memory use with an algorithm that minimizes fragmentation. With the [LD\_PRELOAD flag](https://github.com/jemalloc/jemalloc/wiki/Getting-Started?ref=petercai.com) you can use jemalloc without recompiling ruby. Here is an example Heroku buildpack that [installs jemalloc into the compiled slug](https://github.com/mojodna/heroku-buildpack-jemalloc?ref=petercai.com). ### Take advantage of copy-on-write If you're running multiple puma processes in a cluster, make sure you preload the app before forking. This loads all framework and application code first before creating new processes, taking advantage of [copy-on-write](https://en.wikipedia.org/wiki/Copy-on-write?ref=petercai.com) semantics to share the Rails and application code in the same physical memory among multiple processes. Because of this copy-on-write semantic, in general, it is best to fork as many puma processes as possible on the smallest number of machines/containers: | \# of processes | app code in RAM / # processes | Memory saved (vs 1 proc / container) | | --------------- | ----------------------------- | ------------------------------------ | | 1 process | 1 | no reduction | | 2 processes | 1/2 | 50% reduction | | 3 processes | 1/3 | 66% reduction | | 4 processes | 1/4 | 75% reduction | For example, it is usually a win to run a single container with a cluster of 8 puma processes, as opposed to 2 containers each with a cluster of 4 puma processes. This will be constrained by your high-availability needs though. ### The market for junior software engineers is completely broken URL: https://www.petercai.com/the-market-for-junior-software-engineers-is-completely-broken/ Last updated: 2022-10-11T12:00:00.000Z To say that the US market for software engineers has been broken is something of an understatement. For junior and entry-level engineers, the competition for jobs is intense and miserable -- but many who break through have the strange experience of transitioning from constantly-shunned to perpetually hounded by recruiters in the span of about 2-3 years (hiya Amazon recruiters!). But many would-be entry-level engineers have few opportunities to build the experience that makes them hirable: a classic catch-22. To try to roughly quantify how bad this is, I looked briefly at a sample of several dozen job postings for entry level software engineers (e.g., no prior professional experience required) on LinkedIn over 24 hours and found the following: - Average applications per job posting: 556 - Average applications per week, per job posting: 439 - Average applications per week, per job posting (role is explicitly full remote): 840 - Average applications per week, per job posting (company is a household name): 1,228 These are pretty ridiculous odds - and there's a lot of anecdotal evidence that LinkedIn is uniquely bad - a death spiral of high volume spammy applications AND low quality job postings that never actually result in interviews. But the point remains. To get a sense of how dire this actually looks from a candidate's perspective, I talked to some people in my network + some people online to collect a few dozen data points from recent grads and boot campers. Here's what the average funnel looks like across all the anecdotes I collected: - Applications: 171 - Interviews: 3 - Offers: 1 I double checked these numbers because of how ridiculous they were. And this was a group that was heavily tilted towards those graduating with a CS degree, not majority boot campers. For comparison, I applied to 4 firms out of college and got 3 offers before withdrawing from the 4th interview, and many people in my social circles did similar single-digit-numbers of applications. It seems things have gotten much worse since then! Reddit threads echo with cries of "Why don't companies want to *invest in training anymore?*" This is a great question. Let's dig in. The typical calculus for an employer that needs to hire goes like this: 1. Spend a bunch of money recruiting and hiring a new employee. 2. Spend months (or years) training this employee to be productive and begin generating more value for the business than they cost. 3. Eventually this employee will generate enough value that it pays back the cost of the expensive things you did in step 1 and 2. 4. Hope they don't leave before that, because if they do you have to start over from step 1, *and you're worse off than before you started*. 5. If this happens often enough you go bankrupt. Worse, people might make fun of you on TechCrunch. It gets worse though. At current market salaries and ramp up times, entry-level software engineers are probably the most expensive employees with simultaneously the longest time-to-productivity at a company, and on average the industry thinks this takes -- working backwards from revealed preference -- on the order of 1-2 years. During that time, their value to the organization is often considered *at best* a wash \[1\] - reflecting more future potential than current capability. And so I don't think its unusual for hiring managers to visualize the ROI of hiring engineers like this: ![graph of cumulative return on hiring entry level and experienced engineer](https://www.petercai.com/assets/images/roi-entry-level-engineer.png) Of course, this is a highly stylized hypothetical - many competent entry-level engineers become super productive very quickly, and I'm guessing most firms plan to recoup their investment in junior engineers in far less time than the \~2.5 years it takes in this example. But this graph doesn't have to reflect reality so much as broadly suggest - in a very directional way - a belief in a particular probability distribution. If we're arguing about WHERE the blue line crosses 0, then we have already conceded an important point: > If I'm hiring, it's better business if *someone else* trains junior engineers **and then I hire them away**. Many firms act like this is true and mostly hire only experienced engineers. Because that's what's rational if they broadly buy into this theory of ROI. The final piece of the puzzle is the "changing nature of work." Employees job hop much more often now than they used to; they are effectively incentivized to do so because new jobs usually come with big raises (and I guess it is perfectly rational because companies can usually outbid current employers by virtue of not having paid any of their prior training costs). Software engineers have an average tenure of about 2 years; employers expect this, it is a known thing. So they conclude that years-long training and development investments don't make sense. It's all just a big repeated game of prisoner's dilemma but with a different collaborator each iteration: the best move is to defect, *because defecting is the industry norm*. ## Solutions TL;DR: I think we broadly need a better way to shorten payback periods and de-risk onramps into the industry so companies are willing to create those entry level roles; the existing solutions either have serious drawbacks or haven't solved the problem effectively: ### Lowering entry level salaries One intuitive reaction I had is that junior engineer salaries need to go down for the classic econ 101 reason that there are clearly far more unemployed entry-level engineers than there are jobs for them, and current salaries are very attractive, with lots of people who will happily work for way less than the prevailing wage. | Entry level software engineer base salaries | | | ------------------------------------------- | ----- | | Bottom 10% | $59k | | Median | $78k | | Top 10% | $112k | Data from [PayScale](https://www.payscale.com/research/US/Job=Software%5FEngineer/Salary/4fd947de/Entry-Level?ref=petercai.com) (Sep 2022) Theoretically, this change would fix some of the supply imbalance: some aspiring engineers would decide to become real estate agents instead, and some firms would decide they can make the math work on a junior engineer at $55k (but not $85k). Unfortunately, entry-level salaries have in fact gone *up* over time...probably due to some combination of: - **FAANG**: Top US tech companies have incredible leverage, and can make A LOT more money from incremental engineering hires, including entry level, than most other companies \[2\]. Many of them run college recruiting programs precisely because they're so uncommonly profitable at scale that small optimizations result in big gains; their payback period on entry level engineers is shorter than average and their time horizons are much longer so the urgency to make sure every hire is productive ASAP isn't as high. Also, there's a small subset of people who basically never leave their jobs, so by definition you have to compete for entry-level employees if you want a chance at those people. This pushes salaries up, since FAANG can comfortably afford to scoop up the top 10% of every cohort and shift the entire demand curve. - **The Alchian-Allen effect**: Hiring is so expensive in general for most firms that if there's budget to hire at all, it might as well be an experienced engineer, because the productivity difference is really high, but the difference in total cost between a senior and junior hire can be *surprisingly low* given high fixed overhead (health insurance, hardware, software, office space, management, benefits, taxes, etc. are expensive + don't vary much by experience level of the hire). Because of this, its not uncommon at many firms to budget using headcount, and if headcount is your limiting factor, you aren't going to "waste" it on junior hires. - **Cross-elasticity of demand:** Demand is so high that not everyone can meet their hiring quotas for senior engineers; this generous budget is redirected downward as firms "settle" for hiring more junior engineers than they ideally wanted. This pushes junior salaries up. So salaries stay really high. And any individual firm that offers lower salaries drives away the best applicants, and has to sift through a candidate pool with low signal to noise ratio. Also, I forgot to mention this industry sucks at identifying and selecting talent in interviews, so there's no good way for most firms to cost-effectively assess whether a candidate with 6 years of experience is worth $105k or $450k total comp, nevermind entry level applicants with no work history. So we are probably stuck in this weird spot until some of the above things change. ### Create better materials for self-teaching Self-taught engineers are not unheard of, and all the materials to become a competent software engineer are technically online for free to anyone who cares to look, but we haven't reached a point yet where most people can legitimately break into the industry from self-learning. Without a credential of any sort, its just too hard to stand out from the (very large) crowd. And again, software interviews are so bad that its a running joke in the industry, and salaries are so high that most companies can't afford to interview every random who walks in off the street claiming to know Java without any credentials or formal training (remember: engineers are interviewed primarily by more senior engineers and engineering managers, so discovery costs are extremely high). So credential gating persists; self-teaching isn't so much the problem as motivating students and creating cost effective ways for prospects to demonstrate skills and competencies. ### Fix college education College degrees are expensive and time-consuming to obtain, and the recent economic pullback is suggesting the market can't even absorb all the current CS graduates. CS degrees (by design) just don't teach a lot of skills professional software engineers need to be productive right away, so I doubt there's much room for dramatic improvement here in the short term. Also, lol ### Coding boot camps (students pay tuition) In contrast, coding boot camps were a neat idea with limited initial success, but they rapidly ran through the entire adult population of "has English degree and would be a decent engineer at Google, but currently handling accounts receivables at a restaurant chain due to happenstance." The average person can't learn all the theory, practical skills, and instincts in 12 weeks with no prior background or interest. You just can't. Not in a repeatable way. Also, tuition costs on the order of like...$15 grand or more. There's some innovation here, with income-share-agreements and nonprofits jumping in, but on balance they're making the junior engineer problem worse, not better, by churning out job seekers with few skills and high expectations. ### Industry boot camps (company pays nominal salary) Wayfair and a few other companies had interesting experiments with a salary-paying training program geared towards boot camp grads: It was called Wayfair Labs, and my understanding is they used the program to both train boot campers in engineering skills, and also selected the most promising graduates for full time offers. I suspect these programs aren't sustainable and don't scale because they just aren't that attractive to the boot camp grads: the best engineers don't want to spend 3 months training full time to have a shot at an offer from a single company when they can just…keep interviewing with a lot of companies simultaneously on a rolling basis and skip right to the make-a-lot-of-money part. And if you aren't hired at the end of the program, it's actually perceived as a black mark in a "If even Wayfair didn't hire you after so much training, then why should I?" sort of way. Basically, industry boot camps create an adverse selection problem over time (Wayfair only gets the boot camp grads who couldn't directly get a full time permanent offer anywhere) as well as a signaling problem (those who join the program run the risk of making their resume look worse than if they hadn't attended, if they fail to get or accept an offer from Wayfair at the end). ### Internships Internships seem to be moderately effective, and are correlated with successful outcomes (and often 1 order of magnitude fewer applications to land a job, based on the data I collected), but these are generally only available to college students, and the peculiarities of US higher ed makes this difficult to adapt outside of university programs -- the biggest benefit is that internships end due to the student's unambiguous obligation to return to school, and this provides plausible deniability that it wasn't due to unsatisfactory performance leading to inability to convert to a full time position. Again, this avoids a signaling problem when the candidate applies to full time jobs. ## The side door My current best idea so far for how to fix some of this is to formalize career ladders into software engineering from tangential disciplines like "Solutions Engineer" and "Technical Support Escalation Representative." It's not a pure win because it'd be a hassle for those departments to keep having to replace folks, but these roles are natural onramps because they often involve scripting or coding of some kind, interact regularly with the real engineering team, and crucially, provide a huge amount of context, relationships, and product knowledge that shortens the onboarding period when they transition to engineering. More companies should try this! I hope more candidates consider this route. I hope more companies build these sorts of opportunities into their organizations. I dunno. Otherwise we are squandering an opportunity to grow our field. \[1\] Many people have pointed out to me that this is partly because junior developers suck up a lot of bandwidth from senior engineers/managers for coaching and mentorship - to emphasize: I heard a lot of "hiring junior engineers makes teams slower, not faster." And thus it additionally makes sense why larger companies are more amenable to junior engineers than startups: startups want to move fast. \[2\] There's a lot of evidence that this is true: most other countries think top US tech salaries are shockingly, outrageously high, but that's because they don't have as many hugely profitable tech companies competing for talent (yet), although the arbitrage is closing in recent years. ### Workarounds for hiring discrimination in tech URL: https://www.petercai.com/workarounds-for-hiring-discrimination-in-tech/ Last updated: 2022-09-04T12:00:00.000Z Venture capitalist Marc Andreessen famously argued to New York Magazine that systemic discrimination in Silicon Valley can't exist because of how competitive hiring in tech is: > Let's launch right into it. I think the critique that Silicon Valley companies are deliberately, systematically discriminatory is incorrect, and there are two reasons to believe that that's the case. ... No. 2, our companies are desperate for talent. Desperate. Our companies are dying for talent. They're like lying on the beach gasping because they can't get enough talented people in for these jobs. The motivation to go find talent wherever it is unbelievably high. [Dan Luu](https://danluu.com/?ref=petercai.com) has written very convincingly about why discrimination in tech hiring exists despite this "efficient markets" argument. The efficient markets hypothesis comes from the field of financial economics, and roughly states that market forces cause all available information to be priced into assets. The irony is that it doesn't even hold up in its original field, never mind labor markets: there continues to be an entire industry of people who reliably make profitable trades in financial markets based on publicly available information. Likewise, hiring discrimination persists, despite US laws for protected categories like race and sex. In addition, the literature reveals persistent biases that generally have no legal protection whatsoever, which we'll get into in a bit. To be clear, discrimination shows up in the data. And it persists despite decades of evidence and efforts to address it. For example, annual Stack Overflow surveys of software developers consistently show fewer than 10% of respondents identifying as female, and they are systematically paid less and receive fewer interview callbacks than similarly qualified male applicants in job postings. Eliminating these biases would be not only a "feel-good" measure (e.g., because "it's the right thing" for tech companies to do), it is also unambiguously a pure win by conferring a huge competitive advantage to firms where it counts: recruiting the best people. Let's dive into discrete biases for which we have evidence, and some obvious workarounds that come to mind: ## Names ### Names that encode or imply race Black sounding names get fewer interviews than identical resumes with whites-sounding names \[1\]. ### Names that suggest gender Tech recruiters in one study were 6.47% less likely to reach out to women with comparable qualifications than men, even though female programmers who self-report programming language experience have more experience on average than males. This appears to be partly because women are 11% less likely to self-report knowing a programming language despite working in it. This causes large disparities in outcomes, since candidates with self-reported programming language experience are \~30% more likely to be recruited \[2\]. A study of resume reviews of college seniors of all majors suggested no discrimination (on average) on account of race or gender, but for STEM fields employers displayed "large, statistically significant preference" for white males, equivalent to a penalty of 0.27 GPA for female or nonwhite candidates \[3\]. ### Alphabetical ordering A new research paper suggests alphabetical sorting of names significantly affects early careers \[4\], which disappears by mid-adulthood, "presumably because it is superseded by observable characteristics that are more directly expressive of ability." "Surnames with initials farther from the beginning of the alphabet were associated with less distinction and satisfaction in high school, lower educational attainment, more military service and less attractive first jobs." ### Hard-to-pronounce names A study of PhD students and job outcomes found that hard-to-pronounce names were less likely to be hired in academic or tenure-track positions, and when they were hired, took less prestigious posts overall \[5\]. A longitudinal study of immigrants from the early 20th century shows they experienced larger "occupational upgrading" and higher earnings if they adopted Americanized names \[6\]. **Takeaway:** if you have a "weird" foreign name, a "black" name, or anything that doesn't code as white anglo-saxon male, go ahead and pick an alias that works around these issues. No one should discriminate because they subconsciously think your name is weird, but why give them the opportunity? ## Social class ### "High class" hobbies Resumes from selective law schools to elite law firms showed a higher callback rate when they contained multiple signals of "high class." This included a "high class" last name, and references to sailing, polo, and classical music. This effect only helped men, not women. High class women, lower class women, and lower class men with identical qualifications had a lower callback rate \[7\]. In free-form interviews, almost all attorneys reviewing resumes worried about the "commitment" of "high class" women, worrying they might "leave paid employment entirely." I also laughed a little at a reviewer verbalizing their real-time rationalization for why those hobbies are good signals: > Mark said of the higher-class man: "If you look at the interests, it's classic cultural capital. It would help with being around people who \[he pauses\] work hard." Ah yes, *sailing*, the hobby for the hardest-working of Americans! This study is worth a read and presents more shades of gray than I am letting on here - but the general takeaway is that "cultural fit" (as in having the same prevailing hobbies and interests at a firm) is almost certainly a huge leg up. **Takeaway**: Don't name interests or hobbies that are associated with a "low class" status (e.g., "hoops"); consider picking up and listing hobbies associated with "high class" status (classical music, etc). Actually, you should probably just aim to try everything once and then list the "right" ones on your resume depending on the culture of the firm you are applying to \[8\]. ### Speech and accent Hiring managers who watched a brief "tell me about yourself" video devoid of qualification information picked up on class cues, and judged higher social-class applicants as more likely to be competent for the job, a better fit, and more likely to be hired \[9\]. Some other interesting findings in the research paper, including pronunciation patterns as cues of social class. Listeners could discern from 7 out-of-context words whether a speaker was a college graduate or not, with better-than-random probability (albeit only 55% accuracy). **Takeaway**: Write and speak using "standard english" as might appear on the largest US websites, or like a college graduate. ## Physical Attributes ### Height Employers are reluctant to hire short applicants, and perceive taller applicants as more competent. Additionally, short people are promoted at considerably lower rates than taller peers \[10\]. All else being equal, people seem to prefer to be led by folks who are literally taller. So, basically: Wear a pair of shoes / hairstyle that increases your perceived height. ### Facial dominance The ranks of cadets entering the Military Academy at West Point are known to be related to "facial dominance" (essentially, how masculine the facial features are) based on ratings of graduation photos at West Point. Further research revealed that officers with more facial dominance were also promoted faster, earned more promotions in total, and were more likely to reach the General ranks (Brigadier General, Major General, Lieutenant General, etc) \[11\]. The methodology in the linked paper is pretty impressive - it attempts to control for a large variety of factors to isolate the effects of facial dominance. They also adjusted for whether the candidate was smiling in the photo (which is known to lower ratings of facial dominance). | ![Face with dominance score 6](https://www.petercai.com/assets/images/dominance-score-6.png) | ![Face with dominance score 3](https://www.petercai.com/assets/images/dominance-score-3.png) | | -------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- | | This cadet had a facial dominance score of 6 (out of 7) | This cadet had a facial dominance score of 3 (out of 7) | We select leaders, in part, by how much they literally "look like a leader." Or more accurately, our preconceived notions of a what leader should look like. So, I dunno, I'm not like a makeup expert or anything, but it can't hurt to emphasize how manly your face is, or something like that. ### Weight Obese and overweight women make less money \[12\] \[13\]. The effects are somewhat uneven as the effect appears reversed for black men (emphasis added): > For a two standard deviation increase in weight (64 lbs.), white women are paid 9 percent less. Black men seem to receive *higher* wages with higher weight, and Hispanic men incur a wage penalty. So appear tall and don't be overweight. Got it. ### Age Older job candidates are perceived as less desirable \[14\] and get fewer responses to job applications \[15\]. And when you adjust for experience and education, older workers make *less* than younger colleagues \[16\]. **Takeaway:** if your resume implies you are 50+ (but probably sooner), start truncating some older experience and drop any graduation years from your education. ### Attractiveness Attractive people make more money \[17\] \[18\]. Attractive men and women are more likely to be called when applying for jobs, but the starkest difference seems to be for unattractive women, who are heavily penalized. A set of studies with seemingly conflicting results seems to boil down to: If you are an attractive person, try to make sure the person judging you is the opposite sex. **Takeaway**: Be attractive. Don't be unattractive. ## This is weird. I'm generally against being deceptive or manipulative for personal gain, but is it deceptive if you intentionally mask an attribute that shouldn't be a factor in a hiring decision? It's not like your interviewers would be able to consciously object - "I offered you the SQL developer job on the basis that you were taller" just doesn't come out right? It should go without saying that the existence of these biases is unfortunate, and the onus shouldn't be on candidates to work around them. But I just wouldn't blame anyone who adopts any of these workarounds (or similar ones) to their personal benefit. \[1\] https://eml.berkeley.edu//\~crwalters/papers/randres.pdf Involved 123 firms from the Fortune 500 for a total of 84,000 job applications. Methodology was "audit" of job postings involving submission of fictional resumes \[2\] https://pubsonline.informs.org/doi/10.1287/mnsc.2021.4077 . Sample size: 170,886 candidates, 7% female \[3\] https://faculty.wharton.upenn.edu/wp-content/uploads/2018/09/KesslerLowSullivan\_Revision1.pdf \[4\] https://www.colorado.edu/faculty/zax/sites/default/files/attached-files/alphabetism\_-\_cauley\_and\_zax\_-\_6-26-17.pdf \[5\] https://papers.ssrn.com/sol3/papers.cfm?abstract\_id=4031991 \[6\] https://www.journals.uchicago.edu/doi/abs/10.1086/692531 \[7\] http://www-2.rotman.utoronto.ca/facbios/file/RiveraTilcsik.pdf The researchers specifically picked law because it's a field requiring education and credentials; its unclear if this generalizes to other fields like STEM. \[8\] If you try something once, I don't feel like listing it as a hobby on a resume is a lie. And if someone brings it up in an interview, you'll have some experience at hand to talk about. \[9\] https://www.pnas.org/doi/epdf/10.1073/pnas.1900500116 \[10\] https://opencommons.uconn.edu/cgi/viewcontent.cgi?article=1432&context=law\_review \[11\] https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.878.6872&rep=rep1&type=pdf . Results over 289 career officers who were in service for 20 years or more (e.g., analyzing only those who made a life commitment to the career) \[12\] https://www.bls.gov/osmr/research-papers/2007/pdf/ec070130.pdf \[13\] https://papers.ssrn.com/sol3/papers.cfm?abstract\_id=2379575 \[14\] https://academic.oup.com/esr/article/37/1/49/5900768 \[15\] https://www.tandfonline.com/doi/full/10.1080/13504851.2011.581199?src=recsys - or if you believe https://www.tandfonline.com/doi/full/10.1080/09585192.2011.540160?src=recsys, age discrimination starts in the late 30s. \[16\] https://www.tandfonline.com/doi/abs/10.1080/08874417.2004.11645817 - the abstract is really good, but unfortunately I couldn't find a free version of the paper itself to read. \[17\] https://press.princeton.edu/books/paperback/9780691158174/beauty-pays \[18\] https://journals.sagepub.com/doi/10.1177/0022185608096808 ### Watch your replays to level up your programming URL: https://www.petercai.com/watch-your-replays-to-level-up-your-programming/ Last updated: 2022-08-06T12:00:00.000Z In video gaming, its standard advice to record your gameplay and review your own games to spot mistakes and areas for improvement. Reviewing recorded games is extremely important to improving because: - You can spot game-losing tactical mistakes, and practice the specific sub skills that you need to avoid those mistakes. This might be a mechanical mis-play, or a lack of coordination or awareness about where teammates are / what they were trying to accomplish, and so on. - With the aid of in-game statistics, you can see where you were wasting your time doing things that weren't effective, but that you didn't know weren't effective. - You can identify game knowledge gaps and remind yourself what you need to study - e.g., "oh I didn't know my opponent could do that at 3:45," or "I didn't know it could hit me from that distance." - In most games it's extremely common for all but the highest level players to not realize when they've lost, and to end up blaming some other event happening near the end, long after the outcome was inevitable. A replay with complete information can help you understand what your opponent was doing that you couldn't see at the time, creating a more complete strategic picture of what led to a loss or victory. Competitive gaming modeled itself after professional sports, where experience teaches us "coaching" and "reviewing your own performance" make a big difference. Not everyone can afford a coach, but self-review is free (even if not as good), so there's no excuse not to self-review. And yet "watch your own replays" is *still* common advice to people asking for tips to improve, because lots of people don't do it. If you don't do reviews of any kind - either on your own or with the help of a coach - then you just aren't that serious about improving at your game. Which is OK! But still, many people clearly want to get better...but default to grinding more games, or watching more Twitch.tv, instead of reviewing their replays. What I'm getting at is this: I think we're overlooking "reviewing performance replays" as a way for software engineers to improve. I've basically never heard this suggested in programming, even though there isn't any reason why it wouldn't work. Maybe this is based on the (completely unconvincing) arguments about how software engineering is so different from every other human endeavor that there is nothing to learn from other fields. Try it: record yourself programming for say, 2-3 hours, and then watch your own replay on fast forward. Where are you making mistakes? What are you wasting time on? How could you be more efficient? You might find that: - Your typing speed or lack of familiarity with hotkeys is making you 50% slower than you can think. - You're laboriously doing things that an IDE can automatically do for you. - You repeatedly sidestep the same small problems over and over again, instead of fixing them once, permanently. - You're writing too much code before testing that the code is working, leading to wasted time at the end trying to chase down the source of each bug. - You don't know how to use a debugger and are spending most of your time spraying debug statements everywhere. - You don't understand the organization of the codebase *and* how your IDE works, so you spend most of your time searching for things inefficiently. - You don't know how to use the official documentation for your libraries, so you're just constantly reading terrible StackOverflow answers. - You're not reading (or misreading) error messages that tell you what your exact problem is, leading you on unnecessary half-hour goose chases. - You have a bad window arrangement wasting most of your screen real estate, and you can only see a few lines of code at a time. - You're repeatedly confusing yourself about what some code does, by mixing up similarly named variables or methods. - You are constantly trying to guess-and-check your way out of problems instead of approaching it systematically with logging and a debugger. - You frequently lose track of the tab or window that you want to switch to, and have to spend time looking for it, interrupting your flow...instead of using a system that relies more on visual cues and your muscle memory. - You get distracted and go off reading reddit for 20 minutes whenever there's a 2-3 minute wait in your workflow. I've done (and noticed in others) all of these issues here! Catching and fixing mechanical mistakes like this *easily* explains "10X programmer" differences in productivity. But as an industry almost nobody focuses on any of this, *ever*, except maybe, briefly, once, during employee onboarding at a "let me show you how our internal tools work" session. And then they get punted to The Wiki. What about code review? Isn't that part of how we coach programmers to be better? Sure, but that's critiquing the output, not the actual process of the coding. And it's hard to improve the output without drilling down into the process. I think this is a failure, because my experience is that SO MUCH of a programmer's productivity comes from mastering skills, tools, and workflows...not just memorizing a bunch of syntax, APIs, and even design patterns. I suspect this is part of the appeal of pair programming - pairing lets us learn skills and shadow-coach each other in a profession that essentially never does otherwise. There are so many things that I can correct and improve on the spot if I'm pairing with a more junior developer. So in that sense it's actually more like a coaching session than a "pairing" session. But this is suboptimal, because when we are "pair programming" we are usually trying to accomplish a specific task, and "hey let me teach you to (open and save files with hotkeys |use a debugger)" feels like a sidetrack. It's almost better if we focus on coaching the "mechanics" of programming separately. On the other hand, lots of people dislike pair programming because 1) yeah it's kind of wasteful, 2) some people just don't like the pressure of coding live in front of an audience, and 3) live coding with another person can be kind of frustrating if you're not operating at the same speed \[1\], or the person at the keyboard has mechanics limiting the productivity of both engineers. The solution to all of this is to do coaching and skill transfer with replays: that way we have the ability to fast-forward and skip uninteresting parts, and no performance anxiety. Software engineering as we know it is only roughly 50-60 years old at this point - still in its infancy. Future generations will regard us as basically cavemen, programming with primitive tools and espousing superstitions like "typing speed doesn't matter:" ![Reddit, where typing speed doesn't matter for programming, apparently.](https://www.petercai.com/assets/images/typing-speed.png) In the future, I think replays and live coaching will be a key way to train software engineers. We'll look back and wonder why we didn't do it sooner. Well, whenever we start trying to train software engineers. \[1\] In my experience, true pairing works best when both programmers have roughly equal experience and familiarity with the codebase relative to the problem at hand, so they can both contribute and review the work as it's being done live. When there's a big skill imbalance it feels more like a coaching session, but if you ignore this phenomenon and try to "pair" anyway, it can get frustrating for both parties - nobody enjoys "programming at 20% speed but also no multitasking," and nobody enjoys being told what to type without understanding what is going on. ### DeFi is still bullshit URL: https://www.petercai.com/defi-is-still-bullshit/ Last updated: 2022-07-15T12:00:00.000Z So. More than a year ago I wrote that [decentralized finance was bullshit](https://www.petercai.com/more-than-10-years-on-decentralized-finance-is-still-bullshit/). At the time I was thinking of companies like Celsius, since they were obviously, transparently, bullshit. Celsius was one of the biggest "yield farming" crypto loan platforms, offering something ridiculous like 20% yield per year on various cryptocurrencies. I've warned friends away from them because of how obvious the scam was, and how it was just *too good to be true*. This week Celsius filed for bankruptcy. This is weeks after they "paused" customer withdrawals. Celsius and everyone like Celsius is insolvent because it turns out - no, you cannot pay 20% yield indefinitely on cryptocurrencies, because nobody wants to borrow at those rates. There is no counterparty. The yield farms have just been paying out earnings and withdrawals with other people's deposits in a classic Ponzi scheme. There are seemingly hundreds of copycats like Celsius. They're all bullshit, and all freezing withdrawals as they run out of money. I'm not saying "some of them are bullshit" or "most of them are bullshit." Literally all of them, without exception, are bullshit. And like all Ponzi schemes, the perpetrators often convince themselves that "maybe it's a Ponzi today, but any day now we'll come up with a way to make it legit." This actually made it worse, because the scammers themselves often fell for *other scams* that looked on paper like the math would pencil out. So it was easy to fool people! The perps fooled themselves! They believed what they were saying! But its not possible even *in theory* for the math to pencil out, because crypto doesn't have any productive uses. **It is not possible to pay a risk-free 20% yield on a useless unproductive asset**. This is like, an iron law of finance. By definition, there's just no way to turn 1 bitcoin into 1.2 bitcoins (faster than [the risk-free rate](https://en.wikipedia.org/wiki/Risk-free%5Frate?ref=petercai.com)) without taking on some level of risk. And like John Oliver said, "crypto combines everything you don't know about money with everything you don't know about computers." So what did these platforms actually do with their assets, aside from paying out earlier investors? Mostly they ended up (directly or indirectly) lending to **3 Arrows Capital**, a "cryptocurrency hedge fund," which apparently took out massive high-yield unsecured loans in cash and cryptocurrency\[1\] and then…lost it all. But of course they did! Again, there was never any way (*even in theory*) for 3AC to (for example) borrow crypto at like 25% annual interest and get risk-adjusted returns that actually exceed that interest rate. *Cryptocurrencies are not a productive asset*. So 3AC apparently did the only thing they *could*, which is: load up on risky trades and hope for the best. If the bet works out, great, they can repay the loan + interest and keep some of the profit, and everyone makes money. But if it doesn't...oh well, they're insolvent but its their *creditors* that are screwed. Heads I win, tails you lose! Like, this is the dumbest business model ever, and I imagine these lending platforms were like "wow that sounds like a *terrible deal*, but I'm desperate so...YOLO." Or maybe they are all morons and they were like "Sounds great - I don't see any problems with that - what's your wallet address?" I honestly don't know which is more likely. Anyways, since 3AC is insolvent and cannot repay these loans, their lenders, like Celsius, also discovered they are insolvent and cannot repay their "depositors." I use the term "depositor" lightly here, because they don't work like bank deposits: they're more like unsecured non-recourse loans, because there are no rules around what can be done with them. Your bank can't take the contents of your checking account and (directly or indirectly) bet it all on black down at the casino; your cryptocurrency yield farm can. And did! And now it's not even clear where in the priority stack these "deposits" line up in bankruptcy proceedings - you might think that (like in traditional finance) equity and other unsecured lenders would get wiped out first before depositors. But maybe something else unexpected will happen. How exciting! It's important to emphasize this is not "the bad part of town" or some weird corner of the cryptocurrency economy. Yield farming and speculative trading (NFT wash trades, pump-and-dumps, "HODL to the moon!", etc.) encompasses like 99% of all activity in the cryptocurrency/blockchain ecosystem. Some people pretend that crypto is about more than that, but they are either lying or misinformed. Again, this is all very sad. Lots of lots of people who have no business in this industry (and have no idea how crypto/blockchain/finance works) got suckered in and will lose their life savings and such over this. This is not a "crypto winter." It was never summer. It was always a bunch of clowns trying to get rich telling Joe Plumber that *he* would get rich if he bought $SHITCOIN and deposited it in a yield farm that would - through the magic of blockchain! - somehow defy the rules of math and grow 20% a year risk-free. And shouting NOT YOUR KEYS NOT YOUR COINS is useless - rounding to the nearest whole decimal, 0.0% of the general population wants to "self-custody" because - and I must emphasize this, *cryptocurrencies are not a useful, productive asset*. The pain-in-the-ass to benefit ratio is too high for most people. We've literally made no progress on a useful application for blockchain technology since "digital money." Now we're just painfully rediscovering and reimplementing the complete history of financial regulation, deposit insurance, underwriting, etc., all over again. BUT IN CRYPTO. Yawn. Next. \[1\] Here's something I don't understand: 3AC often took large cash loans to, I suppose, speculate on cryptocurrency and make money trading it somehow. Presumably this is mostly a bunch of *long* bets, because they got wiped out when prices cratered. But why did they *borrow* cryptocurrencies? Usually you'd only do that if you wanted to sell them short, betting on a price decline. But we know 3AC didn't (on net) sell crypto short, because they got wiped out on the massive price declines. My speculation: they borrowed the cryptocurrency (unsecured) so they could go and use it as collateral with *another* lender to borrow cash. After all, there was likely LOTS of crypto chasing "yield" and converting this demand into cash at *absurd, stupid high* levels of risk is just the thing these guys at 3AC would do, because...moral hazard? Also, this smells a lot like the inverse of what these yield farms did to leverage up and attempt to juice their returns: rehypothecation of collateral to borrow more assets, and then loaning out those assets to make money. Effectively, you can make money on both ends by charging interest on a loan you make, then charging interest again on the collateral that the first borrower put up (as a loan to another borrower). This works great! Unless prices decline slightly, in which case a chain of margin calls happen and the entire web of players simultaneously find themselves insolvent. ### More than 10 years on, decentralized finance is still bullshit URL: https://www.petercai.com/more-than-10-years-on-decentralized-finance-is-still-bullshit/ Last updated: 2021-05-24T12:00:00.000Z I generally want to avoid talking about the news, but the GameStop fiasco this year has renewed a push for decentralized systems of stock ownership and exchange. And armchair economists who misunderstand how [money and inflation](https://www.lynalden.com/inflation/?ref=petercai.com#supply) works are hyping crypto and decentralized finance as a panacea to looming "hyperinflation" that has supposedly been just around the corner for about a decade. And these inflation truthers are never wrong, its just "the official stats are manipulated," and what about the price of lumber? Where was I? Right - the problem is most of the promised benefits of DeFi still amount to illusory hand-waving \[1\]. The first and most obvious problem with "DeFi" as a concept is that everyone will end up operating through a centralized exchange (a la Coinbase) for convenience anyway, defeating the purpose of "decentralization." And if you *don't*, we're going to lose all the mechanisms that protect us from ourselves and that we take for granted in our current financial system: - **Forgot your password?** Customer service can help you regain access. - **Added an extra 0 to your order? Transferred your money to the wrong recipient?** No problem, transactions can be reversed due to mistakes. - **Your five year old got on and accidentally traded your net worth for doge?** No problem, call in and fix it. The main problem/feature of the blockchain that breaks finance is that it attempts to completely dissociate technical rules from legal ones. It is [well established](https://en.wikipedia.org/wiki/Scienter?ref=petercai.com) that you cannot legally keep money that was sent to you by accident. Most blockchains though, are irreversible by design, even in cases of [obvious fraud](https://ethereumclassic.org/?ref=petercai.com). This poses real problems for law enforcement and umm, anyone who cares about fairness. Dealing with exceptions, reversing mistaken & fraudulent transactions, and the whole web of "human judgment and intermediaries" is treated like unnecessary baggage....and the collateral damage is treated with a shrug. Until those are solved, we're trading one set of problems for another. But crypto enthusiasts refuse to even acknowledge the tradeoffs - at least until they (or their exchange) get hacked, and they learn the actual meaning of "irreversible." And its frustrating because there *could be* real and tangible benefits to cryptocurrency and blockchain technology. Perhaps the greatest embarrassment of 21st century human technology is that we haven't yet solved cross-border payments yet. We might as well try a different approach with crypto. It's just that very few people are actually working on solving real problems like this with blockchain. Overt manipulation, tax avoidance, speculative bubbles, organized crime, and crazy volatility seems to be par for the course. You might think "it seems kind of silly to try and build a financial system on top of such a foundation" and I wouldn't blame you. Luckily, almost nobody is even trying: crypto is mostly used for crypto. Speculative investments, "yield farming," exit scams, and money laundering seem to be the main use cases. Almost every Ethereum project is either an outright scam or Yet Another Way to Manage your Cryptocurrency Wallets. It would be as if the bittorrent inventors could only think of "transferring bittorrent clients to each other over bittorrent" as their main use case. It's a sad joke. \[1\] The one possible exception: if it drags the world financial system into the modern age without changing anything else, it will be an unalloyed good. There's no real reason money transfers can only occur on weekdays, or for stock trades to take 2 days to settle. ### x86 is deprecated URL: https://www.petercai.com/x86-is-deprecated/ Last updated: 2021-01-18T12:00:00.000Z ARM will be the dominant CPU instruction set architecture going forward. In a few years, x86 will be relegated to specialty use cases. All new consumer devices and most cloud services will be based on ARM. Old workloads will probably run on x86 for decades, because nobody will bother to port and upgrade, but a decent amount will run on ARM under emulation with no problem. How did we get here? ARM started out as the upstart competitor optimized for low power and great performance per watt. So we stuck em on smartphones and tablets. Now performance per watt matters everywhere - datacenters and mobile devices are where the volume is. There's nothing inherent to x86 that makes it more power hungry - it's just that decades of optimizing x86 designs for performance has led to path dependency that makes it hard to compete with ARM for efficiency. In addition, x86 is still ahead in many single-threaded workloads, but the performance gap is closing. Just as there's nothing inherent in x86 that makes it more power hungry, there's also nothing inherent about ARM that prevents it from being more performant than x86\. In an increasingly parallelized world, there will be very few cases where x86's shrinking performance advantage is determinative. Most major hardware OEMs are either shipping or looking into ARM. The battery life implications are too hard to ignore. The Apple M1 means Apple is unlikely to release another x86 device ever again. Microsoft has been partnering with Qualcomm to design ARM CPUs for years now, and Windows 10 on ARM has x86-64 emulation in preview as of December 2020\. In other words, we are about 1 or 2 hardware cycles away from the mass availability of ARM devices and platforms that support \~99% of use cases consumers need \[1\]. The major cloud providers will eventually be forced to migrate to ARM for the cost savings. At AWS there's already work underway to switch over internal workloads to ARM. Azure may be secretly testing ARM for internal workloads too. The cost savings is just too great not to do it. Google Cloud will come around to it once they start caring about margins. For customer facing workloads, Amazon graviton2 VMs will soon be a reasonable default for customers, given it can be 40% faster and 20% cheaper than x86\. For many cloud services, the ARM switch will be completely transparent, since HTTP and other network protocols don't care what your underlying CPU architecture is. New Intel CEO Pat Gelsinger is taking over while under assault from all sides. Reports are that Intel has been in trouble for years now, with fat margins that are increasingly unsustainable and under attack from AMD, Amazon, and now Apple. Then you have dedicated fabs (but mostly TSMC) that have been pulling away and are now probably a full generation ahead of Intel. So in summary, Intel makes CPUs on a legacy architecture that consumes vastly more power than the competition, and is slowly losing its performance edge without anything on the horizon that can reverse the trend, with manufacturing capability as much as 5 years behind the state of the art, and can't even produce 10nm chips in mass quantities with decent yield even in early 2021\. Yikes. \[1\] I'm not sure how x86 on ARM virtualization will work, which may be important for some workstation scenarios so developers can build and test all their software. But the single most important event of the last 2 years has been Apple getting ARM laptops to developers, which lit a fire under folks like Docker to get cracking on ARM support. ### Storage prices have sort of stopped dropping URL: https://www.petercai.com/storage-prices-have-sort-of-stopped-dropping/ Last updated: 2020-12-08T12:00:00.000Z Today I want to talk about how hard drive prices have plunged dramatically since 2000 (most data from John C. McCallum's [disk prices page](https://jcmit.net/diskprice.htm?ref=petercai.com)): ![](https://www.petercai.com/assets/images/storage-1.png) We are talking about a roughly 97% decrease in price per GB over 20 years. In fact it's kind of hard to see on a linear scale, so let's switch to logarithmic for this time period: ![](https://www.petercai.com/assets/images/storage-2.png) Yup. Definitely logarithmic. Let's zoom in to the past 10 years or so (linear scale): ![](https://www.petercai.com/assets/images/storage-3.png) For fun we can also superimpose a chart from BackBlaze, a backup service that shared their [average price per hard drive through 2Q 2017](https://www.backblaze.com/blog/hard-drive-cost-per-gigabyte/?ref=petercai.com): ![](https://www.petercai.com/assets/images/storage-4.png) The 2011 Thailand monsoons left a noticeable blip on prices in early 2012, but otherwise the trend is pretty clear: there's been a deceleration of progress, with gains getting a lot slower starting around 2016\. To date there's been a lot of resistance getting below $0.02 per GB. So the era of just assuming storage will get significantly cheaper every year is over, at least for now. In fact, prices don't seem to have really dropped *at all* since mid 2019 when some drives started selling for $0.019 / GB. However, there's a nuance we're missing if we look at only price per GB without accounting for density. Let's look at a snapshot of hard drive prices at various capacities from December 2020 on Newegg: ![](https://www.petercai.com/assets/images/storage-5.png) The prices for 3.5" desktop drives range from $40 for a 500GB drive to \~$645 for an enterprise 18TB. We can conclude that at $40 the cost of the drive is almost entirely dominated by fixed costs, since the price doesn't actually change at all between 250GB and 1TB, and barely changes at 2TB (the cheapest 2TB drive is roughly $50). Because of this fixed cost issue, it's very hard to get good "bang for buck" at lower drive capacities. Here's the cost per GB as a function of drive capacity: ![](https://www.petercai.com/assets/images/storage-6.png) And here's the same chart on a log scale: ![](https://www.petercai.com/assets/images/storage-7.png) For many years the sweet spot was a 4TB drive, but now it looks like the absolute best deal might be the 8TB drives. However, I'm surprised to see some 14TB and 16TB drives get within striking distance ($0.021 per GB vs 0.019 per GB). If the larger capacity drives are roughly as reliable, then going for 16TB drives is a no-brainer, since they'll consume less space and power than two 8TB drives. Because of this, and despite $/GB not really dropping much, big datacenters and cloud providers are likely to continue seeing marginal decreases in total cost of ownership and operational costs per GB-month, but absent new breakthrough technology, not *that* much progress. What are the implications of this? A number of startup darlings have succeeded by betting on trends in underlying technologies to make their business viable (e.g., YouTube was founded 15 years ago and would've remained hopelessly unprofitable if not for the massive drop in bandwidth prices). And many a startup capitalized on the drop in storage prices to build attractive data warehousing and database solutions, replacing technology that assumed a GB cost $15\. But if you're betting on the price of say, AWS S3 pricing to go down significantly in the near future to make your startup profitable, well, bad news: ![](https://www.petercai.com/assets/images/storage-8.png) Amazon literally hasn't reduced prices for S3 since 2016, almost *5 years* now. ### "The codebase should look like it was written by one person" and other lies URL: https://www.petercai.com/the-codebase-should-look-like-it-was-written-by-one-person-and-other-lies/ Last updated: 2020-10-01T12:00:00.000Z I've mentioned that its unproductive to adopt cultural values that don't actually take a stance on an issue (with more than one plausible answer). "Be nice to your coworkers" isn't a cultural value, because it doesn't tell us how to make decisions; there's no trade-off there. Nobody would suggest "let's be mean to our coworkers" instead. But it's easy to fall into this trap by espousing something that *sounds* like an opinionated stance without actually taking one, like "The codebase should look like it was written by one person!" No. Stop. Don't tell me what ideals you want. Tell me what you are prepared to sacrifice for it. Is there a comprehensive style guide? Has everyone been trained on it? Have you adopted a process for how to amend it? Do you make ship-blocking change requests for style (citing the style guide)? Saying you want the codebase to be uniform is like saying you want to be a published author, or a PhD. It's not the declaration of desire that matters. Are you committing the time and effort it takes to bend reality in that direction? It's more important to live up to cultural values than to have ones that are the nicest-sounding. ### We're fast paced URL: https://www.petercai.com/were-fast-paced/ Last updated: 2020-08-24T12:00:00.000Z By custom, all software engineer job postings in the US are required to include the phrase "must thrive in a fast-paced work environment." This is usually not a bona fide qualification, but rather an advertisement ("we're fast paced, so we hope you like that?"). The reasoning is sound: Engineers generally aren't drawn to listings with "must be able to grind through mountains of red tape, bureaucracy, and slow process. We deploy once a quarter." And every tech company thinks it is fast-paced\[1\] the way every driver thinks s/he is above average. But what, exactly, does fast paced mean? No, I don't think you can use meter sticks like "we release every (day|week|month)" - or at least, you leave a lot of room for gaming that kind of metric ("I FTP'd a new config into the production server today"). I think there's a reasonable test that is a good proxy for how fast-paced any engineering environment is, phrased as a question: "Say the app has a **one character typo** in some production code. How long, in the best case, does it take to go from someone in the company knowing about this to it being fixed?" You can learn a lot about the maturity of an organization with this one question, because you can ask this of a specific engineer on a specific team and they're going to start going down the mental checklist of how to actually deploy changes and then you can have a deep conversation about QA, automation, build times, and deployment process. You might be surprised about the answers you get! \[1\] Conflating 'long hours' with 'fast paced' is 😂 ### Low cost file serving on bare metal with tiered storage URL: https://www.petercai.com/low-cost-file-serving-on-bare-metal-with-tiered-storage/ Last updated: 2018-05-13T12:00:00.000Z I once worked a project that stored and pushed hundreds of TBs a month, and cost was an important consideration. AWS S3 is cheap-ish, until you get into multi TB and PB territory. AWS bandwidth has always been outrageously expensive. Google Cloud was cheaper, but only by some factor like 'half the price of AWS.' Keeping costs low involved custom building servers by hand, with RAID storage and off the shelf consumer components. We colocated them in various datacenters around the US. This was bare metal world, and a back of the napkin calculation showed TCO was between 10X and 30X cheaper than a naïve AWS implementation, depending on assumptions. As we grew, one thing that became an obvious problem was trying to distribute service load between multiple distinct servers. Once there were more than a handful, some got saturated with requests more than others, and the data was effectively sharded (you had to request your file from the right server, or the request would fail). At the peak of this, there were close to a dozen heterogeneous boxes - more like pets than cattle. The other problem we had was that the servers tended to need very large RAID arrays to hold enough files to keep the CPU and network saturated (at the time, 10gbps was becoming commonplace). This was expensive, and the RAID gave us good data safety guarantees, but the servers themselves were a single point of failure for high availability — there were no hot spare servers at first, since the architecture described so far would've required doubling the server count. Since having offsite backups was important, we were storing a copy of all files in a separate location anyway. This gave us an idea. We ended up implementing a solution where the backup service actually served as a warm data store for the servers. Whenever a server ran low on disk space, it would 'evict' some of its files in local storage, similar to how an operating system might page out memory. Whenever a request came in for a file it didn't have locally, it would go fetch it from the warm store / backup, then serve it. It seemed like an elegant solution. In practice, this had a number of interesting and unforeseen complications, while other details I obsessed about ended up being nonissues. First, I worried about the eviction algorithm. Keeping a list of least-recently-used files seemed like an obvious idea, but this was expensive to create and update (as most true LRU algorithms are). Using a naïve random algorithm actually worked great, as long as the files were 'aged' - files that were at least 2 months old worked perfectly. The reason is that repeated runs of a very infrequent random eviction tends to perform similarly to LRU in practice, since popular files almost immediately reappear in the hot cache, and infrequently accessed files just stay in the warm store. Version 1 was very, very, deeply problematic. Serving an evicted file seems relatively straightforward - it simply requires fetching it from the store first. The problem with this design is it doesn't tell us what happens when multiple requests come in for the same file. A naïve implementation would end up making parallel requests for the same file from backup, needlessly hogging resources at the precise moment we wanted to complete a request as quickly as possible. In Version 2, a mutex solved this problem, but created another one: massive unnecessary queuing. It turns out that popular files by definition generate a lot of requests, and serializing those requests even for a few seconds while a file is restored could very quickly lead to resource exhaustion. Compounding the problem was that the server was a rails server, and a process-based rails server at that. Some background: As a very efficient file server, this rails app only takes a few milliseconds to do some bookkeeping (checking permissions, updating analytics, and doing a quick spam/abuse check) and then hands it off to nginx to actually transmit the requested file. Each rails worker was a separate process, which is very RAM inefficient \[1\]. We had plenty of ram to handle those requests in rails, and plenty more to run nginx, with no queuing. So using a RAM inefficient process model for rails turned out not to matter in the usual case. But it mattered a lot if requests could occasionally spike to 3-5 seconds in rails, causing a 100RPS server to dip to 0RPS for a few seconds at a time because all the available processes are busy waiting on a lock. The solution I implemented in Version 3 was to make the entire architecture non-blocking. If a ruby process failed to acquire a lock, it meant a different process was in the middle of restoring the file, so it would simply fail the request (I forget what status code, maybe 503?). To the end-user, retrying the download in a few seconds would succeed (our users were very forgiving). Lesson learned: there's a reason rails defaults to a shared-nothing architecture in its web request-response model. In practice, failed requests were exceptionally rare, and was much better than the alternative of causing random latency and availability spikes. This design scaled really well, and is an example of an 80/20 solution that beat out more complicated architectures. There were other complications. A file is 'paged in' by writing it directly to its final destination\[2\]. This leads to potential integrity issues - what happens if ruby is killed during a restart before it has finished writing a file? What if there's a network hiccup? It was possible that this could cause a partially downloaded file to be served on restart. This was solved by checking the filesize and treating it like a cache miss if the size on disk didn't match the size in the database. There was also the (extremely) rare file corruption that would sometimes happen, so the file hashes were occasionally compared to known good values in the database. But hashing the file before serving it every time was expensive and unnecessary, so instead it was simply run with a fixed probability on every request. A hash mismatch is treated as a cache miss, and the file is retrieved from backup. This was admittedly a band-aid, with a number of limitations. It checks the integrity of popular files too frequently, and not frequently enough for files that aren't accessed frequently. If the file is also corrupted in storage, this would lead to an irreparable situation, and a never-ending cache miss loop. This risk was judged to be low enough to ignore, and luckily, I don't know of any instances where it ever happened. This tiered data storage solution worked so well that it ended up being used for load balancing as well. No longer was a file pinned to a specific server - it could be requested from any server, and if the server didn't have it, it could simply treat the situation like a cache miss. This greatly simplified the network and application architecture, since it now meant load could be round-robined, and all the servers could be treated as indistinguishable from each other. Spreading load more evenly also means there is less unused capacity, further lowering both server capex and opex. I'm pretty sure we ended up achieving the 30X cost reduction. \[1\] Puma, a threaded ruby server, wasn't available when this app was first built. \[2\] Yes, I know the correct answer is usually to write the file to a tmp location, and then rename when it's finished, but I no longer remember the precise reason we didn't do this. It *might* have been because tmp was a different filesystem, which means 'mv' required physical copying, not simply renaming the file. ### Why I picked Microsoft over Amazon URL: https://www.petercai.com/why-i-picked-microsoft-over-amazon/ Last updated: 2014-10-15T12:00:00.000Z It's interviewing season, and that means people are going to get offers really soon. I've been wanting to write a blurgh post about my decision to pick Microsoft over Amazon for some time now, and I've been asked for my reasoning a couple times. So maybe I can help others make the right choice. I may be rationalizing my decision in hindsight, but it turns out there were a number of advantages Microsoft has over Amazon; here is the view from 10,000 feet: 1. Substantially better benefits (health, wellness, employee stock purchase plan, 401k matching, perks), and slightly better overall compensation. You can increase your cash income an additional 5-8% risk-free by taking full advantage of ESPP, 401k, and other game-y things with your health benefits. 2. Generous relocation package + annual performance bonus. It makes up for not getting a hiring bonus at least. 3. Stock vesting is substantially faster (for Amazon, stock vesting is all backloaded so the last 80% or so vests in years 3 and 4). At MSFT the vesting is a linear 25% every year. 4. I get my own office, and work-life balance is generally better. Microsoft's median employee tenure backs this up. 5. No on-call rotations\[1\]. Annual performance bonuses in cash, in addition to stock. Did I mention work-life balance? And a handy table I put together, mostly from a combination of sources (stars denote uncertainty) and my highly scientific opinion: | | Microsoft | Amazon | favors | | ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | ------------------ | | Onboarding | | | | | Relocation (from east coast) | all-expenses-paid or $5000 cash, tax-assisted (2011) | all-expenses-paid or $7500 cash, tax-assisted | Amazon | | Signing bonus | None in 2011, there may be a small one now | \~25% base in 2 installments, pro-rated for 2 years | Strongly Amazon | | Hiring Stock Grant | \~60% of base, vesting: 25% per year | \~50% base, vesting: 5% 1st yr, 15% 2nd yr, then 20% every 6 months | Microsoft | | Compensation | | | | | Base salary | 60-75th percentile (on average, industry norm +15%) | 50-75th percentile (on average, industry norm +10%) | Leaning Microsoft | | Base salary increase | 0-9%, 3.5-4% is typical | on average less than 3.5% | Microsoft | | Annual cash bonus | On average 10% of base | usually none | Strongly Microsoft | | Annual stock grants | < 10% of base | Between 10-15% of base\* | Amazon | | Career | | | | | Promotions | see trajectory discussion | see trajectory discussion | Amazon | | Benefits | | | | | 401k matching | 50% of contributions up to 6% of base salary (3% match) | 50% of contributions up to 4% of base salary (2% match) | Microsoft | | Employee Stock Purchase Plan | 10% discount, purchases capped at 15% of base salary | none | Strongly Microsoft | | Other fringe benefits | Prime Card, free onsite health screenings, various health incentives & rewards, charity+volunteering match, discounted group legal plan for routine legal work | 10% off up to $1000 in Amazon.com purchases per year | Strongly Microsoft | | Health | see health benefits discussion | see health benefits discussion | Leaning Microsoft | | Kitchen | Soft drinks, milk, juice, tea, on-demand Starbucks, espresso | Tea, powdered cider, drip coffee | Leaning Microsoft | | Time off | 3 weeks vacation, 10 paid holidays, 2 personal days | 2 weeks vacation (3wks after 1st year), 6 paid holidays, 6 personal days | Microsoft | | Culture | | | | | Location | Redmond | Seattle | Strongly Amazon | | Tools/Platforms | Closed source Microsoft stack, proprietary. Many legacy desktop platforms, lots of new services | Open source Linux stack. Almost entirely services-based, many legacy concerns. Best-in-class deployment tools. | Strongly Amazon | | On-call | Expected of most engineers (unless product has no services component, increasingly unlikely) | Expected of most engineers | Leaning Microsoft | | Median Age | 33 | 32 | | | Median Tenure | 4.0 years | 1.0 years | | ## Career Trajectory The great thing about Microsoft is that there's always a career path for people who want to become valued individual contributors. However, you should be aware that the difficulty level ramps up pretty quickly. Generally, most ICs are unlikely to earn the title of Senior SDE in less than 4-5 years, and Microsoft will rarely consider someone for a lead engineer (the first rung in the management ladder\[2\]) who has fewer than 6-7 years under his belt. However, the promotions don't stop just because you don't want to be a manager - excellent ICs can earn titles like Principal Engineer, Distinguished Engineer, and Technical Fellow who are respected and valued as much as Corporate Vice Presidents. At Amazon, expect a lot of responsibility to ramp up fairly quickly, along with somewhat higher chances for advancement — both because Amazon is growing faster, and because it has higher rates of attrition (I suspect attrition is higher at the bottom than the top; but I have no evidence for this). Three years out of college is not atypical for being offered SDM I (first rung on management track). This is partly because of the horrible retention; by the time you hit 3 years, you're more tenured than about 80% of the company. Anecdotally, I have heard talented Microsoft ICs on the management track note to me that specific Amazon counterparts are progressing faster (to development manager) than themselves. So if management track progression is your goal — pick Amazon, not Microsoft. ## Health Benefits **Microsoft in 2014** - 100% preventative care covered, always. - HSP ($1000-$2500 annual employer HSA contribution, $1500-$3750 deductible; $1000-$2500 coinsurance) or HMO (no deductible/limited coinsurance, copays of $20-$100 for outpatient service) - full or partial dental coverage + payroll credit - vision: free annual eye exam and up to $225 of vision hardware per year; lasik benefit - free gym membership OR up to $800 in cash reimbursement for fitness purchases OR $200 cash - free life insurance - 2x annual base pay - long term disability insurance - 60% of monthly income up to $15,000 - optional accidental death & dismemberment **Amazon in 2014** Documents I got my hands on weren't heavy on details. I'm just going to go out on a limb here and say Microsoft's health benefits are better. Here is a copypasta from their careers page that tells you approximately nothing about how they compare to Microsoft: - A choice of four medical plans, including prescription drug coverage, designed to meet your individual needs, with domestic partner coverage - Dental plan - Vision plan - Company-paid basic life and accident coverage as well as optional coverage at a low cost - Company-paid short- and long-term disability plan - Employee assistance program including dependent-care referral services and financial/legal services - Health-care and dependent-care flexible spending accounts ## Fringe Benefits Allow me a moment to blow you away with the absurd benefits Microsoft offers. Prime Card gives you random discounts on everything from Apple products to local restaurants. It also gets you discounted admission (I think $5?) to IMAX movies. Microsoft hosts free onsite health screenings for general health, flu shots, glucose/cholesterol testing, etc — and even gives away gift cards for attending. They have a charity matching program - they'll match dollar for dollar every contribution you give to registered charities, and also pay $18/hr to any charity you volunteer at to increase your impact. There's a discounted group legal plan that costs, I think, something like 30-40 dollars a month for routine legal work. There's tuition reimbursement. There's generous paid maternity (AND paternity) leave. Amazon discounts 10% (up to $100 off) of annual Amazon.com purchases, which is cool too, I guess. ## Commuting, Culture & Tools Microsoft runs free shuttles to most major residential areas nearby - the largest private bus system in the world, in fact. On top of that, they provide a free ORCA card for unlimited free travel on the local bus system. Amazon also has a free ORCA card on offer, but only a limited private shuttle system between campuses. Amazon is in Seattle proper, where anything vaguely resembling nightlife happens; Microsoft is on the so-called Eastside across a narrow bridge where basically nothing does. This is not an insignificant issue for many people who work at Microsoft but want to live in Seattle - this is likely to extend your commute by at least 30 minutes each way. As far as tools go, both companies have first-rate toolchains. Amazon probably leads here, as they have a very impressive toolset, dependency management system, and deployment process. On the other hand, Microsoft's approach to the software engineering process is both much more disciplined, and less flexible. They produce some of the finest program managers. And almost all their tools are closed-source, so you're unlikely to be using, say, git, unless you work at Amazon. The downside of Amazon's agility is a sometimes chaotic software development process; getting stuck on a team with a mandate to improve a service while simultaneously fixing bad architecture/rush job warts are not uncommon, and unrewarding. Work-life balance is manageable at both companies. I've had a number of 60 hour weeks, maybe even a few 70-hour weeks near shipping time. They were out of the norm. I'm inclined to say Microsoft requires fewer hours on average than Amazon, where people might see 45-50 / week as closer to normal. Everyone will tell you "how much work you get done" matters more than "how many hours you put in." This is a half-truth - you need to put in the right amount of face time, don't be on either side of the bell curve. TL;DR: If you want to work in a fast-paced environment leading the way in services, cross your fingers every time you deploy, and don't mind getting paged in the middle of the night, work for Amazon. If you want to make slightly more money shipping desktop software (or deploy services like you would ship desktop software), and pretend with your 100,000 coworkers that the company is becoming "agile," work for Microsoft. Thanks to all the fine folks who answered my questions and reviewed early drafts of this. **Notes** \[1\] This is no longer true on many teams at Microsoft. For the most part I hear its not as bad as at Amazon, but there can be what I charitably call "rough patches" when a team implements on call rotations for the first time, and invariably screw things up until the alert frequency can be tuned correctly (ASK ME HOW I KNOW). \[2\] Microsoft recently moved away from formal lead positions as of 4Q 2014, bringing it into alignment with most other companies like Google and Amazon. Basically all ICs report to a dev manager now, and a "lead" engineer has no direct reports any more, but has de facto authority over a project or team. This doesn't change the fact that progressing from an IC to a manager at Microsoft is both very hard and takes a long time. ### Raymond Chen's lessons URL: https://www.petercai.com/raymond-chens-lessons/ Last updated: 2014-04-25T12:00:00.000Z A random collection of wisdom from Raymond Chen and The Old New Thing. I plan to keep this updated as I discover/remember more of them. **Windows doesn’t have an expert mode because you are not an expert.** This is just the Dunning-Kruger effect in play: people who are not experts pretty much by definition lack the ability to judge whether they are experts or not. “Expert users” using the advanced features of Windows *invariably* make feature requests that are equivalent to the beginner feature that already exists. **The hatchway is still secure, even if you opened it with the key**. It’s not a security bug if the user has to first give permission to elevate. Bogus security reports of this nature generally go like this: 1. Do something that requires elevation, such as replacing an application’s DLL with a malicious copy. 2. Run the application. 3. YOU HAVE NOW COMPROMISED THE USER’S COMPUTER. Except, it’s not a security bug because step 1 required elevation, and therefore an administrator’s consent. **Eventually, nothing is special any more**. If you create special functions or flags in your API to give them extra functionality, they will in practice become the defaults over time, as programmers cargo-cult their way through programming. Eventually people find that the regular function “doesn’t work” (for various definitions of “work”), and that the special function does. https://blogs.msdn.com/b/oldnewthing/archive/2008/10/06/8969399.aspx **Providing compatibility overrides is basically the same is not deprecating a behavior.** “If you provide an administrative override to restore earlier behavior, then you never really removed the earlier behavior. Since installers run with administrator privileges, they can go ahead and flip the setting that is intended to be set only by system administrators.” **Appearing to succeed is a valid form of undefined behavior.** Undefined means anything can happen, including: returning success, nothing, formatting your system drive, playing music, etc. So it is futile to ask “if the documentation says doing x results in undefined behavior, why does it appear to work?” Also, one cannot rely on a specific form of undefined behavior; relying on it implies the behavior is defined and contractual. **The registry is superior to config files.** Config and .ini files are deprecated in favor of the registry because: 1. ini files do not support unicode. 2. Security is not granular (how do you restrict a group from editing a certain part of the file?) 3. Atomicity issues with multiple threads/processes can lead to data loss on the flat file (the registry is a database). 4. Denial of service issues – someone could just take an exclusive lock on your config to screw with you. 5. ini can store strings only, so if you need to store binary you’d have to encode it as a string. 6. Parsing files is slower, and writing settings would require loading and reparsing the whole file. 7. Central administration via group policy would be exceedingly difficult compared to a registry. **Computer science: do not confuse the means with the ends.** It is often said that the purpose of garbage collection is to reclaim unused memory, but this is incorrect. The purpose of garbage collection is to *simulate infinite memory*. Reclamation is just the process by which this is achieved. For example, a null garbage collector is provably correct if you have more physical memory than your program needs. Similarly, allocating a value type on the stack is an implementation detail. It’s not a requirement that it is on the stack, only that it is always passed by value. https://blogs.msdn.com/b/oldnewthing/archive/2010/08/09/10047586.aspx **Open source isn’t a compatibility panacea.** You don’t get rid of compatibility problems by publishing source code; in fact that makes it easier to introduce compatibility issues because it exposes all the internal undocumented behaviors that aren’t contractual. https://blogs.msdn.com/b/oldnewthing/archive/2014/04/11/10516280.aspx **You can’t satisfy everyone about where to put advanced settings.** This is a specific case of not being able to delight all the people all the time when the audience is measured in billions. Most people prefer advanced settings in one of five categories (quoting Raymond): 1. It’s okay if the setting is hidden behind a registry key. I know how to set it myself. 2. I don’t want to mess with the registry. Put the setting in a configuration file that I pass to the installer. 3. I don’t want to write a configuration file. The program should have an Advanced button that calls up a dialog which lets the user change the advanced setting. 4. Every setting must be exposed in the user interface. 5. Every setting must be exposed in the user interface by default. Don’t make me call up the extended context menu. 6. The first time the user does X, show users a dialog asking if they want to change the advanced setting. Each item is approximately an order of magnitude harder than the last, and the final one is objectively user-hostile. Whatever you decide to implement, the other five groups will call you an idiot. https://blogs.msdn.com/b/oldnewthing/archive/2008/01/14/7103585.aspx **Cleanup must never fail.** Low level cleanup functions don’t have very many options for recovering from failure, so they must always succeed (they may succeed with errors, but that is not the same as failing). https://blogs.msdn.com/b/oldnewthing/archive/2008/01/07/7011066.aspx **Don’t use a global solution to a local problem.** Since an operating system is a shared playground, you can’t just run around changing global settings because that’s how you like it. If two applications with opposing preferences tried this, one or both of them would break; the correct approach is to change the setting in a local scope to avoid breaking other applications. **A platform must support broken apps; otherwise you’re just punishing the user.** Compatibility with apps, including incorrectly written apps, is crucial for platforms because users expect programs to work between versions of Windows. It is tempting to be a purist and declare that the apps should break, which will force the developers to fix them. In practice, the developers either don’t care, no longer exist, or don’t have the source code any more. Users will instead blame the platform and/or not upgrade. **Users hate it when they can’t cancel.** If you have a long running operation or some multi-step wizard, the user should be able to cancel. It should be clear what will and will not be saved or committed when they cancel. https://blogs.msdn.com/b/oldnewthing/archive/2006/03/08/546177.aspx **Geopolitics is serious business.** It can be illegal to have a map with incorrect labels or borders (the correctness of which depends on who is looking), or to call disputed territories (such as Taiwan) countries in some places. https://blogs.msdn.com/b/oldnewthing/archive/2003/08/22/54679.aspx **The USB stack is dumb because it’s dealing with dumb manufacturers.** Some USB devices have identical serial numbers which can cause non-deterministic behavior and arbitrary settings assignment, so Windows has no choice but to pretend every device is unique. This is why if you unplug/re-plug a device into a different USB port, Windows treats it like a new device and forgets all your settings. More generally, Windows could be smarter, but then things would break. https://blogs.msdn.com/b/oldnewthing/archive/2004/11/10/255047.aspx **Avoid Polling** Polling prevents the hot code and all code leading up to it from being paged out, prevents the CPU from halting to a lower power state, and wastes CPU. https://blogs.msdn.com/b/oldnewthing/archive/2006/01/24/516808.aspx ### Hey Engineers, Let's Stop Being Assholes? URL: https://www.petercai.com/hey-engineers-lets-stop-being-assholes/ Last updated: 2013-05-30T12:00:00.000Z There's this toxic idea in tech circles that's starting to get really tiring. And it pains me to have to point this out because I could just blissfully go along with it, and give myself that self-congratulatory pat on the back that most of the tech world is doing on a nearly daily basis. It's the elitism. There's this culture (cultivated by engineers) that worships engineers and shuns everyone else for not 'being technical.' This culture is backwards and counterproductive. It presupposes that engineering is the only thing that matters, the only discipline with smart people, and that therefore everything else must defer to it. There's a reasonable origin for this line of thinking. Back in the dot com days, business majors were raising 50 million to make online wedding invitations and going public because they had a homepage. MBAs were looking for some engineers to "code up this idea quick," as if the tech part of a tech company was just this checkbox that needed filling. As if engineers were these interchangeable cogs in the machine of a startup. Of course, those tech companies imploded and for most of them, technology wasn't the primary cause. But even really great business ideas often failed for lack of technical expertise. It turns out people who don't know how to create software are also terrible at recognizing how important (and hard) it is. But the reverse is also true. And that's how far the pendulum has swung in the other direction. There are lots of engineers proudly proclaiming that everything that isn't engineering is just some checkbox department filled with warm bodies who weren't good enough to be programmers. As those in the know have known for a long time, it turns out things like business development and "having customers" is pretty important too. It turns out people who don't specialize in a non-technical role are also terrible at realizing how hard and important it is. Which brings me to the general observation that everyone thinks their job is obviously the most important and indispensable. Not surprisingly, everyone is wrong, but engineers have convinced themselves that because the MBAs were demonstrably wrong about engineering, engineering must be right. Which is just a logical fallacy wrapped in wishful thinking sprinkled with the chocolate covered bacon bits of all your friends who also happen to be engineers agreeing. This false premise (engineering is everything) leads to all sorts of crazy conclusions. One of them is that everyone should learn to code. This is stupid, and a waste of time. There's no substitute for computer literacy, but saying everyone should learn to code is like saying everyone should learn to drive manual transmission and change their own oil: cars are everywhere! Cars are the future! If you don't drive, you will not be in control of where you are driven! This kind of alarmist propaganda is nonsensical and should be laughed out of the room. The whole point of software engineers is so other people don't have to code! This phenomenon coincides with a related one that also annoys me: the insinuation that being an engineer automatically demonstrates your superior intellect. The recent shortage of engineering talent in the US exacerbates this feeling, because it's easy to conclude that the problem is because people aren't smart enough to become software engineers. Actually, it's mostly\[1\] because 1) most people think programming is about as sexy as mopping floors, and 2) for the past decade, smart people who just wanted to make money could make more for the same hours and less risk - in finance. Software engineering is actually not that hard. There, I said it. Basic computer science type education and work is not much harder, conceptually, than intermediate calculus. The majority of the population is capable of being taught, and understanding, intermediate calculus. We know this because lots of countries teach both in middle school. QED. This means that being a software engineer is not beyond the intellectual capacity of the average joe\[2\]. It also means engineers need to stop waving their diplomas around like they're computer astronauts. It makes us all look like elitist assholes, and it's holding back our profession. \[1\] Ignores our immigration problem, since it's better if this isn't about politics. \[2\] Speaking strictly about proficiency, of course. We all know there's a very high skill ceiling, and being a "great engineer" is a whole other ballgame. But this too has lots of external factors not related to innate skill. ### Understanding Google's Bug Bounty Program URL: https://www.petercai.com/understanding-googles-bug-bounty-program/ Last updated: 2013-04-23T12:00:00.000Z Some people have taken Google's idea of offering security bug bounties (wherein Google will pay cash rewards for reporting security issues to Google), and taken them to their logical conclusion: why stop at security bugs? Why not incentivize reporting of ALL software bugs with bounties? Aren't other companies cheap for not offering bug bounties? Questions along these lines misunderstand how software development works. Engineers don't sit on their hands and surf Reddit after shipping a product. They're already working on bugs. All sufficiently complex software ships with known bugs; more reporting isn't likely to change whether they get fixed or not. So the premise that "reporting more bugs will improve software quality" is speculative, at best. Software quality is determined by what the market will bear. The market usually rewards buggy-but-good-enough software that solves a problem now, rather than perfect solutions that are late to the party. This is partly because of the time value of software, and partly because chasing defects offers diminishing returns. But more to the point: everyday users are not equipped to report bugs. They don't have the training, tools, or motivation to do it properly. At Microsoft (and other software companies), crash dumps already include as much information as can be legally collected, based on the user's consent. So bug information for crashes and many other issues (uncaught exceptions, for example) are already being collected in an automated, accurate way. So really, any well-supported software already has built-in reporting for most high-impact bugs. In addition, you run into other problems with bug bounties: - Common bugs will be over-reported, wasting everyone's time. Customers are already motivated to report these since they want them fixed. - Unexpected, but by-design behavior will be incorrectly reported as bugs. - Problems will be exacerbated by the bounties, increasing volumes and decreasing quality. This actually creates noise around what is really important to fix (e.g., the defects users would report even if there was no bounty). - A bug bounty program isn't free. Someone has to triage the submissions and this could easily be less productive than not having the program at all, leading to worse software quality on account of running a bug bounty program. The only exception to this rule is security bugs, which operate a little bit differently than run-of-the-mill defects: - There is already a market for security bugs, which can be sold to hackers. The developer is simply trying to outbid them to keep the product secure. - This means there's already a set of professionals who are hunting for such bugs; professionals are much more likely to find bugs on account of understanding how software is designed and implemented. - Users are unlikely to notice or report security bugs since they generally don't obstruct functionality, meaning there are fewer dupes to wade through, and bug reports will be of higher quality on average. So paying bounties for security bugs can be effective, but it's not likely bounties for functionality bugs in the general case would be particularly productive. ### The Amazon SDE Interview URL: https://www.petercai.com/the-amazon-sde-interview/ Last updated: 2011-06-09T12:00:00.000Z Amazon recruited on campus at UMass over last fall and I was lucky enough to snag a spot after running into them at the career fair. The rep at the table quizzed me on some generic stuff, recording my answers (what is the algorithmic complexity of the best comparison sort?) on the back of my resume. The interviews are done over several days by 4 interviewers on campus. Each one lasts about 45 minutes. In the first interview, I am randomly assigned to one of the interviewers. I was not asked very challenging questions, but we briefly went over my resume and he described his engineering department and product. Then: - Write the 'merge' function in mergesort. This needs to be linear time, and it is basically the only thing I needed to nail. He is also checking to see how familiar I am with coding; I am only given pen and paper to sketch out the algorithm. - Describe a tree and write a few traversal methods. Find the least common ancestor of two nodes. - The tricky one: Given an array of integers and another integer X, find if a pair in the array sums to X. This is actually a modified subset-sum problem, but it can be reduced to polynomial time by using a hash. My experience at every single company I've interviewed with is that a hash can be assumed to operate in O(1) worst-case time, even though it is not true in theory. The algorithm involves checking for the existence of a “complimentary” number stored in a hash for any given number in the array; if it exists then a solution exists. If not, add the number to the hash and check the next number. I got a call later that evening inviting me back to the second round of interviews (all 3 of them). At this point it is obvious the first acts essentially as a screener. The next day, another interviewer repeated the process, talking about his product and gently bashing Microsoft (a former employer). I had to talk about the difference between a linked list and an array, and when I might use one over the other. For some reason the topic wandered over to the trie data structure, and the interviewer asked if this was something regularly taught at UMass, because it was a generally exotic structure. I answered no, since in fact I simply stumbled on it the day before while comparing notes with other interviewees! My third interview was with the bar raiser. For the uninitiated, Amazon's bar raiser is the designated interviewer who rakes you over the coals. He is tasked with ensuring that the average talent at Amazon trends up instead of down. - How do you detect if a linked list forms a loop? For this question, my original intuition was to modify the node elements with a member that I could flip, 'marking' each element as I passed over it. Unable to change the data structure, the next most obvious solution is to use a hash to track which nodes have been seen. This results in O(n) time and space. He seemed satisfied with this answer; it was not until after the interview that I learned it could be done in constant space and linear time using Floyd's “Tortoise and Hare” cycle detection algorithm (yes, of Floyd-Warshall). - This was the hardest question of all the interviews: Given an array of integers, write a function to find the maximum number that can be obtained by summing up a continuous subsequence. The tricky part is firstly realizing that you cannot start your sum at 0, because if all the numbers in the list are negative, then your final answer will be wrong (because you will incorrectly return 0 as the largest sum). Secondly, encountering a negative number does not necessarily mean you should terminate the sequence, because it could still be part of the record sum. I struggled with this until he nudged me to the answer: the sum must be restarted whenever the running sum goes negative. Tricky question. I was also tasked with analyzing the brute-force approach to the problem, which is finding and testing all possible subsequences (hence n^2). My fourth interviewer dealt almost entirely with design questions. - We started off slow: code a factorial function and analyze its complexity. I did the stupid recursive method because it was easy, knowing I'd probably be asked to improve it. There are multiple approaches to optimization but I decided to memoize since it is a simple time-for-space tradeoff and is easy to conceptualize. - The bulk of the interview involved designing a versioned filesystem in UML diagrams and explaining how it supported the requirements. The first requirement I was given was to make the versioning completely transparent, which I did by using redirectors to wrap the real filesystem objects. He kept piling on more minutiae, seemingly unsatisfied. I was growing a little impatient, each time responding that I'd simply add another data structure or pointer to my diagram to meet the requirement. Finally, he asked me how to support multiple concurrent branches at the same time, and I understood what he was getting at: he wanted me to make my system recursive! I pointed this out, scribbled hastily on the paper to illustrate, and he could not be more satisfied. I got a call literally the same night, inviting me to a dinner sponsored by Amazon in the next few days. He gave me some BS line about how he couldn't say whether I had an offer or not, and in fact our dinner (7 candidates and 4 interviewers) was completely filled with smalltalk and stories of Seattle weather without so much as a single mention of the interview! I got an email a few days later with my offer, and I was suddenly overcome by a realization: if I hadn't made the cut, why would they have bothered buying me dinner? ### Virtualization is bad for database integrity URL: https://www.petercai.com/virtualization-is-bad-for-database-integrity/ Last updated: 2011-04-30T12:00:00.000Z One of the lessons about Amazon's cloud failure last week is that "the cloud" is incredibly overhyped. The cloud doesn't necessarily deliver a lot of the things implicitly promised in the popular imagination – such as improved uptime or reliability, or even cost savings. But **the real problem with the cloud is that you actually get less database reliability** than proper hardware. This is because each layer of indirection introduces an additional unknown that must be accounted for. A database like MySQL or Oracle is able to guarantee atomicity and durability through a number of features. One of these features is a transaction journal (or binary log). These files contain the sequential commands that mutate the database. In the event of a power or disk failure, a database will "replay" its journal upon restarting to ensure that all operations it has guaranteed to be completed are in fact completed. One complication with the transaction journal is that hard disks are actually complicated abstractions of physical media. To achieve high performance, hard disks have buffers and caches that can be dangerous because they are volatile: imagine a scenario where the database is assured that data is written to disk, when in fact it has only been written to disk cache, and disappears on power loss. Any guarantee about reliability that depends on the data being written to disk is therefore invalid, because we can't be sure that it will actually survive a power failure. In fact, most implementations of fsync(), which flushes data to the disk, do not guarantee that the information is actually written to the physical media; it might just be saved in the write-back cache. Another complication stemming from hard disk abstraction is write re-ordering. This is sometimes known as Native Command Queuing or the elevator algorithm. This feature, intended to increase performance, means the hard drive may write data to disk in a different order than the order in which the commands were issued. This is a massive problem for database integrity because it literally makes it impossible to reason about atomicity guarantees if a COMMIT can be written to disk before previously sent data. In fact, losing power during a reordered transaction probably won't damage anything until power is restored, when the bad transaction is 'replayed', corrupting the filesystem. Both of these are not new problems; they are well understood. Highly reliable databases hack fsync() with flags and other dummy commands for every unique snowflake operating system to ensure write order and data durability. However, when you introduce a virtualization layer, **all of this basically goes to hell**. You cannot reason that fsync() means anything in a virtual machine because it's just a virtual hard drive telling you that it has written the data to its virtual persistence. Is the data on the **physical disk**, or is it not? The only way to know for sure is to disable the write cache, **which is not controlled by the guest operating system**. With respect to write reordering, one of the best defenses is to use a journaled filesystem and just pray that a journal write doesn't get reordered at the exact moment that you lose power or crash. Even better is to use a journaled filesystem that has journal checksumming, so that invalid journal entries can be detected and ignored. But again, the success of all these systems **depends on actually having control over the physical disk**, which is exactly what virtual operating systems don't have. What this really means is that with a database on a virtual machine, the best you can hope for is that it is NO LESS STABLE than the same implementation on a real hardware. This is a big leap of faith, considering that it is already extremely difficult to offer ACID guarantees (ex: caches without battery backups, weird fsync() behavior, and the variability of platforms and disk controllers in general). ### The Microsoft SDE Interview URL: https://www.petercai.com/the-microsoft-sde-interview/ Last updated: 2011-04-20T12:00:00.000Z This is the first of several interviews from fall of 2010 that I will document. The Microsoft Software Development Engineer Interview is hard. While no interview process is perfect, they do a lot of things right in terms of setting a high bar, as well as showing off what the company has to offer. It is exactly what you'd expect from a world-class software company that has fairly high standards for engineers. **Background** Engineering candidates are always subjected to a phone screen before being flown in. However, I had an offer from Amazon.com and happened to be in the area, so I tried to set up an interview immediately to avoid having to fly out again. Microsoft initially refused, quoting their standard procedure, but then inexplicably reversed its position the next day. In general, Microsoft arranges travel and a (very nice) hotel, and reimburses meals generously — up to $75 per day. **The Experience** The Microsoft campus resembles that of a university, with a few differences. Free soft drinks (and to a lesser degree, snacks) abound. Hybrid buses and Priuses shuttle employees around. Security is also very tight, although I never got the sense that it was overbearing as with government contractors. The interview process itself is very well organized and structured. Candidate shuttles take you directly from building to building on demand. For the most part, everything went smoothly (except a snowstorm cancelled all cab service before I returned to my hotel, but that is not Microsoft's fault). **Interview the First** My first interview was in the office of a principal engineer lead. The question was to code an algorithm to draw a circle. Of course, the general equation is x2 + y2 = r2\. My first attempt was to do the following: set x = -r, and then continually increment x until it was equal to r. At each step, use the equation to calculate the value of y given x, and then plot it. This gives a semi-circle. I observed that since a circle is symmetrical, I could simply draw the other half of the circle by reversing the y axis. However, this wasn't fast enough, so I was forced to rethink my algorithm. It turns out that a circle is also symmetrical along the y axis, so I only needed to calculate a single quadrant of the circle, and could simply mirror along the x and y axes. This was still not fast enough for my interviewer. After a carefully placed hint, I realized that a quarter circle is actually symmetrical as well - so that only 1/8 of the circle needs to be calculated. The rest of the interview consisted of me refining the algorithm — namely, accounting for any gaps that might appear in my circle (which can be fixed by taking into the account the slope of the line that is being plotted and choosing to either use x or y as the given). **Interview the Second** My second interview was about sorting a large list of numbers in linear time. This was a fairly standard problem with the additional constraint of a very small amount of working memory: 4KB. The problem was further qualified: numbers can only be between 0 and 30,000, and no number appears more than once. Given these constraints, I was able to come up with a simple algorithm: first, allocate a large binary array where the index of the array corresponded to the existence of a number. Then, for each number in the unsorted list, I would set the corresponding array index to the 'on' position. Then returning the sorted results is simply a matter of traversing the array and finding the 'on' elements. I was tasked with implementing this solution in workable code, and told that the smallest denomination available was a byte. So I created a byte array and used bit-shifting to calculate and set the bits correctly (with each element in the array representing 8 numbers). The final solution requires O(n+m) time, where n is the size of the unsorted list and m is the size of the dictionary (in this case, 30,000). Then, in the spirit of progressively making a hard problem harder: I was told to sort the list with 2KB of memory. This means the range of all possible numbers cannot be enumerated in memory, since 30,000 bits requires just under 4KB to store. The solution to this required a bit of trickery: preprocess the unsorted list into two partitions: the first half to contain only numbers 0-14,999, and the second 15,000-30,000\. This can be done in linear time (with respect to the list size) using two pointers on either end of the list: it is the same as the partition algorithm in quicksort where the pivot is the middle of the dictionary. Then process each sublist using the exact same algorithm I described above, paging in and out as needed to stay within 2KB of memory. This can be applied recursively to sort a list in 1KB of memory, etc. **Interview the Third** My third interview was over lunch. This one was more theoretical in nature, asking basic operating system fundamentals: Q: What is the difference between a thread and a process? A: A process is self-contained and gets its own virtual address space. A thread shares heap space with other threads in the same process, making communication easier and faster, and also conserving memory at the expense of isolation and stability. Q: How would you communicate between threads? A: One can communicate between threads using shared memory, or by using a mutex library such as pthreads that allows waiting and signaling on locks. Q: How would you communicate between processes? A: By opening a socket, using named pipes, or the filesystem. Back in the office, I was tasked with describing some data structures and the appropriate use of them - for example, an array vs. a linked list. This is standard fare that should elicit a response detailing the various tradeoffs in time and space complexity in Big-O terms. I was then asked to code a function to swap two nodes in a linked list. This is fairly straightforward. Then I was to expand on this to swap nodes in a doubly linked list. This actually requires modifying up to 6 different pointers. Obviously, a temp variable is needed as well. I struggled somewhat with this because it is very difficult to keep them all straight in your head. I eventually managed it, but there are also a variety of corner cases that you will be expected to account for, such as when one of the nodes is at the beginning or end of the list, or if both parameters are references to the same node. I was then asked to come up with a list of test cases for my function. This is fairly straightforward, although obviously creativity and some more bizarre edge cases would be a bonus. For example — if the two nodes being passed even belong to the same list. I think the point of this exercise, like many of the others, is to demonstrate a desire for clarity in the face of ambiguous problems. It is wise to ask clarifying questions regarding expected behavior and problem scope. **Interview the Fourth** This interview started with more discussion of data structures, mostly centered around trees and the complexity and usefulness of binary trees. I was asked to discuss how to calculate the height of a binary tree, test whether it was full or not, and find the depth of a given node. The height calculation was especially important, because it leads into the main problem. My initial reaction was to simply store the height in the node as it was inserted. However, without such a data field or the option to add it, the second best solution is to do a count by recursively getting the depth of the parent node. This is obviously O(logn) in time, and the best we can do. I was next asked to modify the node structure so that each node in a full binary tree would point to the node on its right. I was able to do this in linear time by using a breadth-first traversal and adding a pointer at each step of the way, except for when the height of the next node is different. We had leftover time, so he asked a brain teaser: fix existing code with a single change to make it count from 0 to 10 instead of to infinity. I needed a hint but got it after some time. **Interview the Fifth** The fifth interview is the championship round. I was now given the choice to either pick the standard question, or the much harder one. Unable to restrain my curiosity, I opted for the harder problem. The interviewer then asks me if I have ever heard of the trie data structure. Slightly relieved, I replied in the affirmative, although I did not remember the exact details. He went on to briefly explain it. Then, he asked me to code the class, including all the members and basic read/insert operations. I was able to do this fairly easily on the first attempt. I was then asked to discuss the various uses of the class, including its suitability for use in a dictionary spellchecker (Although I know Levenshtein, it was not until later that I realized they could be combined to also create spelling suggestions very efficiently). The final question was fairly challenging: How would one compress a trie? I answered that it depended on the characteristics of the data that it contained. While he liked this observation, it was obviously not an answer he was satisfied with. After some prodding I observed that the bottom tries were likely to be very sparse, particularly in words that did not share many common substrings. I created an adaptive algorithm that would try to store the endings of words as strings, reserving the conversion to a trie only in the case where necessary. At this point I was expecting him to unload the Final Question that would make me regret asking for the "hard" problem, but none came. That was the end of the interview. **Aftermath** Microsoft, not surprisingly, failed to follow up with me in the promised timeframe. A few days after the date had passed, the recruiter emailed me to set up a phone call, who informed me that I had an offer. ### The Fencepost Problem URL: https://www.petercai.com/the-fencepost-problem/ Last updated: 2010-03-15T12:00:00.000Z If there's anything I've noticed that plagues many less-experienced programmers, it's what I refer to as the fencepost problem (or fencepost error, if you will). The problem can be most succinctly summed up using this thought exercise: Imagine a fence that is 100 meters long. A fencepost is needed every 10 meters. How many fenceposts are needed? The answer is, of course, 11\. See the image: ![picture of a fence with 10 sections and 11 fenceposts](https://www.petercai.com/assets/images/fencepost-1.png) Notice that there are 10 sections of fence, *but there are 11 fenceposts*. This error is a more specific case of the infamous off-by-one error that occurs in coding in general. Why am I bringing this up? Because major software writers get this wrong all the time. Consider the following picture: ![Doodle screenshot](https://www.petercai.com/assets/images/fencepost-2.png) This screenshot is from Doodle.com, supposedly the most popular scheduling application in the world. Let us assume that I actually did check off 6PM as shown in the picture. Do you know how much free time that denotes? None. The problem is that 6PM is a fencepost, and time availability is "sections of fence," ranges of time. A single fencepost sticking in the ground doesn't denote any length; a second post is needed. The problem is that the visualization on Doodle has *switched the fences and the fenceposts*. Notice how the hours are marked by blocks, and time between the hours is denoted by a white border. The result: I have to color in *two* blocks to indicate that I'm available for *one* hour, and coloring in one block means squat. Ridiculous. But wait, there's more! Now, imagine for a moment that one person's availability was from 5PM to 7PM, and another's was from 7PM to 9PM. What would that visualization look like? ![Doodle screenshot](https://www.petercai.com/assets/images/fencepost-3.png) Argh. Now try to imagine for a moment that I'm trying to schedule a meeting with 12 people (and um, is longer than 59 seconds), and they all enter their crazy, ridiculous schedules into Doodle, and I have to deal with sorting out overlaps that aren't overlaps because they start and end at the same hour, but *look like overlaps* and … ugh, are you listening, Doodle? Let's move on to whenisgood.net: ![Doodle screenshot](https://www.petercai.com/assets/images/fencepost-4.png) As a writer of scheduling software, this irks me. The engineers can't seem to let go of the fact that the hour markings have to take up more room than the time between the hour markings. Luckily enough for me, Google Calendar does not succumb to this problem. I guess that's why those engineers get paid the big bucks. ### Why I don't like CakePHP URL: https://www.petercai.com/why-i-dont-like-cakephp/ Last updated: 2010-02-10T12:00:00.000Z I've used CakePHP for a lot of projects now. There's no comparison to Ruby on Rails. Rails is smartly integrated, well-designed, and sits on top of an elegant language. CakePHP is an admitted ripoff of Rails that sits on a popular but terrible language. It copies Rails - almost randomly - without copying any of the stuff that makes Rails great. And PHP is a language plagued by indecision, delays, and paradigms from the last decade. What makes Rails great? - An ActiveRecord implementation that isn't brain-dead. The Object Relational Model is, you know, actually object-oriented. - Lazy-loading of records that allows as-needed fetching of model associations instead of a *dumb* and unintuitive $recursive variable. - Actual documentation that doesn't suck (not that it didn't suck in the early years). Cake's has been getting better lately, but it's still got a long way to go. - Massive community of great libraries and plugins. What makes CakePHP awful? - Arbitrary and seemingly random nesting of HTTP input parameters. It gets to the point where I sometimes access $\_GET and $\_POST directly, and just carefully sanitize the data myself. - Default routing does not use REST, and is difficult (but not impossible) to coerce into anything resembling REST. - Model instances are represented as PHP arrays instead of objects. So it's like a struct, except uglier, and you can basically set a few class variables on the model, then call a class method, which simulates an object method. Nice. The CakePHP core team's position is: this is a deliberate design decision so users cannot call instance methods in the view. So not only is the decision based on stupid reasoning, it's not even a technical one. - Ridiculous schema manager that has noble intentions, but tries to be too clever. Instead of making you write your own schema changes like Rails does, it tries to infer the changes after the fact (by diffing the actual schema) - with disastrous results. Let's say you rename a column and want to use the cake console to capture that change in the schema manager. Now let's look at the generated schema change that was automatically inferred. Instead of *renaming*, it will drop the column and make a new one, which means - yes - all your old column data is now gone. This is because there's no way to tell with certainty the difference between a rename and a drop/add after it has already happened. But it matters. What they're trying to do can't be done. Instead they try to do it anyway and make users deal with the data loss. Cake doesn't even try to hide the fact that its cloning Rails in PHP, but it's at a disadvantage from the start because PHP's syntax is just so much worse. And they use arrays to represent objects. And PHP's array syntax is terrible. Unbelievably terrible. Which makes the decision to go all-in on PHP arrays baffling. I spend the majority of my time in Cake trying to manipulate arrays-of-arrays-of-arrays-of-arrays that are supposed to represent some kind of shoddy struct, then mangling with model associations that require about 5 nested arrays to define. Add to this the fact that CakePHP tries to "innovate" its way around some of its shortcomings in a less-than-brilliant way (like, um, $recursive) and you get a steaming pile of garbage that's only marginally better than just coding a PHP project from scratch. Rails's smart design has come at a cost. In the past, they haven't been shy about breaking backwards compatibility, sometimes in a big way, to cut out cancerous ideas before they take hold. It's a community that acknowledges mistakes and embraces their eradication. And Ruby is an elegant language with strong design and is under active development. PHP on the other hand has languished for years, most notably with a critical security bug called $register\_globals. On top of that, they've been slow to introduce object support, type-checking, and sometimes make terribly confusing decisions (like the decision to call a feature *overloading* that has absolutely nothing to do with *method overloading* in the traditionally accepted sense of the word). To be fair, here's what I don't like about Rails (and Ruby): - Massive list of reserved variable names, which of course will throw a cryptic and seemingly unrelated error if you use them. - Concise syntax that sometimes resembles perl. Yes, perl. - Bizarre decision to use "end" to close blocks instead of something sensible, like um, anything else. More typing for no good reason. The cherry on top that makes this not OK is that while its illegal to define a variable named "end," its fine to define a function named "end."