1. Observe the system
2. Formulate multiple falsifiable hypotheses
3. Perform experiments to rule out some or all the hypotheses
4. Repeat

It’s heavily inspired by the principle of Strong Inference, therefore I don’t claim ownership, but for simplicitly I’ll call it “my” Scientific Method. It’s been very effective as a diagnostic tool for me, and it differs from the schoolhouse version I was originally taught in some important ways.

Let’s start by stating the schoolhouse version, which only has three phases:

1. Form a hypothesis
2. Perform an experiment to confirm or refute the hypothesis
3. Draw conclusions

First. The first difference is that my Scientific Method starts with observation not speculation. Observing the system gives you a sound basis for hypothesis formulation, and hones intuition. It also immediately rules out many instinctual but wrong ideas before they have time to take up brain space. In many cases you’ll even skip steps 2 through 4 because detailed observation will already reveal the solution you seek. The important part is to explicitly prioritize observation. It’s easy to skip since it’s human nature to start explaining things before understanding them more deeply. You have to actively resist this tendency.

Second. The second difference is in hypothesis formulation, and there are two parts to the difference that need paying attention to. Here they are:

• You should form multiple hypotheses (at least two, ideally several)
• Hypotheses should be falsifiable (rather than confirmable)

I strongly encourage you to read at least the first 2 pages of the Strong Inference article to understand why this is important. Nonetheless, here are a few points to call out:

• Multiple hypotheses avoid the psychological pattern of attaching to one (usually the first) explantion of things;
• Having multiple potential explanations embraces the complexity inherent in most systems,
• And focusing on falsifiable claims produces a stream of concrete truths on which to base further analysis, whereas supporting evidence merely increases the plausibility of something being true (for more on this concept, I recommend Mathematics and Plausible Reasoning)

Third. The third difference is that my Scientific Method formalizes iteration. You will undoubtedly gain knowledge and instincts from the process of executing the phases, and you should keep going until you’re really confident in the knowledge you’ve gained. The only way to do that is to put in the work and keep iterating.

Prose writing has been of great use to me in the course of my life, and was a principal means of my advancement.

Franklin attributes his writing ability to deliberate effort put towards its improvement. This began after Franklin’s father commented on a series of his teenage letters that while “[Franklin] had the advantage of [his] antagonist in correct spelling and pointing (which [he] ow’d to the printing-house), [he] fell far short in elegance of expression, in method, and in perspicuity.”

From this, Franklin sought to improve his writing with a collection of regularly practiced tasks. Here they are:

Reconstruction.

About this time I met with an odd volume of the Spectator … I bought it, read it over and over, and was much delighted with it. I thought the writing excellent, and wished, if possible, to imitate it. With this view I took some of the papers, and, making short hints of the sentiment in each sentence, laid them by a few days, and then, without looking at the book, try’d to compleat the papers again, by expressing each hinted sentiment at length, and as fully as it had been expressed before, in any suitable words that should come to hand. Then I compared my Spectator with the original, discovered some of my faults, and corrected them.

Prose to poetry and back.

I found I wanted a stock of words, or a readiness in recollecting and using them, which I thought I should have acquired before that time if I had gone on making verses; since the continual occasion for words of the same import, but of different length, to suit the measure, or of different sound for the rhyme, would have laid me under a constant necessity of searching for variety, and also have tended to fix that variety in my mind, and make me master of it. Therefore I took some of the tales and turned them into verse; and, after a time, when I had pretty well forgotten the prose, turned them back again.

Reorganization.

I also sometimes jumbled my collections of hints into confusion, and after some weeks endeavored to reduce them into the best order, before I began to form the full sentences and compleat the paper. This was to teach me method in the arrangement of thoughts. By comparing my work afterwards with the original, I discovered many faults and amended them; but I sometimes had the pleasure of fancying that, in certain particulars of small import, I had been lucky enough to improve the method of the language, and this encouraged me to think I might possibly in time come to be a tolerable English writer, of which I was extremely ambitious.

Some thoughts

It’s interesting how similar these are to modern techniques of pre-training machine learning language models.

For example, the fundamental contribution of BERT (which broke 11 records and became one of the most influential language models of all time) was a pre-training technique wherein the model was forced to reconstruct groups of words that had been omitted from a body of text. This isn’t the same as Franklin’s reconstruction task, but it’s similar. In machine translation, one of the most common pre-training techniques is called “back translation” where a source language is translated into a target language and then back again and compared to the original. This is obviously very similar to Franklin’s poetry task.

I wonder what other learning techniques we might devise by studying the way machines learn.

The point of this post is to share some of my lasting impressions. Here’s a list of 5 of them. After the list, I’ll elaborate on each item one-by-one:

• Energetic positivity is a frame of mind that can be brought about deliberately.
• Patient communication leads to smooth confidence which leads to trust.
• People can detect a presence of genuine attentiveness. Conversely, they can tell when you’re not paying attention.
• Every person is enjoyable and interesting once they break out of their shell—it’s your job to get them to show that.
• People feel much more comfortable sharing things about themselves if you share things about yourself.

Energetic Positivity

Regardless of my demeanor, every day I was forced to call and consequently every day I was forced to find ways to get excited. I needed to get excited because the quality of my calls would be much better if I had the energy to make them good. I also saw that if I was positive, the person I talked to would be more positive, and this would drastically improve my chances of enjoying the call.

A consequence of this self-propelled positivity was an improvement in my general attitude. Though at first my efforts to make things positive could seem synthetic, they would often mature into a general feeling of optimism that improved the rest of my day. On walks to the call room, I would take negative things and try to rephrase them positively: for example, I’d replace the thought of “that river is dirty” when walking over Magdalen bridge with “we could make that river much cleaner.” By the end of the 15 minute walk my outlook would completely change.

The big realization was that energetic positivity is not something that’s granted by the day. It’s something that can be brought about with a little bit of effort.

Smooth Confidence

I considered a call to go well if I felt happy with the conversation and the callee donated. Both the conversation and probability of donation improved dramatically with smoothness of communication.

Evidently, an ability to speak effectively, expressively, and confidently is tremendously important and I plan to make a sincere effort to improve my verbal skills further. A single “like”, “um”, or awkward pause—though seemingly innocuous—had the potential to destroy the trust I had built with a person. There were a few times when I stumbled during the payment phase of the call, and on at least 3 occasions, this caused the callee to either (1) withdraw their donation or (2) revoke their decision to complete the process over the phone.

The most effective way I found to fight these interjections was reducing my speaking pace. I found that if I went slow and formed every phrase deliberately, I was much less likely to trip up. I never had a callee abstain from processing their information over the phone in instances with no trip ups, and on several occasions the alumnus even chose to increase their donation from the initial ask amount.

Attentive Presence

There is a presence associated with genuine attention that people can detect even over the phone. In other words, if you’re not giving your full attention to someone, they’ll notice.

Just as well, people will notice when you do give your full attention. I spoke to two alumni who made me feel like the conversation I was having with them was the most important thing in the world to them. Undoubtedly it wasn’t—both of them were in the upper decile of the most accomplished people I spoke to on the campaign (which I suspect isn’t a coincidence) and had much more important things to do. What struck me was their ability to invest wholeheartedly in the conversation in a way that made me feel important.

By the end of the campaign I felt much more attuned to people’s interest. More importantly I felt other people being attuned to my interest. If my attention was elsewhere, it quickly became apparent through a deflation of the conversation and an overall decay of presence.

Breaking Shells

I would frequently begin conversations with dull platitudes, themselves holding no substance. Their purpose was to build momentum in the conversation to get to a baseline level of trust where the callee would feel comfortable sharing some of their more interesting qualities and experiences. It’s like poking around a field until something interesting crops up. Something that I found is that every person had something interesting about them once I found a chord. The most effective technique I found for doing that is the subject of the next point (on sharing).

Sometimes it would take several minutes to find something worth talking about, and some people made it much easier than others. There were some calls where I sat through nearly 10 minutes of prodding before any flames caught and others where the subject was interesting after the first question. In any case, a genuine belief in the existence of an interesting core beneath potentially several layers of dense insulation kept me motivated to keep talking.

There are really two assumptions here: (1) every person is interesting if you really reach them, (2) it’s my job to realize that truth. The first point helped me survive the nastiness of some people in the early stages of the conversation, and the second is a gentle reminder that people aren’t going to open up to you on their own. If you want to reach someone, you have to put in effort.

Sharing

I’ve found the most reliable technique of getting people to share things about themselves is to share things about myself. I think there are two things contributing to the success of this approach: (1) sharing things about yourself demonstrates trust and creates a pattern of volunteering information, (2) there are often subjects to latch onto in personal stories.

By the end of the phone campaign, I would progress most of my conversations by relating something the person said to a personal experience by telling a (short) story and then just waiting. I didn’t ask any questions or anything, I would just share my story and sit there. More often than not, the person would respond with a story of their own and go in a completely unexpected direction. No question I would have asked would have led the conversation to that point, but it would go there naturally.

Sometimes the person wouldn’t latch onto my stories and that was fine. It indicates a good time to ask a question and push the conversation elsewhere. Even in this case the voluntary information was valuable because it added a personal element to the talk that defused some of the strangeness of talking on the phone with someone you’ve never met.

In Sum

These were some of my personal impressions from an experience cold calling. What I didn’t talk about is the character that comes with listening to 56 voicemails in a row in over 2 hours of calling or the humility that comes with asking strangers for money. Those things are important and are the part of the reason I think everyone should try something like this at some point. However, the main learning was that people can be really interesting, and I felt that this experience made me better at having good conversations with people I don’t yet know.

His thesis boils down to 4 points:

what do we need for another [Operation Warp Speed]? 1) Known science—scaling not discovering, 2) Lifting of regulations 3) Big externalities, 4) Pre-existing motivation.

This got me thinking about potential gains we can accelerate with federal funds. Solving the climate problem is always on my mind, and we’re not going to do that without a substantial refactor of our electricity generation system.

The Biden administration recently announced a fund devoted to fusion research which, if successful, could solidify America’s energy independence and open the door to other applications currently constrained by energy availability like desalination and carbon capture. The question is: will this work?

The first thing we can do is evaluate the project on Tabarrok’s criteria. Fusion clearly knocks the last three points out of the park:

• Nuclear power generation is extremely regulated in the US which turns most experimental research into a bureaucratic headache severe enough to compel work to move elsewhere. We need big changes if we want to make a breakthrough on US soil
• The external benefits are huge and obvious: practically infinite, (theoretically) low-cost, clean energy
• We’ve been energy constrained since the industrial revolution (and arguably before that), and the political benefit of energy independence is more relevant than ever

Unfortunately the first point is a miss: fusion energy is definitely more than a scaling problem. We’ll need to address the technical problems associated with stability before we can expect to reach industrial scale. Nonetheless, I’m optimistic we can do it. Here’s (part of the reason) why.

Let’s look at some other historical successes in federal acceleration of groundbreaking projects:

• moon landing
• atomic bombs
• the internet

Most people will be familiar with the government’s role in the former two examples, but some may not know the story of the internet so well. The full story is described phenomenally in M. Mitchell Waldrop’s The Dream Machine, and the tl;dr is that networking projects were funded by ARPA (through the pentagon) under the direction of creative geniuses like JCR Licklider and Bob Taylor (Bob Taylor went on to run the legendary Xerox PARC), which amounted to the creation of the ARPAnet and ultimately the internet.

The point here is that all three of these projects were successful despite the obstacle of discovery: new science had to be created to make them work. Accordingly, like fusion, these projects would have missed the first point in Tabarrok’s criteria.

So why did they work? I’m still not convinced I understand my thoughts on this well enough to formulate a thesis, but a common thread is that all of these projects had an element of international competition associated with them. The case of the internet is perhaps the most subtle—though threats from the Vietnam and Cold wars indisputably influenced America’s decision to fund the initiative as superior communication and computation tech is a huge advantage in wartime. Of course the moon landing precipitated from the Soviet/US space race and atomic weapon primacy secures the deterrent effects they’re designed to have.

The Biden administration is in a great position to frame fusion research as a matter of international competition since:

1. The war in Ukraine (and corresponding threat to the world’s natural gas supply chain) validates the importance of energy independence from Russia
2. China, Russia, and Japan have all recently initiated their own hunts for sustainable fusion

Other thoughts

It’s usually a poor bet that funds alone can produce scientific breakthroughs. Fortunately, research into magnets has been promising and simulations show that near-term reactors have the potential to be energy producing on the net. At this point, a sustained net-positive reaction would be a Wright Brothers moment that proves the concept of fusion energy. Even though we’d still have a lot of work to do making fusion reactors commercially viable (by increasing safety and operating lifetime), we’d have a lot of tailwind once we get the energy going positive.

The focal premise of the article is that we ought to conceive of planning problems as problems of coordinating specialized (and individually incomplete) knowledge clusters instead of determining optimal solutions from a position of complete information. In Hayek’s words:

The problem is thus in no way solved if we show that all the facts, if they were known to a single mind (as we hypothetically assume them to be given to the observing economist), would uniquely determine the solution; instead we must show how a solution is produced by the interactions of people each of whom possess only partial knowledge.

I agree with Hayek’s pivotal assumption, which is that a single actor is incapable of possessing complete knowledge of an economic system at every level of granularity. I admit that China’s success as a surveillance state supports the claim that sufficiently detailed knowledge may be obtained to allow central planning, though I’m not convinced this is optimal. It seems impossible to me that a central agent may possess all the information of a system, which is a belief I base on a heuristic assumption that the whole intelligence of any system must (in principle) exceed the intelligence of any part (I find that Buckminster Fuller’s concept of synergy crops up a lot these days).

Regardless of belief, the planning problem boils down to the following minimax optimization:

1. Maximize coordination
2. Minimize dilution

A point Hayek harps on is that the planning problem is less about strategy and more about coordination. For macro scale growth we require some amount of coordination in the form of alignment to a plan that remains coherent over time (which allows gains to compound). The natural approach to this is to allocate a central planner to enforce a coherent plan, but the salient point of Hayek’s article is that this accrues knowledge dilution risk; in other words, it ignores the second half of the minimax planning problem.

Dilution in this context refers to the lossy compression of local expertise which occurs in the communication channel between localized clusters of knowledge in an economic system. In particular, local knowledge is degraded when summarized for and aggregated by a central planner. An analogy is that the mean and variance insufficiently describe most abnormal distributions, and, for example, a central planner who approximates a distribution by a normal one and therefore neglects the kurtosis of the underlying distribution may undervalue tail risk and thereby expose the system to potential damage.

I think this idea is what Hayek was trying to get at when he wrote

the sort of knowledge with which I have been concerned is knowledge of the kind which by its nature cannot enter into statistics and therefore cannot be conveyed to any central authority in statistical form … It follows from this that central planning based on statistical information by its nature cannot take direct account of these circumstances of time and place, and that the central planner will have to find some way or other in which the decisions depending on them can be left to the “man on the spot.”

Summary

In the end, the things that stick are that:

• we operate in a macroeconomic system with fractured clusters of locally precise knowledge, each one’s information being incomplete
• an optimally planned system will maximize coordination between these clusters while minimizing dilution of their expertise

Beyond Economic Planning

Modelling systems in this way has made me think of tons of other applications, and I’m still clarifying my thoughts about them. I may write about it later, but in the meantime, here’s a list of other stuff this article made me think about:

• the success of garden farming to the growth of east Asian agricultural economies in the 20th century (Joe Studwell talks about this at length in the first part of his book How Asia Works)
• the potential trajectory towards AGI via modular intelligences that communicate
• John Boyd and the comparative importance of responsiveness rather than raw power in combat
• the insane success of the internet’s decentralized routing protocol
• the pattern of history’s greatest CEOs as capital allocators rather than micromanagers
• autonomy granted to researchers at Xerox PARC under Bob Taylor’s direction

What’s the problem? Modern security protocols rely on the difficulty of factoring many-digit semiprime numbers.¹ In particular I’m talking about cryptographic signatures. Cryptographic signing is built on the premise that I can display a public identifier key linked to a private “signing key” so that counterparties can verify that—without revealing the value of the private signing key—indeed I signed a message using the private key linked to my public identifier. In practice this is implemented with large semiprime numbers (along with some other math magic), though its implementation admits a method to compute the private key associated with a given public key, given that semiprime numbers can be factored efficiently.

Quantum computers with sufficiently many stable qubits² can do this time-efficiently. From here the substance of this is derived from the inevitability of practical factoring systems: it’s not a matter of whether physically implemented quantum computers will crack digital signature algorithms, but when. My impression is that modern crypto systems will only survive in their current form if funding for quantum computing research dries up. This I think is unlikely, at the very least because quantum computers are effective simulators for chemical phenomena relevant to drug discovery.

Attack Surface 1: Blockchains

Positing the existence of stable quantum computers with many (many) qubits, what is the attack surface of modern blockchains? I’ve already mentioned vulnerabilities in the private/public key pair system. Another problem is zk-proofs.

The Problem With Proofs

Most Ethereum scaling solutions rely on zero knowledge proof (zk-proof) systems to securely compress a computation into a public commitment that is easier to validate than it is to perform the underlying compute task. There are three systems that dominate this task:

• SNARKs
• STARKs
• Bulletproofs

What matters in the context of this post is that SNARKs and Bulletproofs are susceptible to quantum threats whereas STARKs are (so far) believed immune. Despite their vulnerability to quantum attacks, SNARKs are used the most in practice by a wide margin. Why? The short answer is space efficiency (and the long answer would address tooling and history): regardless of the size of computation, SNARKs take up 288 bytes whereas STARKS take up ~45,000 bytes and scale with computation size like $O(p(\log n))$ for a polynomial $p$.

In the current environment of high Ethereum gas fees, the extra 2 orders of magnitude are very costly. A SNARK can fit into a transaction absorbing ~600k gas, whereas STARKs soak up nearly 2.5 million gas. At current gas prices (roughly $37 \times 10^{-9}$ ETH / gas), that’s a difference of roughly USD 1700 for each zk-proof. As a solution to a problem that doesn’t exist yet, it’s a tough sell.

So what can we do? The quantum threat is inevitable and requires attention sooner than later. Post-quantuming crypto systems is going to be harder than it seems, so we should get started as soon as possible. For STARKs to be adopted, either their size needs to be reduced substantially or they need to be deployed on a chain with lower gas fees. Doing the former seems harder to me.

The tech behind STARKs is new and exciting. It would be great to see more people working on compressing the STARK (or another post-quantum zk system) footprint so we can make steps towards a future version of the internet that will survive the birth of quantum computing at scale.

To solve this problem, we need to introduce a new machine-readable communication stream with higher bandwidth. There are two approaches we can take to this:

1. Create an entirely new communication stream
2. Adapt an existing communication stream with high bandwidth to a machine-readable format

Today I’m going to talk about the second approach. In particular, I’ll talk about what it will take to build an effective semantic parsing system. I’ll likely return to this topic in later Thoughts, but for now, I’ll describe the basics. For one, let’s make it clear what I mean by “semantic parsing”: semantic parsing is the task of converting utterances in natural language to statements in a formal language. In the context of human-computer symbiosis, this means converting sentences and phrases spoken in common languages like English or German into explicit instructions for the computer to perform, usually passing through the syntax of a pre-existent logical or programming language.

The fundamental problem is that human language is fuzzy, idiosyncratic, and ambiguous, which is particularly ill-suited for specifying instructions to machines. Computers require clear, precise instructions that fit a predictable syntax to reliably perform their duties, so if we’re going to build a more natural human-computer interface based on the high-bandwidth channel of human language, we need a way to convert fuzzy statements from natural language into concrete formal structures. This is the basis for semantic parsers.

Before we go about building a semantic parser, we should understand what a good one would look like. To get a rough idea of what makes a good semantic parser, we can evaluate along two dimensions: (1) precision of understanding, and (2) broadness of scope. In other words, the semantic parser should understand what needs to be done and be capable of doing lots of useful things. To get a lay of the landscape, you can plot semantic parsers along these axes:

Our target is to build semantic parsers that sit in the top right: these are the Jarvis-like personal assistants which consistently answer queries correctly and resolve most of our mechanical intellectual needs. Unfortunately, I would say most semantic parsers today sit on the line $y = -x$: we’ve been able to build parsers that are either really good at a specific/idiosyncratic task (if you’re interested, check out Chat-80) or capable of a lot of really simple things without generalizing well to complex inputs.

To fix this, we’ll want to build a semantic parsing system with the following properties:

• You should be able to communicate as naturally as possible—there should be no requirement to learn an unnatural version of English (or any other language) to interact with it
• The system should generalize to any sequence of digital tasks a human could tell another human to do
• It should know when it doesn’t understand an instruction or its instructions are ill-specified

We can look at these one-by-one and try to get an idea of what we can do.

Natural communication

Banning unnatural sublanguages completely eliminates the relevance of rule-based systems. We can’t expect to solve this problem by mapping formal languages onto grammars of common words that admit sentences that look kind of natural. Because day-to-day communication is fuzzy, our models need to be able to handle fuzzy inputs to be sufficiently expressive. This means we’ll have to rely on statistical methods to infer the true meaning of sentences. Fortunately, this isn’t trading off much and statistical methods have improved dramatically in the past 5 years.

Generalization

Most digital tasks humans perform are not easily specified by a single formal language. Theoretically, we could build a programming language that simulates human control of a desktop computer by moving a mouse and sending keystrokes (simulating typing), but this still suffers from a failure to generalize to mobile devices or other machines that are controlled remotely. It seems inevitable to me that a sufficiently expressive semantic parsing system will support several target languages and adapt to new languages that arise as the landscape of downstream applications continues to evolve.

Most computer applications are reasonably complex and require non-trivial formal syntaxes to express the space of possible instructions. For example, the grammar defining VBA (a programming language used to script Microsoft Excel) is roughly 1000 lines long, and of course the space of possible programs is infinite.

To deal with this kind of complexity and dimensionality, we need lots of data. In particular, we needs a spanning set of pairs of natural language utterances and their corresponding formal statements. Fortunately we can generate an infinite volume of formal statements, but these are expensive to annotate: most use Amazon’s mechanical turk service to label data, which in this case would cost at least $0.70 per label assuming you’d hire labelers with a bachelor’s degree (I think this is reasonable since you’d require them to learn the formal language, which may be quite complex).

We can reduce this cost by designing high-fidelity synthesis systems that generate formal statements and corresponding natural language tags. I think this approach is promising if we stay focused on developing automatic synthesis/annotation methods that generalize to more formal targets.

Knowing what you don’t know

Self-consciousness is necessary in any field deployment of a semantic parser. The literature here is comparatively sparse, but I’ve been pretty excited by recent developments in Bayesian methods of machine learning. The scaling problem still needs to be solved for these approaches to be practical, but fortunately they allow us to inject prior notions of uncertainty that are progressively refined as the model observes more examples.

There are at least three huge benefits to this approach:

1. We can define abstention protocols that prevent the model from executing formal instructions it’s unsure about
2. We can get a sense for deficiencies in the distribution of training examples
3. We can bring humans into the loop more effectively as models actively learn in the field.

Most semantic parsers these days predict formal language tags through a probability distribution over a vocabulary of tokens. For now we can use statistics on that probability distribution as a proxy for confidence, but I’m really excited to see what we can do with more sophisticated uncertainty measures as the logistics of Bayesian methods improve.

Summing up

An effective semantic parsing system would propel us much further towards a world of human-computer symbiosis. To do it right, we want a system that understands natural language as it is, is hyper-capable, and knows about its own fallibility. To make this happen, we’ll need a confluence of research initiatives. In the end I think the payoff would be enormous, and I’m really excited to watch the field progress.

What I’m going to do is write a short piece describing the process I use to define problems and make decisions based on formal reasoning. Why? I want my first essay to have a few properties:

• it shouldn’t be long enough to set a trend of lengthy rants in this column (though I plan to allow my entries to stretch as far as they can retain content)
• it should be about something I’m extraordinarily familiar with so that it’s easy to write
• it should be embarrassing enough to make me comfortable publishing organic content in future essays

So let’s get into it. Claude Shannon defined entropy by first conceiving of a concrete notion of “surprise” and concluding that an appropriate mathematical definition capturing the concept should have the following three properties:

• events which are improbable should also be surprising
• your surprise at two independent events happening together should be the sum of the surprises of seeing each event independently
• surprise should increase continuously with improbability

From here you can use some mathematics to show that the logarithm function is the only function satisfying these properties, and hence the definition of “information” was born. Clearly this definition worked very well, because Claude Shannon rode it into a life of fame and leisure as information theory grew into the basis for all modern digital communication. So what went on here? How did Shannon come up with this definition? Crucially, Shannon didn’t start with his definition and then deduce properties. He started with the properties, and then deduced his definition. This is an application of the wider trend in mathematics to “work backwards,” which is a tactic advocated by most of history’s greatest problem solvers.

I find decisions often consist of defining a course of action, therefore I tend to approach decisions similarly to how I would approach making mathematical definitions. In particular, I like to make lists that answer questions:

• what properties do I want?
• what should my answer do?
• what is necessary?
• how should my answer interact with other things?

An underrated side effect of this approach is that if you develop these lists in collaboration with other people, you very naturally develop a canonical language that enables more effective discussions while evaluating alternatives. If the objectives and criteria are developed with organic and transparent consensus from the beginning of the decision making process, it makes it much easier to express opinions that will be respected and understood as intended.