Syllabus

Semantic Scholar integration

I’ve had a consistent desire to avoid tedious data entry in updating my website. This was part of my intention when I moved it to Quarto in the first place: Quarto gives great opportunity to programmatically generate a static website. I can easily mix in rich documents that combine code (R and Python in particular) and easily formatted Markdown text. This is all great. It let me hack together a nice way to display my publications in a consistent format based on data stored in a hand-curated YAML file. The next step was to automate the process of updating the publication records. I used the Semantic Scholar API to fetch the latest information about my publications and then used Quarto’s templating capabilities to generate the updated publication list.

Most of the work of this system is in two functions which, because I vibe-coded it, I’ve never really even looked at.

def fetch_author_papers(author_id):
    """Fetch papers from Semantic Scholar API"""
    base_url = "https://api.semanticscholar.org/graph/v1"
    fields = "paperId,title,authors,year,venue,publicationDate,externalIds,openAccessPdf,url"

    url = f"{base_url}/author/{author_id}/papers"
    params = {'fields': fields, 'limit': 1000}

    try:
        response = requests.get(url, params=params)
        response.raise_for_status()
        return response.json()
    except requests.RequestException as e:
        print(f"Error fetching data from Semantic Scholar: {e}")
        return None

def create_yaml_entry_from_ss_paper(paper):
    """Convert Semantic Scholar paper to YAML entry format"""
    paper_id = paper.get('paperId')
    title = paper.get('title', 'Untitled')
    year = paper.get('year')
    venue = paper.get('venue', '')

    # Process authors
    authors = []
    for author in paper.get('authors', []):
        author_name = author.get('name', '')
        if 'Dimmery' in author_name:
            authors.append("me")
        else:
            authors.append(author_name)

    entry = {
        'title': title,
        'authors': authors,
        'year': year,
        'venue': venue if venue else None,
        'visible': False,  # Default to not visible
        'ssid': paper_id,
    }

    # Add external links if available
    external_ids = paper.get('externalIds', {})
    if external_ids.get('DOI'):
        entry['published_url'] = f"https://doi.org/{external_ids['DOI']}"

    if external_ids.get('ArXiv'):
        entry['preprint'] = f"https://arxiv.org/abs/{external_ids['ArXiv']}"

    # Add open access PDF if available
    open_access = paper.get('openAccessPdf')
    if open_access and open_access.get('url'):
        entry['pdf_url'] = open_access['url']

    return entry

Critically, this is all pretty trivial, but it’s also boring, so I’m glad I didn’t need to think particularly hard about it. It also means I didn’t need to personally dig through the Semantic Scholar API documentation. The code quality is not great, but neither was my own code before!

So now when I build the site, it automatically updates the YAML with my latest papers. I can just browse through them and flip the visible flag to show them once I’m satisfied they’re right.

Listmonk tools

The second main improvement I made was to add an email distribution list to my website’s blog. This was pretty easy because of the great open source email distribution software, Listmonk, and the super easy deployment option at Pikapods. The system I’ve come up with is basically the following:

1. Write a blog post in a Quarto document.
2. Push it to a branch if I want to have sensitivity readers.
3. Once I’m satisfied, push it to main, which automatically triggers a Quarto build of the website and makes the blog post “live” on the website (and RSS feed)
4. Manually trigger a Github Actions workflow to schedule a Listmonk email campaign.

It’s this last part that I think is particularly cool. Basically the way it works is I vibecoded an over-engineered system that pulls the necessary metadata from a given blog post (in the .qmd file), renders the post to an HTML version, extracts the HTML from that, and sends all of this through Listmonk’s API to schedule an email campaign with nicely formatted HTML content. I’m not going to embed all of the scripts for this here, but you can take a look at them in my website’s Github: scripts/.

To actually trigger this monstrosity, I used a nice feature that Github Actions provides: workflow_dispatch event triggers. Basically, what this means is that you can configure a Github Action to trigger manually through the Github UI. It looks something like this:

So basically all I need to do is tell the workflow what post to schedule (the post_slug part), when to schedule it (the send_at). There are also a few optional things I can plug in, such as which list to send to (Listmonk lets me manage multiple lists easily), or I can send test emails.

This is a solution I’m mostly happy with. I think there’s still a bit of tweaking about some of the practicalities. I think there are two things I’m trying to figure out:

1. Whether I should maintain separate lists for different categories of post. It wouldn’t be hard to maintain category-specific lists so that people could easily opt-out of posts about (e.g.) how I built things on my website.
2. How exactly to manage the sending of emails. When I sent out the last email, it only sent to like two-thirds of my subscribers because my email provider is Protonmail setup with a custom blog@ email address for sending/receiving. Unfortunately, there were some rate limits that I didn’t adequately respect the first time.

I’m overall pretty pleased with this system. You’re welcome to subscribe here!

Improvements in design and performance

Within me are two wolves: Sometimes I miss the days of HTML-only websites. Other days, I long for the ability to have nicely reactive and responsive design elements that require richer web frameworks. On some personal projects over the last few months, I’ve learned a bit more about modern webdev tools. One thing I’m particularly taken by is the way that modern webdev tools like Vite manage javascript dependencies.

In short, the problem with a lot of these big web frameworks is that they’re really big (duh)! They have a lot of stuff that you never use in a small personal website like mine, but they still get bundled into the library that is sent down to the website viewer. We don’t need to do that: we can be smarter. That’s basically what tools like Vite do. They only bundle together the parts of code that are necessary for the particular site being built. They also can help with lazy loading (so they’re only loaded when needed).

That’s all a bit of a tangent, but it’s gotten me thinking about why I actually need to load so many Javascript libraries. Quarto ultimately doesn’t seem to be particularly optimized on this front. This is kind of the tradeoff you get: sure, it’s easy to throw something together with Quarto that has a lot of the bells and whistles a data scientist craves, but that doesn’t mean it’s going to be super efficient. There’s a really nice post by Emil Hvitfeldt discussing performance considerations in Quarto.

Remaining concerns

I’m happy with a lot of what has come together. There are a few issues that I think can be improved. I’m considering whether it might make sense to switch to a hybrid system which would use Quarto to build minimal HTML or Markdown documents which are then knit together using a modern web development framework like Astro. This would allow much more granular control over the website and how it loads resources, but could still get the benefits of Quarto where it is most useful (for pre-rendering rich documents full of Python and R code). This is probably a bigger project than I imagine it to be, so I don’t intend to commit to it soon⁴.

I’m also considering whether it would make more sense to switch to a different hosting solution. I’m not particularly pleased with the quality of Netlify’s CDN, which seems to be a bit slow in Europe (I suspect they just have fewer edge locations)⁵. I’m considering whether it might make sense to switch to a system which is self-hosted in a VPS like DigitalOcean/Hetzner (possibly with a tool like Coolify to manage deployment) and then add a CDN like Cloudflare over the top. The benefit to this is that it would combine control (from the self-hosted VPS) with the performance benefits of a reliable CDN through Cloudflare, but it would remain quite cheap (albeit not free).

Tending the garden

There’s something satisfying about all of this tinkering that I find hard to articulate. It’s not that any individual piece is particularly impressive or novel. Rather, it’s the act of construction itself—of shaping raw materials into something that reflects intention and care. McLuhan observed that we shape our tools, and thereafter our tools shape us. The platforms we’ve ceded our digital lives to have shaped us into consumers of content, optimized for something outside of our control. Building my own corner of the web is a small act of resistance against this.

I don’t think everyone needs to hand-roll their entire tech-stack or obsess over font choices (I will tell you fonts are a huge net-negative on my measurable productivity). But I do think there’s value in the process of understanding and improving the systems we inhabit, even if only to understand what we’ve given up by outsourcing them. The web was supposed to be a place where anyone could plant a flag and build something. Somewhere along the way, we traded that for the convenience of renting space in someone else’s domain.

These updates—the automation, the newsletters, the performance tweaks, the typography—are all just ways of making this place more my own. The work is never finished. There’s always another edge to smooth, another inefficiency to address. But that’s the point, isn’t it?

Platforms constrain freedom

Platforms exist to make it easier to do things by creating one streamlined avenue to accomplish a goal. A canonical example is Facebook winning over other social media sites like MySpace. It might be hard to recall now, but one of the main reasons for this was that, in contrast to MySpace, Facebook provided a simple, clean, consistent user experience. On MySpace you would arrive on a page, have music randomly play at you and be barraged by weird fonts, colors and images. Facebook constrained how users could express themselves quite substantially, and this was very popular. It let people ignore all this wacky design stuff created by thirteen-year-olds and focus on what they wanted to do with social media: poke each other, write weird status updates and check out relationship statuses².

I think this is a pretty common pattern for new platforms: They do, in fact, generally make something that people want to do easier! Of course, given the nature of this post, I need to address the Substack in the room. Substack is generally a good platform. It makes it easy to write (certain kinds of) blogs and it makes (certain kinds of) monetization easier. The parentheticals, however, are what I find frustrating about platforms. By their nature, they constrain the user to a particular form of the thing they provide.

Scale means that Substack can’t and won’t cater to all of the small, weird features that people want. If I were designing my own blog system, I would want (a) control over exactly the fonts and styles my content is displayed with, (b) easy integration of math via the great tools like MathJax, (c) the ability to have code-rich documents like Quarto in which code runs in-line and displays output in-line, (d) the ability to embed any kinds of content I want wherever I want it. Ultimately, I think this is the dream of the web: I can have a little homestead on the internet that is exactly the way I want it: a convivial web³. Platforms cannot provide this: their raison d’être is reducing complexity as they make human behavior digitally legible down a particular prescribed path.

I dislike this constraint: I want to use tools that work the way I choose for them to work.

Platforms enshittify

Even if you feel well-served by a platform now, this is no guarantee that the platform will continue to serve you well in the future once you’re stuck there. I won’t belabor this point, because it’s been talked about ad nauseum by others⁴. Platforms, as they mature, need to monetize more aggressively as well as build moats to competitors. This often leads to more constraining of user freedom. I think the real “break-glass” moment for Substack is the ability to collect and export all subscribers’ email addresses. This is, right now, a defining feature of Substack that makes it safe for people to use. This drastically reduces their moat, however. At the moment, switching to Ghost is (technically, at least) pretty trivial. Switching to a Quarto blog deployed through Github Actions and which sends a newsletter through Listmonk (deployed via Pikapods) when triggered through Github Actions is not trivial, but it also wasn’t exactly hard to setup⁵.

The main way that Substack appears to be aiming at constructing a moat is in their push for discovery. That is, they want to make it easy to find Substack writers on Substack. They have no interest in doing this with writers hosted elsewhere. This provides an incentive for people to stick around, even if the actual writing and publishing experience is worse. I don’t think Substack is made up of evil people, quite the opposite, I think they truly wish to help writers make a living (while taking their cut). The problem is that they have few incentives to make it possible for writers to truly experiment. Why would they? Writers will want to stick around because they can get more subscribers there, not because the CMS is uniquely good or they can uniquely realize their vision there. The one exception is that many writers’ visions are simply “to be able to feed themselves through their writing”. I think this goal would be better served by increased experimentation⁶, but I recognize beggars feel that they can’t be choosers. I think this is a false dilemma, because:

AI coding tools make smaller projects feasible

Platforms work by making a particular workflow easy enough that we accept lock-in. An alternative path exists, however, which is to build our own tools to support that workflow. A critical fact of contemporary society is that this is becoming increasingly easy. I think a lot of people have, thus far, only interacted with LLM systems through the lens of the chat interface. While this is impressive technology, I think it blinds a lot of people to what is now possible. Let me briefly detail an example.

For my website, I decided I didn’t want to manually write out the metadata for my publications anymore. So in one afternoon, I vibecoded a system which connects to the Semantic Scholar API⁷, checks for new papers on my profile and if it finds them, sets up the metadata in papers.yaml. It doesn’t display until I flip a switch from visible: false (the default) to visible: true. Thus, I’m not too worried about the system going haywire: nothing is posted without me getting the chance to review it, but it reduces tedious data entry. The technology is there to vastly reduce the time between an idea and a prototype of that idea. For this Semantic Scholar system, it took me just a few minutes to set Claude up with instructions, and I had a complete and functional prototype no more than one hour later, most of which did not require me to be thinking about the implementation (verified via commit log).

It’s substantially easier to build things now. As the effort to build novel computing systems shrinks, it’s easier to build exactly the systems you want for yourself (or for yourself and a few others). We should do more of this, on the margin. Individuals’ capabilities to personalize their computing environment has literally never been higher than it now is. Choosing not to take advantage of this is a choice to let other people (who do not have your best interests at heart) choose your tools for you. The point of platforms is to capture you by making it easy for you to achieve particular goals (but restrict you to those goals they deem worthy), but the thing that’s different now is that it’s also easier than ever for you to build what you need for yourself.

Technological vanguardism

Open source software is a critical part of this story, as it serves as the backbone for building any kind of new software in the current era. This is recognized in a variety of initiatives as part of EuroStack⁸. That is, the idea is that Europe can’t or shouldn’t try to compete with American and Chinese hyperscalers by trying to incubate European hyperscalers as competitors. Rather, Europe should commit to smaller alternatives based on a robust shared digital infrastructure of open-source software. AI-supported individuals able to mix-and-match together the computing systems they want to support their lives is a rare hopeful vision of the future.

A lot of the maintainers of critical open source projects are getting older⁹. I wonder whether the next generation will have the same commitment to the principles that underlie the open source movement. In particular, we’ve all discussed how “the youths” increasingly don’t understand file-systems or whatever your preferred concern is. My worry is that they have become so used to an Internet made up entirely of platforms that they can’t even imagine anything different: an Internet where when you want to create a space to chat with friends or colleagues the only option is Discord or Slack controlled by someone else, never hosting a Ventrilo server, a phpBB forum, an IRC server or, in modern terms, software like Zulip or Matrix.

Extensible tools like this often have a higher entry cost than platforms do: they may require standing up servers, writing some code or doing some non-obvious configuration. This has been a major problem with Open Source Software for ages: they’re built by developers for developers. Open source developers and maintainers of open source projects are a sort of technological vanguard who accept some of these costs on themselves. They build tools, often just for fun, because they can and it scratches a weird itch they have in their own lives. Sometimes, these projects fit the needs of the moment and take off, but many times they remain personal or niche. It’s important that the people who can build such tools do so to pave a (code)path for others to follow, but they are not usually about making that path highly accessible.

The costs of building one’s own systems are lower than they’ve ever been because of AI code assistance. This provides a critical link between the Open Source Software ecosystem (which is the fundamental infrastructure underlying modern software of all kinds) and personal use. That big open source project which was almost right for you? It’s not hard to make minor modifications to it now to make it just right.

The problem, however, is that every person who accepts Platforms into their hearts does create a social cost as well. If everyone only goes to Substack to get blogs, then blogs hosted elsewhere (no matter their quality) will be read less. This is what Substack is hoping to achieve, and I think it’s important for those of us comfortable with forging a different path to try.

I’m a technological determinist: I believe that the technologies we create fundamentally transform our lives and societies. I also believe strongly in the power of human agency: we can reshape the technostructure to harness these technologies the way that we want. We can build the society we want, but that does mean that we need to actually do the building. I, for one, am planning to become much more serious about putting in the work on this. Moving this blog back to my own website is a first step in this direction, but it is not the last.

Image source: https://brooklynrail.org/2017/10/criticspage/Hidden-Figures/

How to respond to a great paper

This was my first reaction to thinking about how to do this. I love the paper, and I refer to it a lot. But a response isn’t that interesting if you’re just saying “ditto”. We started out with Kevin wanting to talk more about how uniform consistency of the SATE was still insufficient for science (plus various other big picture philosophy of science points) and me wanting to discuss different reasons why RCTs are special (e.g. correct specification through shoe-leather)². Regardless, we quickly latched onto the most provocative word in the title: “enough”. What does it mean for a method to be “enough”? Musing on this formed the core of our response.

What was Facebook?

Facebook was an internet technology company from 2004 to 2021. It no longer exists, having been succeeded by “Meta”. At its peak in 2021, it had70,000 employees and 118 billion in revenue, truly staggering numbers that are difficult to comprehend. Around half of those employees were in technical roles³: they directly worked on changing and improving the functioning of the services that Facebook offered. Can you imagine working on a project with 30,000 other people? It’s critical to understand that this is part of what it means to work at Facebook. And it really felt like this, sometimes! Internal tooling changed often (because people were actively working on it!), the use of a Monorepo meant that all engineers pushed code to the same repository and there were meaningful connections in code between projects. To land a commit at Facebook famously only required you to get one other engineer to sign off on it⁴. The number of commits to the repository was so large that you couldn’t expect to have the same ground truth from one second to the next⁵. For a sense of the speed at which Facebook changed, read this post detailing some of the deployment systems Facebook used written by Chuck Rossi, a famous⁶ release engineering guru at the company.

Experimentation at scale

Facebook changed quickly. Temporal validity is a real problem when things change at this pace. But the problem was actually even worse than the sheer scale would suggest! Many of the biggest changes to how Facebook work were gated not by commits of code, but by A/B testing⁷. The number of active experiments at any one time was on the order of thousands. Even assuming each test had only two groups, this number implies that no two people were likely to experience the same Facebook⁸. I wouldn’t be surprised if, given the increase in employees relative to when we roughed out this number in around 2016, that number was an order of magnitude higher by 2021⁹.

It’s worth thinking about how these experiments worked a little bit. One thing the handful of people implementing a single test absolutely could not do was to control everything about how the other 30k engineers run their tests. This is simply the reality of working on a project with this many other people.¹⁰ This led to difficulties occasionally, in which stuff about the platform changed and rendered a prior test uninformative or incorrect about the current state of the world. On my team, this is why we encouraged two practices: (1) long-run holdouts (where for important features, the rollout of the feature would be to ~99% of users rather than 100%) (2) rerunning important tests to verify results continued to hold.¹¹

Everything is evaluated relative to the sum total of other stuff Facebook is doing on the platform.

The reality of massive simultaneous experimentation and change resulted in internal experiments using the term “status quo” condition rather than “control” condition in experiments. Everything was evaluated relative to the sum total of other stuff Facebook was doing on the platform. As an example, check out the software my former team developed for doing complicated experimentation at platform scale. There is no “control” group, there’s just the “status quo”: what the platform is currently doing. The other groups are potential modifications of that behavior. The reason we evaluate relative to this group is because experimentation is a means of optimization¹². Experimentation was a tool to improve how the system worked, so we wanted to evaluate our one change relative to the otherwise current state of the system. We never care about comparing to the state of the system 1 day or even 5 minutes ago.

This is one reason why I welcomed the eLetter by Bagchi et al at Science: it’s important to remember that “Facebook” (the platform) is a moving target¹³. In short, during the course of US2020, the company implemented a number of other measures aimed at combatting bad behavior during the election (“break-glass” measures). A lot of people forget how large and diverse an organization Facebook (the company) is. The “break-glass” measures discussed in the eLetter are a great example of this. One hand certainly can’t control what the other hand is doing, but it often isn’t even aware of what that hand is doing¹⁴. Unfortunately, I think this important message is undercut by another one in the eLetter:

This can lead to situations where social media companies could conceivably change their algorithms to improve their public image if they know they are being studied.

This implies a single-minded agency of the daily practice of Facebook engineers that is unreasonable (and largely impossible). As I tried to emphasize above, the company is huge, diverse and fast-moving¹⁵. The vision of a monolithic company exercising its will is simply inaccurate, especially when considering a project as small as US2020. Lots of things were changing during the course of this study; that’s (a) entirely normal at this scale and (b) why it’s important for us to be humble about its temporal validity (which I think we were)! The thing to be angry about here is not that the company tried to tweak the results of US2020 to make them come out hunky-dory: the thing to be mad about is that we have no truly reliable measures of the effects of these break-glass measures!¹⁶ We need more testing, not less.

The thing to be angry about here is not that the company tried to tweak the results of US2020 to make them come out hunky-dory: the thing to be mad about is that we have no truly reliable measures of the effects of these break-glass measures!

Was US2020 independent or industrial research?

This leads to what I think is a more important question about the nature of this project. Was US2020 industrial research, guided primarily by the needs of Facebook (the company)? The answer is clearly no. The academics proposed the designs, they had final control rights on the resulting papers and their salaries were in no way dependent on the results of these papers¹⁷. This is all critical, as there is an important difference between the Facebook and academic researchers in the project: Academia has academic freedom, while industry researchers do not. Does that mean that these papers are actually independent, then? Of course not. Seeking independence is a category error for this project. The internal researchers have incentives, as do the academics.

Fundamentally, these parties have different things they bring to the project: internal researchers knew how the platform works in detailed ways that no external person could and they had the capacity to actually work with internal platform data. They had been studying the ins-and-outs of this platform full-time for years. External researchers, on the other hand, were not burdened by corporate incentives and they had the academic freedom to speak freely about the project and their beliefs¹⁸. All of these perspectives are necessary for a successful collaboration, but it would be incorrect to suggest that they were equal partners. They cannot be equal, because they contribute in radically different ways.

The best one can hope for is to reconcile competing interests by writing down what everyone wants from the project, having an argument about which things can be agreed upon and then choosing to do those things or not. We already have a great tool for this kind of collaboration¹⁹.

Pre-registration as a contract

The solution, I think, is setting out a clear contract between the parties at the start of the project. We actually already know what these are: pre-registrations. The pre-registration serves as a contract about exactly what you plan to study and how you plan to do that²⁰. Filing it publicly in advance can then tie your hands and commit you to doing what you said. While problems inevitably arise post-data, it’s easy to file amendments as necessary. Cyrus Samii wrote about how he (and EGAP) sees pre-registrations that I think relates very nicely to this view, as he talks about the value of pre-registration as a conversation about the research you plan to do. It very much strengthens the quality of the work you will end up producing!²¹ When parties are collaborating on research and the parties have different incentives, a pre-registration serves as a Memorandum of Understanding. It gives you a specific document on which you can have an explicit negotiation about what you will do in a transparent way. For example, if the parties can’t agree on a definition of the ‘control’ group, then it must be redefined to reflect what can be agreed on. If in the course of executing the project, one side makes a unilateral decision about something substantively important that would not have been agreed to by the other side, then this is an error in the contract for lacking specificity over something important. Take the lesson and apply it to the next project.

This is the most critical part of the design of US2020. Take expert internal researchers who have a clear conflict of interest (but know more about the actual workings of the platform than anyone outside it) and combine that with independent researchers who are not beholden to the company and are only interested in learning something about how it works. Let these two sides negotiate on what would make a mutually beneficial research project, write it down, do that project²². The appropriate reading, then, of these studies is as the result of a dialogue²³. And to the point of the eLetter which sparked this meditation, the pre-registration documents we put together make clear our choice of comparison group which is exactly why these documents are so valuable.²⁴

Early on, it was decided that the project should speak with a single voice. This makes sense given how an academic paper typically works with only a couple authors: consensus works, and every author can speak their mind about the paper. In my opinion, it doesn’t work for a collaboration as large and diverse as US2020. In US2020, only the academics have the freedom to say what they think, and the design of the project means that they are always the ones who have the final determination over what gets written. In practice, this almost always resulted in decision by consensus. There was really very little disagreement because everyone sought the best scientific result. That said, disagreements existed. For example, from Michael Wagner’s report as the rapporteur of US2020:

Specifically, Meta researchers wanted to be able to expressly disagree with lead author interpretations of findings or other matters in articles they coauthored. Stroud and Tucker opposed this effort, noting that collaborators can remove their name from a paper if they have a fundamental disagreement.

From the way the project was designed, this was the appropriate decision (downstream from choosing to speak with only a single voice), but for the design of future projects, I vehemently disagree with this structure. All members of US2020 are expert scientists, many of whom have different perspectives. The remediation for disagreeing with anything that appears in one of these papers is, for Meta researchers, to entirely take their name off of a project which they have poured, likely, hundreds of hours into: the nuclear option. Much better to have some forum (an appendix?) for authors to include a personal statement clarifying their personal perspective on the paper and how that diverges from the main text²⁵. This acknowledges that no written document can perfectly express the views of dozens of scientists, and provides far more information about what the project shows and what may be debateable matters of interpretation. Anyone who has talked with a scientist knows that if you put two together in a room together you’ll have at least two perspectives on any topic under the sun. It also better expresses the reality that everyone involved in the project is a scientific expert: not merely those in academia. This sets up what I think is a common misapprehension about the project.

Facebook researchers were not robots

In the editorial published along with the eLetters, Holden Thorp and Valda Vinson wrote:

In a statement for this editorial, Chad Kiewiet de Jonge, a research director at the company, Meta, insisted (to H.H.T.) that it had been forthcoming about the emergency measures to the researchers.

Reading this, it might surprise you to learn that not only is Chad an author of the published paper in question, he’s in fact one of the lead authors. That is, he has been in all of these meetings where such things would be discussed! To quote from the detailed list of contributions:

N.J.S. and J.A.T. were joint principal investigators for the academic involvement on this project, responsible for management and coordination. C.K.d.J., A.F., and W.M. led Meta’s involvement on this project and were responsible for management and coordination. [emphasis mine]

Frankly, this has been a problem from the beginning. As I said immediately on publication, “This project was a deep collaboration, and generally, I think, not an adversarial one, either. Everyone wanted to get this right!” Nevertheless, it hasn’t always been presented like this. If you read the project as a dialogue between internal and external researchers (the accurate view, I believe), the only way that this could possibly be “independent” as in the standard set by Michael Wagner is if the Meta coauthors were robots who merely carried out the precise will of the academics²⁶. This is clearly bad for the scientific output, and it isn’t what actually happened, as Wagner notes:

On occasion, this, and advice they received from Meta researchers, led the outside academics to reshape what it is they sought to study, or how they sought to study it, as they learned details about what data were collected and structured by Meta for analysis. For example, the outside academics were not precisely aware of how Facebook groups were joined and, in some cases, left, by users. Coming to understand this led to changes in the working paper examining behavioral polarization.

And yet, there is an implication that Meta researchers were trying to hide things, as in the quote he shares from a former employee saying,

Facebook researchers will answer every question they get from the professors honestly; they are ethical professionals. However, they are also corporate employees. If the professors don’t ask the exact right question, Facebook staff won’t volunteer that they know how to get what you (outside academics) are really asking for.

It’s very weird that this is attributed to a former employee rather than either the academics or to anyone involved in the project. From everything I saw, Meta researchers didn’t just narrowly answer direct questions, but bent over backward to try and re-interpret requests to get at what the academics’ questions intended. For my part, on one paper I spent a lot of time on, I wrote dozens of pages trying to redefine an unworkable analysis an external academic suggested to one that would be workable — while preserving the research questions. Other Meta employees did similar work on basically every single paper. That work is, in effect, an act of persuasion aimed getting the academics, who have the control over the content, to adopt the change or redefinition based on their view of the scientific merit. To what extent was it based on corporate incentives and to what extent was it based on scientific judgment? Likely a combination: I believe more of the latter than the former. But the negotiation of the pre-analysis plan is the way that this is adjudicated, and in US2020, the external academics were the only ones with the power to decide.

I think the problem is that people can’t help but see Meta as a monolithic organization which exhibits a single will. This is seriously incorrect in almost obvious ways: employees disagree about, well, basically everything! Dan Davies’ book on the flows of accountability through the cybernetic lens is a useful correction to this vision. The company “Meta” is distinct from the sum of its individual parts: the procedures and processes that it uses to determine how to operate are largely not under the control of any singular person. In short, it is an organization and this imposes constraints on employees. And yet, journalists and editors of Science seem to have trouble understanding this. Internal researchers must seek permission to do this kind of research (a heavy constraint), but conditional on permission to do so, there is no instrument of Facebook’s will hanging over every discussion and decision they make. This is simply not how information is transmitted within the company²⁷.

Specifically on US2020, employees fought for years to enable this kind of research. There was extreme skepticism among executives precisely due to a belief that there’s no way that the project could be “good” for Facebook. The project exists despite the fact that Facebook (the company) was not particularly in favor of it. Rather, the researchers you see listed as coauthors are on there specifically because they thought it was important to do rigorous, transparent social science about the effects of social media. As a personal anecdote, I was advised against participating in the project if I wanted to optimize my career at Facebook: it’s much more professionally rewarding to make the system work better than it is to do this kind of evaluative work²⁸. That said, I don’t mean to paint all internal researchers as exactly like people who choose not to work for Facebook. The internal researchers are unlikely to have seriously negative beliefs about the platform, so they may demand stronger standards of evidence for harm than might be acceptable to people more willing to see the inherent evil of the company.²⁹ That’s why you have the argument before you see the evidence: in a pre-analysis plan.

What does the future hold?

I think there’s a real danger in misrepresenting what this project was, but that danger isn’t about falling prey to Facebook’s manipulation. Instead, the danger is that there may never be something else like it, even though it’s the single best way we’ve managed to study the crucial question of how social media affects societal outcomes. I think that collapsing all of the internal researchers into a single “Facebook” blob risks poisoning the well. This project exists only because of individual people—many working within Facebook—working hard to make it exist. Mark Zuckerberg, or any other embodiment of Facebook (the company), did not ask for this project. He did not push for it to exist, and he certainly did not ask for the rigorous structure imposed on how the collaboration would work. The right way to think about this project is that Mark Zuckerberg was willing to humor the desires of people within his organization for rigorous social science. After all, how much worse could the results be than what people already believed?

For my part, future research collaborations I have with industry researchers will not pretend to independence. Where possible, I will manage conflicts of interest with pre-analysis plans (advocating for what I see as scientific merit), but I think aiming at subordination of these researchers is ultimately self-defeating. When interpretations of results differ, I will resolve disagreements through an appendix (like a “deviations from PAP” section) identifying the different camps of belief: thus bringing it out into the open of scientific discussion. I will bring the dialogue to the forefront, not hide it out of a misguided (and impossible) search for independence.

By far the most important lesson for social science when faced with a complex entity like the Facebook platform is that we should not be content with a single snapshot about its impact. Acknowledging the size and scale of the engineering effort behind this company must mean acknowledging the necessity to run and rerun tests about important potential interventions.

Would the results of the chronological feed experiment hold in a different election? Without the break-glass measures?³⁰ In a different country? The only way we can ever know is if we run more tests. The only way we can run those tests is if we understand what US2020 was and try to improve it so that the next version is better. Either way, when I was at Facebook, we would never leave something un-tested and un-optimized if we thought it was important. Maybe the only way we’ll get such tests is if governments require them.

On October 29, Senator Markey sent a letter to Zuckerberg urging further independent research on the societal effects of Meta in the vein of US2020. A Meta spokesperson just pointed back at the work from 2020 (much of which is still yet to be published), as if this is the final word on the subject. There won’t be a US2024, and it looks like there won’t be a US2028, either.

What is Smalltalk?

Smalltalk was developed by Alan Kay, Dan Ingalls and Adele Goldberg at Xerox PARC in the 1970s. What it was is a little hard to frame in terms of our contemporary categories around computing. It was more than a programming language and not exactly an operating system as we’ve come to understand them. It was the software part of Alan Kay’s overarching personal computing vision (from back before computing was really personal computing). Smalltalk had a lot of things that are now very common, but were very uncommon when it came out. It was an early iteration of Object-oriented programming¹, it had an early graphical interface, all code was just-in-time compiled (and could be recompiled on the fly), and it integrated a development environment into how the user interacted with the system. But Smalltalk was also a philosophy of computing:

look for the distinction between a programming language and a programming system, and consider the difference in providing a system in which the user can feel individual mastery over complexity

(emphasis mine) The intention was to make the vast complexity of a computer system something that end-users could control. The system should allow the complexity of the system to be comprehensible to the user, and allow them to adapt and reorient it at their will.

A lot of what I’m drawing on here is from the 1981 issue of BYTE Magazine in which PARC chose to write a public introduction to the newest iteration of the Smalltalk system. It’s available online and a truly fascinating read that I can’t recommend enough.

Smalltalk was intended to give the User the power of God over the computing system. This isn’t really subtext. In this introduction to Smalltalk, they provide the following illustration:

On the far left, the System Programmer (God) creates a system (the Taj Mahal). The Programmer creates things (a bridge) for the end-user using the things provided by the System Programmer. Smalltalk was intended to democratize the power of creation, moving into the second figure from the left, in which the Programmer creates kits that the User can use to construct what they want. In Figure 3 this goes even further, as Users can create the kits themselves that they can assemble into what they want. On the far right, however, the idea is that Programmers / Users should be able to climb up above God and impose their own will over that of the System Programmer: to become God.

It’s a vision of a radically empowered user who can manipulate every aspect of their computing environment. In my mind, this vision is exactly what it means to have a “convivial” tool in the sense of Ivan Illich: tools which empower human liberty rather than constrain.

There are costs, however, as Smalltalk asks much more of users, as Kay and Goldberg write in 1977:

The burden of system design and specification is transferred to the user. This approach will only work if we […] allow ordinary users to casually and easily describe their desires for a specific tool.

But the benefit is a fully “reactive” system². What did this look like?

Essentially, you had a browser like the above, through which you could navigate through your system, then the panel at the bottom allowed you to edit the code defining how that part of your system worked. You could then choose to re-compile that part of your system and it would automagically update to follow your new instructions. Voila, complete control over the system, manipulable at runtime.

But there’s one last quote, from Daniel Ingvalls from his article about the design principles of Smalltalk:

Natural Selection: Languages and systems that are of sound design will persist, to be supplanted only by better ones.

Our computing is not based around Smalltalk these days, nor does it look very much like this: it died out in favor of opaque systems with a lack of user-control. It seems that users were, in the end, not willing to take the burden of full responsibility for the functioning of their own computing environment.

Empowerment is still about power

I think the question of inevitability is what I really struggle with here. The dream of a system that radically empowers the user is beautiful. But are users willing to actually take this power? Can they? In grappling with this question, I tried to think about affordances of social media and the extent to which they align with Smalltalky principles.

• Bluesky lets users choose their own Feed algorithm.
• The hashtag was created as a tool for users to take control of discovery and categorization themselves.
• Distributed social media services (e.g. Mastodon) seek to allow individuals to choose the community (and, therefore, community norms) they wish to belong to.
• individual choice about network links to create (e.g. friending on Facebook): this is the source of a lot of ideological polarization, rather than something done “to” users by algorithms. In each of these, the user gets pretty profound ability to shape what their social media is. But it’s still not enough! My choice of a feed algorithm doesn’t give me the experience of using Bluesky that I would have if everyone were to use my preferred feed. In order for me to “choose” my experience on the platform, I need the power to choose how everyone experiences the platform. Empowering users doesn’t avoid the critical question of who holds the power to make such changes.

In a previous post, I claimed that there were four main definitional characteristics of microblogging: (1) short texts (2) virality (3) simple engagement possibilities and (4) an infinite feed. The users can’t alter these characteristics on BlueSky or on any other platform. Removing reshares from your own screen doesn’t change whether you experience virality and the associated credibility cascades and everything else associated with virality. To kill virality, you’d need to make aggressive changes like removing the visibility of likes from everyone’s screen, removing ranking based on likes, views, clickthroughs, killing reshares globally and the list of changes goes on. After making all of those changes for everyone then maybe you’ve done a good job of breaking virality. Can you give virality back to a single user, at that point?

It is fundamentally impossible for everyone to alter everything about their social media experience to their whims, while continuing to make it meaningfully social. Even in a world where everyone gets to choose everything about how they interact with social media, they don’t get to choose how everyone else interacts with social media³. Our experiences online aren’t just shaped by our own choices, but by the choices of those around us⁴. We can’t “solve” our experience by changing only our own affordances.⁵ Bluesky can only ever give the illusion of autonomy, it can never provide true control over the environment as envisioned by Smalltalk. That autonomy only works for a computing platform that is fundamentally a-social: it interacts only with you and responds only to your preferences.

Convivial, anarchist empowerment to shape one’s own technical reality doesn’t offer a way out of the shared reality we create with technologies. At best, it can offer a patchwork of small fixes. We cannot fundamentally alter the medium, because the medium is defined by what is shared. The critical question is who has the power to choose the shape of the overall technology, and who gets to study how the world is changed by those choices. Right now, only the platforms have these powers.

The Apparatus, explained

I think it’s worth talking through how change happens in Tech. I’ll focus on Meta, because it’s the place I know best. This is basically the guts of how the Apparatus gets created.

The problem faced by Tech companies is, put simply “how should we decide what to do?”. There are a number of approaches to this, but I’m going to contrast two:

• Build stuff and then ask people how/whether they like it.
• Build stuff and then see how/whether people use it. We have to keep in mind, however, that we’re fundamentally talking about making decisions for a product used by (e.g.) 2 billion people. We have to think about how these two approaches would work in that setting, i.e. asking the question “Will it scale?”

The first approach would rely on large-scale surveys. There’s no real way to build up qualitative knowledge about 2 billion people, so we need to be quantitative, and surveys are pretty much the only game in town if you want to get folks’ opinions like this. There are hard limits here, though. At the limit, you can’t (for instance) ask every user how their experience of the site is every five minutes. At Facebook, the standard “cool-down” period after taking a survey was six months when I was there. This imposes hard limits on the precision with which you can actually measure the opinions of users².

The second approach would rely on directly observing behavior and encoding this into meaningful measures about what people are doing. Most of this is just counting things that people are doing³. Because you don’t ask for any of your users’ attention to do this, there aren’t really any practical limits to your measurement of on-platform behaviors⁴.

Given these practicalities, it’s unsurprising that the result is an almost ideological belief in behavioralism: that the only way to understand what users want is to observe what they do⁵. My understanding is that this was part of an inspiring introductory speech that Chris Cox would give to new hires⁶.

The Apparatus, sustained

This process results in a cycle of iterative improvement. Matt Hindman’s book does a good job of describing the power of this cycle. It’s also the basis of the Agile software development philosophy, expressed at Facebook as “ship early, ship often” or, more provocatively, “move fast and break things”.

This saying has, I think, always been misinterpreted. The point is not that you’re okay with breaking things. The point is that you’re okay with trying new things, because you know that you have systems in place for identifying when things break. Many of the neurotic high-performers that get hired in tech are perfectionists, so you need to encourage them to not spend years working on something before launching it and seeing how it works. This should really be interpreted through a mantra of the Agile Manifesto:

Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.

Over time, trying a bunch of stuff (some of which might seem crazy ex ante) and watching what users do in response to it is a great way to build a successful product. I think this is one of the defining features of the secret sauce of modern Big Tech.

This perspective flows into incentives throughout the company. Software engineers are, in large part, promoted and compensated based on how well their changes result in user behavior that is deemed “good”. The number went up? Congratulations, you get a bonus. You can break apart this monolith, but this isn’t a bad first order approximation⁷.

The Apparatus, examined

This discussion should make it clear that there’s no sense in which any person understands “the Algorithm”, as Kevin argues. Wouldn’t that be comforting? Stafford Beer had a famous saying, “The purpose of a system is what it does”. This isn’t a reassuring sentiment. We have to work extremely hard to see and measure what a system does. There’s no easy shortcut through “intentions” to understand a system. Even with a truth-serum, we can’t just ask Zuck what he intended to build and then decide if that’s good or bad⁸. No engineer fully understands the entirety of the complex system they’re working on, so we can’t judge the worth of what they’ve built solely on what they intended to build.

So what is the purpose of the apparatus? Well, in one sense it’s essentially the aggregate of the incentives for all the people building the apparatus: making the numbers go up. If we want to understand the purpose of the apparatus, we need to encode the things it does into numbers, and measure how changes to the apparatus result in changes to those numbers. There’s only one consistent way to do that: experimentation⁹. And we need to find the right new numbers to measure, because there will undoubtedly be unintended consequences.

But we cannot forget that the essence of the behavioralism in Tech is that we are those numbers. We are as much a part of the apparatus as the engineers who build the system: every post, every click, every message, every linger over a story is part of the apparatus. This isn’t just true in the sense of our being complicit in the system. It’s true mechanically in the way engineers construct the system. The builders of the system want to give us what we want.

Thanks for reading Drew’s News! Subscribe for free to receive new posts and support my work.

What are we doing?

First, a digression. Latour and Woolgar (1979) is often credited with the idea that the main thing academics do is publish papers. Our incentive systems are fundamentally built around this idea: if you publish more papers that appear in good journals, you will be more successful. It will cause you to be paid more, have more prestige and, generally, wield more power. I used to think that this meant that what many academics were aiming to do was to simply write more papers. I am increasingly feeling that this is incorrect. Academics want to produce more papers, but they have no interest whatsoever in writing them².

In the course of events, a major perspective has arisen, probably expressed most forcefully like this:

Andrew is a trained economist who does good empirical work in the way trained economists do it. And, as he is telling you here, he does not care that much about his lit review section. To be a bit more blunt, I think the argument is basically that empirical social scientists aren’t writing papers, we’re producing estimates. We produce a couple of these estimates per paper, and they are the only actual information in the paper. We’re forced to surround these numbers with words because of arcane medieval institutions, but we all recognize that this is absurd and nobody cares about the words, only the numbers. The right way to read a paper is to skip all the textual mumbo-jumbo, look at the charts, see the numbers, then maybe read the methods section if their approach isn’t obvious. This is reflected by the way we do meta-analysis: each paper is one dot on a single forest chart.

To be clear, I think this is a heinous misrepresentation of what we should think of ourselves as doing. If what we were doing actually mattered, I might even say that I think this view is evil.

Ok, but what are we doing?

The work of quantitative social science is to take an irreducibly complex system and reduce some aspect of it down to a 20 page pdf. If we have even the slightest amount of humility, we should instantly understand that this is an impossible task. It is an even more impossible task to reduce that same complex system down to a few quantitative estimates: <20 bits of information³. The reason we have 19.9 pages of text around those numbers is for a few reasons:

• because we’re trying to be humble by acknowledging all of the uncertainty and context around this impossible task
• because we’re trying to acknowledge all of the other human effort that has been put into understanding this aspect of the world before (“standing on the shoulders of giants”)⁴

The idea that either objective is well served by copying from the abstract of another person’s work is starkly offensive. It brings to mind a conversation I was having about Ian Hacking the other day. One of the truly remarkable things he does in his writing is create fluent and meaningful reviews of complex work by other people. That it is so easy to read is because he takes the time and effort to translate that other work to the specific questions and topics he’s aiming to consider. This is a monumental effort! It requires a detailed understanding of both the task he has set before himself (e.g. “what do we know about what science is?”) and each of the distinct efforts before that touch on this question — including all the myriad ways that those prior efforts failed to answer the fundamental question, answered a slightly different question or otherwise went astray.

Is what we’re doing different than this? Fundamentally, I argue it is not. Unless we are completely faithfully replicating an analysis that has been done before⁵, we ought to have some fresh perspective we are bringing to the table: the relationship between the current work being performed and what has been done in the past will not be the same as pre-existing work. I think the discussion about copying summaries of papers back and forth completely misses the task. A much better term than “literature review” is, I think, “related work”. It highlights that what is being done in the section is highlighting connections between works. Why on earth would you just provide a generic summary of someone else’s work in your paper? A good summary of related work should be opinionated and it should deal in comparisons: it should be providing a contribution in its own right as a way of synthesizing existing work to come to conclusions about what is known, what isn’t known, and what the strengths and weaknesses are of the existing body of work.

This is obviously not easy, and for shape-rotators like myself, this isn’t exactly work that is always pleasant. But, like, it’s part of the job, man.

I suppose the counter-argument is that time spent working on related work is time not spent working on the quantitative estimates. That’s true, but there are two major counterarguments: (i) the thing we’re producing is the paper, and the related work section is a meaningful fraction of that: Spend the time to make it better. (ii) Specialization means not everyone has to spread their time among all sections in the paper. If you’re doing a shitty job on one part of a paper, add a coauthor that can do a better job on that part⁶.

In defense of thinking

My point is simple. For historically contingent reasons (not necessarily good reasons), academia has chosen that the primary way that we should communicate is through the medium of the academic paper. Given this choice, I think it’s pretty important that we actually care about what we can communicate through this medium, which has the capacity to share much more than a few quantitative estimates.

An example given by Matt Blackwell⁷ is illustrative:

This was particularly funny to me because I just wrote a paper where this common boilerplate sentence was probably my most highly edited one through the entire revision process. I will soon publish a broader blogpost around this (now forthcoming) paper, but first I will quote the relevant section:

This relationship closely accords with the definition of calibration in classification and regression problems (Kuleshov et al. 2018), with the added challenge resulting from the fundamental problem of causal inference: Labels (i.e., ITEs) are never observed (Holland 1986).

The bolded sentence roughly accords with Matt’s example of how people usually cite Holland (1986). Crucially, the boilerplate version is not useful for the specific point I wanted to make in this selection! I specifically wanted to indicate (by my use of the very machine-learning term “label” when referring to individual treatment effects) the challenge this well-trodden conceptual problem poses for simple supervised learning. This required different words. It’s my contention that this is almost always the case. If you find yourself repeating something that has been said many times before, maybe you don’t actually need to say it!

We should therefore not normalize the reuse of other people’s words (suited to one particular context, audience and purpose) and re-use them for our own (different!!) context, audience and purpose. I think it is both true that we have sleep-walked into reusing text pretty commonly, as in Matt’s example, and also that we should not do that.

Climbing off the high horse

I recognize that it isn’t always possible to achieve the kind of writing that I’m talking about here. We live in a fallen world. I don’t particularly consider myself a good writer or editor, and everyone faces constraints that lead us to behave in ways that we don’t think are fully consistent with our values. But that doesn’t mean that it’s good or neutral when we make those compromises. It is bad. Plagiarism is bad.

The real problem here is, of course, systemic. We are rewarded for producing as many papers as possible, not particularly for having a well crafted discussion on related work. I do, however, want to at least reinforce the norm here: you should write your own papers, even the parts you don’t like (or get someone else to write them). You should care about writing those sections well, even if they aren’t your favorite part of the paper. They are all part of the work you are signing your name to.

We should not make light of the actual work we do as academics by implying that it is fine to reuse text that is not our own. It is not.

All microblogging is fundamentally the same

There are differences between microblogging sites (I’m primarily thinking about Twitter/X, BlueSky, Mastodon and Threads, but I imagine anything else basically fits the same mold). Nevertheless, I think the core properties are basically the following:

• Short text snippets. I think this is definitional. If it weren’t based around this, it wouldn’t be microblogging.
• Virality. Content gets out of the small community in which it’s posted. This is fun (at first) for the poster and fun for the viewer. The nature of a viral platform means when you see stuff from outside your community, it has already been vetted, so it is almost definitionally ‘highly engaging’ for some highly specific definition of those words.
• Simple engagement possibilities. There are very simple, easy ways to engage with the content you see. A “like” or a “reskeet” is just a click of the mouse away. You don’t have to think very hard or construct a complete and cogent thought in order to “engage” with what you see¹.
• A “feed” of posts. You are deluged with an infinitely scrolling progression of more content (making you infinitely content, right?). There are a lot of other features that matter on the margin. Exactly how long is a post? Can you Quote-tweet / Retweet? What content is allowed / promoted? This stuff matters, but I don’t think it really gets at the core of what microblogging is. (As an aside, I think we as social scientists are much better at testing the differences between possible affordances within a single platform than we are at testing the big meaningful differences between platforms).

The thing that I find really clear about this is that these affordances are not built to enable deep and thoughtful exchange of ideas. They’re built to enable surface-level engagement that you’ll come back for again and again.

Some microblogging is different (but not better)

I think Jack may have learned exactly the wrong lessons from Twitter. It seems that the AT protocol which undergirds BlueSky is fundamentally built around the idea that Jack shouldn’t have to be the person that everyone gets mad at. You don’t like how your feed is ranked? Not his fault, that’s a different layer than he controls. Don’t like how content moderation works? Well, you should be on a different application with different rules. This goes to extremes with platforms like Mastodon, where some servers won’t do any moderation whatsoever, while others will ban you aggressively for not giving adequate content warnings. This is different than other microblogging services, but I don’t think there’s much of a way to say it’s better. For instance, the massive CSAM problems there.

How this has worked in practice with AT/BlueSky is kind of different. BlueSky is the only real application built on AT, which means that Jack doesn’t just get to sit around dealing with AT issues, he has to deal with BlueSky issues, too. All of a sudden (and despite his original intentions), he’s having to manage an actual platform rather than just a protocol. This means that BlueSky is (slowly) developing content moderation standards, some of which are actually implemented at the protocol layer (e.g. in AT rather than in BlueSky). As I understand, CSAM, for instance, is a protocol-level concern for AT, it is not federated out to individual applications. I can’t see how this won’t continue to be a giant wellspring of conflicts.

What are we doing On Here?

There are two main things I think happen on microblogs:

• Discovery. There’s nothing I’ve seen that’s better at delivering interesting papers and news items to the front of my face (i.e. “pointing at things”). Microblogging is extremely effective at making serendipitous connections. Getting pointers to things of interest from people within a hop or two of you in a network which you form according to your whims is extremely powerful!
• Jokes/entertainment. There’s a reason that dril is the greatest poster on microblogs. One-liners are a classic joke format, and they’re enjoyable to consume on a feed (perhaps especially a feed that also has a lot of self-important academics on it). What I think doesn’t happen, is anything very much like “discussion”. Discussion requires grace between interlocutors, which is hard to extend given the threat of virality inherent in the platform. Discussion requires context, which is destroyed with virality and its associated context collapse. And important discussions require depth, which is not really possible in short snippets of text. I also think that privacy is often crucial to good discussion². Real privacy is anathema to virality: you can’t really have both. When discussions are ephemeral and low-cost its much easier to try out ideas to see if they seem compelling and convincing.

The easiest discussions to have on microblogs are ones in which the interlocutors already agree with one another and share full context (or which are just pointing to something and saying, essentially “good” or “bad”). In this case, a few words are all it takes to explain what one means and make one’s position clear. As discussions require more nuance and rely on context that is not shared (i.e. on more complex topics), then the goal should not be to write in chunks of a few hundred characters.

I think that a lot of the appreciation of tweet-length communication is a reflection of a failure of academic writing. For a lot of research, the first time someone takes the time to make it broadly accessible is when a tweet-thread is written out³. I’ll bite the bullet, and freely admit that was true for me with our ICML paper about online balanced experimental design. The version we submitted (and which was accepted) was essentially just math. The paper’s structure was, essentially, introduction → math → simulations → the end. Reviewers didn’t love it, but they were intimidated by it⁴. We cleaned it up for the camera-ready version (yes, that’s the easy-to-digest version), but the first time we really thought about how to communicate our results to a broader audience was when I wrote about it for Twitter. That’s bad!⁵ While one solution to this problem is to just write papers that are more accessible, the incentives are currently entirely against this. Being rigorous but difficult to understand is, often, a very good move if your primary incentive is publication.

Twitter, however, has incentives which at least preference some amount of accessibility. If you want your work to be shared and consumed, you have to hook people and encourage them to “engage.” In contrast to typical academicese, I think this is a nice change of pace, and is one of the reason I sometimes appreciate them. “Engagement,” however, is quite different than “accessible.” To be accessible, I think you want things like easy hyperlinks/citation, formatting, figures, quotations and, yes, more than a couple hundred characters. Maybe you also want to throw in some code or math as a treat! We can obviously do better than tweets for this.

What am I doing?

I’m treating microblogging for what it is. I’m essentially indifferent between services, as I don’t think the differences are that meaningful relative to what’s the same. I enjoy finding things on these platforms, so I will peruse them, but I will not expect discussion, and I won’t use them as if I do. Since I think threads about new work are not a particularly good way to provide digestible versions of papers, I will avoid making them. I don’t want to push for “engagement” with my work; I want people to understand it and its context. Instead, I will share new work, and (if appropriate) will link to a longer-form accessible introduction for that work in this space (or somewhere like it)⁶. In short, I will try to make it easy to discover my work on microblogs, but I will not attempt to explain my work in that setting. The platform is not conducive to such explanation. I may also make jokes, but, well, most of those are for the groupchats where I can be spicy.

I think this is ultimately the kind of recommendation Neil Postman makes about television in Amusing Ourselves to Death, too: TV is fantastic entertainment device, but don’t mistake it for something that it’s not. Don’t treat it as a medium of explanation or of news or of education. What it does is entertain.

Microblogging platforms are not built for good discussion. I will avoid a poor facsimile of conversation there, and have conversations in places where those conversations can be at their best. For me, this means that my conversation happens outside of the “clearnet”. I have a variety of WhatsApp / Slack / Discord groups for talking with collaborators or just randomers with shared interests. This is, I believe, a much more productive approach than trying to make microblogging into something that it can’t ever really do well. In these environments, you can set up affordances to enable the kind of discussion you want: you can make it easy to share LaTeX, or code. You can make spaces for highly moderated discussions or free-flowing sharing of memes. But most fundamentally, you can retain so much more control and privacy⁷.

Build alternative communities!

About a year ago I started up a Discord for discussion about experimentation which ultimately hasn’t seen much discussion in it. I’m continuing to hang out there, so if anyone wants somewhere outside of the clearnet to talk about experiments, feel free to join! Or come there to tell me about why this post is bad.

But also, use microblogs to create your own semi-private communities and make them easy to discover! The magic of the internet is that it’s all code and you can create whatever you want within those (weak) bounds. There are a ton of tools out there to create whatever kind of community you want, and you don’t need to troll around for “the next Twitter.” Twitter wasn’t ever all that good, and you can just go ahead and make the better community that you’d rather see! Many of those communities won’t work⁸, and that’s totally fine; the process of figuring out what works is how we find a space for online discussion that’s better than Twitter, which is what we should aspire to, anyway.

What AI thinks “a twitter-like website” looks like. It isn’t wrong.Thank you for reading Drew’s News. Now please microblog this in a long thread.

What do we want to measure?

Many of the most important questions we have about tech’s relationship to society are counterfactuals: If Facebook were otherwise identical but ranked Feed chronologically rather than by an algorithm, would there be more or less hate? If YouTube made different choices in recommendations, would it make people less extreme? A counterfactual, fundamentally, asks what would happen were the world just a little bit different. When thinking about the regulation of technology, we’re asking questions like these: if Facebook were to operate differently in some way, would society be better or would it be worse? This embeds two questions: One is a scientific question: what would society be like if Facebook were different? The second is a question of values: Would that counterfactual society be better or worse? 

The scientific question can be answered through randomized control trials (RCTs). Such RCTs, however, can only truly be implemented by and within these large tech platforms, so under the status quo the question of values can only be answered by people within those walls. Engineers, data scientists, managers and executives of tech companies: mostly good hearted people, but people who work within institutions that require they think about the good of the platform rather than the good of society.

I worked within Meta for years on improving the process through which the company evaluated potential counterfactual versions of itself – in other words, testing changes through A/B tests. The entire business of software development is to try lots of stuff out but only keep the things that work towards the business’s goals. These goals are expressed in numbers (“KPIs” or “metrics”). My team built machine learning tools to realize these business goals. When business values change, so too, can the platform. Look no further than Facebook’s 2018 pivot to “meaningful social interactions”. Once new numbers are chosen as the target, machine learning gears turn and the platform (in many ways a black box to everyone) pivots on a dime to optimize these new metrics.

Many of these business values are widely held societal norms: nobody in tech wants more spam or child sexual abuse material, for instance. As such, companies develop sophisticated (albeit imperfect) systems for detecting and removing such content and ensuring that it is not distributed widely. Other metrics, like the time spent on a platform, are less clear cut: essentially all internet platforms prize this as a KPI, but it does not necessarily align with what is best for society.

Democratic societies have systems for answering questions of values: we elect representatives who enact policies so that society embodies the values we care about. When such values are contentious, there is conflict, of course, but there is a peaceful process for resolving that conflict through our institutions. Tech lacks the mechanism to ensure that it embodies societal values: even market pressure isn’t straightforward, since usage is free thanks to advertising.

Who gets to measure them?

The only real counterfactual evidence about tech’s effects is controlled by tech itself. This is the promise of US2020, a project I worked on while at Meta. The unique power of this project is that it reveals the counterfactuals which—but for the press of a button—Facebook can decide to make reality. Asking ‘Should Facebook remove algorithmic ranking?’ contains both a question of science and a question of values – and this project answers the question of science, showing precisely what would happen if Facebook did no algorithmic ranking on Feed. As a result, the question of values is finally exposed to democratic scrutiny. Do we measure the right things to effectively answer that question? The way forward can only be an iterative refinement of asking the question and trying to improve the answers we get.

In short, the Election Project is a blueprint for the effective regulation of technology companies. The approach is straightforward and consistent with the language of Chapter III, Section 5 of the Digital Services Act. Regulators can require that for important, societally relevant changes, Very Large Online Platforms must run and report the results of RCTs. These are far more informative than audits of the processes used in these platforms or providing data access without counterfactuals, or trying to explain the precise details of how underlying machine learning systems work (which are mostly black boxes to everyone internally, too). To reiterate: in general, many of the most important systems at VLOPs are not truly “understood” by those who create them. Rather, they are refined by measuring what they do, and launching the changes that push those measurements in a direction that those measurements deem positive.

Philipp Lorenz-Spreen has already compellingly argued that Article 40 of the DSA is a way to free the results of internal A/B tests to the public. Tech companies are already running lots of A/B tests, so maybe we can just require them to share those results! This is a very positive step, but it isn’t enough. The problem with limiting ourselves to these tests is that platforms can simply stop doing them on issues they feel present a regulation risk. We’re back in the same place we started where our ability to understand counterfactuals is dependent on the platforms’ goodwill.

Platforms already limit what is measured internally for this reason. A direct example of this is race in the US. Put simply, Meta works hard to limit what it knows about race in order to preclude legal risks like it faced around housing discrimination in 2019. When I left Meta, the primary way concerns about disparate impact were examined were through zip-code level demographics. This process ensures that individual users’ race was not explicitly inferred, so race cannot be explicitly taken into account in decisions. To be clear, this does not mean that systems do not have disparate racial impacts. It does, however, make it very hard to measure when those disparate impacts might exist.

There are also precedents for requirements to run RCTs, such as with the UK’s Competition and Markets Authority requirement that Google test changes to third-party cookies. The logic here was that Google’s change to how cookies work might lead to big changes in how the advertising market works. Without understanding that impact, the regulator can not make an informed decision about whether it was anti-competitive or not. Hence, require Google to measure the impact, since they’re the only people who can.

More traditionally, this is key to the FDA’s regulation of food and medicine. The regulator is not generally performing the clinical trials to demonstrate safety itself: it instead requires that the would-be producer carry out the required tests, and ensures that those tests meet the standards of scientific rigor necessary to ensure safety. We think it’s important to verify that drugs meet at least a minimal standard before pushing them out broadly to all citizens. The same should be true for large changes to internet platforms. Note that the structure of clinical trials under the FDA would be exactly subject to the same critiques that are being made of US2020. 

The point is not who is actually running the test, but who is defining what particular tests must be conducted and how resulting outcomes must be measured. As an insider to this process, let me be very clear: US2020 took incredible care that external academics understood in detail the process by which concepts were operationalized and measured. A lot of the back-and-forth in the collaboration was specifically about this: academics didn’t understand how data is collected and stored internally, and internal researchers needed clear operationalizations to measure the concepts requested. I think I saw someone throw out that there were around 1000 variables to figure this out for.

The DSA provides an avenue for this kind of accountability through counterfactual evidence, but it isn’t through Article 40. Instead, it’s through Article 34. Allow me to quote:

Providers of very large online platforms and of very large online search engines shall diligently identify, analyse and assess any systemic risks in the Union stemming from the design or functioning of their service and its related systems, including algorithmic systems, or from the use made of their services.

By suggesting that these risks arise from the design or functioning of the service, this sentence invokes counterfactuals. The question at play is whether platform design causes these risks. The only valid way to assess this is through counterfactuals as measured through randomized control trials. The counterfactual risk is what VLOPs should demonstrate in their risk assessments. I recognize that this plain-english reading is not formal legal analysis (and I am not a lawyer). Given the latitude of regulators to choose how this law should be interpreted, I hope that they do so in the way that will give them actual answers to the societally important questions of counterfactual risk. US2020 shows them how they can do that. Unfortunately, based on the Commission’s current example of how to apply the risk management framework, this is not likely to be the approach taken to interpreting the DSA.

When are they measured?

Time is a crucial element of this story. The number one complaint by everyone involved in US2020 is the time it took to make it happen. As Kevin Munger emphasizes, it was 32 months after the US2020 studies were actually performed that they were published. Part of that delay was because of peer review (note that the chronological feed paper was originally received by Science on March 7, 2022, but was only published on July 27, 2023. The paper was definitely improved in that time, but it goes without saying that nothing about the intervention or outcome measurement could have been improved or changed as a result of this peer review. It had already happened.

Surely nothing changed about Facebook in that time, right? Well, shortly after submission of chrono-feed, on July 27, 2022, Mark Zuckerberg made an announcement on the Q2 earnings call: “Right now, about 15% of content in a person’s Facebook feed and a little more than that of their Instagram feed is recommended by our AI from people, groups, or accounts that you don’t follow. We expect these numbers to more than double by the end of next year.” Shit. It’s at this point that I’ll note we found clear heterogeneity of the effects of chronological feed on variables like how much uncivil content and slurs show up on Feed, as well as on-platform political behavior on the dimension of inventory size. When there’s more content Feed could show you, there’s more potential for Feed algorithms to change what you see.

Larger inventory is associated with larger effects on on-platform political behavior.Kevin and I have a new working paper showing exactly how poorly agnostic approaches to generalizability fare when reality refuses to stay fixed. When reality is changing at internet speed, then it’s necessary that our knowledge generation process matches that speed. US2020 could do better on this. A lot of work had to be done to mitigate the risks that were inherent in the project as the first of its kind. There was a whole software infrastructure that had to be built from the ground up for protecting privacy, which was done at a pretty incredible speed. There was also a lot of work to build up the internal processes to make everything work both within the collaboration and within Meta (e.g. legal and privacy reviews to make sure user data was appropriately protected). And as stated above, there was a whole arduous process of defining meaningful operationalizations of everything: to a real extent, the scientific language of internal and external researchers wasn’t exactly the same, so translation was required.

If we’re thinking about an ongoing regulatory process, however, a lot of these concerns go away: an infrastructure and set of systems will be set up (by necessity) to streamline processes. Variable definitions will be negotiated over time, so that’s largely just a fixed cost per variable. If the output is regulatory rather than academic, the results wouldn’t need to be gated by the fickle peer review process (just government bureaucracy!).

Defining appropriate implementation of the regulations, too, must be a negotiated and long-term process. Meta has spent years defining specific internal metrics to understand the best dimensions of platform behavior to measure for business purposes. We cannot expect US2020 to have gotten the right societally important dimensions correct on the first try. DSA requires platforms to measure risks “on civic discourse and electoral processes, and public security”. How should these be operationalized? This can only be an iterative process as we see what kinds of measurements are both informative about what we care about in the world and sensitive to the kinds of changes VLOPs make. Iteration on the US2020 model is how to begin this process of refinement.

Wrapping it up

If we want to truly understand a world where tech is different in some way, we must change it in that way and measure what happens as a result. This is how Meta ensures Feed aligns with its business values, and this is the only way to ensure Feed – alongside all other aspects of online platforms – aligns with our societal values, as well. This may require platforms to run RCTs they would not have already run: we may imagine changes that they have not previously tested. At the very least, it will almost certainly be necessary to require that they measure outcomes they might not otherwise measure. In the US, for instance, this might mean requiring that they collect data on race that they are otherwise reluctant to hold.

US2020 has many of the features that are necessary for this kind of counterfactual accountability. It provides rigorous evidence of how Meta’s products would work if we were to change them. The RCTs were designed by external academics with full control rights, whose incentive is not to improve Meta’s bottom-line, but to expose answers to societally relevant questions. Meta had no substantial freedom to say no to such requests. It was not fast enough, nor was it large enough scale (e.g. it was only in the US). Regulators don’t currently have the expertise in-house to do the work that these academics did in asking good questions and co-designing studies to answer them. This is an important problem to solve, but other fields (like the FDA) have shown that such expertise can be acquired by regulatory agencies.

The biggest problem with US2020 is that it only exists out of Meta’s goodwill. And given how expensive it was for Meta, it’s not clear how much of that there is left. Look through the author list for the Meta employees – most worked primarily on this project for the last 3 years, and lots of other engineers and data scientists contributed substantial time as well. These people are all superstars and just their direct compensation over that time is a huge commitment from Meta: easily millions of dollars, likely tens of millions. Given recent belt-tightening, I wouldn’t count on Meta independently finding this a good tradeoff solely in order to be transparent. If we want to guarantee this kind of transparency from internet platforms, then we shouldn’t just hope that they will keep choosing to bring us rigorous counterfactual evaluation. Government should require it.

Think other people should read this? Feel free to share:

Setup

Setting up Quarto was very easy, so I won’t belabor this. The combination of the Get Started guide with the Website Creation guide kept everything very straightforward. I also used Danielle Navarro’s post and her blog’s code to get everything set up.

I decided late in the setup process to add a blog, so I will mention that it’s actually very easy to do: it basically just requires adding a Listing page (i.e. the blog’s index), a folder to contain the various posts and a _metadata.yml file in that folder to describe global settings to apply to all posts. I just created these manually without too much trouble. This is one of the great things about building sites with tools like Quarto: everything is extremely transparent: just put a couple files in the right places and you’re good to go.

Site Design

To demonstrate how I’ve set things up to populate the website from data about my academic life, I’ll focus on my publications page. There are two main files undergirding this page:

papers.yaml

a data file in YAML with standardized information on each publication. I chose YAML because it’s fairly easy to write correctly formatted YAML by hand (and I’ll be updating)

research.qmd

The page which takes the data in papers.yaml and turns it into nicely formatted Markdown / HTML. This is setup as a Jupyter-backed qmd file (essentially a Jupyter notebook).

This idea of separating the data side (information about publications) from formatting is aimed at making my life easier. One of the reasons I often stop updating my website is because when I come back in 3 months with a new publication, I never remember all the details about how I formatted entries in whatever flavor of Bootstrap I happened to be using when I built the website. Moreover, because I know that there’s a barrier to understanding before I can get started, it’s extremely easy to put off (and therefore it never gets done).

By separating out the data entry from the formatting, this simplifies matters substantially.

softblock:
  title: Efficient Balanced Treatment Assignments for Experimentation
  authors:
    - David Arbour
    - me
    - Anup Rao
  year: 2021
  venue: AISTATS
  preprint: https://arxiv.org/abs/2010.11332
  published_url: https://proceedings.mlr.press/v130/arbour21a.html
  github: https://github.com/ddimmery/softblock

import yaml
from IPython.display import display, Markdown, HTML

def readable_list(_s):
  if len(_s) < 3:
    return ' and '.join(map(str, _s))
  *a, b = _s
  return f"{', '.join(map(str, a))}, and {b}"

def button(url, str, icon):
    icon_base = icon[:2]
    return f"""<a class="btn btn-outline-dark btn-sm", href="{url}" target="_blank" rel="noopener noreferrer">
        <i class="{icon_base} {icon}" role='img' aria-label='{str}'></i>
        {str}
    </a>"""

yaml_data = yaml.safe_load(open("papers.yaml"))
pub_strs = {"pubs": {}, "wps": {}}
for _, data in yaml_data.items():
    title_str = data["title"]
    authors = data.get("authors", ["me"])
    authors = [
        aut if aut != "me" else "<strong>Drew Dimmery</strong>" for aut in authors
    ]
    author_str = readable_list(authors)
    year_str = data["year"]

    buttons = []
    preprint = data.get("preprint")
    if preprint is not None:
        buttons.append(button(preprint, "Preprint", "bi-file-earmark-pdf"))

    github = data.get("github")
    if github is not None:
        buttons.append(button(github, "Github", "bi-github"))

    pub_url = data.get("published_url")
    venue = data.get("venue")
    working_paper = pub_url is None
    
    pub_str = f'{author_str}. ({year_str}) "{title_str}."'

    if venue is not None:
        pub_str += f" <em>{venue}</em>"

    if working_paper:
        if year_str not in pub_strs["wps"]:
            pub_strs["wps"][year_str] = []
        pub_strs["wps"][year_str].append(
            "<li class='list-group-item'>" + pub_str + "<br>" + " ".join(buttons) + "</li>"
        )
    else:
        if year_str not in pub_strs["pubs"]:
            pub_strs["pubs"][year_str] = []
        buttons.append(button(pub_url, "Published", "ai-archive"))
        pub_strs["pubs"][year_str].append(
            "<li class='list-group-item'>" + pub_str + "<br>" + " ".join(buttons) + "</li>"
        )

for year in sorted(pub_strs["pubs"].keys(), reverse=True):
    display(Markdown(f"### {year}" + "{#" + f"published-{year}" + "}"))
    display(HTML(
        "<ul class='list-group list-group-flush'>" + '\n'.join(pub_strs["pubs"][year]) + "</ul>"
    ))

Deployment

It’s extremely easy to build a new version of the website locally (quarto render from CLI), but there’s no guarantee I’ll remember that off the top of my head in a month without Googling, so I think it’s worthwhile to setup automatic building after I push a commit to GitHub.

GitHub Actions is incredible. I adapted the example config from Quarto to the following (also on GitHub here):

on:
  push:
    branches: main
  pull_request:
    branches: main
  # to be able to trigger a manual build
  workflow_dispatch:

name: Render and deploy website to Netlify

jobs:
  build-deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-python@v3
        with:
          python-version: '3.9'
          cache: 'pip'
      - run: pip install -r requirements.txt

      - uses: r-lib/actions/setup-r@v2
        with:
          use-public-rspm: true

      - uses: r-lib/actions/setup-renv@v2
      
      - name: Install Quarto
        uses: quarto-dev/quarto-actions/install-quarto@v1
        with:
          # To install LaTeX to build PDF book 
          tinytex: true 
          # uncomment below and fill to pin a version
          # version: 0.9.105

      - name: Render website
        # Add any command line argument needed
        run: |
          quarto render
      - name: Deploy to Netlify
        id: netlify-deploy
        uses: nwtgck/actions-netlify@v1
        with:
          # The folder the action should deploy. Adapt if you changed in Quarto config
          publish-dir: './_site'
          production-branch: main
          github-token: ${{ secrets.GITHUB_TOKEN }}
          deploy-message:
            'Deploy from GHA: ${{ github.event.pull_request.title || github.event.head_commit.message }} (${{ github.sha }})'
          enable-pull-request-comment: true #  Comment on pull request
          enable-commit-comment: true # Comment on GitHub commit
          enable-commit-status: true # GitHub commit status 
        env:
          NETLIFY_AUTH_TOKEN: ${{ secrets.NETLIFY_AUTH_TOKEN }}
          NETLIFY_SITE_ID: ${{ secrets.NETLIFY_SITE_ID }}
        timeout-minutes: 1

This Action requires two pieces of information from Netlify entered as secrets in GitHub. The NETLIFY_SITE_ID may be found in the site configuration settings, while the NETLIFY_AUTH_TOKEN may be found in personal settings (the personal access token).

One thing I have not yet done is set up an renv to ensure dependencies for blog posts are taken care of in GitHub Actions. This means that posts like the experimental design demo can’t be knit via GitHub Actions. I did this for two reasons (other than laziness). First, it’s a pain to get GIS tools working on any environment (ok, so its part laziness). I’ve actually done this before for automated R CMD checking of the regweight package, but didn’t feel like it was worthwhile here.

The reason it’s not worth it is that Quarto has a great feature called “freezing”. Essentially, it knits blog posts or pages, and only re-renders them when something about the source changes. This means that the vast majority of posts don’t need to be rendered on each build. If I’m working on a blog post, I can write it locally, render on my machine, commit that pre-rendered post and then all future builds on Actions won’t get held up by their inability to render that post.

As I type this, it becomes clear that I’ll forget how to do this pretty often (given that there’s been about an 8 year delay since my next most recent blog, I likely won’t stay in practice). But blogs aren’t my main concern on my website: keeping a software and publication list up-to-date is.

Setting up Actions means that simple updates to pages (or YAML files) can actually be done directly in the GitHub editing UI, which further lowers the barrier for my future self. I don’t even need to clone the repository to whatever computer I’m working on to add a publication!

Future dreams

I imagine my CV is similar to most academics’ in that it’s built like a house of cards (and overfull hboxs). Whenever I add something new to it, I have to copy some lines from elsewhere and modify them to fit the new entry. This always takes me way more time than I’d like. If I mashed together my current About page with the Research page, it’s like 90% of the way to a full CV. It should presumably be pretty easy to do explicitly combine them and output a reasonable-looking CV.

This is a project for another day, though. Too much of the Research page directly outputs HTML, which makes it difficult to naïvely import into a CV.

An almost completely naïve approach to directly importing the relevant pages creates this ugly document.

---
title: "Curriculum Vitae"
format: pdf
---

{{< include about.qmd >}}

{{< include research.md >}}

It’s definitely possible to improve on this. The easiest hacky approach is to just write a whole alternative version of the HTML formatting code which resides in research.qmd to output appropriately formatted markup.

For now, I’m pretty pleased with the system I have, but ask me again in three months.

Implementation of methods

writeLines(readLines("https://raw.githubusercontent.com/ddimmery/softblock/master/r_implementation.R"))

library(Matrix)
library(igraph)
library(FNN)
library(hash)

assign_greedily <- function(graph) {
    adj_mat = igraph::as_adjacency_matrix(graph, type="both", sparse=TRUE) != 0
    N = nrow(adj_mat)
    root_id = make.keys(sample(N, 1))

    a = rbinom(1, 1, 0.5)
    visited = hash()

    random_order = make.keys(sample(N))
    unvisited = hash(random_order, random_order)

    colors = hash()
    stack = hash()

    stack[[root_id]] <- a
    tentative_color = rbinom(N, 1, 0.5)
    while ((!is.empty(unvisited)) || (!is.empty(stack))) {
        if (is.empty(stack)) {
            cur_node = keys(unvisited)[1]
            del(cur_node, unvisited)
            color = tentative_color[as.integer(cur_node)]
        } else {
            cur_node = keys(stack)[1]
            color = stack[[cur_node]]
            del(cur_node, stack)
            del(cur_node, unvisited)
        }
        visited[[cur_node]] = cur_node
        colors[[cur_node]] = color
        children = make.keys(which(adj_mat[as.integer(cur_node), ]))
        for (child in children) {
            if(has.key(child, unvisited)) {
                stack[[child]] = 1 - color
            }
        }
    }
    values(colors, keys=1:N)
}


assign_softblock <- function(.data, cols, .s2=2, .neighbors=6) {
    expr <- rlang::enquo(cols)
    pos <- tidyselect::eval_select(expr, data = .data)
    df_cov <- rlang::set_names(.data[pos], names(pos))
    cov_mat = scale(model.matrix(~.+0, df_cov))
    N = nrow(cov_mat)
    st = lubridate::now()
    knn = FNN::get.knn(cov_mat, k=.neighbors)
    st = lubridate::now()
    knn.adj = Matrix::sparseMatrix(i=rep(1:N, .neighbors), j=c(knn$nn.index), x=exp(-c(knn$nn.dist) / .s2))
    knn.graph <- graph_from_adjacency_matrix(knn.adj, mode="plus", weighted=TRUE, diag=FALSE)
    E(knn.graph)$weight <- (-1 * E(knn.graph)$weight)
    st = lubridate::now()
    mst.graph = igraph::mst(knn.graph)
    E(mst.graph)$weight <- (-1 * E(mst.graph)$weight)
    st = lubridate::now()
    assignments <- assign_greedily(mst.graph)
    .data$treatment <- assignments
    attr(.data, "laplacian") <- igraph::laplacian_matrix(mst.graph, normalize=TRUE, sparse=TRUE)
    .data
}

assign_greedy_neighbors <- function(.data, cols) {
    expr <- rlang::enquo(cols)
    pos <- tidyselect::eval_select(expr, data = .data)
    df_cov <- rlang::set_names(.data[pos], names(pos))
    cov_mat = scale(model.matrix(~.+0, df_cov))
    N = nrow(cov_mat)
    knn = FNN::get.knn(cov_mat, k=1)
    knn.adj = Matrix::sparseMatrix(i=1:N, j=c(knn$nn.index), x=c(knn$nn.dist))
    knn.graph <- graph_from_adjacency_matrix(knn.adj, mode="plus", weighted=TRUE, diag=FALSE)
    assignments <- assign_greedily(knn.graph)
    .data$treatment <- assignments
    attr(.data, "laplacian") <- igraph::laplacian_matrix(knn.graph, normalize=TRUE, sparse=TRUE)
    .data
}

assign_matched_pairs <- function(.data, cols, .s2=2, .neighbors=6) {
    expr <- rlang::enquo(cols)
    pos <- tidyselect::eval_select(expr, data = .data)
    df_cov <- rlang::set_names(.data[pos], names(pos))
    cov_mat = scale(model.matrix(~.+0, df_cov))
    N = nrow(cov_mat)
    knn = FNN::get.knn(cov_mat, k=.neighbors)
    knn.adj = Matrix::sparseMatrix(i=rep(1:N, .neighbors), j=c(knn$nn.index), x=exp(-c(knn$nn.dist) / .s2))
    knn.graph <- graph_from_adjacency_matrix(knn.adj, mode="plus", weighted=TRUE, diag=FALSE)
    E(knn.graph)$weight <- (-1 * E(knn.graph)$weight)
    mwm.graph = igraph::max_bipartite_match(knn.graph)
    E(mwm.graph)$weight <- (-1 * E(mwm.graph)$weight)
    assignments <- assign_greedily(mwm.graph)
    .data$treatment <- assignments
    attr(.data, "laplacian") <- igraph::laplacian_matrix(mwm.graph, normalize=TRUE, sparse=TRUE)
    .data
}

# library(tibble)
# data = tibble(
#     x1=runif(10),
#     x2=runif(10),
#     x3=rbinom(10, 1, 0.5)
# )
# library(dplyr)
# library(tidyr)
# library(ggplot2)
# data %>% assign_softblock(c(x1, x2)) -> newdata

# ggplot(newdata, aes(x=x1, y=x2, color=factor(treatment), shape=factor(x3))) + geom_point() + theme_minimal()

# newdata %>%
#     attr("laplacian") %>%
#     ifelse(lower.tri(.), ., 0) %>%
#     as_tibble() -> adj_df
# names(adj_df) <- paste0(1:ncol(adj_df))

# adj_df %>%
#     group_by(id_1=as.character(row_number())) %>%
#     gather(id_2, weight, -id_1) %>%
#     filter(weight != 0)  %>%
#     mutate(id_2=as.character(id_2), id=paste(id_1, id_2, sep='-')) -> adj_df

# locs = newdata %>% mutate(id=as.character(row_number())) %>% select(id, x1, x2, x3)

# edges=bind_rows(
# adj_df  %>% inner_join(locs, by=c('id_1'='id')),
# adj_df  %>% inner_join(locs, by=c('id_2'='id'))
# ) %>% arrange(id) %>% ungroup()

# pp = ggplot(newdata, aes(x=x1, y=x2)) +
# geom_line(aes(group=id, size=1), data=edges, color='grey') +
# geom_point(aes(color=factor(treatment), shape=factor(x3), size=2), alpha=.9) +
# scale_size_continuous(range=c(1, 3)) +
# theme_minimal() + theme(legend.position='none')

# print(pp)

Data Preparation

This demo will be based on North Carolina data because their Board of Elections makes it very easy to get precinct level data. I’m also only going to use historical data from the most recent election to avoid needing to match precincts across elections.

With access to a full voter file, this section could be drastically improved by incorporating other important elements into the design like demographics.

url <- "http://dl.ncsbe.gov/ENRS/2020_11_03/results_pct_20201103.zip"
zip_file <- tempfile(fileext = ".zip")
download.file(url, zip_file, mode = "wb")
spec = cols(
  County = col_character(),
  `Election Date` = col_character(),
  Precinct = col_character(),
  `Contest Group ID` = col_double(),
  `Contest Type` = col_character(),
  `Contest Name` = col_character(),
  Choice = col_character(),
  `Choice Party` = col_character(),
  `Vote For` = col_double(),
  `Election Day` = col_double(),
  `One Stop` = col_double(),
  `Absentee by Mail` = col_double(),
  Provisional = col_double(),
  `Total Votes` = col_double(),
  `Real Precinct` = col_character(),
  X16 = col_skip()
)
results <- readr::read_tsv(zip_file, col_types=spec)

New names:
• `` -> `...16`

Warning: The following named parsers don't match the column names: X16

url = "https://s3.amazonaws.com/dl.ncsbe.gov/ShapeFiles/Precinct/SBE_PRECINCTS_20201018.zip"
temp <- tempfile()
temp2 <- tempfile()
download.file(url, temp)
unzip(zipfile = temp, exdir = temp2)
nc_SHP_file <- list.files(temp2, pattern = ".shp$",full.names=TRUE)
shapes <- sf::read_sf(nc_SHP_file)

results %>%
    filter(`Real Precinct` == 'Y') %>%
    group_by(County, Precinct) %>%
    summarize(
        total_vote_pres=sum(`Total Votes`[`Contest Name` == 'US PRESIDENT'], na.rm=TRUE),
        dem_share_pres=sum(`Total Votes`[`Contest Name` == 'US PRESIDENT' & `Choice Party` == 'DEM'], na.rm=TRUE)/total_vote_pres,
        gop_share_pres=sum(`Total Votes`[`Contest Name` == 'US PRESIDENT' & `Choice Party` == 'REP'], na.rm=TRUE)/total_vote_pres,
        total_vote_senate=sum(`Total Votes`[`Contest Name` == 'US SENATE'], na.rm=TRUE),
        dem_share_senate=sum(`Total Votes`[`Contest Name` == 'US SENATE' & `Choice Party` == 'DEM'], na.rm=TRUE)/total_vote_senate,
        gop_share_senate=sum(`Total Votes`[`Contest Name` == 'US SENATE' & `Choice Party` == 'REP'], na.rm=TRUE)/total_vote_senate,
        total_vote_gov=sum(`Total Votes`[`Contest Name` == 'NC GOVERNOR'], na.rm=TRUE),
        dem_share_gov=sum(`Total Votes`[`Contest Name` == 'NC GOVERNOR' & `Choice Party` == 'DEM'], na.rm=TRUE)/total_vote_gov,
        gop_share_gov=sum(`Total Votes`[`Contest Name` == 'NC GOVERNOR' & `Choice Party` == 'REP'], na.rm=TRUE)/total_vote_gov,
        total_vote_house=sum(`Total Votes`[grepl('US HOUSE OF REPRESENTATIVES DISTRICT', `Contest Name`)], na.rm=TRUE),
        dem_share_house=sum(`Total Votes`[grepl('US HOUSE OF REPRESENTATIVES DISTRICT', `Contest Name`) & `Choice Party` == 'DEM'], na.rm=TRUE)/total_vote_house,
        gop_share_house=sum(`Total Votes`[grepl('US HOUSE OF REPRESENTATIVES DISTRICT', `Contest Name`) & `Choice Party` == 'REP'], na.rm=TRUE)/total_vote_house
    ) %>% ungroup() -> results_agg

`summarise()` has grouped output by 'County'. You can override using the
`.groups` argument.

inner_join(results_agg, shapes, by=c('County'='county_nam', 'Precinct'='prec_id')) -> df_joined
DT::datatable(df_joined %>% sample_n(size=100) %>% dplyr::select(-geometry), rownames = FALSE, options=list(scrollX=TRUE, autoWidth = TRUE))

df_joined$geometry %>%
    st_centroid() %>%
    st_transform("+init=epsg:4326") %>%
    st_coordinates() -> latlong

Warning in CPL_crs_from_input(x): GDAL Message 1: +init=epsg:XXXX syntax is
deprecated. It might return a CRS with a non-EPSG compliant axis order. Further
messages of this type will be suppressed.

df_joined$longitude = latlong[, 'X']
df_joined$latitude = latlong[, 'Y']
area = df_joined$geometry %>% st_transform("+init=epsg:4326") %>% st_area()
df_joined$area_km2 = units::drop_units(area) / 1e6 # convert m^2 to km^2
df_joined <- df_joined %>% mutate(vote_density_pres = total_vote_pres / area_km2)

Assign treatment four ways

start_time = lubridate::now()
df_joined %>% assign_softblock(c(
    longitude, latitude, area_km2, vote_density_pres, # geographic
    total_vote_pres, dem_share_pres, gop_share_pres, # 2020 presidential
    total_vote_senate, dem_share_senate, gop_share_senate, # 2020 senate
    total_vote_gov, dem_share_gov, gop_share_gov, # 2020 governor
    total_vote_house, dem_share_house, gop_share_house # 2020 house
)) %>%
rename(treatment_sb=treatment)-> df_joined

Warning: The `normalized` argument of `make_lattice()` provide normalization instead as
of igraph 2.0.3.
ℹ `normalized` is now deprecated, use `normalization` instead.

softblock_weights <- attr(df_joined, "laplacian")
end_time = lubridate::now()
print(end_time - start_time)

Time difference of 0.7473469 secs

start_time = lubridate::now()
df_joined %>% assign_greedy_neighbors(c(
    longitude, latitude, area_km2, vote_density_pres, # geographic
    total_vote_pres, dem_share_pres, gop_share_pres, # 2020 presidential
    total_vote_senate, dem_share_senate, gop_share_senate, # 2020 senate
    total_vote_gov, dem_share_gov, gop_share_gov, # 2020 governor
    total_vote_house, dem_share_house, gop_share_house # 2020 house
)) %>%
rename(treatment_nn=treatment)-> df_joined
nn_weights <- attr(df_joined, "laplacian")
end_time = lubridate::now()
print(end_time - start_time)

Time difference of 1.482411 secs

start_time = lubridate::now()
df_joined %>% dplyr::select(
    longitude, latitude, area_km2, vote_density_pres, # geographic
    total_vote_pres, dem_share_pres, gop_share_pres, # 2020 presidential
    total_vote_senate, dem_share_senate, gop_share_senate, # 2020 senate
    total_vote_gov, dem_share_gov, gop_share_gov, # 2020 governor
    total_vote_house, dem_share_house, gop_share_house # 2020 house
) -> df_qb
qb_blocks = quickblock(as.data.frame(df_qb), size_constraint = 6L)
df_joined$treatment_qb = as.integer(as.character(assign_treatment(qb_blocks, treatments=c('0', '1'))))
end_time = lubridate::now()
print(end_time - start_time)

Time difference of 0.03046703 secs

# This is extremely slow.
# start_time = lubridate::now()
# df_joined %>% assign_matched_pairs(c(
#     longitude, latitude, area_km2, vote_density_pres, # geographic
#     total_vote_pres, dem_share_pres, gop_share_pres, # 2020 presidential
#     total_vote_senate, dem_share_senate, gop_share_senate, # 2020 senate
#     total_vote_gov, dem_share_gov, gop_share_gov, # 2020 governor
#     total_vote_house, dem_share_house, gop_share_house # 2020 house
# )) %>%
# rename(treatment_mp=treatment)-> df_joined
# mp_weights <- attr(df_joined, "laplacian")
# end_time = lubridate::now()
# print(end_time - start_time)

Power simulation

To do power calculations consistently across methods, I will do the following:

1. Calculate the implied regression adjustment for each design, applied to vote share in 2020 as the outcome.
2. Find the residual for each point.
3. Permute residuals over units.
4. Estimate power for a given effect size by pulling additional permutations.
5. Sweep over a range of effect sizes.

create_power_simulator = function(W, A, Y) {
    dL = diag(W)
    Dinv = (2 * A - 1) / dL
    estimates = drop((Dinv %*% W) %*% Y)
    residuals = Y - estimates
    x_mat = cbind(A, 1)
    xlx = t(x_mat) %*% W %*% x_mat
    bread = MASS::ginv(as.matrix(xlx))
    detect_effect = function(effect=1) {
        sim_outcome = estimates + sample(residuals) + A * effect
        coefs = bread %*% t(x_mat) %*% W %*% sim_outcome
        r = diag(drop((sim_outcome - (x_mat %*% coefs)) ^ 2))
        meat = t(x_mat) %*% (W %*% r %*% W) %*% x_mat
        vcv = bread %*% meat %*% bread
        upr = coefs[1] + qnorm(0.975) * sqrt(vcv[1])
        lwr = coefs[1] + qnorm(0.025) * sqrt(vcv[1])
        lwr > 0
    }
    detect_effect
}
create_power_simulator_qb = function(blocks, A, Y) {
    estimate_by_block = tapply(Y[A==1], blocks[A==1], mean) - tapply(Y[A==0], blocks[A==0], mean)
    estimates = unlist(purrr::map(blocks, function(b) estimate_by_block[as.character(b)]))
    residuals = Y - estimates
    detect_effect = function(effect=1) {
        sim_outcome = estimates + sample(residuals) + A * effect
        result = quickblock::blocking_estimator(sim_outcome, blocks, A)
        upr = result$effects[2,1] + qnorm(0.975) * sqrt(result$effect_variances[2,1])
        lwr = result$effects[2,1] + qnorm(0.025) * sqrt(result$effect_variances[2,1])
        lwr > 0
    }
    detect_effect
}

simulate_power = create_power_simulator(softblock_weights, df_joined$treatment_sb, df_joined$dem_share_pres)
estimate_power_for_effect = function(effect) mean(replicate(25, simulate_power(effect=effect)))
effects = seq(0, 0.05, length=25)
power_sb = unlist(purrr::map(effects, estimate_power_for_effect))

simulate_power = create_power_simulator(nn_weights, df_joined$treatment_nn, df_joined$dem_share_pres)
estimate_power_for_effect = function(effect) mean(replicate(25, simulate_power(effect=effect)))
power_nn = unlist(purrr::map(effects, estimate_power_for_effect))

simulate_power = create_power_simulator_qb(qb_blocks, df_joined$treatment_qb, df_joined$dem_share_pres)
estimate_power_for_effect = function(effect) mean(replicate(25, simulate_power(effect=effect)))
effects = seq(0, 0.05, length=25)
power_qb = unlist(purrr::map(effects, estimate_power_for_effect))

ggplot(
    tibble(
        effects=c(effects, effects, effects),
        power=c(power_sb, power_nn, power_qb),
        design=c(rep('SoftBlock', length(power_sb)), rep('Greedy Neighbors', length(power_sb)), rep('QuickBlock', length(power_sb)))
    ), aes(effects, power, color=design)) + geom_line() +
    scale_x_continuous('Effect (pp)', labels=scales::percent) +
    scale_y_continuous("Power", labels=scales::percent) +
    scale_color_discrete("Design") +
    theme_minimal()

Analysis

df_joined$outcome = df_joined$dem_share_pres
df_joined$ite = with(df_joined, 0.005 + 0.005 * (plogis((dem_share_pres - median(dem_share_pres)) / sd(dem_share_pres))))
df_joined$outcome_sb = with(df_joined, outcome + treatment_sb * ite)
df_joined$outcome_nn = with(df_joined, outcome + treatment_nn * ite)
df_joined$outcome_qb = with(df_joined, outcome + treatment_qb * ite)

estimate_effect = function(W, A, Y) {
    dL = diag(W)
    Dinv = (2 * A - 1) / dL
    x_mat = cbind(A, 1)
    xlx = t(x_mat) %*% W %*% x_mat
    bread = MASS::ginv(as.matrix(xlx))
    coefs = bread %*% t(x_mat) %*% W %*% Y
    r = diag(drop((Y - (x_mat %*% coefs)) ^ 2))
    meat = t(x_mat) %*% (W %*% r %*% W) %*% x_mat
    vcv = bread %*% meat %*% bread
    list(estimate=coefs[1], std.error=sqrt(vcv[1,1]))
}
sb_est = estimate_effect(softblock_weights, df_joined$treatment_sb, df_joined$outcome_sb)

nn_est = estimate_effect(nn_weights, df_joined$treatment_nn, df_joined$outcome_nn)

result = quickblock::blocking_estimator(df_joined$outcome_qb, qb_blocks, df_joined$treatment_qb)
qb_est = list(estimate=result$effects[2,1], std.error=sqrt(result$effect_variances[2,1]))

tibble(
    estimate=c(sb_est$estimate, nn_est$estimate, qb_est$estimate),
    std.error=c(sb_est$std.error, nn_est$std.error, qb_est$std.error),
    design=c("SoftBlock", "Greedy Neighbors", "QuickBlock")
) %>%
ggplot(aes(x=design, y=estimate, ymin=estimate-1.96*std.error, ymax=estimate+1.96*std.error)) +
    geom_pointrange() +
    scale_x_discrete("Design") +
    scale_y_continuous("ATE (pp)", labels=scales::percent) +
    coord_flip() +
    theme_minimal()

predict.hte.split = function(x, a, y, s, predict.s=4) {
    s.pi = (predict.s) %% 4 + 1
    s.mu = (predict.s + 1) %% 4 + 1
    s.dr = (predict.s + 2) %% 4 + 1
    pihat <- predict(cv.glmnet(x[s==s.pi,],a[s==s.pi], family="binomial", nfolds=10), newx=x, type="response", s="lambda.min")
    mu0hat <- predict(cv.glmnet(x[a==0 & s==s.mu,],y[a==0 & s==s.mu], nfolds=10), newx=x, type="response", s="lambda.min")
    mu1hat <- predict(cv.glmnet(x[a==1 & s==s.mu,],y[a==1 & s==s.mu], nfolds=10),newx=x, type="response", s="lambda.min")
    pseudo <- ((a-pihat)/(pihat*(1-pihat)))*(y-a*mu1hat-(1-a)*mu0hat) + mu1hat - mu0hat
    drl <- predict(cv.glmnet(x[s==s.dr,],pseudo[s==s.dr]),newx=x[s==predict.s, ], s="lambda.min")
    drl
}
predict.hte.crossfit = function(x, a, y) {
    N = length(a)
    s = sample(1:4, N, replace=TRUE)
    hte = rep(NA_real_, N)
    for (split in 1:4) {
        hte[s==split] = predict.hte.split(x, a, y, s, predict.s=split)
    }
    hte
}
calculate_hte <- function(.data, cols, .treatment='treatment', .outcome='outcome') {
    expr <- rlang::enquo(cols)
    pos <- tidyselect::eval_select(expr, data = .data)
    df_cov <- rlang::set_names(.data[pos], names(pos))
    cov_mat = scale(model.matrix(~.+0, df_cov))
    .data$hte = predict.hte.crossfit(cov_mat, .data[[.treatment]], .data[[.outcome]])
    .data
}

df_joined %>% calculate_hte(c(
    longitude, latitude, area_km2, vote_density_pres, # geographic
    total_vote_pres, # 2020 presidential
    total_vote_senate, dem_share_senate, gop_share_senate, # 2020 senate
    total_vote_gov, dem_share_gov, gop_share_gov, # 2020 governor
    total_vote_house, dem_share_house, gop_share_house # 2020 house
), .treatment='treatment_sb', .outcome='outcome_sb') %>% rename(hte_sb=hte) -> df_joined
summary(df_joined$hte_sb)

     Min.   1st Qu.    Median      Mean   3rd Qu.      Max. 
-0.010306  0.004135  0.007536  0.007148  0.009029  0.033580 

df_joined %>% calculate_hte(c(
    longitude, latitude, area_km2, vote_density_pres, # geographic
    total_vote_pres, # 2020 presidential
    total_vote_senate, dem_share_senate, gop_share_senate, # 2020 senate
    total_vote_gov, dem_share_gov, gop_share_gov, # 2020 governor
    total_vote_house, dem_share_house, gop_share_house # 2020 house
), .treatment='treatment_nn', .outcome='outcome_nn') %>% rename(hte_nn=hte) -> df_joined
summary(df_joined$hte_nn)

    Min.  1st Qu.   Median     Mean  3rd Qu.     Max. 
0.005742 0.006825 0.007388 0.007670 0.008596 0.010863 

df_joined %>% calculate_hte(c(
    longitude, latitude, area_km2, vote_density_pres, # geographic
    total_vote_pres, # 2020 presidential
    total_vote_senate, dem_share_senate, gop_share_senate, # 2020 senate
    total_vote_gov, dem_share_gov, gop_share_gov, # 2020 governor
    total_vote_house, dem_share_house, gop_share_house # 2020 house
), .treatment='treatment_qb', .outcome='outcome_qb') %>% rename(hte_qb=hte) -> df_joined
summary(df_joined$hte_qb)

      Min.    1st Qu.     Median       Mean    3rd Qu.       Max. 
-0.0007862  0.0070832  0.0083612  0.0082249  0.0094074  0.0398552 

ggplot(df_joined, aes()) +
geom_histogram(aes(x=hte_sb, fill='SoftBlock'), bins=50, alpha=0.4) +
geom_histogram(aes(x=hte_nn, fill='Greedy Neighbors'), bins=50, alpha=0.4) +
geom_histogram(aes(x=hte_qb, fill='QuickBlock'), bins=50, alpha=0.4) +
scale_x_continuous("Effect (pp)", labels=scales::percent) +
theme_minimal()