In the simulator below, use the drop down options to select from some pre-built patterns. If you want to control the individual joints in the snake, you can expand the 'Angle Controls'.

Unfortunately this simulator may not work particularly well on mobile. For a full screen version, click here.

Code is available on Github: https://github.com/mwkling/rubiks-snake. This was built using three.js and dat.GUI.

Interesting facts about the Rubik's Snake:

• Consists of 24 'wedges', each of which is a triangular prism.
• Not a puzzle like the Rubik's Cube. Inventor Erno Rubik said, "The snake is not a problem to be solved; it offers infinite possibilities of combination. It is a tool to test out ideas of shape in space."^[1]
• Originally released in 1981 just after the Rubik's Cube.^[1:1]
• There are 23 joints and 4 possible orientations for each joint, so a lower bound for the total number of possible configurations is 4^23.
• In reality, some of these configurations are impossible as the snake will intersect itself. About 19% of those 4^23 arrangements are actually possible.

What do we mean by dispersing overlapping rectangles? The idea is to take a set of overlapping rectangles:

Then spread them out in some way ('disperse' them) so that they no longer overlap:

When might we care to do this? I first saw this problem in the context of placing labels on a graph. Any kind of graph (e.g. a bubble chart, scatter plot, etc.) where each item has some text associated with it may have issues with labels overlapping. Most programs for generating these types of graphs (e.g. Excel) do not automatically prevent the labels from overlapping:

Maps with many features to label face a similar challenge. The general problem is referred to as automatic label placement (or point-feature label placement)

Questions on this topic are common^[1]. If you browse through these questions, you won't find any easy solutions. Finding the optimal solution (measured, for example, as minimizing the total movement of rectangles from their original positions) is an NP-complete problem^[2].

This problem has been tackled comprehensively by researchers. This is the best paper I found comparing performance for a variety of algorithms. Below, I implemented and compared some of these algorithms myself.

Greedy Algorithms

I decided to use Python and to visualize the results using Pygame. The first class of algorithms I looked at were greedy algorithms.

The most naive greedy algorithm I could think of was starting at the top of a group of rectangles and working down, shifting rectangles downwards as needed to avoid overlapping any rectangles above them. I'll call this the TopDown approach. Here's an animated example:

You can immediately see that some rectangles end up getting moved far from their original location.

A slightly smarter greedy algorithm might shift rectangles left or right in addition to up and down to prevent overlap. It also might start not at the top, but instead choose which rectangles to shift in a more intelligent way. One option would be to choose to shift rectangles with the maximum overlap first. Here is pseudocode for what I call the WorstFirst approach:

while has_overlaps(rectangles):
    overlaps = get_all_overlaps(rectangles)
    max_overlap = choose_max_overlap(overlaps)
    
    # This shifts one of the overlapping rectangles either down or to the
    # right to fix the overlap - whichever distance is smaller
    fix_overlap(max_overlap)

Here's an animated example:

Visually we can see an immediate improvement from the TopDown approach.

Of course, this type of greedy algorithm could be generalized. You could choose to fix the smallest overlap instead of the largest, the overlap closest to the center, the rectangle overlapping the most other rectangles, or use some other metric to make the choice. Regardless, the basic structure of the greedy algorithm would remain the same.

Separation Behavior

Other approaches can do better than a greedy algorithm. In the Separation algorithm, we iteratively move all the rectangles simultaneously at a certain velocity which is determined by how they overlap with other rectangles. The determination is made based on what is called 'Separation Behavior', originally described, as far as I can tell, for 'Boids' (this approach also has similarities to the Hirsch algorithm described in the paper referenced above).

Here is pseudocode for this behavior:

while has_overlaps(rectangles):
    # First, generate a vector for each rectangle
    # based on it's overlap with other rectangles
    overlap_vectors = []
    for each r in rectangles:
        v = generate_normalized_overlap_vector(r)
        overlap_vectors.append((r, v))

    # Then, translate each rectangle by it's overlap vector
    for each r, v in overlap_vectors:
        translate_rectangle_by_vector(r, v)

The overlap vectors are oriented so that overlapping rectangles will move away from eachother. Here's an example of overlap vectors with 3 rectangles:

The resulting movement looks as follows:

As we'll see below, this approach performs better than either of the greedy algorithms. Intuitively this makes sense, as this approach seems to take on a more 'holistic' view of the entire space.

Simulated Annealing

Finally, I tried implementing a simulated annealing algorithm. This type of algorithm excels at finding global maximums where other algorithms (for example, greedy algorithms, or an algorithm like gradient descent) may only find local maxima. The approach I've used was inspired by this paper, which was explained further in this blog post.

The pseudocode for simulated annealing is a bit more complex:

temperature = INITIAL_TEMP

while has_overlaps(rectangles):
    energy = compute_energy(rectangles)
    
    new_rectangles = make_random_adjustment(rectangles)
    
    new_energy = compute_energy(rectangles)

    # We randomly choose whether to accept the new arrangement or not
    # The choice to accept depends on the change in energy and current
    # temperature
    if random(0, 1) < e ^ ((energy - new_energy) / temperature):
        rectangles = new_rectangles
    
    temperature = temperature - COOLING_RATE

In the pseudocode above, the compute_energy function I chose was simply the total area of overlap of all rectangles. Over time, the algorithm will tend to slowly decrease the 'energy' of the arrangement, which in this case results in less rectangle overlap.

In addition, by slowly reducing the temperature parameter, it gradually becomes less likely for changes that increase 'energy' (in this case, cause more overlap) to be accepted as the annealing process continues. That is the purpose of the exponent containing the temperature parameter.

The choice of make_random_adjustment function is also important. In my implementation, 50% of the time this was a random translation of the rectangle, and 50% of the time this was a random rotation of the rectangle around the axis of it's original position. In testing, adding this 'rotation' option seemed to improve the results.

Visually, this looks like rectangles being randomly perturbed until eventually finding an equilibrium:

The final results may look similar to the results from the 'separation' strategy, but the path to get there is very different.

Comparison

So, which approach works the best, and how much better can we do than the most naive greedy algorithm?

Many factors will affect this: the total number of rectangles, their size, distribution, etc. All of these algorithms can also be adjusted in different ways (for example, the velocity for the separation algorithm, or many of the parameters in the simulated annealing algorithm).

I chose a fixed set of parameters and ran each algorithm for 1000 iterations, then compared the results. Roughly, the parameters chosen were 50 randomly generated rectangles on a 500 x 500 unit space, with each rectangle having width and height between 10 and 75. The 'score' of each iteration was equal to the total distance all rectangles had moved from their original locations when all overlaps were eliminated.

As expected, the most naive TopDown approach performs the worst by far. Separation and Simulated Annealing perform the best, with annealing averaging about 15% better than the separation algorithm for this particular set of parameters. These results are roughly consistent with those in the paper mentioned earlier (see here), where simulated annealing also performed best.

Implementation

The code is available on Github. For visualization, I used Pygame. Generating the GIFs was done with Imageio and pygifsicle.

Improvements

In many of the algorithms above, we are doing O(n^2) work on each iteration to check which rectangles are overlapping which other rectangles. It should be possible to use a more optimal data structure than just a list to improve the runtime complexity of these kinds of tests. Specifically, it would make sense to use a data structure that groups elements based on their spatial relationships, like an R-Tree.

Other data structure improvements might be possible for other algorithms. For example, it might be possible to implement some kind of Priority Queue approach for greedy algorithms (with the 'priority' of a given rectangle in the queue being equal to it's total overlap with the other rectangles, for example).

Significantly more effort could be made to improve the parameters for simulated annealing. The initial temperature, change in temperature, the type of random changes made, etc., could all be adjusted, and I imagine the right combination of changes might significantly improve the results.

Sometimes there's a big difference between the perceived difficulty of implementing a new feature from a developer's perspective and a customer or client perspective.  These are the types of features that sound simple, but in reality are often not worth the extra development time.  Here are a few examples I've run into:

Adding an 'other' option

If you're building a web application, there's a good chance a user will at some point need to choose between several choices in a form.  You might implement this with a select box:

Or maybe using radio buttons:

This implementation is simple, and it works.  The customer is happy, until they think of a potential problem:

Some people might want to choose something that's not on the list.  How hard would it be to add an 'other' option, and let people type in a custom choice?

From the customer's perspective this sounds extremely reasonable.  Why is it harder than expected?  Two primary reasons.

The UI needs to be updated in a non-trivial way

Adding another color choice to one the lists above is (literally) seconds of work for a developer - just copy/paste one line of code.  Adding an 'other' option with a text box is not so easy.  Now the developer needs to add a new input field of a different type and figure out how to style it properly.  If an 'other' option is added to the select box, maybe the text box should only appear after the 'other' option is selected - introducing Javascript to what was previously just an HTML form.

The way the server handles the request and stores the data may need to be changed

What previously might have been a single parameter in the request may now be split into two parameters (the choice from the list, and the text optionally entered in the 'other' box).  If the choice was represented as an enum (and stored as an integer in the database) this data type may need to change, or a new column might need to be added to the responses table.

What's the best approach here?

One choice is for the developer to push back: see how important having an 'other' option really is compared to other priorities, explain that it may be (somewhat) difficult to implement, and see if there's an alternative solution to get you most of the way there (like adding more choices).

Dynamic Form Updates By Administrators

Here's another scenario: a client has a list of fields they'd like to include as part of a user's profile.  This might include name, picture, email address, hometown, and a bio.  The developer builds a basic form where users can enter those fields.  After some additional work, the client realizes profiles also need to have phone numbers, so the developer adds this to the form.  Later, the client decides hometown shouldn't be on the profile, but birthdays should, and the developer makes these updates.  At some point after more back and forth, the client has another request and realizes this process is inefficient:

Let's add a place for users to enter a nickname on their profile.  Also, I might want to keep adding or removing some things to profiles in the future.  How hard would it be to setup a place where I can easily make those changes myself?

This kind of back and forth is to be expected as a platform evolves: consider how much Facebook has evolved, not just with new features, but in the information entered and collected as part of your profile.

In this case, the client, potentially familiar with site builder tools (say Wix, Squarespace, etc.) believes this request is reasonable and will save back and forth time in the future.  Why is it harder than expected?

Forms are not like static pages

Static pages (like homepages, blog posts, other landing pages, etc.) can often be served using a CMS or blogging platform (Wordpress, Ghost, Siteleaf, etc.). The client can interact directly with these platforms to make updates, avoiding the need for back and forth with a developer.

Unfortunately, adding a field to a form is not so simple.  If the client is given access to the HTML code for the page, copying and pasting an <input> tag is going to be error prone without knowledge of how to update the necessary attributes.  In addition, in most cases it will not be possible to update a form without some understanding of how the data is being stored.

In the case of a user profile, does a new field need to added to a User model on the back end?  Does a database migration need to be written to add a column to the User table?  Fundamentally, submitting a form, unlike viewing a static page, is going to change state on the server, and so some understanding of how that occurs is going to be necessary to make updates.

Building a form is nothing like building a form-builder

So why not setup a way for the client to edit the form that hides the hard parts?  In short, this is going to take a significant amount of work and is probably not worthwhile.  At a minimum an entire new interface needs to be built to allow an administrator to add or remove fields from the form, and the underlying data representation needs to change to accommodate these form fields being added or removed dynamically.

Perhaps the best evidence that this is not easy is the existence of specialized form builder tools like Wufoo or Typeform.  Entire businesses have been built to give people the ability to easily create and update forms without a developer.

What's the best approach here?

• Ideally, this problem can be avoided by setting expectations clearly up front.  When working on a new project, I like to make it explicit that there won't be any easy way for administrators to modify forms the way they might be used to with other pages: it will have to be done by a developer.
• In some cases, third party solutions might be useful.  Depending on the customer's goals, even something as simple as a Google Form might work.

Keeping Web Scraping Up To Date

Lots of startups will use web scraping at some point - maybe to gain information about competitors, to populate an initial database of potential customers, etc.  Most basic, small-scale web scraping tasks (e.g. scraping a list of businesses from an online directory) can be completed in an hour or two with a well crafted Python script.

Of course, once a client sees how easy it was to scrape the information the first time, there's a natural follow up question:

Now that we scraped that data from X website, how hard would it be to set something up to check if that list ever changes, and keep the list on our site up to date?

To a client that's seen immediate results from web scraping up front, this sounds reasonable and potentially simple.  Why is it hard?  There are two key issues:

Where will it run?

A one time scraping task is typically a script written on a developer's computer and run by the developer.  If scraping is going to be re-run automatically, say daily, what computer is it going to run on?  Having the developer manually run the script every day probably isn't practical.  If you have access to a server you might be able to set up a cron job, but if you're using a PaaS like Heroku, that's going to take a little more work.

How will errors be handled?

Sometimes websites are down, and sometimes they change how data is formatted.  For a one time scraping task this isn't a problem - the developer can look over the results and make sure that they look OK.  If scraping is automated, and the site happens to be temporarily down when scraping runs, will you end up deleting records in your own database when syncing the results? How will you get alerted that there was a problem?  Suddenly your simple Python script needs to track errors and send out email alerts.

What's the best approach here?

• One again, try to set expectations clearly when scraping the first time: explain that scraping a website once does not mean it will be easy to automatically scrape the same site daily and keep the results in sync.
• How often does the data really need to be re-scraped?  If you just want an update every few weeks, then having a developer occasionally re-run the script might be practical and cheaper than building automation.
• If automated web scraping is an essential part of the platform you're building (say, a price comparison platform), think about error handling and other infrastructure from the beginning.  I've had great experiences with scrapy.

Conclusion

When a developer is working together on a project with someone that doesn't come from a software background, part of the job is to know when to push back and see if the underlying goal can be achieved in a simpler way.  During recent consulting work for several startups, I've found this to be essential.  Really understanding a client's goals can tell you what code can be left unwritten, what parts of the platform can be outsourced to third party tools, and what processes can be run manually instead of implementing unnecessary automation.

Here's another Excel spreadsheet + macro solution to a recent problem.

Problem

Take a written transcript of something (maybe an interview, screenplay, podcast, webinar, etc.) from a Word document (in this particular case received from a transcription service like Rev.com):

and have it appear on your website:

If you're using a CSS framework like Boostrap, the HAML code to generate something that looks like the above image might look like this:

.row
  .col-md-3
    %h4
      Host
  .col-md-9
    %p
      Proin porta, risus a mollis consequat, ligula arcu sodales lectus, et imperdiet ipsum leo vel purus. Quisque at ipsum sed sem cursus eleifend non quis nunc. Etiam sit amet auctor lorem. Duis ut semper augue. Nunc facilisis tempor lacus. Ut et pellentesque turpis, eu mollis nibh. Integer id urna ac nisl porta commodo. Maecenas hendrerit dolor vitae ligula vehicula, vitae varius nunc facilisis. Etiam scelerisque dapibus neque ut euismod. Nullam scelerisque lacinia diam.
.row
  .col-md-3
    %h4
      Guest
  .col-md-9
    %p
      Ut quis hendrerit arcu, sit amet tincidunt tellus. Aliquam finibus ante sed enim facilisis, in eleifend nulla gravida.
.row
  .col-md-3
    %h4
      Host
  .col-md-9
    %p
      Suspendisse egestas vitae nisl in tempor. Proin in laoreet neque, et suscipit est. Sed congue accumsan sollicitudin. Vivamus eu sem sodales, congue tortor sed, facilisis ex. In suscipit gravida odio, vel hendrerit turpis. Vestibulum at elit volutpat, faucibus justo pretium, venenatis nisl.
    %p
      Donec sed lectus eget nibh placerat malesuada. Duis suscipit nunc eu sapien ornare, ac egestas metus imperdiet. Morbi rhoncus nisl sit amet tristique semper. Sed sit amet mattis lorem. Sed nec quam non tortor tristique placerat. Vestibulum at purus orci.

Solution

You could probably do this using a (complicated) regex replacement in a text editor. But, the advantages of a standalone, purpose-built tool include easier configuration changes, and easier extension in the future if we need to handle more complex output.

First, notice we can copy and paste the transcript from Word into an Excel spreadsheet and get a nice breakdown of the speaker names in one column and content in another:

Next, add another sheet with some configuration settings. Some CSS frameworks are similar enough that changing these cells is enough to generate the appropriate HAML (Boostrap vs. Foundation, for example):

And now, add a simple macro to write the HAML code to a file. This macro reads down the sheet where the transcript was pasted and uses the settings to generate the HAML code. Note the addition of appropriate whitespace to the output.

Option Explicit

Sub do_all()
  Dim fn As Long, i As Long

  fn = FreeFile
  i = 0
  
  Dim out As String
  out = Range("out_file").Value

  Dim row As String, scolumn As String, sheader As String, tcolumn As String, paragraph As String

  row = Range("row_class").Value
  scolumn = "  " & Range("scolumn").Value
  sheader = "    " & Range("sheader").Value
  tcolumn = "  " & Range("tcolumn").Value
  paragraph = "    " & Range("paragraph").Value
  
  Open out For Append As #fn
    While Range("contents").Offset(i, 1).Value <> ""
      If Range("contents").Offset(i, 0).Value <> "" Then
        Print #fn, row
        Print #fn, scolumn
        Print #fn, sheader
        Print #fn, "      " & Replace(Trim(Range("contents").Offset(i, 0).Value), ":", "")
        Print #fn, tcolumn
      End If

      Print #fn, paragraph
      Print #fn, "      " & Trim(Range("contents").Offset(i, 1).Value)

      i = i + 1
    Wend
  Close #fn
End Sub

Copy paste the result from the output file into your website source code, and you're all set. Easily run it again by pasting in a new transcript to the spreadsheet and re-running the macro.

The code above is available on Github.

At TrueJob we've spent a significant amount of time building up an email list to use for marketing and outreach. Inevitably, some processes have gotten a bit messy over time - we store contacts in our CRM, but we've been sending out email campaigns via Sendgrid. Sendgrid ends up with important information not reflected in our CRM (bounced emails, blocks, spam reports, etc).

Maintaining good email list 'hygiene' is essential for continuing to get your messages delivered (and it's the law), so we're careful about removing any email addreses from our list that have delivery problems.

Bad Old Days

The workflow when creating a list for an email campaign used to look something like this:

• Download list of relevant contacts from the CRM
• Download a bunch of 'suppression' reports from Sendgrid (separate reports for bounces, blocks, spam, invalid addresses, etc.)
• Copy the CRM contacts list into a new spreadsheet
• Copy all the 'suppressions' into another column in that spreadsheet
• Use some excel formulas (e.g. VLOOKUP) to filter out all the suppressed emails from the list of contacts
• If there were any specific organizations we didn't want to email, search the resulting list for any with that domain
• Copy the final filtered list into a CSV ready for upload

This didn't take a huge amount of time (say 10-15 minutes) but it was annoying and repetitive.

Good New Days

The new process looks like this:

• Download contacts from the CRM and all the reports from Sendgrid into one folder
• Open a spreadsheet and run a macro that does all the necessary filtering and writes out the final file for upload

Why use Excel macros and not a real programming language like Python? Sometimes it is nice to skip the command line and open a spreadsheet with a big button that says 'GO'. More importantly spreadsheets make it easy to provide configuration visually (by updating cell values), and when working with CSV files it's also nice to be able to view and search entries when debugging.

Step 1: Import all the CSVs

The first step was to write a macro to import all the data we need into one spreadsheet. This isn't strictly necessary - but it looks cool, makes things easier later, and leaves all the data in one place if there is any debugging to do.

This macro opens every CSV file in a directory one at a time and copies the contents into a new worksheet of the main workbook.

Sub import_all_csvs()
  Dim s As String
  Dim totalSheets As Long
  Dim thisBook As String
  Dim baseDir As String
  
  baseDir = Range("input_directory").Value
  thisBook = Application.ActiveWorkbook.Name
  
  s = Dir(baseDir + "*.csv")
  
  Do Until s = ""
    If s <> Range("out_file_name").Value Then
      Workbooks.Open (baseDir + s)
      totalSheets = Workbooks(thisBook).Worksheets.Count
      Workbooks(s).Worksheets(1).Copy after:=Workbooks(thisBook).Worksheets(totalSheets)
      Workbooks(s).Close
    End If

    s = Dir
  Loop
End Sub

Note the Dir function - call once in a target directory to get the first matching file, then subsequent calls without arguments return the next matching file (see generators).

The result when you see this run (a rapid opening and closing of spreadsheets) will make you look like an Excel wizard.

Step 2: Read the CRM contacts

In our case, a user's email address might appear in one of several columns that start with email (e.g., 'Email (Personal)' or 'Email (Work)'). The function below handles this very typical data cleaning problem by storing all potential email columns in a Dictionary, then iterating through each potential email column for each row.

Function get_starting_list() As Dictionary
  Dim cur As Worksheet
  Dim c As Long
  Dim r As Long
  Dim key As Variant
  Dim emailColumns As New Dictionary
  Dim emails As New Dictionary
  
  For Each cur In Worksheets
    'In our case this identifies which sheet has the list of contacts
    If Left(cur.Name, 7) = "truejob" Then
      'First find the columns that might hold email addresses
      c = 1
      Do Until cur.Cells(1, c) = ""
        If LCase(Left(cur.Cells(1, c), 5)) = "email" Then
          emailColumns.Add c, 1
        End If
        c = c + 1
      Loop
      
      'Now go through each row and find an email address'
      r = 2
      Do Until cur.Cells(r, 1) = ""
        For Each key In emailColumns.Keys()
          If cur.Cells(r, key) <> "" Then
            emails.Add LCase(cur.Cells(r, key).Value), 1
            Exit For
          End If
        Next
        r = r + 1
      Loop
    End If
  Next
  
  Set get_starting_list = emails
End Function

Step 3: Read the 'Suppressions'

Now, we loop through each worksheet - skipping the sheet we read the CRM contacts from - and build up a list of email addresses to suppress. These are the ones we don't want to appear in the final output. This is straightforward so I'm leaving this code out: on each sheet we find the column with a heading that starts with 'email' and just read down the list.

Of course in our case there is some extra complexity which comes from another common data cleaning challenge: some of the CSVs we download come in a slightly different format with a single column of email addresses and no header. We use a simple and reliable test: check for an @ sign in the contents of the first cell (alternatively, you could use a regex).

Step 4: Write the output file

Finally, we put it all together. One main function (do_all) calls each of the steps in turn, and another writes the output file. It's particular easy to write CSV files that only have a single column (maybe version 2.0 will include additional information like first and last names for more personalized campaigns). Note I've included a convenient ability to filter out particular domains.

Sub do_all()
  Call import_all_csvs
  
  Dim startingList As Dictionary
  Dim suppressionsList As Dictionary
  
  Set startingList = get_starting_list()
  Set suppressionsList = get_suppressions_list()
  
  Call write_output_file(startingList, suppressionsList)
End Sub

Sub write_output_file(start As Dictionary, suppress As Dictionary)
  Dim domainsSuppress As Dictionary
  Dim out As String
  Dim email As Variant
  Dim fn As Long
  Dim i As Long
  Dim domain As String
  
  'get_excluded_domains reads values from a named range on the spreadsheet'
  Set domainsSuppress = get_excluded_domains()
  out = Range("input_directory").Value + Range("out_file_name").Value
  fn = FreeFile
  i = 0

  Open out For Append As #fn
    Print #fn, "email"
    For Each email In start.Keys()
      domain = Split(email, "@")(1)
      
      If Not suppress.Exists(email) And Not domainsSuppress.Exists(domain) Then
        Print #fn, email
        i = i + 1
      End If
    Next
  Close #fn

  MsgBox Str(start.Count) & " initial emails" & Chr(13) & Str(i) & " written to file"
End Sub

Fill in the relevant fields, click GO, and:

Success!

You can find the spreadsheet code on Github. To use it, you'll first need to add it to a spreadsheet with the appropriate named ranges.

The ideal world might see us with a better system to keep our data in sync, but sometimes simple is better and I'm guessing this spreadsheet will keep things working for some time.

The code for this game is available on GitHub. The drag and drop functionality uses interact.js. If you are on mobile, you might prefer to play the game by itself here. If you have feedback, leave a comment!

The Rules

In this game, you start with a set of initial digits and a goal number. Your task is to combine the initial digits using different mathematical operations to reach the goal number.

You might have heard of or played this kind of game before (I've listed some variations below). In this version, all the basic operations are permitted, which includes (+, -, *, /) as well as exponents (a^b). The key differentiator is the inclusion of concatenation, and an inverse that we'll call 'digit splitting'.

What do I mean by 'digit splitting'? This means the original digits or any digits that are part of intermediate results can be split apart, rearranged, and recombined into new numbers in any way. For example, given the number 23, you can use the digits 2 and 3 individually, combine them with +/-/etc, or rearrange them to make 32.

This gives you a lot of options. Starting just with the numbers 2 and 6, you can make:

$$\begin{align}
2+6 & =8 \\
6-2 & =4 \\
2\times 6 & =12 \\
6/2 & =3 \\
6^2 & =36 \\
2^6 & =64 \\
2||6 & = 26 \\
6||2 & = 62 \\
\end{align}$$

From here, you can take the digits from these results and make new numbers. For example, you could take 64 and split this into a 6 and a 4 to make 24, 46, 4⁶, and more.

To show solutions to particular puzzles, I'll use a set notation with arrows. For example, suppose our initial digits are 7 and 8, and the goal is 65. One solution is:

$$\{7,8\} \xrightarrow{\times} \{56\} \xrightarrow{split} \{5,6\} \xrightarrow{\Vert} \{65\}$$

Background

This is a mental math game I've been playing for many years^[1] but the original inspiration may have come from a worksheet I did sometime in middle school called the the Four Fours puzzle. The idea is to combine four 4s using basic math operators to make different numbers. If you try this out, you'll see you can make all the numbers from 1 through 9 using only the four basic operators (+, - *, /) and parentheses.

Show solution

However, using only these operators you'll get stuck at 10. We can get to 10 by introducing something new, like concatenation: $$(44-4)/4=10$$

This doesn't get you far - now you'll be stuck at 11. To continue, you can introduce square roots, factorials, logs, etc, and if you bend the rules in this way enough, you can actually generate any number with four 4s^[2].

I don't love this solution. Functions like square roots or logs operate on just one number, instead of two numbers like our original operators. For this reason, I'd prefer to add exponentiation (a^b) to the toolkit. This plus the digit splitting operation (which acts as a kind of 'inverse' to concatenation) resulted in the game.

So, in this game can we get to 11 starting with four 4s? Here is one possible solution:

$$\{4,4,4,4\} \xrightarrow{\times} \{4, 4, 16\} \xrightarrow{\times} \{4, 64\} \xrightarrow{split} \{44,6\} \xrightarrow{-} \{38\} \xrightarrow{split} \{3, 8\} \xrightarrow{+} \{11\}$$

Variations

I haven't found a reference to someone playing this game with these exact rules before, but there are many similar variations.

One is the 24 game. In each round, a playing card with four numbers is dealt face up, and a group of players tries to figure out how to reach 24 using the four numbers on the card and basic math operations. The first person to do so gets a point.

This is a good example (image from here). Create 24 using 2, 9, 6, 6, and basic math operations.

Show solution

Another is the British game show 'Countdown'. In one portion of the game, players are given 6 starting numbers (usually including a few small numbers, and few large round numbers like 25, 50, or 100) and a goal number between 0 and 1000. Here's a video of a particularly tricky case:

In this video the goal is to create 952 using 100, 75, 25, 50, 6, and 3.

Show solution

There are weirder variations. One mathematician decided to find ways to generate numbers using each of the digits from 1 through 9, with the added requirement that the digits must appear in increasing or decreasing order reading from left to right. For example:

$$\begin{align}
60&=1+2\times 3\times 4 + 5 + 6 + 7+ 8+ 9\\
60&= 9+ 8+7+6 +5\times 4 + 3^2 + 1
\end{align}$$

This paper contains solutions for every integer up to 11,111 - with one exception^[3].

A proof of completeness

It turns out that in this game there are only a few types of starting sets where you'll be 'stuck', and unable to create any possible number given enough steps. This might make sense intuitively: as long as we have 2 digits of a reasonable size, we can use the exponent operator to create a large number with lots of digits to work with, and if necessary keep repeating the process. Let's call starting sets of digits where you can create any possible number complete sets.

The first step in the proof is noticing a key pattern. Play with this game for a bit and you'll notice certain numbers and operations can create 'loops':
$$\{25\} \xrightarrow{split} \{2, 5\} \xrightarrow{a^b} \{25\} \\
\{3,7\} \xrightarrow{a^b} \{343\} \xrightarrow{split} \{3,4,3\} \xrightarrow{+} \{3,7\}
$$

Here's another example that's not quite a loop:
$$\{3,6\} \xrightarrow{a^b} \{216\} \xrightarrow{split} \{2,1,6\} \xrightarrow{a^b} \{1, 36\} \xrightarrow{split} \{1, 3, 6\}$$

Notice we returned to our original set but with an additional '1' digit. By repeating this process, we can continue to generate more 1s, and thus create any number. Therefore, with our terminology, any starting set containing the digits 3 and 6 is a complete set. In addition, any starting set that can be used to reach a set with a 3 and a 6 is also a complete set.

So, when are we able to reach a set with a 3 and a 6? First let's consider all starting sets with 2 digits (1 digit starting sets are always stuck where they started). There are 100 of these. It's not too difficult to check by hand that all except for a few can reach a set with a 6 and a 3 and are therefore complete. For example:

$$\begin{align}
\{2,6\} &\rightarrow \{36\} \\
\{2,7\} &\rightarrow \{49\} \rightarrow \{4,9\} \rightarrow \{36\} \\
\{2,8\} &\rightarrow \{256\} \rightarrow \{2,5,6\}\rightarrow \{3,6\} \\
\{2,9\} &\rightarrow \{512\} \rightarrow \{5,1,2\}\rightarrow \{6,2\} \rightarrow \{36\}\\
\ldots
\end{align}$$

The exceptions are:

$$\{0, x\}, \{1, x\}, \text{where } x \in \{0, 1, 2, \ldots 8, 9\} \\
\{2, 2\}, \{2, 3\}, \{2, 4\}, \{2, 5\}$$

You should be able to convince yourself that the only starting sets with 3 digits that might not be complete must contain two numbers from one of the above sets and one additional number. Try adding an additional digit to one of the above sets and you'll see almost all become complete. For example, by adding a 0 to the set {1, 2}, it becomes complete:

$$\{1,2,0\} \rightarrow \{2,10\} \rightarrow \{1024\} \rightarrow \{10,2,4\} \rightarrow \{10, 16\} \rightarrow \{26\} \rightarrow \{2, 6\} \rightarrow \{36\}$$

The only sets of 3 digits that are still not complete are:

$$\{0, 0, x\}, \{0, 1, 1\}, \{1, 1, 1\}$$

By the same logic, the only sets of 4 or more digits that are not complete are:

$$\{0, 0, 0, \ldots, x\}$$

That covers it -- every initial starting set other than those listed above is complete and can be used to reach any goal number.

I spent three weeks traveling through New Zealand, starting in Auckland at the top of the North Island, then traveling by bus, ferry and plane towards Queenstown in the South Island.   Along the way I met locals, ate at fast food chains and nice restaurants, hiked in several national parks and gradually accumulated knowledge of differences between New Zealand and the United States.  It has much in common with the United States: both are former British colonies, English speaking, with fully developed economies.  With so much in common, the differences are all the more surprising when they appear.

1. European New Zealanders are still referred to using a native Maori word

The first thing I noticed was the inflight magazine on my Air New Zealand flight from Los Angeles to Auckland:

The title of the magazine, ‘Kia Ora’, is the native Maori people’s word for ‘Hello’.  Inside I found reference to another term – ‘Pakeha’.  I noticed this word again in newspaper articles, generally in a sentence referring to ‘Maori and Pakeha’.  When Europeans first arrived in New Zealand several hundred years ago, the native Maori people used the word ‘Pakeha’ to refer to them, a word meaning ‘foreigner’ or ‘white person’.  Incredibly, this word has widespread usage by European New Zealanders today when referring to themselves.  John Chambers in ‘A Traveller’s History of New Zealand’ explains how unprecedented this is:

“This usage, whereby the majority and more economically powerful and formerly imperial ethnic group unremarkably uses a term given them by the native people is perhaps unique in history.  It is as if modern United States citizens of European descent, naturally and unselfconsciously in everyday language were to call themselves ‘Palefaces’ or some native-American term.”

The story of the Maori people mirrors similar stories of native peoples across the globe.  Maori leaders signed a treaty with the British in 1840 turning over some land rights (the Treaty of Waitangi) and disputes over land began to occur.  These conflicts, as well as disease, led to a decline in the Maori population until the 1900s.  Today they make up about 15% of New Zealand’s population, and continue to struggle with higher rates of crime and health problems.  If you’re a rugby fan, you’ve heard of the All Blacks, New Zealand’s national rugby team.  Before some matches they perform a Maori Haka, a traditional war chant.

2. World War 1 was a huge deal for New Zealand

Just outside the center of Auckland, at the top of a hill surrounded by grassy fields, sits one of Auckland’s most iconic buildings, the Auckland War Memorial Museum.  One of the first things I noticed inside was the heavy emphasis on World War 1 — even more so than World War 2.

The Great War has steadily faded in the American consciousness for the past hundred years.  Hitler, D-Day, Pearl Harbor, Nazis, and the Holocaust are remembered vividly, but what do we remember from the First World War?  Before I went to New Zealand, my memory from high school world history was a hazy mix of the Archduke Franz Ferdinand, trench warfare, and black and white photos of soldiers wearing gas masks.  If Vietnam and World War 2 are in the top tier of 20^th century American conflicts ranked by their presence in our collective memories, World War 1 is on the second tier, above but not far from conflicts like the Korean War.

In New Zealand, World War 1 has top tier status.  This is mostly a result of timeline and geography.  New Zealand is a younger country than the United States, and had existed only briefly when the First World War began.  It is also an isolated island country and avoided regional conflicts.  Though they had been involved in other conflicts prior (e.g. sending troops to South Africa in the Second Boer War), World War 1 was their first large scale international conflict.

Since the signing of the treaty of Waitangi in 1840 by the native Maori leaders, New Zealand had been a British colony.  When WWI started, many New Zealanders still had close ties with Britain and considered themselves proud to be part of the British Empire.  When Britain declared war in 1914, many New Zealanders were prepared to help the Crown.  Just over 100,000 New Zealanders served overseas during the war–10% of the population.

The casualty rate for these soldiers was 58% – 17,000 killed and 41,000 wounded.  Almost two percent of New Zealand’s population was killed in the war.  While this is on par with the United Kingdom, and less than France or Germany (deaths were closer to 4% of their population), it is huge compared with the United States, which lost ten times as many soldiers but had a population 100 times as large.  Though the United States played an important role in the war, they entered very late.  New Zealand was present from the beginning, which meant they took part in the failed invasion at:

Gallipoli

Later in my trip I visited Wellington, which sits on the south end of the North Island.  In a prominent position on the waterfront is a museum referred to as ‘Te Papa’, a Maori phrase meaning ‘Our Place’.  There is currently a major new exhibit there devoted wholly to Gallipoli.  I tried to visit my first day in Wellington, but with a wait of well over an hour, I came back at open time the next day (and still had to wait almost a half hour).

In 1915, New Zealand forces joined Australians, British, and others in a now infamous attempt to capture the Gallipoli peninsula, which controlled access to the strategically important Dardanelles Strait near Turkey.  At first glance an American might find some things about this campaign ripe for comparison with our best remembered military battle: the D-Day invasion of World War 2.  Both were large allied invasions, starting with a beach landing and moving inland to capture key territory.  Winston Churchill had a hand in both — he originally proposed the Gallipoli Campaign.

(picture from here, landing at Anzac Cove on April 25th, 1915)

By every measure Gallipoli was more brutal than D-Day.  New Zealand and Australian troops landed at Anzac Cove on April 25th, 1915.  The beach had some protection, but was in range of Turkish artillery, and heavy machine gun fire prevented them from moving inland.  Trenches were dug on both sides and the battlefield slowly took a similar appearance to the Western Front.  The Allies launched several major offensives over the following months, all of which suffered devastating casualties, and all of which were pushed back.  By the end of 1915 the Allies had almost entirely evacuated.  Casualty rates on both sides were close to 60%.

No one disputes the bravery or sacrifice of the soldiers on D-Day, but however brutal it was, that invasion succeeded quickly.  The Allies suffered approximately 10,000 casualties in the D-Day invasion.  At Gallipoli the Allies suffered 250,000 casualties in a prolonged eight month conflict which ended in retreat.  The stature of these battles in the respective national consciousness of each country is similar.  New Zealand and Australia still celebrate ANZAC day on April 25^th, the landing day at Gallipoli, as their major day of remembrance.

New Zealand also suffered major casualties later in the war on the Western Front and in other battles.  In nearly every town I visited there was a World War 1 memorial of some kind.  There were no American troops at Gallipoli, so for us it is largely a historical footnote, but its impact on New Zealand as a young country still forging a national identity cannot be understated.

3. ‘Free Wifi’ might mean ‘A little bit of free wifi’

I’m including this as a note of caution to future travelers.  After leaving Auckland, I took a bus south to Rotorua.  I had booked a single room in a cheap motel that advertised ‘Free Wifi’.  At check in, the man behind the desk asked if I needed Wifi access.  I said yes, expecting he would give me the password to connect.  Instead, he handed me a small slip of paper that said ‘100 MB Wifi’, with a login code.

(these Zenbu vouchers were common)

100 MB of Wifi is fine to check email, but disappears quickly if you are scrolling through pages on TripAdvisor, and even more quickly if you are uploading pictures.  I had the same data rationing experience in most places I stayed.  At one Queenstown motel I was asked if I wanted ‘a little, a medium amount, or a lot of Wifi’.  I chose medium, which was worth 300 MB of Wifi credit.

That said, everyone I met working at hotels and hostels was incredibly kind and accommodating.  The one time I ran out of data, they found me an extra 100 MB of credit.  But if you’re taking a lot of pictures, don’t expect to upload them all until you get to a country where unlimited Wifi is common (or else, expect to pay up).

4. New Zealand was once the world’s greatest bird paradise

When I failed to visit the overcrowded Gallipoli exhibit at the ‘Te Papa’ museum on my first day in Wellington, I turned next door to an exhibit on New Zealand’s natural history.  My knowledge of New Zealand animal life was mostly the Kiwi, a symbol of New Zealand most people recognize, but it turns out that’s just one relic of the country’s history as a bird paradise.

The landmass of New Zealand was formed 80 million years ago when shifting tectonic plates caused it to separate from the supercontinent of Gondwanaland.  Notably, this separation occurred before mammals had evolved.  The Tasman Sea, which separates New Zealand and Australia, slowly widened.  Until humans arrived, the only animals that could reach New Zealand were birds.  The result?  New Zealand became a bird utopia.  The only mammals that existed in New Zealand were bats.

In isolation and without the presence of mammals, some of the largest birds to ever exist evolved in New Zealand (a phenomenon known as island gigantism).  When flight no longer proved to be a competitive advantage, many birds became flightless.  The Giant Moa that lived on the south island (#4 below) was the tallest bird species ever known to exist, a flightless herbivore that could reach plants up to 12 feet from the ground.  The only predator of the Moa was the Haast’s Eagle, the largest eagle that has ever been discovered, which had up to a ten foot wingspan.  The Kiwi, now the country’s national symbol, evolved as another flightless bird.  It is a strange, nocturnal creature distinguished for laying eggs six times larger than a chicken’s–the largest in proportion to body size of any bird.  They are still wild in certain parts of the country, but I saw one only in captivity.

(picture from here, scale of different Moa species)

What happened to all of these species?  When the Maori people arrived in the 1300s, they hunted the Moa to extinction within 150 years.  The Haast’s Eagle lost its primary source of food and quickly went extinct.  The Kiwi survives, but along with many other native birds, it suffered from the introduction of species like ferrets and stoats.  These fed on Kiwi eggs and necessitated conservation efforts which continue.  The most common places for conservation are islands or fenced off peninsulas, utilizing the surrounding water as a natural barrier.  Kiwis are common on Stewart Island in the Pacific below the South Island.  While hiking around Lake Waikaremoana on the North Island, I passed by a fenced off peninsula on the lake dedicated to Kiwi preservation.  Metal fences are driven several feet below the ground to prevent the ferrets and stoats from digging underneath.

Today, many outside plants and animals have been introduced.  One tour guide I spoke to said, “If it walks on four legs, or loses its leaves in the winter, it was probably introduced by humans.” The most obvious is sheep, which have long outnumbered humans and cover the countryside.

(Sheep – introduced by humans to New Zealand)

5. New Zealand was almost the last landmass on Earth to be discovered

What about human settlement?  After wandering through the exhibit on the natural history, I found some information on the first human settlers, the Maori people.  For 80 million years New Zealand was home to mostly birds, a few reptiles and amphibians, and surrounded by vast oceans.  It was not until 1300 AD that humans arrived, most likely on large canoes that had traveled from other Polynesian islands far to the northeast like the Society or Cook Islands.  Americans might often be tempted to group New Zealand with neighboring Australia, but this is a point of stark contrast: in Australia, human remains have been dated to 50,000 years ago.  New Zealand was the last major landmass to be discovered with one exception – Antarctica.  (A New Zealander, Alexander von Tunzelmann, was a member of the first group known with certainty to have set foot on Antarctica in 1895).

The map below shows approximate times a few islands in the Pacific were settled.  Areas like New Guinea close to Southeast Asia were accessible and settled early.   Polynesia was settled much more recently.  Dates are estimates, and migration paths are derived by looking at the movement of cultural objects like pottery as well as by analyzing linguistic similarities between different island peoples.  Interestingly, there is no evidence of the Maori ever returning home to islands in Polynesia.  The trip to New Zealand was probably one way.

(original image from here)

6. New Zealand is a ‘Nuclear Free Zone’, which occasionally causes drama with the United States

While exploring Wellington, I came across something unexpected: a Japanese lantern housing a Peace Flame.  This flame was gifted by Hiroshima, Japan in 1994 and originally lit from fires started by the atomic bomb.

(Peace Flame in Japanese lantern, located in Wellington)

Wellington declared itself ‘nuclear free’ with the passing of a resolution in 1982.  Starting in the 1950s, the public was concerned about nuclear weapons testing occurring in the Pacific.  Public opposition to nuclear weapons grew throughout the Cold War.  Many towns passed resolutions declaring themselves ‘nuclear free’.  The rest of the country followed suit in 1984, when the prime minister declared that nuclear ships or submarines would not be permitted to dock in New Zealand.  Three years later, legislation was passed establishing New Zealand as a ‘Nuclear-free zone’.

Prior to this, New Zealand and the US had a long history of cooperation.  In World War 2, more than 400,000 Americans were stationed in New Zealand before famous battles like Saipan and Iwo Jima.  After the war, Australia, New Zealand, and the US signed the ANZUS treaty, a three way defense pact established in 1951.  New Zealand sent troops to 20^th century conflicts including the Korean War and Vietnam (which was very controversial, as it was in the US).  But as the 20^th century progressed, New Zealand grew more independent and less reliant on both the UK and the United States.

When the prime minister declared New Zealand ‘Nuclear Free’, the United States was not pleased.  The US refused to confirm or deny if their ships or submarines were nuclear.  Effectively, no US ships could land in New Zealand.  As a consequence, the New Zealand-US link of the ANZUS treaty was terminated (both links with Australia still remain effective).  This was a low point in relations, which have since since improved.  New Zealand sent troops to the first Gulf War and Afghanistan, and they remain a part of the ‘Five Eyes’ intelligence sharing alliance (comprising the US, Canada, UK, Australia, and New Zealand).

Interestingly, the recent leaking of diplomatic cables suggests that the banning of nuclear warships and subsequent termination of ANZUS with the US was driven by economic considerations as well as ideology.  A cable from the US embassy in Wellington read:

“one of the considerations favouring the policy was that it would lead to New Zealand withdrawing or being pushed out of ANZUS, thereby lessening the country’s defence spending requirements at a time of fiscal and economic crisis”  (Quote from here)

7. New Zealand’s landscape is as diverse as a continent

I was stunned by the variety of landscape packed into New Zealand’s two islands.  In this small space, New Zealand is home to:

• Volcanoes, some of which are still active (see e.g. Tongariro National Park, which I did not have time to visit)
• Temperate rain forest (on North and South Islands)
• The Southern Alps, a mountain range formed by the collision of tectonic plates
• Pristine sand beaches (see Abel Tasman National Park, or Ninety Mile Beach on the North Island)
• Fjords carved by glaciers (see Fjordland National Park, and Milford Sound)
• And of course, Hobbiton (near Rotorua)

Beach at Abel Tasman National Park

Southern Alps while flying into Queenstown

Milford Sound

Hobbiton

Visit if you can!

I recently left a job on Wall Street where some of my former coworkers donated large fractions of their income to charity.  At least one was motivated to work on Wall Street by the possibility of earning more to donate more.  I wondered: what does that say about my choice to leave?  Not only was I not donating any significant amount before leaving, but even worse, I was giving up potentially much greater future earnings that could have been donated.  Was my choice bringing indirect harm to those that would have benefited?  Admittedly this is not something most people have considered.  A person struggling to get by is not going to feel guilty about not donating a sufficient fraction of their income, so posing this question at all is already privileged.  Before deciding we need to start with the movement that motivated my coworkers: effective altruism.

Thought Experiment 1: The Drowning Child (Wall Street version)

You are a successful Wall Street trader.  Walking to work one morning, you pass a shallow pond where a child is drowning (there are no ponds near Wall Street, but let’s pretend).  If you wade into the pond to save the child, you will ruin your $3,000 suit and need to purchase a new one.  Are you still obligated to do so?

The next day you are walking to work and pass a booth requesting donations for an anti-malaria charity.  The sign states that a $3,000 donation will buy enough mosquito nets to save one child’s life in Africa.^[1] You know the person manning the booth, and have already researched this charity, and you believe this claim to be accurate.  Are you obligated to write them a check?

I’m paraphrasing the story of the Drowning Child originally from philosopher Peter Singer, who argued that both cases are equivalent, and that we have as much moral obligation to write the check as we do to jump in the water.  The distance of the child in Africa and the difference in nationality make no ‘morally relevant difference’.

This idea is just one piece of a broader philosophy and social movement known as ‘Effective Altruism’.  In his Ted talk outlining the central tenets of this philosophy, Singer describes effective altruism as an attempt to combine the ‘heart and the head’, meaning we should combine our empathy for suffering children with a logical consideration of how we can most effectively reduce it.  One consequence is a careful consideration of which charities to donate to.  We should favor those that have proven they are most efficient at converting dollars into saved lives and reduced suffering.

Singer argues it should also induce us to donate significantly more than we currently do.  Say your income is $50,000 per year.  Does donating $3,000 to reduce that to $47,000 cost your well being more than the value of a child’s life? When weighing the importance of first world desires against third world suffering, Singer concludes that donating 10% of your income (or more, with some in the movement going beyond 50%) comes at a small cost.

With this in mind, why not choose a career that will allow you to earn more and thus donate more?   This is the concept of ‘Earning to Give’.  An effective altruist weighing two career choices might choose the one with greater earning potential.  This idea is unusual enough that examples of it have received publicity: the New York Times article “The Trader Who Donates Half His Pay” discusses my former coworker Matt Wage who chose a career on Wall Street over academia and donated $100,000 to charity in 2013.  Or the Washington Post article “Join Wall Street. Save the world.” where Jason Trigg describes his choice to work on Wall Street and donate half of his salary.  Trigg said:

“A lot of people, they want to make a difference and end up in the Peace Corps and in the developing world without running water, and I can donate some of my time in the office and make more of a difference.”

If, as Trigg says, working on Wall Street is making more of a difference than if he had joined the Peace Corps, what does this imply for someone choosing between the two careers that decides to go to the Peace Corps?   What does this imply for someone like myself, who chose to leave Wall Street without planning to go work for the Peace Corps?  Does ‘Join Wall Street. Save the world’ imply that leaving is akin to abandoning a moral obligation?

The interesting parts of this question lie in the details.  To dig into them I present another thought experiment (a variation on the Trolley Problem).  The numbers involved give us something to work with later, but they are not critical.

Thought Experiment 2: The Altruistic Trader

You are a wall street veteran and effective altruist at work on the trading floor.  You are staring at your computer screen, waiting for a large trade that might happen at any moment.  If a certain set of numbers appear on the screen, you’ll need to press a certain set of keys quickly to execute the trade.  You expect the profit on the trade will be $1mm.  As part of your contract, you receive 3% of the firm’s profits as a bonus each year, so if you execute the trade, your bonus will increase by $30,000.  As an effective altruist, you will donate half of this money to the anti-malaria charity, saving 15,000/3,000 = 5 children’s lives.

While staring at the computer screen, you see through the window a child drowning in the pond outside.  It appears no one is going to help her.  If you step away from your desk it is likely you will miss the trade.  Are you obligated to ignore the drowning child?

The answer is: it depends.

Some Assumptions

You protest, ‘Isn’t there lots of waste in charities?  How do I know my money is getting where it is supposed to be?  How do I know that $15,000 donation will really save 5 lives?’  A valid concern, but a distraction from more interesting questions.  Organizations like Givewell have made significant investment in determining how effective particular charities are.  By doing this, they help improve confidence in these claims.  There is uncertainty and they will make mistakes, but for now let’s take it as fact: donating that money will save that many lives.

‘How do you know you’ll make $1mm on the trade?  Isn’t trading like gambling, with probabilities and no certainty?’  Again a valid concern, and accurate – I don’t remember ever waiting for a trade with a certain $1mm profit, but to keep things interesting let’s assume such a trade exists, the profit is certain, and so is the effect on your bonus.

‘There is no definite moral right and wrong.  It will always be grey, not black and white as you have described it.  There are too many variables and we can never know what the right thing to do is.’  I find this difficult to argue with.  In reality there will always be too many variables and too many second or third order effects from every choice we make to really understand the consequences.  Maybe you believe there are no ‘moral obligations’ at all, and questions of morality depend entirely on an individual moral framework^[2].  Right and wrong is always subjective, and with the right context any action could be justified.  I believe this to an extent, yet I’ve still made the effort to have the discussion below.  Why?  First, because it is interesting.  Second, because even an imperfect model making these assumptions might still teach us something.  Through exposure to new ideas and perspectives it can still influence our actions.

Easy Cases

What if you can’t swim?  Jumping in the water means you, the drowning child and the five children in Africa die, which seems to be the worst possible outcome.  In my own case, if I believed my only skills to be those needed for trading, I would be doing a disservice both to myself and the world by leaving a career on Wall Street.

What if the office has no windows, and you saw the girl drowning on a security camera feed?  Perhaps next to that is a video feed showing African families, likely to contract malaria, who will receive the mosquito nets if you make a donation.  Singer argued that changing our visibility of or proximity from suffering does not change our moral obligations, and I agree.  These superficial changes should not affect our actions.

Why not yell at someone else to go save the child?  Obviously you would do this in reality.  In fact, you should be willing to pay someone to save the drowning child – but not more than $3,000, which would save lives more efficiently with a donation to the malaria charity.  Let’s assume no one else is able to help out.

Replaceability

If you leave your desk to go and help the drowning child, and the trade occurs while you are gone, where do the profits go?  They don’t disappear – someone else at another trading firm will execute the trade and reap the profits.  How bad is this?  We can do some math with this toy example to come up with the damage.  Suppose someone at another firm executes the trade, and their bonus also increases by $30,000.  They are a bit less generous and donate only 20% to charity.  They also donate to a less effective charity called the ‘Staff Parties and Anti-Malaria Charity’, which uses half the money donated to throw lavish parties for the staff and the other half to buy mosquito nets.  In total there are 6 lives at stake.  Had you stayed at your desk, 5 would have been saved.  Since you went to save the child, now 1 will be saved by you, and we can calculate that $30,000 * 20% * 0.5 / 3,000 = 1 life will be saved by the donation of the other trader.  Net, 3 more people will die because of your choice to go save the child.

It might not have to occur this way.  Perhaps you have a coworker who can’t swim but can watch the computer while you’re gone.  Assuming the coworker does execute the trade successfully, the profits will contribute to your firm and bonus.  As long as the coworker is 80% likely to execute the trade correctly in your place, the total expected lives saved would be 1 (drowning child) + 5 * (0.8) = 5, the same as if you had stayed at your desk.  The question becomes: how replaceable are you, and how confident are you about it?  The more uncertainty in how well a coworker can perform the task in your place, and the the more uncertainty in how likely you are to save the child, the less obvious the ‘right’ choice becomes.

In a blog post about replaceability, Ben Kuhn argues that for this particular career (quantitative trading) one person employed does not necessarily replace another.  In the recruiting process, the firm is limited not by roles to fill at the firm, but by the number of people found with the necessary skills.  Effectively, the firm is always understaffed, and additional hires will always lead to more profits.  The only factor limiting firm size is how difficult it is to find the right people.  If this is true someone leaving the firm will not change the pace of recruitment or growth (they are already recruiting as much as they can!), and the firm will simply be one person smaller than it would have been otherwise.  How should this influence a decision to leave?  In my own case, while not donating a large portion of my own income, I knew some coworkers did (e.g. Matt Wage mentioned above).  I might reason:

(1) If I leave the firm will be effectively smaller than if I had not

(2) The smaller firm will be less profitable

(3) Matt Wage and other effective altruists will receive less compensation

(4) Fewer lives will be saved as a result of their donations

(1) and (2) are not so clear: Maybe I am quite easily replaceable, or perhaps others at the firm could just work harder to close the gap.  If (2) is true I believe (3) will be also, but without knowing the inner working of the firm it’s not clear to what extent.  Even if (4) is true, it would need to be offset by gains from good I might choose to do elsewhere.

Wall Street is Bad

You protest, ‘Wall Street has ruined the economy.  The system is doing more harm than you can make up for by donating money.  You are part of a system that keeps people in poverty.  By sitting at your desk and working as a trader you support that system.’

The simplest argument against this (and my personal belief) is that while not perfect, Wall Street is a critical part of the economy.  We can improve it, but trying to tear it down and threatening the global financial system would do much greater harm.  The type of quantitative trading firms often recommended for the purpose of earning to give are providing important (though hard to define) functions like ‘providing liquidity’ and ‘taking on risk’ that improve the market.  At worst the marginal value they add is small due to competition between firms, but some are not only profitable but also have leadership that advocates for improvements to market structure.

My evidence for this is partially anecdotal and I haven’t provided any data to back it, so suppose you reject it completely.  You believe HFT (high frequency trading) and other trading practices keep people in poverty and lead to more suffering in the world.  What if you have this belief, and were a skilled mathematician offered a job working for a trading firm?  If you take the job, you might be able to undermine or slow the advancement of new trading strategies internally.  You might be able to better advocate for changes to the ‘system’ from that position.  If you do not take the job and the firm hires someone else, you believe they are more likely to zealously seek profits with no regard for negative externalities.  Therefore, if you are a utilitarian you conclude you are obligated to take the job.  This type of thought experiment comes from philosopher Bernard Williams, who objected to it and called it ‘an attack on his integrity’.

In his book ‘The Most Good You Can Do’, Singer discusses the argument from Williams and shows how it might be used to reach less palatable conclusions.

“It implies, for instance, that the guards at Auschwitz were not acting wrongly if their refusal to serve in that role would have led only to their replacement by someone else, perhaps someone who would have been even more brutal toward those who were about to be murdered there.”

We can fall into a philosophical rabbit hole debating different ethical frameworks, rule utilitarianism, collective responsibility, etc, but some might just accept this as a consequence and say yes, some of the guards at Auschwitz were not in the wrong.

Is there another alternative?  Surely we are not forced to accept either Wall Street is good, or it is bad but if we are able we have an obligation to work there.  Returning to the thought experiment:  why is this child drowning?  Was there no fence around the pond, or lifeguard if people swim there often?  Perhaps the best use of your time is not to be a trader, but to appeal to government to improve policy that will prevent future drowning deaths.  To their credit effective altruists discuss policy advocacy as a career possibility.  In his book Singer states,

*‘Advocacy for policy change has obvious appeal to those who are concerned that more traditional forms of aid are putting a Band-Aid on the symptoms of poverty rather than dealing with its deeper roots.’ *

Measuring the effects of policy changes numerically is probably harder, but maybe sometimes possible.  Imagine data exist that 15 children have drowned in this pond during the last year, and the cost of hiring a full time life guard for the year is $30,000.  If the lifeguard prevented all of these deaths, it would have been at a cost of $2,000 per child, a better return than donating to the anti-malaria charity.  If you had spent the day convincing lawmakers to enact this change, you would have saved 15 lives instead of 5.  Most causes, unfortunately, are not so easily quantified.

Self Interest and Legacy

‘Sure,’ you say, ‘I can see the numbers, but to coldly compute that I should sit and look at my computer screen to earn money I can donate feels wrong to me.  That sounds unpleasant and unsatisfying.  Shouldn’t you find your passion and follow that, and try to do what good you can?’

Perhaps while waiting for the trade to occur and allowing the child outside to drown in the pond you are having second thoughts.  Two futures come to mind.  Decades from now, at your funeral, a friend reads your eulogy.  In one version, he speaks of your accomplishments as a trader.

‘He was an excellent and efficient trader, able to sit at his desk in complete focus for hours at a time.  In his career his total salary plus bonus was approximately $23.7mm, of which he donated $12.7mm.  At a cost of $3,000 per life this saved an estimated 4,233 lives’

In another future, he speaks of a different path.

‘His life was changed one day when in a great act of bravery he jumped from his desk and dove into the Wall Street pond to save a drowning child, who sits here with us and her three children today.  This experience changed his whole view on life, and he left Wall Street to pursue his newly discovered passion for work as a lifeguard.  Thanks to his bravery and courage, he personally saved nearly 100 swimmers from drowning over the course of his career.’

Perhaps your friend is wrong, and the braver path is to look at the numbers, see that 4,000>100, and conclude that the thousands of additional lives saved outweighs your personal legacy preference.  Some might prefer the first regardless of the numbers: they could find trading satisfying work they are talented at, and also find the idea of earning to give and effective altruism inherently satisfying.

Examples of those earning to give in Singer’s book express this view.  Matt Wage stated that he finds the work itself interesting, and “I’m extremely happy with my life.  I’d still do this altruistic stuff even if I thought it was making my life worse…”  Another effective altruist donating a large fraction of his pay, Alex Foster said his job was “insanely fulfilling – mores satisfaction than any other period in my life.  Despite heavily reduced social life.”

Matt states he’d still do this ‘altruistic stuff’ if it made his life worse, but it is difficult to find stories of people in unsatisfying careers gritting their teeth through it because they value the lives they are saving above their own suffering.  In an article entitled ‘Is it ok to make art?’, author Rhys Southan interviews a number of effective altruists while trying to come to terms with the idea that choosing to be an artist might be very selfish.  In the article he quotes Robert Wiblin, the Executive Director of the Centre for Effective Altruism, as saying:

• “It is true that Effective Altruism would sometimes say that the thing you most enjoy isn’t the most moral thing to do,”* later describing how he used to be “extremely interested in evolutionary biology” but unfortunately, “couldn’t really justify it on the effects that it has on helping other people“.

So much for following your passion.  While Singer does not advocate this, he points out this type of situation is not uncommon in another context:

“…it is, unfortunately, the fate of many people to spend their lives supporting themselves and their families by doing work they do not find intrinsically interesting or enjoyable.”

If many people do unsatisfying work for the sake of supporting themselves or their family, why should I or anyone not do so if it allows us to save lives through additional earnings?

One answer is essentially publicity: for effective altruism to work as a movement, it must seem appealing from the outside.  Sure, taken to its limit the philosophy might demand people work on maximally profitable jobs they don’t enjoy and live in tiny apartments.  But, if you are trying to recruit others to join you this is probably not optimal.  Better to convince ten people to give 10% of their income than zero to give 90%.  On her blog ‘Giving Gladly’, Julia Wise writes:

• “Effective altruism is not about driving yourself to a breakdown. We don’t need people making sacrifices that leave them drained and miserable.”*

If working on Wall Street is driving you to a breakdown, then it doesn’t make sense to continue for many reasons.  Burn out or breakdown might affect your ability to do other work in the future.  Better to give half as much a year over a career spanning decades than to give a huge amount up front working on a job you can only sustain working at for a few years.   In a response to the article on Jason Trigg, David Brooks of the NYT writes,

“I would worry about turning yourself into a means rather than an end. If you go to Wall Street mostly to make money for charity, you may turn yourself into a machine for the redistribution of wealth.”

Brooks implies this might lead to burn out, and could be ‘corrosive’ over time.  He does not acknowledge that some might find turning themselves into the most efficient and productive such machine deeply satisfying.

Weird Considerations: Existential Risk

Lets consider a final objection. ‘Forget about trading, or saving the drowning child.  If we don’t do something about global warming soon, hundreds of millions will die and the future of the human race will be in jeopardy.  That is what we should be working on’.

I mention this argument because I find it fascinating.  Existential questions, like global warming, nuclear war, or an earth destroying asteroid, might outweigh most other considerations like malaria because the destruction of the earth would result in the deaths of billions of people, and all future generations.  Even ignoring future generations, how much should we be willing to spend on projects to prevent things like the destruction of earth by an asteroid?  If we can save the life of 1 child spending $3,000 to prevent malaria, we should spend that money on anti-asteroid technology if we believe it will reduce the chance of such an impact (in the near future) by 1/7 billion = 1.4*10^-10.  Would $3,000 accomplish this?  How well can we measure the likelihood of an asteroid impact?  How much will technologies to prevent it cost?  Estimating answers is difficult.  To their credit effective altruists have considered these questions and Singer devotes an entire chapter of his book to preventing human extinction.

With this view, perhaps the greatest effective altruists of all time have not been those that donated the most money, like Bill Gates, or an advocate of nonviolence like Gandhi.  Instead I would nominate JFK, who helped bring the world back from the brink of nuclear war during the Cuban Missile Crisis and who (while certainly not alone) helped prevent the destruction of all current and future human life.

What’s Next

While working on Wall Street I was not an effective altruist and did not donate a large portion of my income, but some around me did.  I respect their choice to do so.  Career paths are not certain but those individuals can estimate with some confidence the total dollars and thus lives they can save over the course of their lifetime.  Effective altruism has been criticized for this obsession with measurement.  Donate X dollars, turn that into Y mosquito nets, save Z lives.  This to the detriment of other causes that might not save lives but are still ‘worthy causes’.  When digging deeper, you’ll find there is nothing compelling about this argument.  The movement has been open to radical ideas of what causes might be ‘worthy’ – ideas like helping to locate and track earth threatening asteroids.  It is open to careers outside of earning to give like policy advocacy, when those choices are carefully considered.  In addition they are happy to admit that the current ‘best’ causes might not be optimal, and will change which charities to support in the face of new evidence.  Just because we can never really know what the ‘best’ cause is, we should still work to measure it and act accordingly, lest we make perfect the enemy of the good.

My own conclusion is one that can be applied to all philosophies or social movements: take what you like and leave the rest.  Will I look at Givewell and consider what they believe are the most effective charities before donating money?   Yes.  Will I stop donating to my alma matter because that money could be used more efficiently elsewhere?  No, though I’m less likely to donate a large sum in the future.  Will I feel riddled with guilt that by quitting a job on Wall Street I have forgone the opportunity to consistently earn a high salary and bonus that I could have donated to save hundreds of lives each year?  No.  The advantage of my own future being vague and uncertain is the impossibility of comparing it to the path I could have followed on Wall Street.  Who is to say I might not do even more good in the world?  I accept that I am selfish to some degree, like everyone.  If you find Singer’s arguments convincing, then no one is doing enough.  Why did Matt Wage donate only $100,000 last year and not $110,000?  Should we measure his, or my own selfishness by the number of dollars at stake?  Perhaps it is arrogant to believe that you could ever do better than donating the money earned from a consistently high paying job on Wall Street.  Even so, what good comes of feeling guilty about it?  When the philosophical debates are over the question that remains is what motivates people to action, and by this measure effective altruism is seeing success.

Update 5/4/15 7:27pm: For some reason I had accidentally referred to Peter Singer as John Singer, now fixed.

The invention of the airplane at the start of the 20th century, and the subsequent introduction of air raids and strategic bombing in warfare led to devastating results.

After listening to the excellent Hardcore History podcast by Dan Carlin discussing the context and story behind the dropping of atomic bombs at the end of WW2 (see here), I was struck by some of the numbers involved.  Before WW2, only about 20,000 people had ever died as a result of bombs being dropped from the air^[1].  Some of these early air raids seem ludicrous in hindsight.  In 1915, as WW1 began to enter the extended stalemate of trench warfare, the Germans actually used Zeppelins(!) to bomb different locations in the United Kingdom (and ultimately stopped as the Zeppelins proved too vulnerable).  The British and French also executed their own limited bombing raids.

(picture from here)

The idea of Zeppelins dropping bombs on cities now seems absurd, but had I looked up at the time to see a giant enemy airship filling the sky, it’s hard to imagine not feeling helpless and afraid.  Though no more than a few thousand people died worldwide from bombing raids in WW1, it traumatized citizens, and was the first step towards massive aerial bombing in WW2.

Between the world wars, many air theorists believed that aerial bombings would be a decisive factor in the next large scale conflict.  Giulio Douhet, an Italian air theorist, was a major proponent of bombing, believing it might be used to win a war quickly.  Here is a simplified outline of how Douhet believed a war would proceed:

• Build a massive, overpowering air force
• When war breaks out, immediately launch devastating attacks against populated enemy cities
• The morale of the citizens will deteriorate
• The citizens will rise up against the government and demand peace
• You win the war

One great benefit – the war will be short!  One devastating attack against Berlin or London, and the people are sure to rise up and demand peace!  After prolonged and deadly trench warfare in the first world war, anything that kept the war shorter was worth considering.  Not only that, if you don’t follow this plan the enemy will!  In 1932 Stanley Baldwin, a British politician, said in regards to aerial warfare:

“…the bomber will always get through, the only defence is in offence, which means you have to kill more women and children more quickly than the enemy if you want to save yourselves…”

World War 2

After WW2 broke out and bombing raids began, the ideas of Douhet and others were put to the test, and they were very wrong.  The German Luftwaffe began the London Blitz in September of 1940.  While about 40,000 civilians were killed, morale of Londoners did not collapse, and there was no general uprising demanding peace.

(picture from here)

If not the British, perhaps German morale could be crushed under sufficient bombing!  Britain’s “Bomber” Harris certainly believed so, stating:

“There are a lot of people who say that bombing can never win a war.  Well my answer to that is that it’s never been tried yet and we shall see.”

Harris led British bombing attacks on German cities starting in 1942, which at times included thousands of planes in individual raids.  He viewed these attacks as retaliation for earlier German attacks, and believed that enough bombing would force the German surrender.  He has since faced criticism (along with others, and the entire system of aerial bombing), particularly for the destruction of cities later in the war.  In February 1945, he said:

“I do not personally regard the whole of the remaining cities of Germany as worth the bones of one British Grenadier.”

Despite these campaigns and later with the help of the United States, German morale did not collapse from the bombing, nor did that of the Japanese.  The chart below is my attempt to visualize the human cost of the major air raids on cities during of the war.  If you have not seen a bubble chart like this before, each bubble is positioned along the x-axis based on the date it happened, and the size of the bubble indicates the (estimated) number of civilians killed.  They are spread out vertically simply to keep them from overlapping.

A few caveats:

• The Warsaw bombing was not exclusively an air raid, with damage also coming from artillery and street fighting, but aerial bombing was a major component.
• The bombing of London was more spread out (over about 8 months) than most of the others shown.
• There were other major bombing raids that took place, for instance allied bombing of Berlin, which was spread out over the entire war and I’ve chosen not to show.
• I’ve also chosen not to display the numbers of casualties and only focus on the relative size of these bombings.

However, I hope this graphic can help to tell the story:

• While the Axis (in particular the German Luftwaffe) dominated the first half of the war, the Allies (RAF and USAAF) dominated the second half, when the Luftwaffe was kept busy with the Defence of the Reich.
• The Tokyo fire bombing, on March 9, 1945 (Operation Meetinghouse), is believed by some to be the single most deadly air raid of the war^[2].  Estimates vary widely, and measuring casualties in the nuclear attacks is made more difficult when trying to account for later effects of radiation, but at the very least it was on par with Hiroshima.
• Allied bombings starting at the beginning of 1945 were concentrated and devastating until the end of the war.

I’ve left out much of the story, for instance the incredible inaccuracy of these bombs (particularly early in the war), which were often many miles off of target, and how this contributed to the targeting of large urban areas.  None of this has mentioned the massive areas destroyed in these bombing raids.  Sixteen square miles of Tokyo burned to the ground in the 1945 firebombing – an area nearly the size Manhattan.

Some of this might help us at least understand the dropping of atomic bombs, if not justify it.  Henry Stimson of the US said (paraphrasing), “The only reason I approved the atomic bomb was because I believed it would stop the firebombing.”  At the same time plans were being made to drop these bombs, the US military was making plans for a full scale invasion of Japan (Operation Downfall).  General Lauris Norstad estimated it would cost “half a million” US lives.  A study by American physicist William Shockley estimated “400,000 – 800,000 killed“, and “we shall probably have to kill 5 to 10 million Japanese“.  This certainly changes the bubble chart.

Is it OK to kill a few hundred thousand civilians, to possibly save millions?  At the post war Nuremberg trials, Germans questioned whether the mass bombing of civilians by the Allies constituted a war crime.  Carlin argues that they were – but that through the course of the war they had become accepted.  At Nuremberg the chief counsel said in response:

“The ruin of German and Japanese cities were the result not of reprisals, but of deliberate policy, and bore witness that aerial bombardment of cities and factories has become a recognized part of modern warfare as carried on by all nations.”

These questions of morality and the weighing of some lives versus others are the kinds of ‘Logical Insanity’ questions Dan Carlin discusses in his podcast.  If you find these questions interesting or want to hear more firsthand accounts about the experience of living through these aerial bombings, I would highly recommend it (here).

Thoughts on Data Visualization

Taking rows of data and turning into a chart does not suddenly make it interesting.  Someone tasked with creating a graphic on civilian casualties of various air raids might dutifully transform a dull list of numbers into an equally dull chart:

This chart is terrible.  The labels are far from the data they represent.  We’ve totally discarded the timeline.  Comparing the relative quantities is difficult.  Whatever we choose to do with the data, we’ll always suffer from Stalin’s maxim “one death is a tragedy; a million deaths is a statistic”, but this does not mean we shouldn’t organize and make an effort to display those statistics in the best possible way.  A million deaths do not happen in a vacuum, and to display them carelessly is just disrespectful.

When looking at the data behind air raids in WW2, a few things struck me:

• Most air raids early in the war were done by the Axis, and essentially all later in the war by the Allies
• The concentration of air raids in the last year of the war
• The scale of the Tokyo bombing (and other firebombing) relative to the more well known atomic bomb attacks

By identifying the interesting parts first and then trying to create something to reflect it, I think I got a better result.

• Colors for the Axis/Allies
• Use of bubble chart for easier comparison of relative sizes
• Use of the x-axis to show position of the events in time

At the same time, I worried about trying to do too much at once.  Originally I planned to show all the major conflicts of WW2 on one graphic, with each circle partially shaded to show the fraction of casualties on either side.  Quickly I realized this would obscure the hope of making any individual point clear.  Leaving off specifics like the exact dates and number of casualties was also an attempt to reduce clutter.

Better data visualization can help us understand the how and the what of the war, and even hint at the why (as I tried to show with the decision to drop the atomic bombs).  But to understand the data is not to understand the war.  Nothing in the numbers relates the experience of someone on the ground that lived through a fire raid, which is why other sources (e.g. first hand accounts, or artistic interpretation in books and film) are critically important.

References

Most of this is taken from the Hardcore History podcast by Dan Carlin (linked again here).

The wikipedia page on Civilian Casualties of Strategic bombing has a good overview of the data I used for the graphics.  I attempted to increase my confidence in these numbers by digging a little further into the sources (and comparing against e.g. the podcast), but in most cases casualty estimates are vary widely, and I intentionally tried to keep specific numeric claims to a minimum.

I decided recently to indulge my interest in the US pop music charts.  The most interesting is the Billboard Hot 100, which ranks singles in the US and is published weekly.  Created in 1958, and with other similar data going back to 1940, it provides an interesting data set of manageable size (75 years * 52 weeks/year * 100 entries/week = around 400,000 total entries).

Trajectory of Singles on the Hot 100

Let’s start by plotting the paths a few hits took on the chart.  In each graph, the x-axis is a date, and the y-axis is the rank on the Hot 100.

Lose Yourself – Eminem

Released in 2003, this track follows what I found to be a typical pattern for pop hits – a rapid rise, a period on top, followed by a rapid decline.  Lose Yourself had a 12 week reign at #1 which ties it for 4th longest all time, but after leaving the #1 spot it dropped off the charts quickly.

Party Rock Anthem – LMFAO

From 2011, this track spent only 6 weeks at #1, but an incredible 29 weeks in the top 10 (longer than the entire chart run for Lose Yourself), and 68 weeks on the Hot 100.  Compared to ‘Lose Yourself’ this single had remarkable staying power.

I’ll Be Missing You – Puff Daddy & Faith Evans featuring 112

Some singles don’t need time to rise to the top of the charts.  Released in 1997, this is one of only 22 singles ever to debut at #1.

Beatles dominance in 1964

In 1964 The Beatles totally dominated the Hot 100. On one graph, here are all Beatles singles to chart from 2/1/1964 to 6/1/1964:

The Beatles occupied the top 5 positions on the charts on April 4, 1964 (no other artist has ever held more than the top 2 spots simultaneously).  Not only that, they had an additional 7 singles in the top 100 that week, and 2 other tracks in the top 100 referred to the Beatles in their title!

Scoring a Single’s Success Based on its Chart Run

A natural way to score a single’s success on the charts would be to take an integral to find the area under the curves we’ve drawn.  I decided only to count areas where the single was in the top 10.  In the below graph, the area of the region above the horizontal line would be found to give the score (more precisely, I took the sum of max(0, 11-chart rank) for each single over all weeks):

Top 15 all time Hot 100 singles: Area Method

1. Paper Doll – The Mills Brother (1941)
2. Smooth – Santana (1999)
3. How Do I Live – LeAnn Rimes (1997)
4. Vaya Con Dios – Les Paul & Mary Ford (1941)
5. Chattanooga Choo Choo – Glenn Miller & His Orchestra / Tex Beneke & The Modernaires (1941)
6. Un-Break My Heart – Toni Braxton (1996)
7. We Belong Together – Mariah Carey – (2005)
8. Blue Tango – Leroy Anderson (1950)
9. The Twist – Chubby Checker (1961)
10. Because Of You – Tony Bennett (1951)
11. Yeah – Usher featuring Ludacris & Lil Jon (2004)
12. Somebody That I Used To Know – Gotye featuring Kimbra (2012)
13. All About That Bass – Meghan Trainor (2014)
14. Macarena (Bayside Boys Mix) – Los Del Rio (1996)
15. Mona Lisa – Nat King Cole (1955)

Top 15 all time Artists: Area Method

1. Elvis Presley
2. Perry Como
3. Mariah Carey
4. The Beatles
5. Madonna
6. Whitney Houston
7. Janet Jackson
8. Usher
9. Michael Jackson
10. Boyz II Men
11. Patti Page
12. Elton John
13. Katy Perry
14. Dinah Shore
15. Stevie Wonder

What’s interesting here?

• Perry Como?  I was surprised to see him near the top.  Looking at the data, he had a steady string of hits from 1945 – 1958 so perhaps his place on the list is deserved.
• Related to this – why are there three singles high on the list from the 1940s? Technically the Hot 100 did not exist in its current form until late in 1958, however, in the 1940s other charts existed (Best Sellers in Stores, Most Played by Jockeys, and Most Played in Jukeboxes).  The data source I’m using goes back to 1940, but it’s not clear to me exactly what source it uses prior to 1958.
• Nothing here would suggest it, but the third highest ranked song, ‘How Do I Live’, while on the charts for 69 weeks, actually never hit the number 1 spot, peaking at #2.  So, this method of scoring singles at least tells us something that just looking at the number of weeks a single was ranked at #1 wouldn’t.

Another Scoring Idea: the exponential method

How else can we score a single’s success?  Two obvious choices:

• total number of weeks in the Hot 100
• total number of weeks at the #1 spot

These are already easy to look up so I’d prefer something more interesting.  What about an exponential function?  For some value of p, where p < 0, I consider a single’s score for a given week with rank r to be p^r.  We score every single each week, and sum up a single’s score across all weeks it was on the chart.  As p approaches 0, this approaches the “number of weeks at the #1 spot”.  As p approaches 1, this approaches the “total number of weeks in Hot 100”.  I’ve chosen three different values of p.

Top 15 all time Hot 100 singles: Exponential Method

Top 15 all time Artists: Exponential Method

• Notice how which list a song appears on can tell you about what type of success it had.  The song ‘Radioactive’ was in the hot 100 for an incredible *87 weeks *(the record) but never hit the #1 spot!  As expected it tops the list for p=0.99, and doesn’t appear in the other lists.
• Contrast this to a song like ‘We Belong Together’, which held the #1 spot for 14 weeks, but subsequently fell off the charts quickly, not appearing at all in the list for p=0.99.
• Looking at artists, compare Elton John to The Beatles.  The Beatles do better with a lower exponent, while Elton John does better with a higher one, suggesting The Beatles had more high charting hits, but Elton John beat them for quantity of singles with more modest chart success.
• Beatles vs Stones?  The Stones get close with p=0.99, but trying a few other p values I don’t see the Stones ever beating the Beatles.  In terms of chart success for singles, The Beatles are a clear winner.

Plotting an Artist’s Career

Using one of these scoring functions, we can accumulate the total score of an artist over time.  On each of the plots below, the x-axis is time, and the y-axis shows the artist’s total cumulative chart scores at that time.  Periods when an artist has more hits should have a steeper slope, and periods when they are absent should appear flat.  The accumulated scores use the exponential method with p=0.9.  The actual values on the y-axis are unitless but  they can be compared across different artists assuming the same scoring function.

Elvis Presley

An incredible run starting in the mid 1950s, you can see the decline of his chart success starting in the 1960s.  There was a small ‘comeback’ in the late 60s corresponding with the so called ’68 Comeback Special’ that aired on NBC and produced a few hits, but it was nothing compared to his earlier chart success.

Britney Spears

Britney hasn’t (quite) made it onto any of the top artists list, but her career follows an interesting trajectory.  After a meteoric rise in the late 90s with ‘Baby One More Time’ and ‘Crazy’, much of the 2000s was relatively quiet for Britney.  The small bump in early 2004 corresponds with her hit ‘Toxic’.  In the late 2000s and early 2010s, a series of her releases experienced huge success on the charts, arguably much greater than her original success in the 90s, with three of her four lifetime #1 singles coming out in 2008 or later (Womanizer in 2008, 3 in 2009, and Hold it Against Me in 2011).

Pop Stars of the 2000s: Rihanna vs Katy Perry vs Taylor Swift vs Lady Gaga

In the mid 2000s several new pop artists emerged simultaneously.  I’ve overlaid these four artists’ careers. You can immediately see the recent dormancy of Lady Gaga, and the rise of Taylor Swift in the last three years.  For T Swift in particular the two recent steep sections correspond with the release of her last two albums.  Katy Perry and Rihanna have been consistent hit makers, though Rihanna has not released an album since 2012.

Concentration of Hits

Is there more or less variety in the Hot 100 today than in the past?  I considered two ways of measuring this.  First, do current singles rise and then disappear more or less quickly than those decades ago?  Secondly, even if the rate an individual single rises and falls has remained the same, perhaps there is more concentration in a smaller number of artists.  Among the top spots on the Hot 100, do we now see more or less artists represented?

To answer the first question, I used the area method from above to score every single.  I then looked only at singles with a positive score (appeared at least once in the top 10).  For each year, I took the average score of all singles from that year.  A high average should suggest singles from that year tended to stay in the top 10 longer, and vice versa.  Nothing conclusive, but it does appear a bit lower from 1960-1990 than the last 25 years.

To answer the second question, for each full year in the data I computed two values.  The first was the total number of distinct artists to appear in the top 10, and the second was the total number of artists to have singles that hit #1.  Both are plotted below.  Again nothing earth shattering, but it seems consistent with Billboard diversity peaking during the 1970s and remaining lower since the early 90s.

Methods & Caveats

All of the data was retrieved by scraping the Ultimate Music Database website.  Python code is below.  After gathering the data, I did all work in Excel.  The size of the dataset is ~300k rows, which is still small enough for Excel to handle (with some help from VBA).  A few issues did become obvious while working with it:

• Beatles vs. The Beatles: both versions appeared in the data, and similarly for many other artists.  For any artist where I found names of X and The X, I replaced all instances with The* X*.
• Spelling errors/other different spellings: I happened to notice that the Britney Spears song ‘Oops … I did it again’ also appeared as ‘Oops! … I did it again’.  Unfortunately these strings don’t precisely match up, and the simple fix of making everything lower/upper case doesn’t fix it.  I noticed a few spelling errors as well and there are surely many I didn’t notice.
• Multiple artists appearing on a track: For any track with multiple artists, the score did not contribute to the career scores for *any *of the artists involved.  Partially this was due to uncertainty about how to handle it (divide the score between them? count the score fully for all artists involved? how much less should it count to be a featured artist as opposed to a lead artist?).  The other issue comes from the data itself.  Consider these artist strings that appear in the data: - Bing Crosby & John Scott Trotter & His Orchestra
• Nicki Minaj featuring Drake, Lil Wayne & Chris Brown
• Peter, Paul and Mary
• Daryl Hall & John Oates
• It gets tricky fast.  The ampersand is sometimes used to divide distinct artists, is sometimes part of the name of the artist, and in some cases a single track uses it with both meanings!  The comma is no better – it can also be a divider or part of an artist name!  Maybe this could be addressed but for now I skipped it.

I’d love to create an interactive tool for generating these types of graphs dynamically, but I currently lack the Javascript ability to do so in any reasonable time.  Instead I’ve uploaded all the raw data here, in PSV format (as mentioned song names/artists sometimes have commas, but none so far have featured a pipe character) so feel free to have at it.

Scraping Code

Fortunately everything on the Ultimate Music Database is nicely formatted in HTML tables, so pulling it out using the python Beautiful Soup library is not too difficult.  A small note – using the default python html parser did not work, but BeautifulSoup docs suggested using html5lib, which fixed the problem.  Code is available on Github.

Sources:

Wikipedia

Ultimate Music Database

Beautiful Soup