Cypherpunk Cogitations

When do Bitcoin Node Operators Upgrade?

Jameson Lopp -

Recently, when compiling the metrics for my Annual Bitcoin Report, I came across this 8 year chart of historical user agent counts for reachable nodes. It's interesting to see the patterns of how node operators update their nodes, since it's a rough measure of how responsive these operators are and how much they're paying attention to new node releases.

However, as you can see, the chart isn't particularly granular for releases more than 2 years ago. So I reached out to Addy at Bitnodes to see if he could provide me with the raw data for a deeper analysis. Thankfully, he did!

Why Should We Care?

While we often talk about how Bitcoin is a very stable protocol and you can run really old versions of Bitcoin clients and still reach consensus with the rest of the network, the flip side of that is the fact that software isn't perfect and developers are constantly improving it. If you read the Bitcoin Core documentation regarding their software lifecycle, it goes like this:

  • 2 major releases per year
  • minor releases that backport important bug fixes from recent major releases
  • bug fixes will only be backported to the past 18 months of releases

As such, it's not considered safe to run Bitcoin Core releases that are more than 18 months old, as they may have unpatched vulnerabilities. If a significant number of nodes are running very old, unpatched software, it could pose a systemic risk to the network. Why not patch releases that are 18+ months old? Because it requires more maintenance resources to do so and the project maintainers are already quite busy. If someone wants to volunteer to backport patches and maintain older releases, feel free to step up!

However, it's a fundamental aspect of Bitcoin's security model that node software does NOT automatically update. Auto-updates would create a single point of failure that could cause a malicious update to quickly spread to a supermajority of nodes on the network. As such, manual action must be taken by node operators to keep their node software up to date. This creates a point of friction and means that node operators must maintain some awareness of project development.

Methodology

I wrote this script to sift though the decade of weekly snapshot data.

There's a ton of noise in the raw data. One reason for this is that it's possible for anyone who knows what they're doing to set their own User Agent string. For example, my Statoshi nodes present their user agent as "Statoshi" rather than "Satoshi" like the Bitcoin Core project does. There were nearly 200 unique user agents in the data set, such as...

bcoin
Bitclassic
Bitcoin
Bitcoin ABC
bitcoin-php
bitcoin-ruby
bitprim
Blockcore
blockd
BTC-2MB
BTCC
btcd
btcwire
BTCXChange.ro
Classic
Cornell-Falcon-Network
Gocoin
JavaBitcoin
libbitcoin
Multicoin
PseudoNode
Satoshi
Satoshi-UASF-Segwit
Statoshi
TestClient.0.0.1
thebitcoin.foundation

It's clear there are many different node implementations - though many of them are not even fully validating nodes, but are rather just minimal lightweight clients. After some consideration, I decided to only focus on looking at Bitcoin Core nodes. Why?

  • Over 90% of Bitcoin nodes run Core
  • None of the other node implementations have consistent release schedules
  • There were a variety of "fork" clients during 2015 - 2018 that further mess with the regular cadence of node lifecycles

Results

After reviewing the visualized data, I chose to break it up into 3 distinct charts.

  1. Major version releases, years 2015 - 2020
  2. Minor version releases, years 2015 - 2020
  3. Major version releases, 2020 - present

Why no charts for older releases? Bitnodes' data history begins on June 7 2014; Bitcoin Core 0.9 was released March 2014 and 0.10 was released Feb 2015. As such, we can't see the entire node deployment lifecycle of releases prior to Bitcoin Core 0.10.

Let's begin with the behavior exhibited by node operators running major releases from 2015 to 2020.

What strikes me from this chart is that it looks like:

  • In 2015, 2016, and 2017 most of these node operators are updating their software twice a year, with each new major release.
  • In 2018 more node operators are slowing down and only updating their node annually

What if we look at nodes running minor release versions?

When looking at the lifecycle of nodes that are updated to minor releases, unsurprisingly it looks like their node operators are updating even more frequently; often more than every 6 months. This makes sense, as minor releases tend to be published between major releases.

What happens when we look at releases from 2021 to present? Node update frequency appears to be decelerating. This is easy to tell if you focus on the peak node counts of each version. Note that the peak counts in both of the previous charts were always less than 6 months after the actual release date, but now the peaks are more in the 6 to 9 month range.

Also, note that the peak node counts are 150% - 200% the peaks in previous charts. Can these phenomona be explained?

I suspect that the increase in lag time for node operators to update is the same reason of why we see node counts higher in recent years: the rise in user friendly plug and play node hardware and software. And at least some of that adoption is a result of people running Lightning Network nodes, which generally requires also running a Bitcoin node. That is: there are a higher number of less technical node operators now and they aren't quite as enthusiastic about updating the software on their node machines.

How Long are Bitcoin Core Releases in Widespread Use?

As noted earlier, the Bitcoin Core release lifecycle only really recommends running a given release for 18 months before it should be considered past end-of-life. But it's not possible to force anyone to update their node software - there's a very long tail of outdated client software still in operation. How long is it taking for 95% of the nodes running a given release version to update to a different version? I ran the numbers:

Note that this chart ends with the 0.20 release from 2020 because none of the releases after that have seen 95% of their (peak count) nodes upgraded. We can see that before July 2018 it generally took 1 year for 95% of node operators to update their software. But after 2018 it doubled to 2 years. And it's entirely possible that we're now in an era of it taking 3 years for 95% of node operators to update!

Recommendations

While we shouldn't change nodes to auto-update their software since it would greatly weaken Bitcoin's security model, I do suspect we can improve the awareness of node operators.

  1. It would be great if Bitcoin Core and full stack projects that make it easy for folks to run Bitcoin Core nodes prompt the user to sign up for the Bitcoin Core Announcement List when they install the software.
  2. Plug and play node software should explore alternative notification methods to warn users that their software is out of date. I suspect that few hobbyist node operators are logging into their node dashboards and thus they could easily miss UI notifications. For example, it would be neat if you could set a nostr npub as the node's "admin contact" on setup and the node could effectively send you software update notifications that would get pushed to your phone.
  3. I'd like to see more warning / initial setup documentation from node software that informs users about best practices and ongoing maintenance responsibilities.

Bitcoin is a constantly evolving adversarial environment: node operators should be aware that they should not just "set it and forget it" if they want to retain a strong security posture. Running a node to validate your funds against the rules of the network is the path to financial sovereignty, but it also means that you're taking on the same responsibilities as a server administrator!

Feel free to reach out if you have other suggestions!