The Future of Statoshi
I am a software engineer who has spent the past 7 years scaling my employer’s web application from a several server infrastructure capable of sending a hundred thousand emails a day to a several hundred server infrastructure that sends a hundred million emails per day and processes billions of individual metrics that are created as a result. We never would have been able to accomplish this feat if we had not adopted a DevOps mindset, and it is this perspective that has led me to create Statoshi.
Statoshi’s objective is to protect Bitcoin by bringing transparency to the activity occurring on the node network. By making this data available, Bitcoin developers can learn more about node performance, gain operational insight about the network, and the community can be informed about attacks and aberrant behavior in a timely fashion. Any DevOps engineer will tell you that the first step is to measure everything, after which you can build reporting and automation around the raw metrics in order to maintain and improve the system. Statoshi is this first step.
You can read about Statoshi here:
https://blog.lopp.net/announcing-statoshi--realtime-bitcoin-node-statistics/
A public dashboard is available at http://statoshi.info
My public Statoshi node has been running for several months now and has gathered sufficient data to determine baselines for normal behavior on the network. However, this is still much work to be done. Some of my ideas for next steps:
1) Build a pool of Statoshi nodes that act together in order to monitor a larger portion of the node network and thus provide a more complete picture. I’m currently working on a node connection management daemon that would facilitate running a pool of nodes that maximizes their connectivity to the network mesh by reducing redundant peer connections across nodes in the pool.
2) Sending alerts to Bitcoin developers about aberrant behavior on the network, such as changes of more than one standard deviation.
3) Automated quality assurance tests / sanity checks. For example, adding alerts that would fire if the sum of bitcoins in existence ever changes by an amount other than the block reward in a single block.
4) Graphing the data collected by Statoshi nodes is but one of many possibilities. It could be useful for us to provide batch files of the raw data collected so that other engineers can analyze it, or we could build an API devoted to making it easy to query the data.
5) Automated defensive measures / spooling up honest nodes to combat certain types of attacks.
The Statoshi project doesn’t have an end date for development as it is meant to be the catalyst to an ongoing iterative process. I will consider it a success if developers say that it helps them gain a better understanding of the network. For example, I was recently able to use Statoshi to contribute to a conversation about “network abuse” by node crawlers:
http://sourceforge.net/p/bitcoin/mailman/message/32669128/
As an engineer, I believe that that realtime monitoring of the network activity is an area that has been mostly overlooked. Given the collaborative nature of DevOps, I would even consider the goals I listed above as being flexible — I would wish to confer with the developers before embarking upon a large project in order to ascertain their needs and pain points. Are you a Bitcoin developer or a company whose business relies upon the continued reliable operation of the Bitcoin network? I’m eager to hear any ideas you may have and welcome your support!