This week I’m using the teaser text to ask DJ Crayola to chainflip-ify Magikarp for this edition’s image. It’ll make sense when you read on, promise.
Progress Since Last Update
Development progress has been excellent in the last couple of weeks, here are a few of the highlights:
- Successfully tested contract upgrades on multiple devnets
- Finished up dynamic set size resolution, for a smarter approach to collateral maximisation
- Removed some wonky assumptions about RPC implementations
- Resolved several security issues relating to witnessing
- Made our P2P code more generic in preparation for multi-crypto support
Lots of awesome work being picked up beyond this too, I’m very excited for the next few months. Every line of code that we’re merging is tighter than the last. The new Ethereum contracts will help me sleep a lot better at night, too.
Now to the less fun news. Since we hit that critical bug a few weeks back, it would be optimistic to say that the network has been crawling. In reality, a closer comparison might be to a Magikarp out of water. I’ll try to provide a sequence of events here, for those interested:
Saturday April 02
Reports of log messages about Equivocation reports hit the Validator Discord channel. Upon checking the network, we notice that roughly 2/3 of all nodes (not just the Validator nodes) have crashed at block 1502695, and are unable to recover. We discover that it’s possible to get back online by purging the chain and re-syncing from an uncrashed Validator as bootnode, but don’t advise this due to the long sync times and desire to freeze the network state for forensic purposes.
Monday April 04
I post a question on the Substrate StackExchange in an attempt to get some answers. There are a couple of responses which don’t help too much, and then a couple of days later Gav asks me to raise the issue against the Substrate repo.
Wednesday April 06
I raise the issue with a few more details on the Substrate repo. Andre (developer of the Substrate GRANDPA client) initially dismisses the issue, but then realises that there is the possibility for an unhandled exception within the client.
Friday April 08
We get confirmation that Andre is working on a fix.
Wednesday April 27
We get wind that the fix has been applied to Substrate thanks to the watchful eyes of our Validator community.
Thursday April 28
We backport the fix and release an update, hoping to see the resumption of finality once we get 2/3 of the Validators on the new version. Since finality halted, the network had been producing large blocks, and it takes most Validators an excruciatingly long time to sync.
And that pretty much brings us to now. The head of our chain is about 120k blocks past the point at which finality stalled, but finality itself has only caught up about 57k blocks (which it did all at once, mysteriously).
I’d be lying if I told you that I knew how to get finality back on Soundcheck. I think we could probably do it, but it’s time and effort that I would rather put into the remaining work that we know we need to do for a production launch. It is pretty frustrating to watch the network thrash around, since our original intention was to keep Soundcheck alive and use it as a proving ground with respect to network upgrades (we might as well get practice at them whilst we can, before there’s real value on the line).
Alas, this is where we find ourselves.
I’m aware that not everyone has had the chance to knock off all the tasks for the Leaderboard yet, so this is the rough plan to make that right over the next few weeks if we don’t have finality by Monday May 09 (subject to change):
- Spin up a new network based on the Soundcheck codebase, with a new $tFLIP token
- Airdrop everyone who did not hit the maximum 70 points with 1000 $tFLIP tokens
- Let the network run for two weeks with a fresh leaderboard
- End the program
- Upgrade network to PARADISE-spec that we’ve had behind the scenes for weeks now
- Kick off the tunes and let everyone back in for the smooth journey to SANDSTORM
We’ll do the necessary cross-referencing to ensure that nobody can double-earn or lose out in the new network.
Goals for the Next Fortnight
Some quick bullet points detailing the major milestones that we hope to achieve by the time the next dev update rolls around.
- Broadcast fee witnessing (ensure Validators don’t lose money on eth gas)
- Remove defunct witnesser-api pallet
- Kick off the AMM build
- Kick off the swapping app build
Big shout out to everyone in the #active-soundcheck channel for continuing to bring the goods. You’re all welcome here in the HQ at any time. We can teach you how to skull Club Mate.
Until next time.