Join Delphi Research today and immediately get access to our full Member Portal!
Join Delphi Research today and immediately get access to our full Member Portal!

THORChain: Hardening the Protocol with Halborn, Immunefi, Nine Realms and Chad Barraford

Sep 6, 2021 · 56 min media

By Tom Shaughnessy

The Delphi Podcast Host and GP of Delphi Ventures Tom Shaughnessy sits down with Chad Barraford (Technical Lead at THORChain), Gavin McDermott (Founder of Nine Realms), Robert Behnke, (CEO of Halborn), Steven Walbroehl (CISO of Halborn), and Duncan Townsend (CTO of Immunefi) to discuss THORChain, details surrounding the recent hack, strengthening THORChain’s security, the role of bug bounties and much more.


Interview Transcript:

Tom (00:02):

Hey, everyone. Welcome back to the podcast. I’m your host, Tom Shaughnessy, one of the hosts of the podcast and I help lead Delphi Ventures. Today, I have a really galaxy brain panel here. It’s actually in a conference room for those watching the video. I have Chad on from the THORChain team, Duncan from Immunefi, Gavin from Nine Realms, and Robert and Stephen from Halborn, here to talk about THORChain, previous hack, how the protocols become a lot hardened and safer, and more secure in the future. With that, we’ll go in order here. Chad, why don’t you give your quick background? We’ll run through everyone.

Chad (00:37):

Yes. Sure. So my name is Chad Barraford, I’m the technical lead on the THORChain project. I’ve been with it since, I think it was July 2018, something like that.

Tom (00:49):

That’s awesome. Duncan, you want to go next?

Duncan (00:52):

Yeah. So I’m CTO of Immunefi. Immunefi is the immune system of crypto. We’re really DeFi’s last line of defense against attacks. We’ve got the world’s biggest bug bounties.

Tom (01:06):

I love that.

Gavin (01:08):

I’m Gavin McDermott, founder and CEO of Nine Realms. And Nine Realms is effectively bringing institutional infrastructure and some of the extension of the protocol to the THORChain ecosystem.

Tom (01:20):

Love that.

Robert (01:22):

Hey, there, I’m Rob. So I’m the CEO and co-founder over at Halborn. Halborn is hyper-focused on securing the convergence of CeFi and DeFi. So we typically sit right in the middle, work on the more centralized aspects of penetration testing, technical diligence, and so on. And then DeFi world, working on all sorts of things, including our most recent engagement, working with THORChain.

Steven (01:49):

Yeah. I’m Steven Walbroehl, partners with Rob, CSO for Halborn. So I do all of the technical work and work with the engineers at Halborn, to test and secure THORChain, among other launching protocols. So I do all the tech stuff and some of the offensive security work.

Tom (02:10):

That’s awesome, guys. Thanks for introducing everyone. We’ve had a lot of great deep dive podcasts on what THORChain is. Disclosure, Delphi is an investor. I have to throw that out there, happily. So why don’t we start, guys, with a recap of what exactly happened to THORChain over the past couple of months, hack wise? Let’s dive into just an overview on the hack and then we can go from there, free for whoever wants to grab it?

Steven (02:34):

Yeah, yeah. I can start. I’ll go into the details at a high level with some technical aspects of the hack, and you guys are familiar with THORChain or if not, the bridge, essentially they call it the Bifrost, that allows transactions between different protocols, was the target of the hack. And what happened the first time was, somebody actually figured out they could wrap one of the bridges routers’ contracts, and they sent a message essentially saying, “Hey, I’m going to transfer Ether to this.” It’s in a wrap contract, which then forwarded to the bridge and essentially tricking the bridge into saying that inflating the value of the underlying tokens, and the message value is actually zero. So what happens was, this allowed the attacker to manipulate what the price was and enable arbitrage between a lot of the token prices here, and tricked out because of the way that it was parsing, the message of the value that was sent.

Steven (03:43):

So essentially, by wrapping that router, they were able to send a value, which the router would then allow it to change the price of the token with zero and then extract it, to allow your arbitrage and the price slippage there. So there is also another attack that happened right after that with again, the memo of fields. So if you guys ever see the transactions that are occurring, the memo field is actually something that is parsed and split up. And by manipulating the memo fields, they were able to almost inject arbitrary messages to cause different payouts. So it’s pretty interesting attack vectors that were done. Maybe Chad can go into some of the things that happened about halting the validators and not letting some of the flash spots come to do the extraction from it. So I know there’s a lot of questions around how that approach was handled.

Tom (04:41):

I really, really appreciate the overview, Stephen. Chad, definitely dive in?

Chad (04:47):

Yeah. Another way to think about the attacks is, just to add to what Steven was just saying, somebody was able to figure out a way to trick the Bifrost to think that it received an asset, like Ethereum for example, but it didn’t actually receive. The person was able to swap in with 10 Ethereum, for example, to get something out on the other side, but they never sent in the 10 Ethereum to begin with, right? So they tricked the network to think that it received funds, that it didn’t actually receive effectively. That’s a very simple way of explaining it. And so once that was happening, they were able to swap large quantities of Ethereum, fake Ethereum really, into the network to receive real tokens out on the other side of it.

Chad (05:30):

I think all of a sudden, around $14 million out of those two attacks on the network, was extrapolated. And these incidents, anybody that was affected by the incidence, any liquidity providers, are all going to be made whole again. So in the end, everybody was providing liquidity to the network, will receive every coin back to them effectively. That’s what’s the process of happening now. Also in the process happening now, is a whole myriad of security focused changes to the protocol, to the network, to procedures and policies, to expanding the team, to having more internal people to be monitoring security, and external people too. It’s a pretty wide and long range of action items that the team and the creative community took, to better secure the system from any kind of attack, not just these specific attacks, but any kind of attacks that matter, that we’re happy to get more details with that. I don’t know where you want to start in all that kind of stuff?

Tom (06:32):

No. No, these are great overviews. And guys, just to zoom out a second, I was on your Discord when this was all happening and I saw the community obviously be shocked by it, but I also saw them come together in a really fast way, and try and get to the bottom of this. What would you guys say was the strengths and, I guess, weaknesses of THORChain’s response to these acts as they happen?

Gavin (06:56):

I mean, one of the strengths I’ll say, it’s a loose collection of folks who are operating nodes. And the response time hadn’t improved over the quality of those two weeks, when the exploits were happening. And what happened over really over the last four to six weeks, up to this point in time, is the community response. And I’ll speak from our side specifically, we realized some of the gaps in like, “This is actually where networks are made.” Right? What gets you to this point, and then what gets you beyond this point, are two different things. And so the way that the community showed up in response to these things, was something that’s blown me away.

Gavin (07:34):

And so that’s one of those areas where, if anybody’s paying attention to just beyond the surface level, there’s some really great work going on, that is what’s going to allow this protocol to mature. And all things considered, the fact that the treasury can handle this, and just the nature of this exploit is what it was, I don’t know if there was another event that could have happened so early in this protocol’s lifetime, that would have allowed all of us to come together to focus and understand security, and start changing the posture of the network in this direction, without it being a much larger exploit or something that was far more serious. And so, from that standpoint, I look at it as like, “This is probably the best time for this to happen.”

Chad (08:13):

Yeah. [crosstalk 00:08:13] Sorry, [crosstalk 00:08:17]. Go ahead. Go ahead.

Steven (08:20):

Yeah, to add to his 2 cents, I think it’s pretty amazing the Discord community, I’ve never seen anything like it before, as far as the knowledge level of the people, the people that are running validators and participating in the network. I mean, even us as we were doing audits, we got ideas from a lot of the people that are participants, whether they’re staking or just in the Discord channel. And I was pretty surprised at the level of knowledge and passion that the community had, just to help motivate and drive us, but also give us ideas, and help and assistance for us, so we can nail react faster to it. So there’s a lot of individuals, that was almost like, “Hey, you want a job here because you’re really good? You’re doing security auditing here for it.” Because they’re invested in the health of the network and finding these problems.

Steven (09:11):

So that was really motivating and encouraging as well. And then one thing I would to answer to the other side of it, I would say weakness, but not really a weakness because I actually admire the fact that it’s almost one of the few true decentralized type of networks, where if there’s an incident going on, everybody had to take hold, you have to fit the systems, manage your validators in a certain way, which shows this is something that is true consensus. It’s not like a single admin key that somebody comes in and says, “Oh, I’m going to hit the pause function.” And then everything stops. That’s centralized and it gives you a good sense of protection in case something happens. But in the case of this, it was decentralized management of handling the network. So I think I admire that because you don’t see that too much in the space right now, in this stage.

Gavin (10:06):

And really quick side note on that one, on the Nine Realms, even you guys, our teams are collaborating now. Two months ago, we were operating in the same domain, but we hadn’t been talking and now behind the scenes, there’s a whole lot of work going on, in terms of shared Testnets, testing environments ways, and so the cross-pollination, that’s now taking place in this ecosystem as a result, [inaudible 00:10:29].

Steven (10:30):

Awesome. Absolutely.

Duncan (10:34):

And to heap more praise on the community, I haven’t seen a community respond to a vulnerability event like this, ever. The passion, the drive, the determination, and the belief in THORChain as a protocol, as a concept, it’s really incredible. Just the sticktoitiveness and the passion that everyone’s demonstrated.

Tom (10:58):

To be devil’s advocate here, what would you guys say is one thing you wish was improved about the response? Just to ultra critical, what was one thing you guys would change to do better, just to play the devil’s advocate side of the table?

Steven (11:12):

There’s one thing that we were wondering here, and this is not that to be offensive to anybody in the community here, but it’s actually almost a compliment is, a lot of the smart people that are on there, at first my sixth sense was like, “Huh, I think it’s hackers.” Definitely somebody in the community because they know too much about it. And because it’s so transparent with what we’re doing for our auditing, about TAO, if we find vulnerabilities, let’s talk about it on a Discord publicly.

Steven (11:41):

I think that sometimes we should think about that, because if we do find issues and immediately say, “Hey, here’s the issue.” Then it could possibly be exploited right away before we find a fix for it. So a little bit more discretion in the auditing that we’re all doing, as security partners of THORChain, if we see something, maybe take care of it, find the mitigation, put in the good place, then talk or announce it. So I was a little paranoid about that at first, but I think now we have the right processes to go from finding, to patch, to disclosure.

Chad (12:19):

Yeah. I mean, just to attack part of all that I’m very dearly connected to, give just one second. Well, I think the team could have done a lot better on, is bringing more eyes into the code earlier on. Really the number of people who actually look at this code and participate, and comment on the changes and all these things, is actually very, very, very small quantity of individuals. And that’s not a good thing, right? We want more eyes, more smart, brilliant people to get involved. That’s one of the things that Bitcoin Core has, that’s amazing about it, they’ve had over 800 contributors to that protocol, which is fantastic. It’s amazing. It’s awesome. And I would love to see THORChain heading in that direction, and unfortunately we’re not there quite yet. But there’s some really good action items that was taken over the last few weeks, to push it in that direction.

Chad (13:12):

We’ve got a lot more eyes, very smart eyes, looking at the code to find any problems or exploits that you can possibly conceive of. I know that some members of the community, some members of the team, myself included, got very tired and was operating on very little hours of sleep, extremely little hours of sleep. And I certainly made some mistakes myself, personally, which I certainly will regret, but we have to move forward. We’re going to be moving forward. We’re going to be relaunching this network. It’s going to be a bigger, badder, and stronger, than it ever was before, for sure.

Tom (13:44):

I really appreciate-

Robert (13:45):

And that’s just indicative.

Tom (13:45):

… you guys being honest about this. Yeah, no, it’s great call out. Sorry.

Gavin (13:50):

Yeah, yeah. Honestly, that was the biggest thing. The transparency and communication, because there was so much work going on behind the scenes, that sometimes the public communication was lacking. And that’s honestly one of the commitments that, I think, everybody here on this call has probably gotten around to like, “Okay, in this next phase, we’ve just got to be just a little bit better, in terms of information throughput and transparency.” Everybody was scrambling and the job is getting done, but that’s one of those areas where improvement could definitely can happen.

Steven (14:21):

Yeah. I think we’re definitely… At first, it was a reactive impulse. We got everybody involved. We’re trying to fix the problem, and get the network back to how it was. So the next steps as far as future, and to build on those improvements that Chad has just mentioned, would be to have more standards and procedures, maybe not cowboy code, I guess you could say. So you learn a lot from your mistakes and I think we should take some of the mistakes have been fixed and now build better security into it. The whole DevOps methodology of shifting left and have maybe documentation around this, build a set of practices as far as security and even co-development, that THORChain’s community and the main developers could take, and that’s going to really impact security in the longterm, because now as you’re following these practices, you’ll find issues before they get exploited in the wild.

Tom (15:19):

Well, one question I have for the panel is, you have a hack and it’s an inflection point where you’re able to put a lot of eyes, a lot of power, Nine Realms, Halborn, Immunefi, the Core Devs, the community, everyone’s looking at the code. How do you guys sustain that level of depth for the next year, two years, three years, right? Because everyone’s looking at the code now, but you guys are going to be adding since, there’s going to be new chains at it, you’re going to have cross chain value transfer for potentially every chain, and be an important piece of infrastructure. How do you maintain what you guys have now, what you’re doing now, how do you make sure that it’s the same way a year from now?

Gavin (15:59):

I’ll throw something out there, just to jump on this one, because this is actually what I think is the most exciting piece of where we are right now. Think about THORChain as a tier one exchange, right? We’ve got to have the response capability and the delivery in terms of security posture, all this stuff is a tier one exchange, except we’re building it from the inside out. So everything is public. Whereas a centralized exchange can just hide the mistakes, that delta is an interesting thing to just be aware of. And so THORChain at its core, is a very creative engineering endeavor. And as a result, what was required for security, also needs to meet the engineering challenges creatively. And so going above and beyond that standard, is what we have to create here, right?

Gavin (16:43):

That’s why some of these things have been stood up. The unique engagement with Halborn is one thing, the bug bounty, and the relationship that’s being created with them Immunefi, going forward. That’s another thing. And on the other side, THORSec, that’s a specific function that is project specific, that stood up for the longterm to do a very specific thing. In the end, as we start to do this, because THORChain’s topology is varied and it functions as this large L1 decentralized exchange. We’ve got to create the incentives and the models to secure this thing, in a way that a lot of DeFi protocols don’t. And that’s part of the creative challenge here. But I mean, I wouldn’t be on this call if I didn’t think we were up to it.

Gavin (17:19):

And also, some of the things that have come together over the last six weeks, I think start to show the first prototype steps of that. But also some of the people that are getting involved on this, we’ve got the firepower to make this thing very viable in a different way over the next 12 to 18 months. And so I think that’s some of the commitment from all these people that are on this call, but also the people who are going to be coming in, is the acknowledgement and recognition of what the nature of this project actually is and can be.

Steven (17:45):

Yeah, there’s two things there… Oh, go ahead, Rob.

Robert (17:50):

Well, I mean, this is why it’s been exciting to, these are seeds that are now being planted, that are only going to sprout over the next many months, many years, at this point. So this is why we’re excited to get started with, not only our first round of penetration testing, which was absolutely necessary from this incident, but really what we essentially look at as security advisory, which we’re essentially calling it advanced persistent protection services, or whatever, because there’s this idea that no matter what, the bad guys are out there and being persistent, they’re always there. They’re always looking for new holes. They’re always waiting for the next upgrade, for the next get push, for the next for the next repo to be pushed out there, so that they can look for the things that haven’t been found just yet.

Robert (18:42):

And so the idea being is that, now THORChain has the… Basically, you’re putting together, not even really an army. I look at this almost as they’re putting together a SEAL Team Six, to just go in and infiltrate ahead of the bad guys, and make sure that things are secured throughout all the upcoming deployments as well. And so that’s why we’re excited to be able to, basically, a ramp-up period over the next three months of getting some of our… Halborn’s constantly growing. So all of our best new hires, the guys coming in that are fantastic already with Golang and have already have a couple of zero days under their belt, outside of this, we’re putting them directly into this ecosystem. And what you’re doing there is, you’re creating expertise over time. So it’s almost like something that is exponentially growing, as time goes on.

Steven (19:44):

Yeah, definitely. You can consider it a blessing and a curse with THORChain, because it does lots of things. There’s many bridges, lots of functionality that enables, which is why it’s so popular and there’s such a big community around it. But complexity is also the enemy of security. So the more complex, the more functionality that you enable, the more holes that can come out and they become more elusive over time as well, as you start adding modules. The environment grows and you have these economical type of attacks, are very hard to find. So it takes a lot of, I would say, tribal knowledge in becoming familiar with the way the system works. So this is something that, when you have a dedicated team, from us at Halborn, engineers training for it, from Immunefi, and everybody’s there for the longterm, you start to learn how the system works and how THORChain operates.

Steven (20:35):

And you see all of the hidden functionality behind the scenes and you become very familiar and intimate with it. And that’s where you really hit the nail on the head on, really good security is, you have a dedicated team that knows the complexities and knows the common patterns, and a lot of the different models and compositions that you can DevOps formal verification on. Whereas, if you had somebody, “Okay, we have this new module, let’s go ahead. Let’s do an audit in two weeks.” Nobody’s going to learn the system in two weeks and find all the holes in there. It’s massive. The code is complicated. So what THORChain and everybody here on this panel, is getting involved with now, is the right way to do it. To have experts that are familiar with it, for the longterm. And month over month, train individuals and become THORChain offensive engineers that know how to find these issues before bad guys do.

Tom (21:30):

Yeah, no, it’s great. Great context. And I want to talk a bit about Immunefi, Duncan. It’s a different approach to finding bugs. You guys hit the ground really hard, really fast. Give us a sense of what Immunefi is and how you guys got engaged with THORChain?

Duncan (21:45):

Yeah. So it really is a different follow-on security approach to traditional auditing, totally complimentary to that system. The idea here is, your independent hackers bring a fresh mindset to these sorts of things. We are seeing people surfacing novel kinds of attacks through Immunefi, it’s also a way of [inaudible 00:22:10] economic skills in favor of the project, right? Yeah, I’m sure everybody knows that the hacker of the first exploit, published that a 10% bounty could have prevented that. I mean, that’s basically Immunefi’s bread and butter, right? The idea is, you post that 10% bounty, black hats take notice and they go, “You know what? I’ll take 10% rather than a 100% because it’s clean money.” So as far as our involvement, my co-founder, Trayvon, got involved with THORChain pretty much immediately after the attacks, really expedited our onboarding process to tailor that bug bounty program to THORChain’s needs, provide a strong front to the community and really assure everyone that THORChain does take security seriously, and they’re going to put their money where their mouth is.

Gavin (23:00):

Actually, that’s an area to clear the air too. Immunefi stood up immediately, it’s day zero of the hack, Trayvon was on the line and we were getting things stood up. And that was one of the areas where, as the community was starting to surface things and talk about exploits in the Discord channels, there’s a lot of noise we were getting that stood up. So I know that there was a little bit of some people are like, “What’s the nature of the bug bounty program?” But one thing is, I think we smooth things out as fast as possible, but really that was only possible because of Immunefi stood up so quickly and expedited the process. It happened on a Friday and we were on the line with Treyvon Friday night, Saturday, and by Sunday, the proposal was ready to go and everything was just beautiful. So just awesome work, you guys.

Tom (23:41):

Yeah. No, I mean, hands out to both of you guys, I mean, Immunefi and Halborn, you guys got started quick and THORChain’s better to have both of your teams jamming this down and fixing it. It’s amazing. I’d love to chat a bit about THORSec. Let’s dive in there?

Gavin (23:57):

Yeah. So also, so on the Nine Realms side, that was realizing that the security of this network needs something a little bit beyond the typical engagement, and we saw that coming together. We had some friends in our network, who are world-class white hat hackers. I mean, they’re the top of the board with some network exploits and that’s how they’ve made their living. They do it for fun but there are two guys who were just extraordinarily good. They’ve been interested in THORChain for a long time, and as this actually happened, this provided the fertile soil for us to create this unique thing called THORSec, where we have two world-class white hatters who would focus on effectively being offensive security, help understand.

Gavin (24:40):

So we’ve got this team now, that is reviewing every pull request. They’re talking with Chad and the core team on a daily basis, helping set up environments for testing, and exploits, and everything that’s going through. So as new security features, and new modules, and things get plugged into THORChain, we can have a new type of testing environment for doing things. This becomes useful to everybody, Halborn, us, everyone. And so we looked at it from our perspective of, we’ve got the resources. These guys are some of the best in the world as well, and they want to focus here. And so it made sense to just create this unique function. If you’re looking and talking about a tier one centralized exchange, they’d have a security team. And so we looked at as, “Well, we need something then that is funded by the treasury, that is a part of the core protocol, but it’s distinct from the guys writing the core code.”

Gavin (25:25):

So they think about it differently. They think about it from the security standpoint. And then, there’s a really tight connection, but it’s a loose coupling. So it’s just basically the THORChain focus security aspect, that will grow over time as the surface of the protocol grows. But as a starting point, these guys are just quite capable. So they’re some of the best in the world. And so now that we’ve got this mesh of security wrapping THORChain from an array of perspectives. And so THORSec was our approach to just throwing something into the mix, that would become a core part of the protocol, going forward.

Steven (26:05):

I like how you mentioned they array of a mix of things too, because Rob and I, at Halborn, we always say that the pen testing and offensive security is more of an art form. Everybody has their own canvas, their own methodologies, their way of doing things. And the more perspectives you have of bringing an experience from Nine Realms, from Immunefi, Halborn, anybody else, you’re going to get different attack patterns in a different attack services. Because of that perspective and the experience that people bring, you’re going to get way more coverage. So having that diversity, is going to be huge in the long run as well.

Tom (26:42):

No, I totally agree. Having a spectrum is really important, because you have eight plus devs within the community with THORSec, you have how Halborn, you guys are incredible, and then you have the community with Immunefi. I guess my question for you guys is, it’s hard to talk about results since you guys have all started, because it’s highly technical and you don’t want to share anything that could be exploited somewhere else, but can you guys all give a taste of, I guess, your work so far? If somebody’s boss came to you and said, “Hey, what did you guys done for THORChain lately?” What would be the answer there?

Chad (27:13):

Well, so I was on the team, by the way, too. Trail of Bits’ also been finishing up their own audit of the network as well. So we have Halborn, you have THORSec, you have Trail of Bits, you have a lot of different hands going into it. But one of the things that I’ve been involved with, is building these blanketed protections for the protocol itself. So the network now checks for insolvency. So if somebody sends in a fake ETH, like last time, the network can instantly detect that, “Hey, we thought we got 15 ETH from this guy, but we really didn’t get anything. Let’s just hold everything, figure out what the problem is and then resume things when…” The chain autonomous does on its own, without any human interaction whatsoever. That’s the first one. That’s protecting the ingress, the outgress, funds getting sent out, there’s a new feature being added, where when there’s a large volume of trades tapping through the network, it actually slows down those trades or slows down the output of those trades.

Chad (28:10):

It can slow a trade up to an hour effectively. And that gives the community time that some sort of attack happens and someone’s trying to pull out $8 million out of the network for some reason, or whatever. The network would just naturally just slow itself down, and giving opportunity for the community to see what’s happening, like, “Oh, wait, this is an attack. Let’s come together, let’s stop the attack from happening.” So they don’t actually pull out any funds out of the network, hopefully.

Chad (28:36):

Our approaches are more a catch-all approach to any type of exploits you might find. If you’re trying to find every single exploit, you may or may not find them, and you don’t know how many there are, right? Inherently, you never know, no matter what you. But now we have these systems in place, that will protect the network from a whole myriad of possible attack factors, whether you’re doing it this way or that way, on this chain or that chain, doesn’t matter, vast majority of situations, these two blanketed protections can protect the network from such exploits.

Steven (29:09):

Yeah. One thing that they have on data, an unique approach that helped us find some issues is, it’s one thing to look through the code and see what it’s doing and find vulnerabilities of static analysis. But we actually realized, “Hey, we need to create our own validator and connect it to the chaosnet and you’ll interact with it dynamically.” So our expertise, like Rob mentioned before, there’s the CeFi and DeFi. We have a lot of enterprise security experience in DevOps as well. So we created our own validator infrastructure, and part of the scope that we’re doing, besides a lot of the contract and code, is looking at things the Kubernetes deployments, and the node deployers, and the way that these values are created and looking.

Steven (29:51):

So when we were going through that, there’s a few issues that we could see, “Yeah. This might be an insecure way to deploy nodes. Or it’s not hardened correctly.” So providing extra findings, just around the security of the infrastructure, not just the consensus of the code itself. So now interacting with it dynamically, and attacking something on a live network and seeing how it responds, has been huge for us as well. So yeah, we’re participating in the network and adding funds to find those things that you get. It’s very hard to imagine how it would react in a live environment, and now we’re getting our hands in there and try to wrangle the beast now, just live on the network.

Robert (30:33):

Also, in every other industry, outside of this blockchain ecosystem, it’s absolutely not. It’s if you’re in the middle of doing a penetration test, none of that has ever seen the light of day. If your penetration testing infrastructure of a bank, you’re keeping that throughout the entire process internally. Meanwhile, we’re completely flipping this on its head with our engagement with THORChain, where literally our penetration tests, our Gantt chart, our progress, our entire scope, is completely public. In fact, it’s tweeted weekly. So even in the blockchain industry, it’s not a very normal thing where you actually see every day what auditors are doing. So that’s just from a high level, “Here’s precisely what’s being worked on.” And then I’m trying to think of the positive terminology around death by a thousand cuts, right? Because it’s this idea that every single day we keep on having findings and talking internally with the team about, like, “All right, we’re just adding and compiling to the report.” So I feel just every day, it’s just a ton of little things that continue to just stack up at the end of this. Even just this first six week engagement.

Steven (31:50):

But before we break, having the access to it [crosstalk 00:31:55] too.

Duncan (31:57):

The phrase I would use is, with enough eyes, all bugs are shallow. That’s really Immunefi’s approach to it. Right? We’re bringing the largest number of eyes for a [inaudible 00:32:10] no, makes sense of protocol.

Steven (32:20):

Yeah, for sure. And having direct access to talking with the developers and architects, and any of the community members, helps expedite what we’re doing too. So if we maybe have a sixth sense, “Oh, I think this might be an issue here,” rather than spending a lot of time independently doing something, and then a couple of weeks later giving the report, if they, “Hey, I’m trying this. Has anybody else tested this before?” The guy’s at Trail of Bits or Immunefi, want to look at the memo for some reason, “Try this. Have you done this yet? Or what’s your take on it?” They can almost say like, “Oh, yeah, that’s a good idea.” Or like, “Oh, no, I tried this before. There’s a protection that is put in here. So don’t worry about it.” Now we’re like, “Okay.” Now we can react with a lot of agility and take care of the things that actually need to be looked at, rather than spending countless nights, just hacking away at something and find a roadblock or a dead end.

Tom (33:14):

No, that’s really important. I guess my dumb non-dev question for you guys is, Chad mentioned the solvency checker, you guys are adding a lot of protections in place. Does that make the system even more complex, and introduce crazy intense, high-level type bugs down the road? How does the system manage all of the safeguards in how they interact with each other? And maybe I’m just not thinking about it the right way, but I want to stay as the devil’s advocate here.

Chad (33:45):

No, it does. It’s a good question to ask. So the answer to that, the simple answer is that, yeah, it does add more complexity and it can introduce new bugs into the system. Right? And that’s why you need to do testing, testing internally right now, we’re testing in Testnet very soon, this week, and seeing how that goes. We’re getting our friends and partners around the world, to start slamming on it, finding any bugs, if there are any, and then adjusting as we need to. Managing complexity is always a difficult thing in any software project, in general. In a crypto product, it’s more difficult typically, because crypto products tend to be much more complicated to write in code than web two stacks, in a manner of speaking. And in THORChain’s case, that’s considerably more because THORChain is very, very complicated protocol, much more complicated than probably any and all DeFi products out there, I would say. I’ll say that with a good amount of confidence too.

Tom (34:41):

I agree with you. I’ll support that.

Chad (34:45):

Yeah. Take a look at the code, they see how very complex it is. I mean, even TSS thresholds by itself, just TSS thresholds by itself, is more complicated than most DeFi products out there. Just that one aspect by itself, not to mention the rest of THORNode itself, Bifrost, manipulating but not moving, but interfacing with multiple chains, protecting yourself against double spends and reorgs, and all sorts of crazy situations and doing that completely autonomously, that’s mind-numbingly difficult and complex. But to your point, Tom, it does add slightly more complexity, not considerably more. For example, the delayed outbound is like adding a couple 100 lines of code, for example. But it’s important, because the question always should be, is the return on investment worth the additional complexity? How much complexity you add and how much of value to get out of it, right?

Chad (35:46):

Now on the delayed outbound transactions, the feature we’re talking before, that’s in part designed to say that if somebody did attack the system, and were successful to do so, they would extrapolate less value in funds than if they just went to Immunefi and say, “Hey, I got a bug. Can I have my $300,000 bug bounty?” If they were to actually attack the network that we get, maybe 250,000 or 200,000 or some number, and this code is designed to help protect the network from that kind of scenario. So it does add some complexity, but I would argue that that complexity is warranted, given how much of value and security that it cakes into the [crosstalk 00:36:28], keep the score.

Steven (36:30):

But, Chad, you also get reputational benefits from that as well, too, not just financial is, some black hat hacking the network and stealing it, and having all the updates, that all 13 was hacked. It’s a lot better to have a white hatter or a bug bounty to discover that, just for the optics of it, that adds value to the long term.

Chad (36:52):

Yeah. And I would say [crosstalk 00:36:54] if you aren’t an attacker, honestly, at this point, it’s probably more valuable to you, to be away and you actually make more money as a white hat, than you would have as a black hat [inaudible 00:37:04] network probably. So, “Come with us, help us. You’ll get paid.” All these things, “It’s a great opportunity for you.”

Gavin (37:11):

Well, and your peers are now other world-class white hatters, who have flipped their own hats in their time, which is another thing to say. But there’s another salient bit to distill here, which is the nature of what’s going to come out of the testing, right? Chad’s talking about testing, setting up even the environment to test for exploits and things that, every time you roll something out, the state of cross chain DeFi testing tools, is absolutely behind where it needs to be, just for the entire ecology. Flat-out. And so this project is going to push that. And so part of what we’re all engaged in right now, is creating this unique test environment where you can spin up these different things and you can have the ability to test this code in a different way. Because where we are and where we’re going, are two different places, and so we’re effectively going to have to build our own roads, that will benefit everybody.

Steven (38:03):

Yeah. Back to Tom’s question about the complexity too, of how do you manage this dynamic changing environments? There’s really two approaches for it. And one of them is looking at it by components, which just started that, “Let’s hone in on just the code here.” But then there’s the whole composability aspect of it, where, all right, when you look at this one, like a router, or the Bifrost, or the TSS, alone, there’s no vulnerability. It’s code looks fine. It’s working as expected, but once you introduce the way it’s used in context with the other components, that’s when the vulnerabilities come. And composability is very difficult, because it’s not just something about like a standard, the way that the code is, it’s actually functional type of issues. And that requires understanding, and also requires on state of the network at the time too.

Steven (38:53):

You think a lot of these DeFi-type hacks where it’s like flash load attacks or anything, that all depends on how much liquidity is on there a certain moment, and how much can you borrow, and how much is outstanding? And it’s something that can’t really be tested, but more about what the risk level is, and do you have mitigating controls put there, to either react or defend against that risk? So I think that’s where we’re going with those blanket-type of mitigating controls that Chad has mentioned earlier, and things that we do as we stand up and work with the infrastructure live.

Tom (39:25):

No, no, that’s really helpful context, guys. And continuing on my questions on the other side of the table, do you guys think that THORChain lost ground during this? Or do you think THORChain is going to be ever hardened? And I ask that question in the context of the community and also in the context of, THORChain is a pretty critical piece of infrastructure, my opinion. You guys are building real actual trustless, cross chain value transfer. It’s important. And to Halborn’s point earlier, it’s not a closed bank, right? You’re building everything out in the open. It’s very important that it is secure, but I am curious, and maybe I’ll direct this towards Chad. What’s your take on the community here? Is the community as strong as it was? Stronger or less strong, because it’s a critical piece when something takes a while to fix and to harden?

Chad (40:15):

Yeah, absolutely. To speak objectively and honestly, I think when the attacks first happened, there was this moment of pile on, right? We were seeing on Twitter, we were seeing all these mediums, even Max Keiser got a few jabs into the fray here. And so it did feel a dark hour for the project, for sure. And I think a lot of people in the community were definitely feeling that at the beginning. But I also do feel that the strong individuals that we have within this community, came together in a way to iterate and to say, “Let’s take a step,” and Gavin was a good example of this, just coming in, “Let’s take a step back, let’s reevaluate everything. Let’s bring a lot more hands into it. Let’s bring a lot more people into it.” And inevitably the result is, this code is and the community is, extremely much more hardened than it was before. I would argue a good magnitude larger than it was before, in that sense.

Chad (41:20):

The community is definitely behind it, in a very strong and very obvious way. We’re seeing that today through social media, but also in private channels. People are talking to me and absolutely people much more hardened in their view. I would argue that the project is definitely in a much better position today, than it was the day before the attacks actually happened. Even though the network hasn’t fully restarted yet, they’re not doing trading quite yet, I can see the road down the street and where the project is heading in, in general. I’m very happy and very excited to see where the project is going. I can just see a clear path, we’re going to be a much, much better place in a few weeks than we were a few weeks before.

Tom (42:03):

No, Chad, I really appreciate-

Steven (42:04):

I would agree with that.

Tom (42:05):

… the authentic answer there. Oh, sorry, keep going.

Steven (42:08):

Oh, yeah, no, no, I completely agree with that too. And there’s a story behind it, where it validates what Chad’s saying too. A couple years ago or a year ago when GoDaddy got a big hat. And a lot of my friends were coming up to me saying like, “Oh, GoDaddy just got hacked. Should I move my domain to somewhere else?” I’m like, “Actually, now GoDaddy is probably the most secure domain you could be on.” Because unfortunately, when an incident happens, people, whether it’s actually technical people, or management, or their executives, now all of a sudden security is front and center in their mind. And investment comes in, and skills come in, and protection come in, and you get the highest level of security maturity after an incident. So I would definitely say that’s the case with this too, where now the response that we have and all of the team members, communities, and the companies involved to help protect us and get the eyes on it, THORChain is absolutely, probably, in the strongest position from a security standpoint, right now.

Tom (43:12):

No, I love that. And guys, one other question I had for you, we have Trail of Bits, we have Halborn, we have Nine Realms, we have Immunefi, we have the Core Devs, how do you guys all coordinate in a way that’s productive for all of you? How do you check problems? What if you find the same bug, which would be pretty interesting too, how does that happen?

Duncan (43:35):

[crosstalk 00:43:35] Finding the same bug is absolutely something that’s happened.

Gavin (43:38):

Discord it.

Robert (43:41):

Yeah. Look, I think at this moment, Halborn’s working on a couple dozen projects at any given time, with our team that’s constantly growing, and there isn’t a single other project that we work on at this moment, where we are in direct contact with other auditing firms, other firms that we normally be vying for similar forms of business with. So it’s been really eyeopening and amazing to basically requests from the team, right when we got hired to come in, it was like, “Here’s a crazy idea. What if you throw us all into the same room together? And then we collaborate while we’re doing our separate things. Why don’t we collaborate as well and let’s just see what happens.” It’s been amazing. We’re learning every day from each other at this point, and new relationships are being formed. So it’s as simple as the fact that there is a separate, private Discord channel, where we are having these conversations, but a lot of it’s very public as well.

Steven (44:51):

Yeah. And all of a sudden turns into a capture the flag events, as we hackers I like could do it. It’s a bit encouraging too, because we are peers in this industry and having multiple eyes on it, is recommended. We’re never offended by anybody saying like, “Oh, we want a second opinion on something.” Have another auditor to come in. That means that they actually take security very seriously, so it’s admirable. And the results are going to be speaking for themselves pretty much, from this, and working directly with this, it’s not like we’re trying to get the business from our competitors, like, “Okay, great.” No, it’s not about that. We’re both here to help THORChain become more secure.

Steven (45:35):

And it turns into a mix of assistance towards a friendly competition in a way, it’s like, “Oh, if we find this bug,” like, “hey, we didn’t mean to announce it right away too.” So we look cool for the credit for it, but also it may be that they already found it, and now we’re wasting our time and resources on it. So I mean the whole process of having multiple auditors communicate with each other on this, I think the positives just speak for themselves. It’s been a very validating and fun experience as well.

Gavin (46:09):

And one of the benefits is, we’re going to see the benefits of this, I think, expand over time. Because we’re just now getting to the place over the last month plus, where we all share context and language, and that’s taken a long time to even build up. If Chad was the one coming in here with all this context and understanding, we’re all getting ramped up into this thing and that just starts to pay dividends over the long haul. This is all just getting turned on, but we’re going to see the benefit of that shake out, as this thing starts to ramp back up.

Chad (46:37):

Yeah. That’s something that I’ve really happy about, is to see the context that I have in the code base, being somebody who, one of the deeper understandings of the code base, and getting more eyes and more brains into it, and then spending my time to talk, to answer questions, train, get more mind share of this code base and how it functions, how it works, to get more people whether it’s Halborn, whether it’s Trail of Bits, whether it’s THORSec, all these different separate teams, all these different individuals, and deepen their understanding of how this very difficult and very complex code base actually functions underneath the hood. So that it’s not just me and one of the guys, so to speak, who have a full idea of how this code actually works. There’s a whole myriad of teams that are deepening their understanding just to make it even more of a larger community around the project, from a dev perspective.

Tom (47:28):

So guys, let’s sort through fun stuff. What is the roadmap release schedule? What should people look out for? What’s next?

Chad (47:39):

Yeah. So the team, along with the community, has worked very hard to get the network back up and running. It’s a multi-phase approach. It’s a little bit of taking our time and doing it right. The first phase was just to get THORChain up itself. Just the chain itself, and just committing blocks. Almost all functionality just disabled, just to get the chain up and committing blocks, and having blocker wars being admitted to LPUs and node operators as well. Second part was to enable THORChain-specific functionality, like sending RUNE and adding Bond, or take one bonding and these kinds of things, adding it to the THORChain network itself, that’s phase two, which has already been done. And then phase three, which we’re in now, is to re-enable Bifrost and to start just one chain, the BNB chain first, and get that one up and running, trading. People can add, withdraw liquidity, whatever they want to do. That’s the current phase that we’re, as a community, working on.

Chad (48:46):

And so I think this week it’ll be deployed to the new changes to make that possible, to be deployed, to Testnet. Once Testnet has validated all that, and everything looks good, if there’s any bugs and we’ll go and fix and patch them, continue on. Then the next phase at that is opening up the UTXO Base Chain. So that’s Bitcoin, Bitcoin Cash, Litecoin, for example. And then the next one after that, is that the biggest, baddest and the most difficult and complex one, which is Ethereum, largely because it has ACE, which is arbitrary code execution, which makes it more difficult to secure and walk down in a sense. We’re leaving the baddest for the last, in the sense to get the chain up and running.

Tom (49:30):

That’s awesome. No, I’m really excited. And I want to zoom out as we close out here, I want to talk to you guys. You guys are all pretty intimately involved with THORChain, the community and the vision here. I’ve already said that we’re an investor and happily so over the past several years, so we have our own views. But what does THORChain mean to you guys today? And I’d be interested to hear how it’s changed, because it’s changed a lot since the initial Testnet. Now that we have a real multi-chain future, people leaping into Avalanche and Solana NFTs in the last week. We may not have a lot of NFTs yet, Avalanche Defi, but people are APE-ing into place, it just shows that a cross chain future is now a reality, right? Something like THORChain has to exist. Right? And it has to just a grand scale, but I’d love to get everyone’s thoughts on what THORChain means to them. And then I guess their future for a multi chain or their thoughts on it?

Steven (50:26):

For me, I think a THORChain is setting an example, and it’s almost brave. When you think of just Thor, the character itself from Marvel, or the Norse god, he’s just going out there and smacking the hammer down. And I think that’s what we’re doing here, it’s so transparent and so public, and it’s setting an example of, “We’re doing this because we’re serious about it.” We’re not going to give up, like, “Oh, we got hacked. It’s all over.” It’s like, “No, we’ll persist.” Whatever, you brush your shoulders off, go forward. And everybody’s getting involved, we you get through this and come out stronger. And it’s going to give a lot more safe into the network as a whole, and they always say that there’s no such thing as bad publicity either. So I think that after this is all done and we’re back to normal, it’s not such a great long-term thing that’s going to be making THORChain persist for many years to come.

Tom (51:27):

Absolutely. Anybody else have any closing thoughts before we close up?

Chad (51:32):

Yeah, I think for me, to take a more esoteric perspective maybe, THORChain has a lot of technological innovations to it, that there’s doing the things that nobody’s ever done before, right? Whether that’s using federal signatures to lock up hundreds of millions of dollars, or even protecting against MEV attacks, for example. There’s all sorts of things actually happening within the stack, that are innovative. But for me, what I actually see the value of THORChain from a higher level perspective, it’s actually, it’s a human rights push, right? Bitcoin to me, was the first human right that was granted to all peoples across the entire plant. No matter what side of an imaginary line that you happened to be born on. The first time in human history, everybody had access to a basic inalienable human right, which was the ability to hold your own wealth in a completely sovereign way.

Chad (52:24):

That was like, “Whew.” I don’t think a lot of people recognize how massive of a contribution to humanity the Bitcoin was. Right? That was absolutely astonishing. And I think THORChain is taking that idea of moving just a little bit further and saying that, not only can use for your own wealth, as Bitcoin did, but you can actually exchange your wealth to another asset of your choosing, without being requiring some sort of set entity in the middle of it, all to allow you to do it or not allow you to do it, and these things. It’s to me, that’s a massive push for human rights. And it’s partially what gets me excited about the project, is the idea that no matter where you are on the planet, you can actually transact and move your assets from one chain to another chain.

Chad (53:06):

You can actually provide whatever value you have, at whatever amounts you have, into a system and get a 30%, 40%, 50% APUI. That’s amazing without needing to KYC or not. It doesn’t matter if you’re a man woman, Black, white, who gives a fuck? Sorry for swearing, but who cares? That’s a massive, massive thing. That’s the biggest movement in income equality the world has ever seen. That is an absolutely massive thing and we should all be applauding this. We should all be getting behind this. This is a world shifting technology, not just THORChain, but Bitcoin and other contributors to the space. It’s absolutely astonishing and I’m super proud to be a part of it.

Steven (53:51):

It as a big Bitcoin maximus myself. First of all, I got goosebumps, thank you for that. My favorite quote, you may have heard it before, if not, but they always say that Fiat is the government’s money. Gold is God’s money, and Bitcoin and THORChain is the people’s money and its self sovereignty and giving to themself. So I think that’s right in line with the spirit and the principles that you just went through.

Tom (54:15):

Yeah. Guys, this is incredible. I really appreciate the authentic and devil’s advocate conversation here. I’m really excited for THORChain. I do think it’s a critical piece of infrastructure, and to have Trail of Bits, obviously not here, but shout out to you guys as well, to have Immunefi, to have Nine Realms, to have Halborn, to have the Core Devs. It really is nice to see you guys all come together to harden the protocol. So I am very excited for you guys. Shout out for going through all those sleepless nights to clear things out when it happened. But I really appreciate you guys all coming on tonight.

Chad (54:51):

Thanks, guys.

Gavin (54:52):

To leave a parting note on that side, if you guys think the last 12, 18 months was good, just wait till the next 12, 18. This is where it happens.

Robert (54:59):


Tom (55:00):

Hell, yeah. Excited, guys. We’ll talk soon.

Steven (55:04):


Show Notes:

(00:00:00) – Introduction.

(00:00:08) – Guests’ backgrounds.

(00:02:10) – THORChain’s hack overview.

(00:06:32) – Response to hack.

(00:15:19) – Sustaining THORChain’s quality in the next few years.

(00:21:30) – Immunefi overview and relationship with THORChain.

(00:23:41) – THORSec initiative.

(00:26:42) – Panel’s role in THORChain.

(00:33:14) – System’s management of safeguards.

(00:43:12) – Team coordination / solving problems.

(00:47:28) – THORChain’s roadmap.

(00:49:30) – Closing thoughts on THORChain.