[Music]
and I also lead the uh War Games
Initiative for the security Alliance um
I'll give a brief introduction to just
what the security Alliance is and what
the various initiatives are um because I
think it's actually pretty relevant also
after the first talk um about ens and
data sharing and so the top initiative
here is actually called the seal ISAC
which is not me I'm seal Isaac this is
the information uh uh security something
something which basically this is like a
uh a shared database between many
different companies where they can share
data on uh like dprk threat actors or
fishing data or bad domains um there's
seal war games which is the initiative
that I lead where uh we prepare protocol
teams for incident response through both
tabletop simulations and then live
simulations on forked environments um
seal also operates seal 911 the
emergency hotline if your protocol is
under attack or if you have discovered a
vulnerability a protocol and need to get
in touch with um a security expert as
fast as possible um The Safe Harbor uh
agreement um which is legal protection
if you're a wh hat hacker and you're
acting benevolently in an urgent
situation to uh rescue funds from a
protocol think uh Nomad Bridge hack a
few years ago if you were trying to save
uh money but maybe you were worried
because you thought that you'd be sued
for hacking during uh because you it be
impossible to distinguish the white hat
Safe Harbor agreement exists to protect
white hat uh hackers uh in those
situations and a legal defense Fund in
case they sue you anyway um so today
what we'll talk about primarily is war
games so what are these war games and
why do we do them um I'll have some
links to resources on how you can work
with us uh there's a lot of open source
toolkits and we also run these um as a
as like a public good uh throughout the
space um we'll be talking through some
best practices that I've learned from
working with some of the major protocols
in the ecosystem um and then just
briefly show you what this new toolkit
is that we released so you can run these
yourself uh if you would like so what
are war games these are cyber secur
exercises for web 3 um I've heard uh
traditional cyber security folks call
these something like a purple team
exercise where we're not exactly red
team blue team but we're basically just
trying to stress test these teams and
understand how they behave under
pressure and so we want to help teams
practice for these um high pressure
situations before they actually happen
um we want to be able to like test both
the technical side of their team and
their social resilience and we're not
just trying to help like cause them to
panic but we want them to practice
emergency response um a lot of the uh a
lot of the reason why we started doing
this um was a few years ago um so the
security lines was started by Sam sun
and he was pulled into so many War rooms
where it seemed uh like I guess in uh uh
from what I've heard uh teams just
completely melting down and not really
being exactly uh prepared for what would
happen so what could we actually do to
help put them in that situation before
that inevitably happens um so War Games
take place through three phases um I do
a lot of uh uh intelligence gathering
which really just means reading every
document that the protocols ever
published understand their contracts
understand how it works uh we do a
tabletop exercise where we talk through
various scenarios with them um and
understand how they would have detected
them what they would do who's in charge
um and how would they have fixed
something and then what I think is the
most fun is we do a live simulation so
we set up a forked environment we run
all of their contracts we run all of
their uis we run all of their monitoring
um and then they actually have to coord
in real time to uh uh defend and respond
as if it was a real incident and we tend
to find a lot of things that they can
improve through that process and it also
just helps a team build trust in each
other so that they know when an incident
goes down for real I can trust my team
uh uh to be there so a big reason why we
do this is because more and more hacks
are due to operational failures not
smart contract bugs that's a really I
think good sign that we're making fewer
and fewer dumb mistakes on the smart
contract side how however as all of the
stuff that we're building is becoming
such a critical component of Global
Financial infrastructure the stakes are
higher to actually operationally be
strong as well and so whether that's uh
whether that's things like supply chain
attacks or uh or social engineering uh
it's much harder now than just like not
uh including dumb bugs in your smart
contracts so these are cross functional
exercises we're not just working with a
development team they're of course a
core part of it but we're also tending
to work with their with work with their
Auditors if they're a big protocol that
has a guardian multisig that's in charge
of pausing the protocol in case of
emergency we of course work with them um
Communications team is actually
essential so that we know like what
would they be communicating out to the
public and when during an incident and
legal so that they can say okay if we
were to take this action perhap to
perhaps to protect the users of our
protocol can we do that should we do
that and what should we be saying the
conversations between Communications
legal and devs should really be that are
figured out in advance so that during
the incident you're not thinking like do
we tweet out that we're under attack
right now or do we say something cryptic
or what do we say like these are things
that you can just have in a Playbook so
that you don't have to think about it uh
in the moment we've worked with a number
of products uh of protocols uh so far uh
but the purpose of this slide is also
just to show you like why teams are
working with us it's because like teams
are growing uh these are Global teams
and like it's really hard to like
prepare for this type of uh this type of
thing um there's many ways to engage
with us if you're a protocol that wants
to work with a security Alliance and
have a drill performed with you you can
uh there's a form on the on the website
and there's also open source uh uh
resources for you to be able to run
these yourself so uh this is a quote
that I had a really hard time
attributing uh the internet said it was
either like a Greek poet or the Navy
Seals but I think that it really like uh
stands to to show like why we do this
and it's uh that Under Pressure we don't
rise to the occasion we sink to our
level of training uh yeah I think that
that speaks uh to why we do this so each
one of these builds is quite different
um a lot of protocols uh a lot of
protocols have like various sorts of
like custom infrastructure and
monitoring so every time we do this we
end up having to like build a lot of
things and we've tried to combine all of
those tools and things that we've
learned on how to run these simulations
um into this uh drill template that you
can see here it's on the security
Alliance GitHub repository but what are
the things that we do to make the
environment feel realistic so we of
course have to run a network Fork we're
usually either doing that with Anvil or
with uh tenderly we have to run a block
Explorer so we're either running block
Scout um or using a virtual test net um
we're running monitoring so it's
essential to mirror the same monitoring
that you would have on Main net so if
you're using a product like hyper native
hexag or Forida you should be we should
be setting all of that same stuff up
against our forked environment we write
Bots to behave like users so if it's a
lending protocol we have a lot of noisy
transactions of users depositing
borrowing Lending spping um doing
everything that would exist on mainnet
so that the only TR so it's not like the
team only sees like two transactions and
they're an exploit and then it's too
easy we try to make it a little bit
harder um we have an exploit which uh an
exploit bot which ramps up intensity
over time and then if there's any sort
of miscellaneous things that they need
to pull in we try to provide that um as
well so this takes us usually a couple
weeks to build for each protocol but
we're trying to get faster and faster
each time and part of that is uh uh
thanks to this uh open source template
that we're that we're working on
so what are some of the scenarios that
we've uh that we've worked on so uh as
you've seen we've worked with various
like lending protocols and decentralized
exchanges and l2s and so the things that
we're work that we like try simulate are
not necessarily uh like bugs as in like
a smart contract bug because we're not
Auditors we're not trying to just find
like a mistake in your code um and in
fact like if we do that's like that's
like not what we're that would not be
the goal we're trying to figure out what
are the things that can happen in if the
protocol works even how you designed it
so what if some sort of external uh
dependency fails what if the way that
you're importing Oracle data is weird or
what if like some token uh some token
contract gets upgraded and starts
behaving differently so what are the
things that are outside of your
protocols control that can uh that can
change what if your upgrade goes poorly
how would you detect that um what if
there's something strange happening with
oracles and also uh one thing that I
think was actually pretty fun is in some
cases we did like a a malicious
governance proposal uh C f with some
protocols where we just like kind of
flooded them with a bunch of governance
proposals that looked like normal
funding requests for events or
sponsorships or just uh parameter
updates but some of them also included
like malicious data um like how tornado
got uh hacked or I think a number of
protocols how how they got hacked just
by having like sneaky additional call
data um that uh that ended up having
wide uh bad
implications uh I've pulled out a few
best practices of what I've learned from
working with with uh working with these
teams that I just want to go over
briefly and so one is just maintaining
multiple communication channels to reach
your teams uh one thing I like to tell
teams is like imagine uh like during a
tabletop like imagine this scenario
happens but now imagine that this
happened and telegram is down on the
same day that this happened can you
actually reach the Guardians do you have
another way to reach them can you reach
them on slack Discord can you page them
uh sometimes we find that like when we
work with a team like a a paging system
that they think is set up is just
deprecated and hasn't been working in a
while and they can't actually reach the
teams that they want to or they reach
the team uh and they don't have their
Ledger with them and so like okay cool
we got in touch with you but you can't
actually sign the pause transaction
because you're camping like these are
things that you want to test um also
risk isolation this is a we learned a
lot I think from the uh from how the UR
team um prepares and isolates risk
between the different strategies and so
there's there's a a big lesson in like
minimizing the amount of contagion that
can spread
uh from like one strategy to another and
also the diligence process before any
strategy was employed there was always
an emergency response of like with this
change these are the things that can go
wrong and these are the steps you would
take to recover funds before they become
uh unrecoverable so like Risk isolation
with urine uh similarly like Asset Risk
isolation with a we found was very
interesting how you can see how there's
all these different configuration Flags
based on how potentially risky a
different collateral asset is is they
have different properties as far as how
they can be uh how they can be used in
the protocol so I think that these are
all really interesting lessons from
different like larger dii protocols
about like how you minimize potential
contagion and and and things from uh
things from
spreading uh finally uh or additionally
um incident post-mortems are really
important so uh this is another call out
to like the urine GitHub repository
where you can see that every time
there's been something that has happened
you can uh see what they what the team
did how they found out and then also how
they'll be improving in the future to uh
to avoid that sort of
issue and monitoring uh mirroring
monitoring I think is quite important
like a lot of teams only will run like
their critical monitoring against their
main net smart contracts um and that's
handy but uh the things that are
actually going to trigger to maybe like
pause your smart contract in the event
of emergency maybe it will never happen
or maybe it will happen like once a year
and so how can you really be sure that
it's going to be there the moment of and
a lot of the these these tools are great
and I'm sure that they have some
internal simulation capabilities but if
you can also run your monitoring against
these like simulated networks so that
you can make sure um that it's all
functioning the way that you think it's
functioning um I think that's that's
pretty
key so in the final few minutes um
before I'll leave a few minutes uh we'll
have a few minutes for questions I just
wanted to show you the the open source
repository uh that I la that I released
a couple weeks ago with all the tools
that I use to run these for protocols um
so it includes like a Foundry and hard
hat setup for like validating different
like war game scenarios um on a local
Fork it contains a bunch of
configuration scripts for setting up
virtual test Nets with block explorers
to run these uh templates for the
tabletop exercise uh monitoring and Bot
Services which we forked from um an
optimism repository and also like a
stack for setting up like a Prometheus
grafana instance so like this is all
that I use when I run a when I run a war
game so this should be all uh that you
would need to potentially use if you
want to run one of these with your own
uh with your own
protocols so this is just a a brief look
at like what the tabletop script looks
like and so we'll be on a call usually
uh it's like an hour hour and a half
with like the whole with the whole
protocol team or with like if it's a
very large protocol maybe it's a subset
of the protocol uh a subset of the team
um but always cross functional and we
just talk through like in this exercise
we're going to um discuss various
scenarios that you'll need to detect and
respond to issues what we want to like
uh understand is typically like uh do
you know what the key dependency of your
protocol is and you know what would
happen if it fails is there sufficient
monitoring in infra to detect and
respond to these failures um do you
understand who has admin capabilities
within your within your protocol um are
there backups in place so these are the
this is almost an additional information
gathering step for us um so that we can
design the best uh live scenario and
then the questions that we ask in each
one of these scenarios is always um like
how did you discover this issue so if
we're looking at maybe an an L2 and
somebody submits some sort of fraudulent
uh withdrawal proof how did you find out
did you wait until somebody rate like uh
reached out on Twitter or Discord and
said hey it looks like you know XLT is
under attack right now or did you have
some internal system that would have
alerted you within seconds and so first
how did you find out do you know who to
gather do like does somebody become an
Incident Commander here do you have like
an on call team or is it like whichever
Engineers awake somewhere in the world
is the one that has to to deal with this
like is that uh how does that work do
you have a response plan for this if not
why not is it just too rare um and then
who took action and how and so all of
these questions are how we can
understand like how well positioned a
protocol is um to uh actually uh to
respond uh and then for the live drill
um the setup steps this is always the
part that takes me the longest but we
create the fork run the Bots brief the
team I'll send them some sort of like uh
atic message of like you know you're now
in like a very you're now in a very
realistic simulation where this really
bad thing's about to happen you have to
treat it as if like a everything was uh
as if it's all real life um but you
typically don't need to encourage that
like they'll go into stressful mode and
uh everybody wants to show that their
internal tooling really works really
well and so um uh and so yeah teams we
found always take it very uh very
seriously um I always love love to
observe kind of the especially just the
conversations between like Dev and comms
where it's like Communications prepar is
like a statement developers are like no
we can't say that at all or developers
want to say we're pausing the protocol
and Communications is like no we can't
say that like so that's always really
interesting uh to to observe how that
works uh we have um uh this is just a
view of like a tenderly virtual test net
so I found that this is like I used to
set up like an anvil fork and a block
Scout uh drop uh and a block Scout
Explorer um that would always take me
like two days um So lately I've been uh
using these tools but you can run these
Forks using any any tools but this is
like what it looks like when I set up a
a fork of main net um and then this is
like my grafana setup where um I'll have
like an in just a basic invariant
monitoring thing where um we're checking
for certain conditions like you know
more tokens were withdrawn from an escro
contract than we're actually locked up
for a user and why would that happen and
and how would we fix
it uh and then the uh we the team
actually has to submit the transactions
on the test net um we try not to we try
to uh do a couple things to keep safe
like change the chain ID of the test net
so that the um so that the actual like
response transactions couldn't actually
be replayed on Main net because we
wouldn't want to be able to like take a
pause transaction from our test net and
then actually pause a major protocol on
Main net and so we change the chain ID
um and then we also try not to uh make
them access like keys for safes and
stuff if they really don't need to and
so we'll kind of spoof the process of
uh of doing like a multi transaction
just in case like there's all sorts of
security procedures around accessing
these Keys um yeah so we try to try to
keep team safe doing that uh here's a
link to a few resources the first one is
uh actually related to an event that was
two days ago that we hosted uh here in
in Bangkok but I left it there because
we I think we'll be doing more of these
like Live Events where you can compete
with each other uh to try to white hat
rescue funds from protocols uh the
second is the GitHub repository uh we
have the security Alliance website where
you can uh sign up for a drill sponsored
by the security Alliance um and you can
also sign up for drills um on my
company's website Shield 3 uh if you're
on a an accelerated timeline uh or feel
free to chat with me if you're just like
hey I want to run war games for
protocols uh I'm happy to
share so thank you all right thank you
very much
Isaac appreciate it even though we
started late you finished well within
your time I really appreciate that and
I'm happy to say that our questions are
now working so let's turn our attention
to the screen I'm going to read out the
first question with the most votes is it
possible to have a Seal Team Member be
part of a protocol Guardians multii
somehow y yeah um a lot of us are on
some of these uh multisig or on security
councils we actually talked about this
recently like if seal itself should be
like members of security councils we
came to the conclusion I mean this might
change in the future that like us
individually can also be like on
security councils uh but seal itself is
not but like many many seal members are
open to being like on these uh uh on
guardian multisig and uh and stuff like
that so feel yeah just find me and we
can we can help all right thanks for the
answer thanks for the question next
question that got voted to the top do
you share threat intelligence data on
chain if no how or where do you share
threat intelligence data yes so the
security Alliance website should have a
link to the cal- ISAC which is the
information sharing tool um there's a
there's a a large amount of data that
actually is available publicly but then
also if you're like a wallet company or
an exchange where also um there's like
even more sensitive data that can be
shared in a more private way um so
that's the best place to look for like
uh for just data being shared across
teams it can be it there's they're even
starting to gather a lot of information
just about every time like a a North
Korean developer applies to a protocol
they try to just like profile it and
understand like how they're doing it
who's getting in just so that we can
like share data on like how teams are uh
how teams are being like trying to be
infiltrated thanks seal ISAC yeah thanks
for the answer great all right let's go
on we've got time for more questions
does the drill occur unexpectedly I.E
the team knows it'll be happening or but
doesn't know when uh I've gone back and
forth on this a few times initially it
was like we would give them an 8 Hour
window and then say like okay it's going
to happen at some point in this 8 Hour
window um but then I was just sitting
there like stressed waiting for it to
start so I always just started it right
at the beginning of the window um so I
don't know if like uh it might be fun to
do that I think at at some point we're
talking we're thinking about like can we
do like more surprise drills um on these
longer running test Nets where we like
submit a faulty or like we submit like a
a faulty proof to like an L2 rollup or
something like that just out of nowhere
um just to surprise a team um but yeah
typically they know that it's going to
happen on a certain date um but it I it
still maintains a pretty good amount of
realism right thank you very much
surprise you've been compromised anyway
uh let's move on to the next question I
think this is an important question it's
got quite a few votes as well what's the
most common way teams fall over I think
one of the most common things is like
having your internal tooling just not be
up toate um so it's very like maybe like
one developer got on a was very inspired
a year ago to make all of like these
really great internal tools but then
like the protocols changed a bit over
the last year and the internal tools
don't work so I think like maintaining
the internal tooling that you need need
for these scenarios is is very important
and dedicating engineering resources to
doing that so that like it's not like oh
I really hope this tool Works um when
you're actually in the incident
excellent answer fantastic we got a few
a little a little bit more time let's
try and do one or two more any way to
react to layer one protocol incidents uh
yes so I think that we want we're
thinking about some fun things to do on
the layer one like maybe related to um
like major protocol major like L1
protocol upgrades thinking about what
could happen if there's major chain
splits or Forks or bugs in a certain
consensus client but not another one
like what chaos could we simulate across
the whole ecosystem if upgrades perhaps
go really poorly or if there's bugs or
supply chain attacks into major like uh
validator code um of course like these
are things that shouldn't and probably
won't ever happen but it's fun to
practice uh and I think worthwhile to
practice uh the response excellent Isa
you got one minute for the last question
do you go through the worst case
scenarios in war games for example you
got hack money's lost what now do you go
to contact Law contact with law
enforcement uh typically I actually
that's a really good question which I
didn't touch on which is like we try not
to just make these like an instant
wrecked scenario because it's no fun if
we just say protocol was upgraded
there's a bug all the money's lost you
can't do anything like that's no fun for
anybody so we try to make it something
that's like recoverable that they could
actually respond to um but uh I think
yeah actually on the last one um uh
where we had like some money stolen they
were like okay now what we would how
would we actually try to like Chase down
this money that we got that was that was
uh was stolen and the answer is like
that's something you can go to seal 911
for uh because they have really good
like tracing people that can like say
okay now we're going to see where the
money's gone and try to recover it in
other ways um and so uh yeah if you get
hacked uh uh first fix the protocol but
then yeah we can also help you uh track
down the money excellent wow we finished
all the questions in record time let's
give Isaac a big round of applause well
done thank you so much Isaac well