[Applause]
so
[Music]
difficult so here's a little bit today
I'm just - morning this is not going to
be super technical talk I couldn't hold
up to github the shooting or super
support so the first thing I always have
to talk about generally is what is
security how does it work
why is it there how long has it been
there we'll talk a little bit about our
first main that security and did I will
refer to this is story time and my team
who are here are probably tired of
hearing the stories but if go time so I
or how we started thinking about these
things I'm talking a little bit about
some of the things that come up when
you're trying to manage to centralize
yet coordinating in response especially
when you have lots and lots of
stakeholders and
in our case you know Adam poor people
might people in Syria and then and also
we'll talk a little bit about how
governance is very uniquely positioned
to be able to help us when we're trying
to be more proactive and prepared when
responding to sue areas
so just to front just to talk a little
bit about what security is I will skip
over about ten thousand years of
examples of human seems horrible and
managing keys there's some great stories
from ancient Rome about how you if you
had a key would have invented a piece of
jewelry
let's get that today but security is a
scary face it's the seven principles
that are extremely important to player
and implement correctly and the right
places to make sure that you are
protecting and defending what's
important to you your communities and
the conditions right for one there
especially public activity there's
opportunity rigor minimization and the
compartment ation is extremely important
making sure that you're not putting all
of your most valuable things in one
place but you're distributing them
personality and what we do a little
extra things but these principles
especially if you help improve your
thinking about practice in the security
space there's some other aspects of
security that are incredibly difficult
technical piece of us get very excited
about it but there's also a human thing
because humans are very very difficult
to Courtney and organize and manage when
we have very strong intense emotions
like fear which is kind of a big deal
when you're in the middle of a security
emergency and you think this thing you
might have done is zero day there's also
the question of being proactive about
security meaning that you try to stop
bad things from happening and you
identify risk early and you step in and
you take time to mitigate problems
before they really become emergencies
and then there's reactive security
incident response is an area where you
can be proactive and
run books and planning for emergencies
that will naturally come up in the
course of running the watching but
reactive because you basically have to
detect that this badness has happened
where this incident is occurring and
then spring into action
there's also a very very deep line
between offensive security and defensive
security in the security space
I am hardcore blue team you name I wanna
build defenses I'm going to protect
people I want to protect a news I want
to care for and maintain things a lot
but offensive security can be very
important are in space freighter that
much work and attacking things and going
after them and deeply understanding
there are they break can inform how do
you run your defense some of the ways
that you see security happen in planning
organizations tend to become security
programs depending on what your
organization does you can have product
security could have application security
or infrastructure security they can all
be in one room they would be separate
programs it really depends on how
critical each of those things
and the operation of your network but
then there are also things that we don't
really cover as much that are just as
important as the technical ones
operational security is one of the
things that watching space and very
significantly behind where it needs to
be at the airport penguin assessments
are difficult I run security on it's in
the last two years okay well it would be
important to make sure that you have
someone independently evaluating your
codes all of the business negotiation
that goes online and scoping out and on
it and defining the testing you need to
do these are things that are hard they
are immensely difficult to do if you've
never heard before like how to think
about them in a comprehensive and
programmatic way to make sure that that
bought it you're releasing to the public
actually does attest to the things that
you're claiming it does and we've chosen
the correct remediation steps to close
bugs or issues that were surfaced within
me on it and then there's education
that's also extremely difficult if
you're in a smaller company an incident
response which is the one for me that
has been the most interesting and
watching space because other people in
conventional technology but I also get
challenges that they maybe wouldn't know
how to solve even if we told them which
is kind of exciting and I personally am
very competitive
with security like I want us as a space
to win I want to I want to be like
looking
while everyone is making stupid
blockchain jokes so this incident
response areas thanks for thanking me a
huge opportunity to shine Wow and then
there's just a couple of other things
about security that I've really learned
and taken to hearts and practitioner
security is very much a shared
responsibility there's only so much I
can do is core maintainer or a core
developer to make my community secure
there's lots of stuff that I need them
to do and to be set up to do to help so
that we're able to quickly resolve
issues and build immunity and security
is definitely much much more than
funding all of the bugs in code the next
time I'm in it tells me it's my job to
find all of the bugs I'm going to get a
level of staff see that it's relatively
unmatched for me and give them a lecture
about stopping the bugs from definitely
more than anything else though
Security's really don't enable in and
one thing that's really important we all
know Security's never going to be a late
100 percent guaranteed but it's more
successful when we make it more
expensive for an attacker to launch a
successful attack all we have to do will
be one is where the attackers are and
make them jump through hoops and lots of
money and have to do things that are not
male individual scalable I'm too
exhausting and maybe if not make them go
away
make them leave us alone for a while
there are things that we can do in order
to plan and prepare for incident
that other organizations and other
technologies can but more than anything
else it's really important to recognize
the decentralization might make parts of
security more complex it doesn't require
us to reinvent the wheel if we are more
busy reinventing the wheel than anything
else we're gonna miss a lot of
opportunities to get very small very
nuanced very important things right so
this is the story of time once upon a
time I made a fatal error of accepting a
keynote in Australia and so I was like
17 and 18 time zones away from my core
development team now I made it a point
to make sure I still showed up for my
meetings and did all the stuff I was
supposed to do but in the middle of this
week that I'm in Australia and like I'm
getting late chewed up by the ocean and
spit right back out at the beach I have
a high severity security incident
planned in my security app in once so I
have been planning for this for a year
and a half at this point I'm like our
parts go time
all the stuff that I do it works
somebody give me a bug oh my god I'm so
excited and a few of my colleagues but
this is not exciting what is wrong with
you but how did you come in and it
looked at it and I realized uh-oh this
is a problem basically the issue Croker
security model in that it would allow
for someone to be able to find bond
their stink and not have to wait three
weeks which is something that is
built-in
to our perfect to economic layer and it
under my practical II anything that you
would want to do on the network on not
later so what we realized was to fix
this we were going to have to do an
emergency one er ability coordination
exercise meaning you're going to have to
patch the code we were going to have to
develop some willing to investigate
exploitation because if exploitation
happened
I better be ready to show people where
it happened why it happened is how it
happened and additionally we had to
think through notification so my office
wall at home has all of these sticky
notes on it it looks like kind of looks
like it would be received earring up
fine basically what my office walls does
for me is best it's got the process
mapped out just document to the three or
four different places and I because they
already planned on having to do this
knew what was next
[Music]
how may I have a moment but what we knew
we wanted in the case of any security
emergency was if we wanted to be able to
be the best dependency that we could
mean to people who rely on our code we
wanted to make sure that we reduce harm
we wanted to make sure that we've
provided visibility
transparency and the way that we
specifically do that case is very
comprehensive incident report post that
I put up within a week or so of
something bad happening that requires
emergency exercises a couple of things
we knew we were going to come up against
we knew that we time was really really
important here we are monitoring for
exploitation we were able to see where
the person who reported the
vulnerability had been exploiting the
buyer just to check and see if it works
they did this in a way that didn't cause
harm but we also knew if one person knew
about the spot it was likely other
people could have Co discovered it and
just didn't say anything because we knew
time is of the essence here it was
especially important for us to make sure
that our communication and coordination
was fast and it was good and it was
clear and when we were asking people are
recommending that our validators do
specific things we're not telling them
what to do but we specifically aren't
making sure everything we say is
actionable and clear one thing that we
committed to very early on was making
sure that we pre notified people who are
who rely on our code meaning instead of
just dropping a security bug into
Tuesday Monday I send our message
through our forum through a mailing list
or through every nail any other channel
that we need to send it through to say
hey heads up you're probably going to
need to patch we've got a bug here and
if you're high or critical disparity but
the patch will land around this time
part of the reason we do that is because
we have an extremely global community
and if we have everything decided on
Pacific Standard Time
always waking up at 2:00 in the morning
and somebody's always you know having to
jump away from the dinner table and go
deal with some security patch that's
incredibly important for them to apply
so in addition to this we knew we were
going to need a little bit of discretion
we were gonna have to handle this
privately we can't release an exploit to
the entire world look what we found
because badness happens when you do that
so we realize we have to quickly stand
up a safe space where we can talk to our
network operators in this missive
encased we already know the topology of
our network pretty well so we created
telegram room which was a moment that we
want to hide a new table but we have new
opportunities to do things much better
than getting 100 people in a telegram
room and talking about explanation of
scenarios and security flaws one thing
that was particularly rough in this
specific spot though when we were trying
to clean notify people but we have a
critical emergency in the hopper we were
trying to go and make sure that we reach
out to all of these different projects
and all of these different companies for
environment are toast
more than anything else the most
frustrating thing but there were no
operational practically no operational
Security Act email inboxes when a human
behind them there was no way for
companies to get a straight security
message from us and a lot of cases when
we were trying to proactively reach out
to exchanges they told us to file
customer support tickets which is not
available and it put me in a position
where I had to hope that when we have
emergency part forking happening in 24
hours
some exchanges going to pay less
attention in public process or in
advisory from support all the way to
engineering their security in enough
time for them to act but leave and not
be harmed because realistically the
moment attach plans in your code
Facebook and your github repo not really
trivial if you're good at rehearsing
hearing to figure out what changed and
to find and exploit and then turn them
around and automate attacks against
anyone who hasn't passed yet
it's basically relatively simple
reconnaissance and so if you happen to
know anyone in a position at an exchange
to set up a security act if it company
who relies on blockchain code if you're
a developer who you know works with a
bunch of different projects Holy's
security a Kitfox but some form of
obnoxious alerting so that instead of
just having that be a reactant place we
can at least quickly good to you and
tell me what's up so we have this giant
perspective and basically what happened
as part of the retrospective what's this
I had to figure out a bunch of ways to
make the process that we had to go
through this first time much much much
better and I will be I'll make sure I
post these slides online and it links to
all of the things that I've screen right
here but basically what we realize this
is we built a public watching and it's
probably time to figure out how and
where the firefighters and our emergency
services so
that was interesting about me being an
Australian Spence I happen to be
speaking at a conference that was their
national conference or their computer
emergency response teams which I wanted
to go there because we wanted to learn a
lot about this public-private academic
partnership of a bunch of organizations
who get an emergency can pop up and they
immediately help organizations of any
size of any kind
all of these problems they come when big
security problems happen
this particular sir even though hard to
eat alone I think all had some fantastic
stories about how they helped their
members and how they were able to help
civic governments and university is
basically patched and quickly identified
what we have public stuff it's important
so what we realized is we should
probably stand this up we can leave it
someone decentralized you probably need
some folks from our management probably
need for developers we definitely have
to give our network operators as a table
you don't just have to have everyone
around for an emergency we can also use
this group as a way to make everyone's
murder about insecurity instead of being
in a position where we're the ones
doling out advisories and maturity
models and all kinds of tips for how to
run your AWS staff or your GP
put out proactive educational
information to help mature the entire
network so here's what we came up with
as potential core refinement it's
probably best when you are responding to
an incident and not have to deal with
like 107 people at once it gets busy
it's usually better than five your end
as less than 10 I was thinking about fun
outfits to plan for when we have to
respond to emergencies spacesuits make
it really hurt too tight so I killed
that idea but that being said we
basically decided you know what we are
going to take this group together we
need to make sure that we've got people
who are from different parts of the
world that aren't just North America and
we probably need to make sure that it's
distributed across multiple kinds of
stakeholders so we have valid leaders
from tons of stake and we have small
validator community aren't as mature on
the security front and who need a little
bit more support from us to be doing the
best big canvas particular mix those who
realize we probably should outline
expectations for service so initially we
were thinking maybe running here that we
would have people as a part of our
certain which we are working on coding
now but we wanted to make sure that
we're keeping them accountable
especially in the areas of potentiality
and respond if direct or thirteen
and you cut access to the internal 'ti
information I don't have to be worried
that you realized you couldn't use some
learner ability that you have access to
to exploit you know a validator in a
single environment post too much and
telegram channels so there's a couple
things we still like to do to make this
successful we have to encourage
everybody to start the security act
email address so we kind send out our
signal flare when bad things happen we
are working on adding a security
perimeter to the cosmos SDK basically a
place to state your security I email
address where I can query the network
daily for our time note and have a
Security Response list instead of going
through like 110 websites and any given
time and hoping and by the way to get to
human and also realized we should
probably go ahead and start working on a
model that allows us to elect
individuals to be represented in this
instead of just picking on a bunch of
people I know that's probably not the
best way to make this particular group
work especially because of the need for
trucks what's really interesting about
this is we internally have been playing
this on the movie called inquanok
receipt and we've been thinking about
how floor to garments can be a tool to
help us accomplish things in a wider
context but I'm pretty proud that I got
to work with a copy of our protocol
apparently this is really the first
application of that liquid democracy
principle that we have been talking
about over the past
when a delivery to a new module is we're
making sure that we are writing the code
that will govern the ability to turn out
transfers if there's an emergency we
have ten people and we need to go
interesting but there's one particular
thing where basically they have a
complex ceremony the other goal if I
flip the room key among seven
that's the example that I'm looking to
engineer so finally really quick plot
chains and forward mediation tools more
than anything else
nine-tenths of security is making sure
that you're using the right tool to get
the job done and happen to think on the
incident response part we could build
the security incident response that we
want to be in the world that being said
I wanna make sure that I have a shout
out to my core developers who have been
helping make this idea come to life
especially rifle into the cloud he is
one of the first people to get excited
about liquid democracy and whose natural
state is just one so thank you very much
if I have any questions and rant over a
little bit but happy to take them in the
hallway
[Applause]
you