foreign
[Music]
we love you all right guys hey everyone
so uh we're here to talk uh what web3
projects need to know about web 2 cyber
security uh and really I think this is
going to be a great kind of follow-on
talk to the bridge security framework
talk that Patrick just gave
um this is kind of the compliment right
so he went over really some of the smart
contract and protocol layer aspects of
bridge security uh unfortunately a lot
of bridges have been hacked though not
from you know the hardcore cryptography
but from Basics from cyber security
Basics okay
the cyber security Basics matter
they matter because they have
geopolitical significance
so before we get started on the panel
take a look on the screen there was a
great article by CNET that came out a
couple days ago
and it really talked about how Lazarus
Lazarus Group which is affiliated with
North Korea is seeking to infiltrate
this ecosystem by setting up front
companies and then having applicants job
applicants apply to these front
companies and then the engineers get
fished right that's actually how the
Ronin Bridge attack happened okay so the
fundamentals matter and it doesn't just
impact people in this room the people at
this conference it impacts geopolitics
because then it funds things like North
Korea's new program or you know
transnational criminal organizations and
stuff of that nature
Okay so
what do we need to take away from this
panel so a lot of fundamentals from the
last two to three decades of web 2 cyber
security
there's Cloud security risks that we
have to understand and mitigate
a couple with that there's front-end
security risks right there's operational
security risks there's also corporate
security risks and then obviously big
topic with some of these Bridge hacks
has been Key Management
endpoint detection and response and
again management of individual
individual nodes and validators
um
doesn't mean we need to Discount smart
contract security risk what it means
though
is we need to shift the security
conversation in this space from very
siled you know individual audits to a
holistic end-to-end scope of the
Project's entire architecture right
and not just their technical
architecture but the people architecture
and that's something that you know we
have some newer entrepreneurs in the
space and that's something that as we
kind of get some Battle Scars as an
ecosystem I think we're going to learn
time and time again
okay so who are the panelists
um we'll start left to right we got uh I
guess we'll start over there uh Naseem
from a16z crypto we have Taylor from
from metamask
we have Corey from openc
special guest Hudson from crypto Twitter
and then another special guest Moody
from also from crypto Twitter oops no
polygon polygon says cool
all right guys so we'll start
um with kind of a basic question and
I'll ask it to NAS first and then the
rest of the panel now as if you could
kind of describe
you know from the context of just
normal web3 or like a you know
representative web project in the
ecosystem
what are the web two parts of other Tech
stack and then what are the web three
parts of the of the tech stack and then
what are the security relevant parts of
of each yeah I actually feel like it's a
it's a fairly uh long answer that is
required for that but you know we we do
spend kind of quite a bit of time going
from the early days of the company up
until kind of like the ongoing updates
to
um you know decentralized protocols
I would say that
um you know it kind of like starts with
the people it's essentially just like a
set of people who just want to you know
design a component uh build it uh deploy
it make it available and just like
maintain it over time so there is kind
of like the everything that it that it
encompasses so that means you know you
essentially start introducing devices to
build the components on you start in
involving kind of like the entire kind
of like
you know secure
software development lifecycle that is
relevant to every single person kind of
like Building Technology in general
um and I'm trying to cover cover like
anything that is you know kind of common
to what to enter through here
um
so you're going to have like this part
then you know this these people need to
build an organization right and like
with that column you know kind of like
corporate security
um as you as you mentioned earlier kind
of like anything you know from like a
endpoint uh you know
security standpoint device security and
and so on
I would say that like the appsec is also
like very uh very similar you know
always kind of like
you know analysis of your code as you
kind of like develop it and and
obviously you want to iterate uh pretty
fast so you're gonna build your CI you
were going to integrate you're gonna run
you know security testing and bond
scanning and all the the stuff that we
all know in love
um doesn't really matter whether you're
building for web 2003 yet at this point
right
um
I think that yeah like in my in my mind
everything that that spans over you know
network security
um
essentially like network security uh
corporate security appsec security
engineering
key management and by the way like the
key management is the part that is like
highly overlooked like a lot of people
think like okay it's okay I can deal
with this later you know I'm just gonna
have like you know like a single key on
like a either letter or like some
you know a fairly not robust you know
key key storage and then kind of like
push it out to uh to later but you know
Key Management is like a fairly complex
thing that like we have to deal with in
in web 2.
um yeah I would think I would say that
like these are probably like the the the
majority of the things that we see and
obviously you know comes with like you
know physical security I think that like
it kind of like ties into you know key
management and the security of the
people as well
um because it's not just about the
phones it's also on that note we've
heard stories of people walking around
uh certain places with you know crypto
swag and exactly having their phones out
and so on so that probably ties into
this exactly as well right um yeah and
we had like quite a few conversations
actually with uh you know various teams
of our portfolio that were coming here
and you know
because decentralization is a spectrum
and like you try to kind of like start
obviously centralized with like just
very few people and kind of like you
know centralize over time uh there is
like a point in time and we're probably
at this point in time like right now
where uh any person that kind of uh gets
hacked attacked like physically can kind
of like result in either assets getting
in trouble or even like reputational you
know damages that can be created as well
or potentially physical access physical
actions company or exact protocol and so
I think this but yeah the repetitional
damage is actually something that like
is is highly overlooked as well it's not
just about your protocol it's like you
know everything that comes around that
do you guys have your stuff squared away
or not effectively yeah pretty much
anyone else on the panel
um good
so
so when I think about web2 versus web3
security especially right now how early
we are with web3 and ethereum and
blockchain there is going to be a need
for web to
stuff in your stack uh not even consumer
facing necessarily but like who here
actually like doesn't run their project
on GitHub and uses something like matter
most instead of Discord or slack or
other non-hosted you're awesome whoever
that one guy in the back is uh but yeah
uh yeah good for you but if you're
unless you're doing that like you really
need to be very very careful about where
your data is going and keep up with that
something else is that I think is
overlooked a lot especially when with
web3 and how open it is and how open it
is to contribute are two things uh
watching your GitHub uh to make sure
that the you know PRS that are being put
in uh are actually from your team
members that you trust and that you've
vetted uh Sushi swap had a bad incident
where someone joined their join their
team and then put something in that like
I think stole keys or something on their
web page Insider thread yeah Insider
threat basically and then secondly
um this is something I worked at the
ethereum foundation as the org security
lead for three years and and something I
learned especially as I was leaving and
hadn't reflected back on there were just
a handful maybe two people in the
organization that had Master access to
just about everything and that's
incredibly bad uh it's even it was even
worse earlier so just bear with me yeah
like what happens if those people get
run over by a bus right not even that
they could literally take down the
ethereum website post on the ethereum
official Twitter post on the YouTube
pretend that vitalik died crashed the
market like there's there's like a lot
of incentive there and there's a montage
everyone has a price right yeah yeah so
the thing is if it wasn't that we just
really liked and trusted the people
doing it like we got lucky but really
there should be different ways and
there's you can look up ways that like
people have theorized how to do this
best for years separate the people who
have access to the most critical parts
of your organization and have an audit
Trail have like a foolproof audit Trail
they cannot wipe uh and so yeah that's
that's kind of what I think of because
it's like web 2 security seemingly has
mastered this in some levels but web3
like forgot about it so that's what I
think
oh so on the sushi incident first of all
glad we are not talking about the horse
incident but off limits yeah it happened
a while back
um so what happened actually was one of
the contractors had push access to the
code base uh there was no PR review
approval process and it just got uh they
like got angry with something post
malicious code nobody noticed it got
pushed out and the key takeaway from
that is you should always manage access
people should only have access to things
that they really need in a production
environment no single person should have
Push access uh it should always be
approval based have at least one or two
reviews required and expanding that
further uh really like just minimize
access to only things people really need
for example like uh access should not be
considered as uh like a pride or
something so uh co-founders for example
or founders of a company shouldn't
really have access to critical things
like your emails servers or whatever
unless they are actually doing something
of it just because they have a higher
precedence or whatever doesn't mean they
should have access the thing is like
people in People Are People no matter
what people will get compromised one way
or another so you have to have that
second layer of security where even if
someone gets compromised the impact is
minimum so a practical example in the
Iranian case uh the bridge hack
their employee got compromised
that that I can understand like if
someone a phishing attack he was like it
was a very well orchestrated attack so
uh I'll give it to him like okay sure he
got caught but the issue there main
issue was that that one single employee
had access to everything like that one
single employees access could have
trained 600 million which should not
have happened in production that was
that that was the key problem in that
heart not that deficient happened that
is gonna happen you just have to protect
from the impacts
cool
go ahead
all right um so I'll just add that um I
think uh if you want to look at the
different uh what's missing in the web
three on versus web 2 is just look at
the web 2 security Award chart uh you
know you have your instant response
teams you have your uh you have a
corpsec group you have a production set
group you have an app set group you have
your uh secured reviewers you have your
vendor security audits you have your uh
you know all those roles are there
because that's how you achieve security
you just have Samsung yeah yeah well not
right now but Sam doesn't see it yeah
yeah Sam doesn't scale though right yeah
yeah yeah and I and so I think like if
you want to know what you're missing
like look at the Delta between your org
chart and you know any any
well-established web 2 company
so that's an interesting question then
so
in
the various life's lifestyle like life
phases of a crypto project what needs to
be internal
versus external right so obviously
everyone goes out gets an audit it's
kind of a commonplace thing
um I still have audits right I'm the
co-founder of an auditing Marketplace so
we love audits but it's not the the
pansia so I guess core to you first and
the rest of the panel like when do you
need to hire an internal Sizzle uh we're
starting to see later stage projects
higher Sizzle but is this something
where you know you raise Venture money
and then on your use of funds like
employee three is an internal ciso
uh
well uh I have a bias that yes this
place maybe playing two actually but uh
you know I think
I think an under underappreciated
property of security is that it's uh it
is not something that's added on later
it is an emergent property of your
design and so if you don't start from
the beginning thinking about security
you're not going to have security at the
end uh and I think that like so yes I
would I would argue that you should have
them early
sorry yes I would like 100 agree with
that because
um
security is really about like the whole
system and it's like the social system
it's the people system it's it's how you
interact with your community uh it's how
you hire it's who you hire it's how you
vet them
um it literally touches like everything
that you do
um and if any one piece is like not
secure that's an entry point and the
thing about specifically like these
crazy hacking groups that are coming out
of North Korea and other locations is
that uh let's say that they just get
access to your slack right so it's like
your internal slack you got like 20
employees right they will then use that
just that access and they will sit there
for like literally months and they will
read everything that you do and they
will understand how you operate not
creepy At All by the way it's super
crazy uh and they will use the
information they glean there to then
like escalate to fish you Spearfish you
via email or Spirit issue via telegram
or LinkedIn DMS or whatever they
whatever you use they're gonna get you
and and they just over the course of
like months and even years they'll just
like keep going and going going
um and so it's like these little things
that we don't take seriously because you
don't have anyone on your team early
that's thinking about security that is
the problem right and the solution is
not to get a freaking security audit I'm
sorry I love security audience no no to
be clear like it's a it's a secondary
check yeah that's for just a secondary
check yeah audits are not going to solve
your internal security problems for you
projects have to solve that yeah and if
you especially if you hire like a good
security person early on they will uh
indoctrinate your team as it grows and
they will uh design your team right your
org chart your organization your
products how you work right they will
help design that from the get-go and
therefore every layer of your sack as it
gets bigger
um as the responsibility gets spread out
across more people as you scale up as
the financial incentives get greater
you're just gonna be in a way better
position and I I really cannot emphasize
enough like what Corey said like you
cannot add Security on after like it's
not a thing you can't be like oh yeah
we'll just be lazy now and then we'll
like fix it later that's not how it
works like security is a core part of
your entire organizational design your
product design and your system design
make this a little spicy so oh let's go
I think like these guys are just
securing their jobs right now but in
terms of I mean to be fair we all are
but you know yeah but like just being
practical if you are just starting out I
don't think a second employee needs to
be the security guy but what needs to
happen is your first developer needs to
have a security mindset your first it
hire that will probably happen when you
are 10 to 25 people that needs to have
some security background how do you
qualify that like the security mindset
when you're high so
that's a great question yes it's not
something I can like uh like Define
qualities or something but if you look
for me if I look at someone's code the
way they are writing it the way they are
documenting it I can tell like it's just
intuitive that this guy is thinking
about security uh if you just talk to
someone like look uh ask them their
priorities uh see how well tested their
code is uh even the code syntax and
everything it will tell you uh how
serious they are about security uh so
yeah your first developer have someone
with a security mindset uh let them
train others then your it guy probably
will be your next time get someone with
price security experience but does it
need to be a dedicated security people
and then will you scale Beyond maybe 20
people that's when you should start
looking into a dedicated security person
for your own before that just have
internal people with security mindset
plus hiring or not hiring but like just
Contracting external service companies
to provide you with security help
external companies they don't only do
audits they will help you with internal
security as well because the thing is if
your org is below 20 people the security
guy won't really be utilized fully
unless you are someone like FTX with
like 10 uh 10x Engineers that's pushing
the whole code out then it's a different
story but talking about normal logs yeah
security people can come in a bit later
maybe the 20 metric is there for
projects at the application layer yes
okay obviously l2s and I just want to
chime in on that that the point that you
made about like
um the security mindset like what is the
security mindset
um I think one reason why web3 sort of
struggles with this security mindset a
lot of times is that we are like so
generally optimistic and like Visionary
and forward thinking and we are thinking
about the potential and we are like
thinking that nothing is impossible and
so so much of our time is spent in this
sort of idealistic future world that
we're trying to build and that's
necessary like we have to be that in
order to build these crazy things that
we're building
uh however
um like I would say that I have a
security mindset but I would also say
that I don't spend most of my time
thinking about like the potential of the
future I spend a majority of my time
thinking about like how we're gonna
it up
so that's like the security mindset on
that note how do you ensure compliance
then right I think it's something we
talked about before is like
getting people to actually do the
fundamentals even simple things like MFA
being enabled right or
um Corey would talk about this you know
endpoint detection and response actually
being used and if people aren't using
that then you kick them off the network
how do you how do you get people to
actually do the basics so like when we
were starting up right like and I I Grew
From Me and My co-founder to 20 and now
we've got uh like sort of over 100 on
the core team uh we got like there's a
thousand at consensus right now
um it's a when we had like two people
three people and started bringing people
on the thing that we did from the gecko
was we uh we had good documentation
about like what was required and what
was expected of our employees uh it
talked about like what they needed to do
and it need they needed to do it across
like all of their accounts like there's
no personal professional split in crypto
so like you have your personal Twitter
like that needs to be security if you're
gonna be a part of our company
um the other thing is a lot of these
services like GitHub you can force like
if you want to be a contributor to my
repo like you have to have your MFA on
uh we bought everyone uh multiple like
all the Little Hardware doodads right
everyone has a ledger everyone has a
tresler everyone has uh all all of them
the Google ones the the
the yeah the Yuba keys and the Google
one the Titan one
that one was that one was pretty slick
uh still go back to my ubiky all the
time though
um and these things right from the
get-go like like the first employees
no matter what
um then as you grow you like your
procedures are sort of in place
um but then your early employees also
like establish this like floor of like
this is what's expected right and and
that's the culture like that's how you
establish the culture like nobody is
going to step out of line one is already
there right and and you call each other
out like when you see someone being a
little bit lazy when you see someone
doing something that's a little bit you
know even the founders right I mean it
starts oh it wonders honestly I feel
like the like the the First Security
people that need to be hired are like
literally the founders because like I
mean they need to have like this
adversarial mindset like from the get-go
as Madrid was saying it's like
I feel like almost like web 2 versus web
3 is kind of like the wrong kind of like
comparison because like we're really and
just the the business of um security
critical software we had this
conversation yesterday with Corey it's
like you know it's like everyone here is
writing code like whenever we you know
unchain code is essentially like ring
zero in a kernel like there is no safety
net you know besides like whatever
you're gonna put in place essentially no
one is going to do essentially like the
security for you and so it's just like
coming with this adversarial mindset as
a founder and just like thinking through
like everything that is going to go
wrong just like I was saying is like the
most important thing so probably like
starts from the top the the culture and
the DNA of the company needs to be built
around that like there is no way for
your company to be successful if they
don't have that because like the the
Founders need to kind of be behind every
single one of these decisions where like
obviously security always comes at a
cost of other things right like it's
always the the security it doesn't live
kind of like on its own right like in a
corner it actually always is a balance
between you know convenience or like you
know product features and like user
experience versus you know the safety of
of doing these actions so I think that
like just uh just having the founders uh
kind of like come with this mindset like
right away is the most important thing
is this something that VCS in their due
diligence process should include like
asking the founders about their views on
security
um I think that like we don't want to be
uh too prescriptive about like the
mindset that people have before uh
before initially like investment but
it's more like as soon as like we meet
with them we kind of like do the rundown
of everything and we kind of like
provide you know insight into all the
threats that they're exposed to and kind
of like try to really if they don't have
it already like this mindset uh just
make sure that like after a discount do
the first conversation they're already
kind of like think about like everything
that that is going to go wrong because
it's only a matter of time right you
were saying it's like it's a matter of
price and time essentially so that's
that's like how we how we approach this
interesting kind of switching from the
project perspective to the user
perspective you brought this up before
what the projects need to do to educate
end users on their security hygiene
exactly so that's one thing this
ecosystem needs to understand that uh at
the end of the day security is really
about users even if your protocol is
secure but nobody knows how to properly
use it and everyone just keeps getting
compromised then you have failed like
you create uh like you use the public
private key architecture but you never
tell anyone how to secure their private
key everyone just copy paste that on
Google everyone gets compromised you
have failed your mission so user
education is super important metamask
here is doing a great job at it but we
need a lot more of it and to get
yeah and to get that like we need to
really understand like in this space we
need to understand one thing that it is
uh not always about getting the uh like
getting your mindset uh very crypto
focused or whatnot you can be a crypto
Anarchist but you have to be practical
in the sense stop telling like 60 or 60
year old grandmas to manage their own
keys
they are much better off using a
custodial Service uh not your keys not
your coins everyone's favorite logo but
if you don't know how to manage your
private Keys it doesn't matter if your
key is yours or not it will just get
compromised so unless you know what
you're doing it's often better to just
go with a service product just a
custodial product uh rather than trying
to learn what you don't know uh this is
why even in traditional Finance like
earlier in the days you saw like bonds
and everything were paper based but
eventually everything got digitized and
everyone just uses a custodian nobody
really ever buys shares on anything
directly anymore wherever you go you are
buying them through a platform through a
custodial platform because traffic has
understand that it's
it's not ex like if you want to scale
you can't ask everyone to just manage
their own keys or secure their whole
things you have to make it much easier
but what we in web 3 what we need to do
is give users the choice so if someone
is tech savvy enough if they want to run
their own node if they want to manage
their own Keys uh they should be able to
do that but we shouldn't be like shaming
others for using infuro or other
products or like using custodial
products we just have to change that
mindset and educate people on both ends
I uh uh so one thing I would add to uh
the ux which 100 it's it's ux is such a
big uh part of uh security and helping
people do the right thing uh but I think
an unappreciated part of it ux is um uh
is that it's actually a um you know it's
a team game it's it's every one of us in
this room every product every project uh
that works on web3 uh the uh the the ux
is not just how good your ux is it's how
consistent is your ux with everyone else
uh you know the way users uh the way
they learn what is safe what is unsafe
is consistency uh when they uh otherwise
they have no prayer of going on uh to a
website and knowing if this is going to
scam them or not because they're gonna
have a whole new ux and every single
time and maybe this website just does a
little bit differently and there's no
way of telling uh just you know using
your
um you know your lizard brain on in fact
right when you're on that website uh
what your what you should or shouldn't
be doing
um you know I um
uh my favorite uh example of this is uh
logging in uh you know on a d app uh you
connect your wallet and you know some
websites change and like start start
treating giving you data about like
what's associated with your wallet uh
some don't yet or they'll immediately
challenge you some will let you click
around and then you then challenge you
um and you don't you don't have this
concept of like am I authenticated or am
I not like how like where where is that
what is it what is expected of me and
all of a sudden you click something you
just pop up asking you to sign something
and you know who knows what that
something is hopefully it's science
ethereum but you know
um and
you know compare that Tool uh passwords
uh you know users have three decades of
experience with passwords and knowing
how to log into a website it's consider
every website's exactly the same uh they
they know exactly what the experience is
they know where they know what
authenticated state is versus an
unsending estate like that is you you
we're con we're in Ground Zero with user
education and it's not just how good
your product is it's how good we can be
as an ecosystem at being consistent and
showing the same experience over and
over again teach users the right moments
um and I think that's something we
really need to internalize and start
pushing more as standards
yeah I was just like I was exactly
actually what I was gonna say but I was
also going to show Taylor's project
because they might not uh if you want to
look at good user documentation and
education around every little thing go
to my cryptos facts and their like
documentation it's so good because it
like will walk you through like the
mempool it'll walk you through what a
private key is and all these other
websites that are built on ethereum
assume prior D5 knowledge assume prior
Key Management knowledge and if you're
really building for people outside of
the ecosystem and including other crypto
ecosystems that have dissimilar key
structures and things like that you
should really spend just a little bit of
time invest in a tech Rider and or even
just point to other people's pages that
give that education for the users and
like if that sign like sign this message
box pops up I mean there's so many
people who don't realize that doesn't
cost money that is not a transaction
that is signing a message from your
public from your keys keep here so like
like we're just so far behind in that
because the protocol layer can't be the
ones to teach that users aren't running
CLI Geth and learning that from scratch
anymore that's where I learned it but
that was way long ago where it's very
different now uh so yeah just look at
really good documentation and try to
emulate that in your project
yeah I want to emphasize like
the amount of docs that I've written but
uh it's worth it because
um you will be surprised at like how
much people want to learn and so like
there's sort of like this track where
like we can make the ux better and we
could give users more choices that
better serve them like we should
definitely do all of those things but if
something is like the current way of
doing it and it's not clear and it's not
obvious take that opportunity to tell
the user just like literally tell them
don't treat them like they're idiots
right treat them like they're smart
capable adults and tell them what's
happening because they will carry that
knowledge and then they will actually
pass that knowledge along to other
people and it's uh it's quite remarkable
you would think like the docs I've
written like what what what comes with
that nobody reads docs like it's weird
when my words like show up in other
people's like Tweets in other people's
documents right people reference myself
like I'm like wow there's people really
reading this
um it's it's amazing I also want to say
like back to what Corey was talking
about
we do need to work together better uh
I'm told nadav is in the audience
love you nadab nadab called us out
recently like now we're getting spicy
finally Open Sea called metamaskow
because there is so much use or loss
happening right now
um and a lot of the user loss is
actually literally with users of openc
who are using metamask and we should be
working together better to uh make sure
that every single thing that we're doing
and showing the users is uh way clearer
and we should be in lockstep so that
when openc releases a new sighting
mechanism or a new way of handling
something metamask can actually like you
know not show gibberish
literally not show gibberish please
anything but gibberish
um and so we have some like exciting
things in the works that like hopefully
we'll uh we'll get into production soon
that that will ultimately serve not just
metamask users not just openc users
right it'll serve everyone in the
ecosystem but specifically right now it
will prevent the loss that we're seeing
and we are seeing an immense amount of
loss what's the way to solve it is it
just point to point interactions between
like the metamask and Open Season do we
need like industry working groups I know
I know that's a can of worms a lot of
people don't like that we have industry
working groups they fall off time and
time again because the new shiny thing
comes out but now tokenizer man just
like that's just so meditative
economically tokenized the incentives to
economically there's the incentives
repositories of this layered protocol so
but yeah no I I you might be able to
answer a better tail but all the groups
I've seen some of them have worked
really great some of them floundered
because they're the teams especially the
ones starting out but they have a big
impact are small they don't have the
time to commit to going to a bi-weekly
Zoom meeting to look into to that and
then there's a lot of competing um
what am I trying to say there was like
for a while there was even competing ERC
standards you know a few years ago up
until now with even the basics of how
we're doing messaging and signing of
transactions across a theory that's a
very interesting history to look back on
but uh to fully answer it I guess yeah
just pretty much direct communication
with the teams like if something's
happening just telegram them and then
don't just leave them for a few days
like literally get back to them soon
especially if it's like money loss
security related that kind of that's
where you need to have a dedicated
person who's doing at least security
infra or ciso or whoever who could be
like oh let me grab my lead devil think
this is in 10 minutes it all comes back
to the point of someone on the team
hopefully from the founders side to
NASA's Point earlier has to own security
right yeah one thing yeah oh good okay
one thing I'd like to add is kind of
like on this conversation I think that
like we should learn from a lot of the
the great kind of like uh ux efficient
security that like that has been put out
I think that like apple is probably like
top
you know like the the kind of like top
company in in this space that like has
been not inspections like in the tech
space in general uh in terms of like uh
ux security and you know if you take
something as simple you know I I used to
work there and
um in the security team and kind of like
saw a bit like the mindset from like
internally and you can see for example
like you know
um
logging in for example like used to be
like unlocking your phone used to be
pin-based and you know we all know that
like people just like hate to remember
things so what did people do you know
when they were allowed to bypass it they
bypassed it right it was kind of like I
think for only 14 of the people used to
have uh depend on and everyone else had
just like the fully or even before the
passwords right password you can exactly
one two three ABC or whatever exactly
yeah and then the moment they switched
to face ID this that actually Rose to
over 90 percent
so you know it it like if you're
thinking about kind of like the
mechanism to just like make security
easy and just like flow with the the ux
people will just like come along with
you in the process and so I think that
like again we're discussing this with
because yesterday I think that like the
the you know the the user interface can
do a lot more for us as of today you
know when we're thinking about
um when we're thinking about making
obviously like educated decision you
know metamask and and other wallets are
starting to to put like great features
in order to kind of like show you you
know what are you essentially achieving
by submitting this transaction to the
network what is what is going to happen
essentially uh ahead of time but
something that is super interesting is
like there is more to be done you know
like you have uh approvals for your nfts
these are permissions right similarly
for like your uh fungible token
allowances that could be managed
actually directly within the wallet the
wallet should be able to do that just
like your phone actually manages your
permissions you know like giving access
to you know cameras and like Bluetooth
and like location services and so on
like all of these things are things that
we can learn from just you know great
user experiences on like you know mobile
devices and so on uh and and I'm really
hopeful that we're going to see more of
these things as part of wallets and
similarly for like you know fishing when
we see like a lot of people that kind of
like get scammed sign the wrong like set
approval for all and get kind of like
all of their wallet trained even kind of
like wallet management should be
possible like within the within the
wallet itself like you know think about
uh you know your wallet essentially just
like spinning off like you know
generating a new address on the Fly uh
you know providing essentially really
like allowing you to just like send
funds like minimal amount of funds to
just like pay for the gas fees for the
operation and not have access to any
other asset that is part of kind of like
your main address right like just trying
to think about like all of these things
that don't necessarily need to show any
complexity to the user which is like
help them in the journey essentially of
like there's a startup doing that right
do you all know about this there's like
a thing where when you launch a metamask
transaction there's a second window and
it takes the byte code and puts it into
human English so there's uh hexagate
that's working on that there's a second
uh redefine we Define yep yep
yep
so something I would add um so 100 agree
uh I think all that stuff's amazing and
I think um you know we're going back to
you know this whole thing sort of web
two right uh I think one of the things
we can also take away from the web 2
world is like the depth of security
controls that have been implemented
um you know no one is um you know going
back to right now we're kind of running
and bring zero
um and uh when you when you sign you
sign with your wallet
um the equivalent of an operating system
like um you know adding those kind of
controls would be awesome ways to put
some sandboxing some capability based
access controls and extra reduction
mechanisms
um but you know there's more to
um there's more layers you can add right
um I think one of the biggest areas in
my mind right now is um uh and I've
learned this from talking to victims of
phishing attacks is they don't they
often don't even know where they signed
the malicious transaction at
um they they're they're at a loss um and
um there's no introspectability uh we're
missing um you know we're missing logs
we're missing uh and I think that we're
also missing like a really like like key
linkage for like on-chain and off chain
uh transactions so not only do we need
monitoring login on the project side
yeah we need to log in on the on the
user side as the user side yeah we if
we're going to trust you we're going to
say your key your your uh your Authority
you also owe we also owe them every tool
they need to be able to do that
correctly and that logging and
monitoring is a big part of that
yes
um specifically one thing that we've
like experienced um at metamask that
openc has called us out on and continues
to call us out on is
um for the longest time there were
transactions and the transactions have
like Financial value and it's pretty
easy to like you know like teach users
like this is a thing and like there
might be some bad things that happen
depending on if you click this button
and then
um
now a lot more stuff is just off chain
right or it's not on chain until after a
certain period of time if the order is
matched or whatever right
um and because of this like super early
assumption that like everything is
there's a key and then you have
transactions and the transactions go on
a chain because of that super early
assumption that we made uh we did not
see this coming we're now a like just a
signature right just like an off chain
signature can actually lose you like all
of your nfts or all of your tokens or
allow permission somewhere else right
and so now we're like playing catch up
while open sees like sprinting ahead
like making sure that it's usable making
things scalable we're playing catch-up
to try to like map back and like in
reality we should have never ever
treated a transaction like an on-chain
transaction signature differently than a
message signature and because of that
very early design Choice like at the
core architecture level we are now like
remapping everything and it's like uh
it's just a it's a stupid amount of work
to be honest and it takes way longer
than it should and it's especially
painful when users are are losing money
because of this every day right like
when we are hearing about celebrities
losing their nfts because of this thing
that like we know should be better that
is that's horrible
um I also want to comment that like
um my talk tomorrow I'm gonna be diving
into like some of the early early
choices of like wallets but also of the
protocol itself uh and diving into like
a lot of the stuff we're talking about
here because I think that the
the biggest sin of it all is that it all
traces back to this like private key and
this private key gives full authority
over everything right like that's the
world that we've like like started with
and like fundamentally I'm so hopeless
like it is so broken like why is the
single thing that you can't change you
can't do anything with the thing that
grants like so much permission and I
think that like
um we are now seeing that like at the
token level we're trying to like
Implement permissions and restrictions
but ideally like that would actually
live like out the very core layer right
where like the first thing the thing
that holds your stuff is actually like a
default like you don't do anything and
the first thing you have to do is Grant
the permission and then you can take the
action
um so yeah come to my talk tomorrow uh
we'll we'll dive into this but you know
I think that this is why I really want
to emphasize like
have conversations with like the people
that are around you learn from your
users listen to them like the most
valuable thing that I've ever done is
have literal one-on-one conversations
with people who've lost all their money
and like just ask questions and let them
talk because you will learn so much
about where your product is failing
um and you will be able to serve them so
much better if you carry those stories
with you it's not super fun but it is
like the most valuable thing that I
think I've ever done uh to this day
you agree with that uh on on the bed dot
private key search I completely agree
that refers back to the point I was
saying like uh theoretical security is
one thing uh people have built products
that are secure in the best case but if
they are not practically secure if
nobody knows how to use their product if
nobody knows how to manage their keys
and so on by just failing at your goal
and then the other thing on monitoring
and alerting that's another super
important bit so not just about users
but about your product as well if your
Bridge loses 600 million you shouldn't
have to wait six days before you realize
that so definitely always have and that
was because of no monitoring yes yeah
someone literally message them on desk
or hey something's not right so yeah
anyway you should
you should have proper alerting and
monitoring but yeah also don't go too
overboard don't start logging private
keys of users you have to find the right
balance there uh but all of that aside I
want to like take a step back and like
say we just like never talk really
talked about the title which is web 2
versus web 3 same or different because I
think we all just assume they are the
same web3 just sprinkles a few things on
top of webto security but if it wasn't
clear like at least in my opinion they
are basically the same things Spencer
here has been trying to guide us towards
a bunch of I think 20 questions or
something he prepared if we have managed
to get through maybe three no just wait
and turn on random spree yeah but yes
great job yeah speaking of a curveball
question so uh we were sharing more
stories outside
um yeah there's a reason why we're not
going with the prep questions uh so
Taylor you're you were sharing one about
response because you know obviously
after you do monitoring to logging oh
crap something happened right
maybe you were uh you know what do you
do next right so what are the Lessons
Learned for that so
um
this piece is decentralized and a lot of
the teams are there for fully remote
uh we use telegram we use slack we use
Discord we use signal we use like
there's like 8 000 chat apps I don't
know about you but like for me uh my
notifications are basically like off
because otherwise I'll just be like
constantly inundated with information
and I wouldn't be able to think
um however if something happens
like how do you let that person know
that something happened so for example
this one time Hudson was giving a talk
and you got Sim swap while on stage
uh and The Sims offers actually talked
to his wife
uh and I think told her that he was
kidnapped
because they're because they're
Sim sloppers
um and because me and husband have like
a relationship that goes back and
because Twitter like immediately alerted
that Hudson was uh was Sim swapped and
was like damning asking for money
because of that like I had the actual
sort of like in real life Network and
the actual phone numbers in place to be
able to like contact right same thing
has happened with my own team like I
don't know having each of those phone
numbers and making like old school phone
calls that break through like the Do Not
Disturb mode is like critical almost
every single like incident response
includes like tracking down the person
who has the person's phone number who
has the critical access to stop the
thing yeah do two things number one I
think both of you are on a list so like
if I call you I go through Do Not
Disturb mode right exactly same here so
like I have that set up yeah we have
that set up yeah yeah so like if one of
us because another time someone posted
something on my Twitter because I had
like an API key out there anyways the
second thing is when I got Sim swapped
and as soon as it got on there I I
walked I went back home when I was
everything was fixed there were 10 very
critical group chats that just said
Hudson Sim swap don't listen anything he
said Hudson swim stock don't listen to
anything he said and my startup at the
time open Innovations immediately shut
out all my access cleared it cleared it
completely the ethereum foundation in
less than 10 minutes after the Sim swap
happened clear had a round table
discussion a live War Room Zoom call and
got all my access taken away in minutes
did you guys have a plan for that or
just kind of uh we did have so by that
time so I talked earlier about how not
great the EF stuff was by that time we'd
improved so much we had a whole incident
response plan ready for this for the
most part and we had a dedicated CSO
role type deal that we have kept over
the years and years and years and we
have a great one right now who replaced
me uh after I left or not before I
before I left the EF and I was doing
other things but either way yeah that
was one of the scariest things because
like the cops were called my spouse is
crying one more thing and this is
something just not related to this
exactly but um something I'm really
proud of little it for my spouse
wouldn't I finally got a hold of her the
first person after resetting my
passwords and I went to the like you
know the phone store I called them and I
said hey this is Hudson and they were
crying and they were like I'm here with
a police officer I know this sounds like
Hudson I'm going to give you two
questions that only you would know and I
was like thank you I've taught you upset
oh my God
so uh so the day to do it was me and
everything worked out from there yeah
yeah I I just wanted uh like really high
at one point uh you had there which uh
well is like having the plan uh but I
think a key is practicing the plan uh
rehearsing doing tabletop simulations
making sure that uh uh what you know
documenting where the failures were
white what was hard and smoothing it
over before you actually need to do it
in uh in real life uh actually one of my
favorite stories of this was uh we had a
simulation where we had to disconnect
our Corp from our prod Network previous
company
um and we had one room that we could get
into but that room kept getting smaller
and smaller because you know we never
need that room uh and then when we in
our simulation we did uh we realized
there's only one ethernet Jack and we
had no way of getting all of us into
that room to be able to connect to prod
and to keep it up keep everything up and
then like so we're like yeah so it's
just one of those like there's it's
always the thing you don't think about
that fails you
um in an incident and you want to know
that before you actually before it
counts something uh you know sweating
training versus bleeding and War yeah
another another thing I think that like
is important to to take into account is
that
um a lot about
um incidents response and like Incident
Management is about time right like time
works against you the person is you know
someone or like some entity some group
already kind of like perform like found
the issue and kind of like try to
execute it so your role is also to kind
of like expand as much as possible the
time window that you have where time can
kind of work for you right it's kind of
like
and and you know we kind of like always
discuss these things well like you know
uh governance protocols and like you
know uh Bridge security it's it's about
like you know time locks and just like
freezing funds for like a specific
amount of time so that like you actually
give yourself some kind of like
breathing rooms and kind of like your
teams to kind of like go over the logs
go over like everything that happened
understand exactly like is like is this
something legit or not if this is not
legit you know obviously if you have
other mechanism in place great otherwise
kind of like get ready for you know kind
of like freezing assets across you know
chains and so on so I think that just
like making time work for you is also
one of the most critical things that you
that you can do and uh a lot of people
have been kind of like theorizing about
this uh it's um how much time
essentially like the uh bad guys need to
spend in your trenches
um in order to kind of like you know get
the assets and then get away with it so
that's that's something that is very
very important and obviously you know uh
will never uh emphasize enough that the
need to kind of like you know baking
time whenever you have like a new
governance proposal and just like uh
whenever you have like specific
thresholds or like limits that are hit
you know making sure that
um these things like take time and that
you have like some some freezing periods
where you can actually like cancel
operations and just want to hit on this
point again uh this is a this all the
stuff layers together right you hold on
your systems you may uh you you make
sure you patch systems and you find them
uh and you've audit them and then but
then whenever things do happen you time
box along the attacker has in your
systems that we have you can quickly
respond like these all the stuff layers
and that's how you get a great security
defense and depth on that one little bit
so with incident response I agree time
is very critical so what everyone needs
to do is have an a year plan have a
disaster recovery plans literally have
playbooks of every scenario you can
think of of anything that can go wrong
have playbooks of what to do so that
even if like your out sake someday and
some other security or someone else in
the article doesn't really know as much
about the product can just see your
playbook follow the exact steps and be
all good and like you need to practice
these playbooks as well it's not just
about having the content out there you
need to practice all of these playbooks
and everyone should be aware like what
needs to happen when needs to happen how
to follow the Playbook and only then you
will be able to properly respond in a
timely fashion
uh and another bit I would say like
coming back to the swim swap thing so
there I wanted to add a little thing so
switching swaps in fact not just Sims
wife but any product where the support
department of the company can uh like do
whatever with your access or wherever
like you don't have control on your
product which is most of the webto
products
don't depend on either get something
like highly secure or don't depend on
the cheapest one so for Sims almost all
of the public providers like they can
just same swap any of their support
engineer or uh whatever can same swap
you so never use mobile numbers for 2fa
or recovery or anything permission what
do you think of a funny that's the one
uh supposedly non-sen swappable cell
phone provider I'm not Lumen into them
too much but they're like a regular I
mean they're like legal requirements for
them to like allow specific operations
and like porting you know like you're
lying to something else so like so long
as like this operation is like required
by law from from these providers there
is an entry point right and it's like
this is just like a matter of like how
much Social Engineering you can do like
people are saying that like Google 5 was
actually not sim swappable because there
is no one to talk to on the customer
service but you know it it could happen
you know like no one is kind of safe
despite because even the more secure
cell phone providers yeah exactly like
you know you want to aim for obviously
you know the regular kind of like
non-sms to fa base but like if you're if
it is mandatory go for like you know
Google Voice number for example like
there's no sim to swap essentially so
you know go for for services that do not
have like the same associated associated
to it yeah and back to like the whole
stack right and like layering this on
top right get the most secure Sim and
then
have nothing like to have nothing on
that Sim right or like only the the two
services that like actually have to be
on that Sim and like have that video
number that isn't really your primary
number right like all of these things
layer and like that's and then you're
not signing up for like marketing
databases yeah
uh for this conference and this is like
this is why we talk about like the
culture right we talk about the
organization we talk about the people
and like how important that is from the
ground up it's like every little
decision that you make has to be the
most secure one right and so that you
have all the different layers on top of
each other so that it's less likely that
things happen that are really bad but
even if those things happen you can
mitigate the loss you can respond to
loss quickly uh we are being we're
getting ushered off the stage thank you
for coming love you all thanks everyone