[Music]
hello my name is Martin Hollis Wanda and
I am the security lead for their Terry
foundation today I'm going to talk a bit
about avian forensics and managing
attacks against the network and how
we've been working on that so I've been
security lead for one year I started
just before DEFCON 2 in Shanghai last
year which started off with the Shanghai
attacks roughly one day after I started
my new role it kept on for a month we've
also seen here in the last year we have
done three hard Forks we've had one
unintentional consensus split there was
a DOS attack against specifically the
client there has been thousands of ether
stolen in more or less more and less
sophisticated attacks both on Shane and
off chain we had the test not totally
brought to its knees and then
resurrected again and of course there's
the standard IT incidents with leaked
databases and someone taking over
someone's phone number and their account
and attacking our github but stuff like
that
so we should all be very clear about
where we're out at this is crypto land
and we're all in crypto land and it's
like Australia where anything with the
heartbeat will try to kill you
and if you make a mistake you're
probably dead so meanwhile for attackers
they have never had it better they no
longer need
to hack point-of-sales computers and
trade carding details over shady forums
they can just hack a computer and or
somehow get some cryptocurrency and
immediately turned into value and and
it's so it's like a Wild West in
Australia right now these are the
Shanghai attacks I'm not going to talk
that much about them
the first Chiang attack is a little blip
down there and then it just kept on
going for a month and it was a lot of
different attacks mostly targeted
towards gap but when the dust is settled
after incidents happen then that's when
you can actually do something about them
and think about how can we be better
prepared next time something similar
happens when I can we prevent it so how
can we improve the readiness and the
resiliency so for readiness it's about
detecting attacks and performing
analysis quickly so we started improving
that with some monitoring adding up some
monitoring nodes that we run in the
cloud and adding some graphs turns out
there was some inherent issues which
hadn't been noticed before
with transaction propagation
inefficiencies which over the course of
a few months in the beginning of from
January to March we managed to bring
down the overall network traffic with
about northern magnitude just by
removing valid transaction propagation
from the clients on these monitoring
nodes we also added some interface so
that we can extract very detailed
information about what are the canonical
blocks in the chain and if we see a
consensus splits we can get very
information about the receipts and
differences in these and and quickly
point out which transaction cost this
consensus issue so here you see as a gas
master and gas develop and the parity
note and right in this image their
differing on two fields marking red
there and that's because parity RPC
interface exposes a few different fields
than gas now as we're going to analysis
talk a few words about the EVM because
there's they might be conception that
any minor difference in the
implementation of the UM will
automatically result in a consensus
failure and that's not quite true
because there are some things some parts
of the ABM which are ephemeral such as
the memory on the stack and which do not
necessarily issues but they're very
interesting because they can be used to
trigger consensus errors and in order to
really measure EVM side by side and
detect implementation differences in
idioms we need an kind of up by up view
of the internal state so we push kind of
hard to get common output format for
EVMs so that after each instruction made
in a VM it would output JSON blob with
internal state as you can see on the
left and also a capability to use
arbitrary state in Genesys configuration
with the raw EVMs so one problem that
can arise is if we hit by an attack
which blows the node out of the water
how can we analyze that because I now
just died right
how can analyze transaction if the
transaction crashes my node well if we
have a standalone EVM what we can do is
we can just fetch the pre state about
the sender and the receiver is just
to accounts we can execute that locally
in our EVM and then analyze the trace to
find did we miss anything was there
anything else we should have had here an
external references and fetch those and
start over and if no crashes then we're
just successfully reproduced the API the
transaction and for this we only need a
web 3 standard API without any debug
specialities sorry so I'm going to
demonstrate quickly how we can do
analysis of the jump test attack which
we were hit by in on June 1 so I'm
running this little reproducer here I
pipe in the hash used in the attack the
transaction telling it I'm gonna use my
local EVM not through darker and it
basically sets the right fork rules for
that particular block and executes it
and it has some intermediary traces here
we can take a look at those so let's go
directly and set for the final trace I'm
showing this and what I call the OP
viewer or retro mix if you like it's
remix like a debug viewer for the JSON
output format that I showed earlier and
it's a good start for analyzing what's
happening in a transaction so you can
see this particular transaction it
doesn't Xcode copy
and the Xcode copy fills the memory with
five B and it does it repeatedly and as
you can see the memory is growing and it
keeps doing this for about 600 steps I'm
gonna go a bit faster here until it has
filled up the memory with half a
megabytes all 5b which happens to be
jump test then it puts some more code in
there and this is looks like actual EVM
code 6003 56 5b and all of you I'm sure
recognize that that is the push one jump
jump test and stop so it it's just
executed to create with that code and as
you can see the size of the create is
the full half megabytes okay so now we
know that the attacker is doing creates
and he keeps doing it repeatedly one
part of the memory changes between each
invocation it's a little counter down
there and I'll skip forward with so it's
all just create and it ends on create
number 105 it goes out of gas so by this
time you can be kind of have an idea so
it's it's doing creates a lots of times
with a large memory segment totally
filled with jump tests it changes one
little bytes each time so obviously
bypassing and caching mechanisms so by
reproducing it and viewing at the trace
in this fashion we can do a very quick
analysis of what happened and we can
benchmark it
right now it's running at 300
milliseconds and if I compare that to so
this is the EVM get EVM with the patch
applied after this attack I can try it
against the EVM without the patch for
the jump test analysis and as you can
see it took nine seconds so this tooling
makes it possible for us to do quick
analysis and then to check does this
patch work and I can I can share it with
the co-workers and they can try out
various patches and see which one is the
best I can also run this in a web-like
format and do the all the same things
and investigate other on chain events
for example the take that was the same
pair 2 volt attack
and there we have the parity wolves
attack reproduced and you can run it
locally or you can check and annotate
the trace of what happened there in the
Perrault attack and for example yeah so
here's the fated infamous delegate call
and the node attack if you want to if
you want to analyze that more in depth
so the EVM lab which I showed you part
of makes possible to do some EVM
assembly pythonic Li and investigate
these kinds of issues and yeah dissect
attacks on a really low level we had two
hard folks also and in preparation of
those we ramped up the testing and
introducing parameterize tests or
generalized test which Dmitriy talked
about yesterday in the breakout room and
also put it all into high high vis PDC
log is super cool framework for running
notes and in a blackbox fashion and just
synthesize the the environment the
genesis and the blocks and everything
and then you can compare the the
expected post state after a sequence of
blocks and this makes possible to run it
runs about 24,000 test cases against
Pythian parity gets and cpp and it runs
it 24/7 365 it removes the dependency of
the developers to perform tests as part
of the test process so now testing can
be a totally separate process which
doesn't really rely on the developers
per se the fall out however after the
first heart sorry second heart fork is a
consensus issue
which was yeah definitely not what we
wanted manually crafted tests are great
but there's no way to scale it due to
the inherent complexity of the EVM we
can't just have enough people know that
much about it to to be able to scale it
up so we wanted more coverage for
Byzantium and started looking into
fussing one way of doing that would be
to generate test cases randomly execute
them on each EVM use this shared output
format to compare the internal state
after each operation and just repeat it
this can be done fairly quickly you can
do a couple of million tests per day if
you draw binaries and you can use these
for clients the second track is based on
Lib foster where we got in touch with
the Guido Rankin who has done a lot of
fussing and is a real expert on live
foster so live foster is the core of
American fossil op its foster developer
mikhalev scheme and it's a bit more
sophisticated because it uses code paths
and instrumented binaries to detect code
paths for any given input and mutating
those inputs to maximize the code
coverage and since everything is
instrumental in compiled into one big
binary is an order of magnitude faster
to perform this test so I can do about
100 million tests per day and there was
a spectacular and kind of unexpected
success in this we've had seven or eight
consensus issues found most of them
before the heart Fork one of them
slightly after the hard work it has been
fixed and patched and released and the
clients today I would say are more
thoroughly tested than they have ever
been in the history of ethereum and
we're still running fosters 24/7 and
it's been millions of tests done on the
tests off and billions of
based on live posture naturally the
Castella consensus issues or denial of
service issues and if that's really
really concern of you of yours then you
should run multiple clients and try to
detect mismatches and you can use the
debug method and guess to find out if
guess has tagged one of parities connect
canonical blocks as bad and key takeaway
here is that all everyone here are
target's for attacker it's involved
because it's important so be paranoid
and be proactive and work on improving
the security and your resilience and how
you can handle tax that's about it for
me thank you
[Music]