foreign
[Music]
if you're in the wrong room this room is
talking about testing the merge with
parallel universes
my name is David Searle and I'm the head
of Amir for a small startup that's been
working with the ethereum foundation for
just over a year now
who likes Bugs okay I brought some bugs
along with me today so you're going to
have to be like who would like a bug
because I've got some bugs that actually
I'd like to give to various members of
the audience anybody you'd like one okay
here we go just after a little you're
gonna catching so little bugs we've been
spending the last year kind of really
stress testing for my boat so you see
this is why you should stay at the front
I've got three more to go there we go
see
so the it talks about testing ethereum
and is is basically looking at using a
pretty sophisticated and unique testing
methodology which is based from a
company called antithesis which I belong
to
um the the company is involved in using
deterministic simulation techniques to
allow us to basically explore a whole
host of iterations of of complex
distributive technology that's running
in a simulated environment so let me
just get my my little Clicker
I brought a few uh Stars along to the
show um obviously I've just explained
who I am
um little did you know that uh doctor
strange and Cher are also going to
participate in this this presentation
um I've got about 25 years worth of
experiences in the tech industry
um for the last two and a bit years now
I've been working for this business
antithesis and they've
um you know basically made me into
somebody of that there's a bug Hunter
buyer by trade um I never thought
testing could be fun but there's nothing
like hitting a seg fault and actually
getting a real kick out of it and that's
what I do on a day-to-day basis this is
my house in the UK you can tell from my
my accent
um and as you can see I've mostly got a
few gpus I've got to sell so if you're
interested please let me know
um no joke but some the the day of the
merge came round and we've been doing a
lot of work with with all the various
clients
um all that hard arduous testing making
sure that we call every last bug
um what was going to happen was it
really going to be smooth sailing yeah I
think we saw this morning uh which
obviously is great news um but um yeah I
was wondering if we were going to see
this you have to press play on the uh
the video
there is sound as well but uh
um I couldn't believe this when I saw
this but uh this is real
yeah it's pretty bad eh
um not my fun Garden again so
um yeah testing is really important and
this is where you know we talk about it
but the merge was a significant event
that's why we had so so much celebration
across the world about kind of what was
really going on uh when when the modes
took place
um so congratulations to anybody that
was involved in the merge you know
phenomenal job and uh yeah I think we
can you know hold our heads up high
there
so I'll say testing is hard and you only
got to use Cura to basically you know
ask a question you know what you know
what makes software testing difficult
yeah it's a bit of a high level question
you know apologies for that
um but you know these two really um kind
of uh sort of struck me here like the
first one you know this is the basically
fundamentally this is an impossible
requirement because absence of evidence
is not the evidence of absence and so
often we run these simulations in
parallel universes hitting you know
hitting a whole host of code and it's
clear and you're like well is that good
who knows like you know we we can only
really understand how much coverage
we've got and it may be maybe good maybe
bad who knows so that's that's a
phenomenal kind of you know kind of you
know requirement the the second one is
then about kind of understanding like
testers have to think about all those
possible scenarios where issues may
arise and ensure that they are handled
by the code and you know that's just not
it's just like how do you how do you
approach that with a distributed you
know kind of architecture like we have
with ethereum
um we've got a whole host of different
clients using different architectures
we've got the execution layout we've got
the consensus layer it's just it's
amazing to see it actually working and
and but the complexity of that and the
is just is just phenomenal
um
if we then look at kind of what ethereum
has done to approach this task and this
is you know this is not just looking
specific antithesis we are a part of the
equation
um and I think we're very much of a
complementary is is that ethereum is
using unit testing test Nets we've got a
list of Shadow forks and test net
mergers occurred
um some technologies that are being used
called Hive and ketosis
um that do similar
um heaters to the equation we then have
antithesis that is then doing this
deterministic piece which allows us to
to Really you know get into the detail
of of all the different iterations
um that are possible and then we we have
a host of different fuzzing technologies
that the ethereum are using
um to to really try and isolate and kind
of hammer areas that are of concern so
you know throughout the entire year you
know a huge amount of testing has been
done and on top of that they've also
used static analysis to just you know
help with doing code Audits and making
sure that we catch things before the
merge was to take place I'd step back a
little bit in terms of kind of the
testing as hard statement you know it is
a phenomenal kind of like you know
obstacle to try and get over like how
how do we find every iteration and how
do we find those hard to reach bugs that
you know may not be common ground but if
they do occur will be catastrophic
um and so if we just look at the right
hand side
this is a representation of a
distribution Network so on the right
we're going to we're going to call this
node we're going to call this node
um you know using Lighthouse and Geth so
the lighthouse and gether running on
this node they're running on an
operating system you know leveraging CPU
um we've got a file system we've got
different processes happening and then
inside those processes we've got
different threads being used as well so
just in its own sort of node there's a
lot of complexity about what's going on
and we can we can test the nodes we can
test maybe a combination of Lighthouse
and guests working together making sure
that we're kind of you know pushing the
boundaries but
if we then think about how we then have
a network and we have a whole host of
different nodes running together and may
have other underlying pieces like
databases other pieces kind of actually
on on top of that
um the the complexity of all those
different communication channels
operating just means that the search
space for Hidden Away Little Gems I call
them gems but you know they're bugs
basically
those those they can hide away for a
long time and you may never find that
search when you're looking at testing
you may never find that that particular
condition where the network was slow and
there's some there's a bit of thread
pausing going on in a certain node and
you know the the the the number of
things that have to happen could could
have a bug manifest and so this is
really where where we step in because
you know we then we then have you know
the complete picture and this is what we
build inside our simulation
um we have the ability to kind of run
the entire collection of consensus
clients and execution clients in in a
simulation run it and and understand
exactly kind of every way in which we
can actually kind of see outcomes
um
that's pretty intense you know how do
you do that and fundamentally how do you
do that in a deterministic fashion if we
find a certain situation where
Lighthouse crashes well can we represent
that again can we replay it can we yes
the first thing you do right can we
reproduce it and often bugs are not
reproducible they often you know they
might try something different they might
move around what like how do we how do
we debug this and and this is again
where we step in
so what do we do we leveraged the
ability to Auto generate networks inside
the simulation environment and stand up
all the containers that were necessary
to bring up ethereum
um we then were able to then use Network
faults and other types of faults to just
inject chaos into that running
environment
um obviously it's great to see the
software handle that it's designed to
handle these types of things so you know
under duress we see things going wrong
um and we we basically use fuzzing
technology to to basically hit the
entire system
um not just individual pieces of
software but the entire system is fuzzed
and that allows us so basically kind of
deterministically replay the complete
orchestration of a situation not just
one particular application but we know
that in this combination different
clients under these Network conditions
something goes wrong
um and so we use strategies inside the
system to allow us to seek rare events
and so when you when I start looking at
the numbers and kind of what we've been
doing over the last year you know we
hear a huge amount of code edges like
you know a huge amount and so how do we
how do we find and allow ourselves to to
seek out those edge cases that you know
if they do happen again it can be can be
pretty catastrophic
um but you know get to them and and
uncover them to help the client teams um
debug and fix so we have we have the
ability to have all this wrapped up in
in a a tool set um that is available to
ourselves and we share it to the rest of
the client teams and the F and that's
been really really useful I think if we
look at it we have we have the the
individual testing going on which is
fantastic we continue to promote that
and say keep keep going with that we
have the test Nets and Shadow Forks
again
I think you know we can all say that
they've been tremendously successful and
useful
and then we have
our parallel universes we have the
ability to run not just one simulation
but we're running simulations literally
every day that are generating and using
you know exploring the various search
spaces that exist across any number of
different branches inside the uh the
repositories so
what does this mean for us well there's
a little Little Help from Doctor Strange
here
hopefully you've seen it because it's
not playing here we go
this is what I look like when I'm doing
the work
[Music]
for it in time
to view alternate Futures to see all the
possible outcomes of the coming conflict
how many did you see 14 million 605.
how many we win
[Music]
one
so so for us winning is finding these
edge cases right now your winning isn't
like oh all tests passed yay um we want
to find those those really intricate
kind of piece you know combinations and
iterations where we've covered 14
million different scenarios and we've
got you know maybe not more than one but
we at least have one you know we can
actually hold our hat on so that's
that's an example of what we're doing
and we think that kind of represents you
know pretty pretty good sort of analogy
um if we move forward we've seen it
again
here we are this is kind of what it
looks like in our world
um you know we're looking for bugs
um this this is an example of an output
of just one run so we we ran this um 13
hours worth of water wall clock time
which is if you look at the wall and
there's a clock on it um it will last 13
hours
um but uh it actually allows us to to
exhaust 536 hours worth of testing
um and you can see here we are we're
talking about an enormous amount of
edges being seen here um the branches
are kind of how we get decision points
we we get like an if then else on a kind
of statement inside the code very simple
example but just just take it
um we can basically Branch off at that
point and exhaust both Avenues and
actually see what happens in both
situations underneath those and you know
under those test conditions
um that's an example of a branch and
obviously the code The Edge is seen is
we've got the instrumentation happening
so we can see what kind of functions and
and pieces of code are actually being
executed
pretty pretty insane you know that's one
run I've got 180 000 edges that we're
seeing across across the entire network
it's a busy Network
um and obviously with all the different
clients happening that's uh there's a
lot going on
so we see ourselves as a complimentary
piece we you know we we've we've been
looking at this um as a you know a great
example of a project where we can
basically work alongside all the other
pieces that the EF are kind of using and
different client teams are using um so
we're just another layer in in the
growing list of of of strategies being
being used and I think that's sort of
Testament to the approach that EF have
used
um to to Really hold on to this
resilience you know making sure that we
take advantage of every way we can we
can make sure that you know nothing
untoward happens in the future
um you know more more and more upgrades
coming I'm sure so you know we've only
got 4844 around the corner I'll say
around the corner we'll see um but uh
it's certainly something to look at in
terms of
um you know making sure that uh we're
involved at every step of the way
so this is a very simple sort of uh
illustration of what we've been doing
we've been building all of the clients
um giving them in in place and actually
establishing Genesis so we actually have
been using this at Genesis block and
then moving it forward towards the merge
in our in our world we would have the
merge kind of positioned um we obviously
start fault injection we've established
a chain and then we start fault
injection
um to get to a point where we've got a
whole host of different faults you can
see them here labeled below so anything
from partitioning
um that's a great one obviously to
encourage forking of the chains
um we've then also got you know delays
happening drops of packets
um the nodes themselves
um we can stop you know we do we do we
put this thing through its faces we stop
things we pause things we kill things we
bring them back up I suppose in some
respects it's quite realistic you know
things do get rebooted and violators
come back up again we we have all that
kind of like happening inside the
simulation and then from that we then
have you know any number of threads
being paused released there's there's
yeah it's it's busy
um every type of this sort of
configuration of faults that we put into
a simulation is complete again
completely deterministic so if we know
that this ended up with a segful we know
we can literally put exactly the same
into the into the environment it's
basically creating a pure function of
the entire system and we know we can get
the exact same output which I think is a
tremendous piece of technology which we
have under the hood
what the numbers like well yeah you know
in one year we've we've conducted 31
years worth of 24 7 non-stop testing
um that that has processed and and
explored over 50 million edges of code
um you know out of that and you can see
there you know out of that we've got 45
validated errors some of which were
painted for but 30 of the 33 of them
were logged bugs
um that were pretty catastrophic um
things that really would bring either an
actual note down so panics seg faults um
we'd have you know nil pointer
exceptions we'd have there's there's one
I'm going to bring up in a minute with a
little little example but
um things that you don't want to have
like living in your code base and and I
think that's you know we would forward
those through to the different client
teams and they've been able to eradicate
them and uh and obviously through the
successful merge
um it's been great to see our efforts uh
put to put to the test
as he wants an example or I'll give you
I'll give you some
um so here's here's an output of our of
our sort of one of the runs this would
come through from an email and hopefully
you can see it's Biven our chart you've
seen the side of the select the stuff on
the right hand side but down here you
can see a whole host of assertions that
we've got running across across the
entire network
um this this is actually flagged up by
fail and they've got a fail on a seg
fault this is this is you know something
that is is absolutely real zigzag V
illegal storage access attempt to read
from nil question mark
I guess I'm really interested ones you
get like you please report this oh
please repeat please fix this
um so this is just an example here and
we're like What do we do now how do we
how do we take it to the next step
because it's great saying well you've
got a problem
um this you can see here is actually for
the node that's running Nimbus and Geth
so we know there's an issue there so
okay fine have you got any other
occurrences on any of the combinations
we do
um so okay that's cool
um well let's let's look at an example
uh of an actual kind of log entry that's
coming through so this is this is a
unified log the serialized activity
that's occurring across the entire
simulated environment
bit of mouthful so you see you can see
here on the third column the blue column
here again this is just showing us all
the different combinations so we've got
Cloud we've got prism client Aragon
we've got um there's nimbers never mind
nimbuspessu all kind of serialized and
you can see that we've got a concept of
time running through so this blown up
piece is basically the stack trace of
showing us kind of what is happening and
Nimbus is having some issue it's
receiving some Json trying to figure out
not doing too well for it so because it
basically crashes the entire node
um what you know so we go okay this is
cool this is this is where I kind of go
yeah you know there's got something
something of Interest here
um another example here this is just
from from Prism
um again
I don't love sort of seeing panics but
like you know it does it does show that
there's something like of value here
that we're really stressing
um the the entire environment so again
invalid memory address
yeah okay cool what do we do now you
know do we descend it to the client team
well yeah you can and they might go I I
wrote that code I can see what's going
on here
um
well she had to do it didn't she yeah if
she could turn back time well the great
news about deterministic environments is
you can turn back time right so we're
going to basically jump into the actual
environment and actually say okay right
how when does this manifest like is it
okay it crashes
was it a second ago was it five seconds
ago what happened in the execution pass
of all the different Opera operating
nodes with all the fall injection going
on or or was it
um yeah what where where do we where do
we see this like actual kind of thing
manifest
and so we we want to look back so many
seconds we may want to turn on car
packet capture pack capture isn't by
default just because the amount of data
but it isn't turned on by default we
want to be quite selective about kind of
where we see packet capture happening
and and then we can look at the data see
if again we can rerun it
um again obviously determinism you're
stressing a little bit of kind of will
it still be deterministic if I suddenly
turn on packaged at a moment in time
let's have to look at that but but what
we've seen is actually yeah you
generally if you've seen something you
know manifest then we can we can
actively kind of kind of turn on things
to help kind of debug and work out
what's going on there's some really
interesting stuff coming through next
year which I can't share but really cool
so this is what we can do today this is
what we've done with the with the uh is
basically look at this mountain of data
like you think of all the stuff that's
going on all the different scenarios
branches Paths of execution that have
happened we can we have all these
available in a big massive data set so
we can we can look at this kind of
analysis and look at the x-axis which is
your time versus the probability of the
bug occurring on the y-axis now the top
right is where the bug is seen
we can then look at the common routes
that are happening across all the data
and try and then bring it together to
see what is happening you know where
where does the execution occur where
suddenly the probability of this
actually becoming a bug yeah happen and
and lo and behold we have a huge jump
here from literally
0.05 of kind of the the kind of bug
occurring to over 50 this jump here in
the middle we can start to go okay right
we know how many seconds to go back now
we can actually Replay that simulation
and turn back time I won't sing it
um and and and see what's going on there
because otherwise you're looking for a
needle and a haystack like how do you do
that and that's really been really
valuable
um in in our in our efforts
so
what's next well
merger's great love it we're doing some
Pokemon testing on some stable branches
which isn't you know we've got you know
things in a fairly good order there
um but we know ethereum's not standing
still we've got obviously EIP 4844 just
around the corner I say again
um thanks sharding withdrawals you know
a whole host of new capability which is
you know brand new products brand new
code brand new testing
um there's other pieces in there around
you know actually the one that's
interesting is using things like
malicious clients you know what about if
I brought up an environment that has a
client that is you know not doing what
it should be doing
um you know can that cause issues you
know how does that kind of part of the
of the code for the rest of the the
network handle handle that client and
does it does it operate in a different
manner so
other pieces here you merge clothes
clean up you know there's changes to you
know kind of established pieces of code
that are in in place now
all those changes introduce you know
probably good and it could appear uh you
know Downstream so pretty cool
um we're working actively with the
clients
um and open to really you know kind of
broadening that relationship further so
that's the end of that q a
um any questions yeah um you showed us
uh a slide with the probability of
finding a bug how did you calculate that
probability which model you were using
or how complex is you know the
calculations and that stuff that's a
good question so the just because the
amounts of data we've got available we
we know every kind of path execution
that's occurring that the outcome
doesn't occur with a bug and we have
then obviously all the different
outcomes that do translate to the bug
happening
um and so
where the ability to you know do some
simple calculation we can see that you
know the path the path that's sort of
trodden and worn and you know this but
this if this is if we see this happening
then we can calculate the bug occurring
there are obviously other branches down
that that well-trodden path where my
virgin's gonna go somewhere else and the
bug doesn't manifest so we that's that's
how we at every point in that graph we
can we can calculate if you know if it
does contribute or not contribute to the
bug how do you kind of isolate unique
bugs so you know it's like the same bug
that's a good question and so we we see
a huge amount of bugs that do manifest
and show us like you know duplication
like you know like the one you saw with
nimbuson and Geth
that we can we can run that literary day
in there and then it will you know with
the right commit code which is in
history but we would we would be able to
see that and we'd see we we have counts
on that occurrence so we can see very
quickly this isn't just like an edge
case that's just happened on one sort of
combined client set it's it's like it's
a problem across the board and so
um we have all of that wrapped up into
our reporting you know what do you think
about applying formal verification in
this huge environment is it
um do you think
um possibility or
I'm usually not the best person to
answer that question but bring it bring
it to us at the end because I've got
some people that can answer us that
question for you
okay
well please great chance of us keep keep
you know keep the conversation going and
uh I hope you enjoyed the talk