foreign
[Music]
about the quests for the best tests
today we're going to look at all the
different testing tools that we used and
I'm going to be talking about the bugs
that we didn't find on mainnet if you
were here for two talks before mine
Mario spoke about the bugs that we did
find so I'm gonna do the contract
um the first part let's talk about what
we're actually doing with testing why is
the merge complicated to begin with well
we have roughly 20 different client
combinations and regressions sneak in
very easily you might see a regression
in one of the client combinations but
none of the other ones and that's very
tricky to pinpoint
the specs for an active development
there were quite a few of the early test
Nets where we kind of didn't pin the
spec version so someone was implementing
something on a different commit so of
course the test net broke so we had to
actually figure out how to treat
modifying specs along with modifying
test Nets along with 20 different
combinations all changing at the same
time
um communicating and debugging it's
great that we have a decentralized
environment it's horrible if I have to
wait for the Australians to wake up for
anything
um it does take quite a bit of effort
and quite a bit of planning on our side
we had to kind of schedule around people
picking up their kids from school where
to schedule around people waking up in
Australia we have to schedule around
Americans a lot of different things
and figuring out how to do all of this
in a reliable Manner and on a timeline
was crazy
um the last one was debug knowledge we
were really surprised the type of
debugging you need to do for CLS and
years are totally different we had to
see how to bring all of that competence
in one place
and how to actually figure out when
something goes wrong and they did go
wrong a lot in the beginning
um and the nice part is if you figure
this out once we figure it out for
future test Nets so happy that that
worked out
what could possibly go wrong so the
merge has two parts the consensus layer
and the execution layer and they
communicate via this thing called the
engine API
so if you mess up the consensus layer
you're going to have a network that
can't agree on anything if you mess up
the execution layer you have a network
that can't do anything so those are the
two high level problems that we're in
trying to ensure that never happen
um just to enumerate a bit from the
regular testing world what sort of tests
even exist so we have unit tests and it
is in our decentralized uh world with
all the client teams the client teams
take care of this themselves we don't
have to do anything no coordination work
from our side uh this makes sure that
there's no regression but these are
localized regressions something that
they might have seen on their client
based on how they've built their client
we have integration tests part of them
are done by client teams so for example
the Nimbus team spins up a local Nimbus
Network to make sure that their stuff
can always talk to each other
then on a high level we do some
interrupt tests so these might be
devnets or whatever they are
then we have system tests and this is
where external coordinators come in so
I'm going to say the EF was part of the
external coordinator for the testing
efforts so these tests end-to-end
functionality we spin up a test net we
run transactions we withdraw we set up
faucets we look at explorers work and so
on
and then we have production tests this
is something you guys might have seen
with shadowfox so production tests sort
of work on a prod like environment and
I'll go into what shadow Fox are later
on but on a high level we inherit all
the complexity of mainnet and this also
includes public test Nets so if you guys
remember Kiln or kinsuki that falls
under this bucket so you have everyone
all over the place testing their things
layer tools are deploying over there we
have random D5 protocols deploying over
there and these find issues that only
happen on real world workloads you
probably can't simulate them any other
way
so the second part I'm going to talk
about what different tools we had and
just give you a brief overview and just
to tell you how this is going to go once
you have an overview what sort of
testing tools we have I'm going to talk
about what we actually did with them and
what we didn't find
so the first one uh starting at a high
level we have spec tests it's the great
thing about the consensus layer they
have an executable spec they have
specifications with a ton of tests
client teams can then import these specs
and make sure that they can test them in
their local CI this means whenever
they're making a release at least you
know that they're coherent to the spec
uh this is largely a sanity check it's
not meant to find any massive bug but it
ensures that there's no regressions
happening
we currently have the spec tests I think
running every night on on a new CI
machine
the second one is hype tests you might
have seen this being referenced a couple
of times
um so Hive tests run with the simulator
and they essentially start up the
clients and then run the tests against a
predefined interface these are a couple
hundred tests these take anywhere
between I think a day or two days to run
everything
and a brief example of how this is is it
sets up a tiny instance of another mine
node it sends it two terminal blocks and
asserts how the transition happens
this is a lot of awesome work by Mario
he should be somewhere in the crowd
shout out to him and follow him on
Twitter
we found a lot of edge cases in this and
once we do find an edge case it's always
in this so that we make sure in in
future updates whatever it doesn't
happen if people check the website of
course
then we have this thing called kurtosis
it's an external tool that we're working
with and kurtosis obfuscates all the
complexity with setting up a test net
you don't have to worry about how
Genesis works you don't have to worry
what format that never mind needs its
Genesis file in nothing you just Define
it in a yaml file you say I want a five
node network with this Docker image and
that's what happens for you
um we time this actually nightly and
they think we've gone through the merge
at least a couple hundred times this
year
um a lot of them with issues but a lot
of them without which is great to see we
use this also to rapidly iterate ideas
so once there's a new spec version or if
you want to try out a new testing tool
we throw it in there first we also had
map boost integrated into kurtosis so we
could test it out and do a lot of cool
things there
um a general view of kurtosis is it it
checks the happy case if you can't
figure out the happy case there's no
point checking the rest of it so it just
starts up a test net make sure
everything is fine if it's fine it shows
you green on your CI
the next one is sync tests
um there's no point on a network if you
have nodes that can't sync up to the
network so what we do is we spin up
nodes I think at a week's notice right
now
um they sync up to the head of the chain
and then they assert whatever we specify
over there so if you notice over here
you can say is execution healthy is
consensus healthy are both synced are
they both reaching head so you can kind
of Define what side a type of syncing
you want to do here
um and as you can see on the right there
are a ton of different options you can
do and a ton of Networks and the cool
thing is you can also assert bad cases
you can say start your El stop once it
reaches head then start your CL and then
build weird scenarios there
um shout out to Sam for building this he
was also running it and sending a
summary I think he even presented it
once on Orca devs
um so we could make sure that at least
when we're making releases and when we
merge that we can sync the network
and this is the meaty one we had test
Nets and shadowfox so a shadow fork and
testnet help us coordinate all client
teams in one place and we check
compatibility largely we take whatever
assumptions we have in the spec and we
assert if those assumptions are true
on a large level what we're doing is
we're taking the Genesis configuration
of any one network and then we modify a
couple of values here and there and what
happens when those modified values are
hit is we split away from the main
network but we continue staying sync
connected on the gossip Network so we're
importing all the transactions and we
have the old load but it's on our side
we it's it's just a side Fork it runs
parallel to the main Network and no one
cares about it except for us because we
can find dozens and dozens of bugs there
this allows us to stress all of our
assumptions and we've done a lot of them
I have a summary on how many we actually
did in the end
um I sort of look at this as a release
test so it's kind of one of the last
things we would do on any of the future
Forks that we have it's kind of when
we're almost figuring out are we ready
to go ahead with this are we ready to
move forward with with committing to
this Fork is there any unknowns that we
don't know yet and then we have fuzzles
and in external organization called
antithesis so antithesis is a
deterministic hypervisor that allows us
to perform Network splits packet loss
all sorts of really weird edge cases
we don't necessarily expect the network
to be put in these edge cases but if
they are we know that the clients can
handle it on the right side you could
you can't even actually see it but each
of those is a 256 thread machine we have
three of them all running fuzzles it was
insane we had way more uh I've never
seen that much compute in my life we
actually had the IBM data center go out
and buy more CPUs because we bought up
all of them
um yeah various fuzzles various teams
running out all of them uh super cool we
found a ton of bugs referred to marius's
talk if you want to know what we
actually found there and we hope that
some of those bugs change the spec to
make it a lot more stable or sometimes
than implementation issue
so to give you a brief idea on the
testing life cycle let me tell you it's
never that clean but that's what we want
to hope to achieve so you have the
client releases happening and once the
client releases happen they go into the
integration tests so we make sure that
testnets can be run that integration
tests work fine and once that's done we
move on to test Nets and then we move on
to stress tests and we push whatever we
find on these stress tests onto
specifications we do regression tests
and then we do fuzzing and hopefully
whatever we find in the last couple of
stages go back into client releases so
that's a kind of a nice way to look at a
full life cycle of how testing would
work we're hoping to adopt the same life
cycle for future test Nets maybe we
moved around a tool here or there but
that's the general idea
and the end game what did we actually do
um it was really hard to find a graph
that actually fit all the test Nets we
had that's how many
um so we had we started off in April of
2021 with ryanism and since then we've
had
four public test Nets that anyone could
permissionlessly join into we had six
devnets meant for all the client teams
we had
um five Gully Shadow Fox and in the end
13 midnight Shadow Fox and after all of
this we had three test Netflix and only
then we hit mainnet so the whole number
of just testing hours in the merge is
insane I'm quite sure if we added it all
up it would be at least a couple ten
thousand hours put into this
um so
what didn't we find
this is a really interesting part
because even though we have all of these
Cool Tools there's still always going to
be something we didn't find this 99
participation 98 participation or great
blocks being produced awesome but I want
to know what we still didn't find so
first one
um we had in-memory databases that were
too low to process main net blocks it
just so happened that we did too good a
job of deciding which machines run our
test Nets which means we didn't have any
resource constraints and we kind of
missed the people running nodes on 8 gig
ram machines or people running on 16 gig
ram machines we didn't account for that
or we didn't account for older versions
of ram being used whatever it is
um another one that we missed is
non-optimal block production
um we were super focused on making sure
that we didn't see any zero transaction
blocks on the network we didn't compare
that to what the optimal blocks on the
network could be so it is an
optimization problem of course and it is
an awful that on mainnet we have some
reduced load but it is something we
missed and something we probably should
look into in the future
um the next one there was a really
specific way in which the terminal
blocks could arrive that broke
Netherland and it broke Netherland by
causing missing receipts and we only
noticed that issue when there are
deposits being made so we completely
missed this and for what we still yet to
figure out why but this is mainly an
issue for lodestar never mind and no
other combination or at least we didn't
get the uh Memo from other breaking
combinations so lots of another mind
figure that out on day one of the merge
and it I think it was passed like a day
later
um another one that we didn't think of
all of the Shadow Fox didn't have any
nodes syncing up to the network after
the shadowfuck was done
which means they weren't serving sync
data and all they were doing were
keeping up with the chain which is great
but constantly thinking on minet also
adds load on the machine so we want
accurately simulating that so
potentially on future shadowfox we're
gonna have to add a bunch of syncing
nodes that start up later on to see what
could happen there
and the last one that we didn't find was
a failover beacon note scenario a lot of
people are really obsessed with making
sure they don't miss a single
attestation which means they have
multiple Beacon notes set up and that's
something we just never did so a lot of
the requests were just being sent to the
primary Beacon node and not the backup
so when the failover happens the backup
wasn't ready
um I think this has also been patched
right now but it was a really tricky
thing and we should try
more failover backup scenarios in the
future I guess
um one cool thing about why all of this
wasn't found
most of them are optimization bugs
um as you can see merge still went fine
but we could there's still some room for
improvement and it's a really hard
trade-off to make between do we want to
spend more depth time in fixing these
optimizations or did we want an earlier
much
and that's kind of the tricky thing do
we want earlier fox or do we want
completely buckless free free
um free Fox no idea but we're definitely
going to be adding resource constraint
machines to the mix next time we're
thinking of just getting a bunch of
Raspberry Pi's and doing some Shadow Fox
on there see what happens there
and another reason some of the bugs
appeared but they didn't appear on the
shadow Fox is last minute commits
um The Last Shadow Focus I think a week
before the March and the last releases
also happened a few days before the
March
so yeah it's a hard trade-off
um if you want to join the testing
efforts please join Mario Vega
his email address is here send in your
information there and I think I supposed
to okay I was supposed to have one more
slide but I will just talk about that
other slide over here
um so we have a bunch of testing tools
that we've completely open sourced if
you're someone who wants to run that's
nothing kubernetes if you're someone who
wants to just run kubernetes nodes if
you want to set up your own networks we
have easy ways to set up Genesis if you
want to
um yeah use the same stuff we use for
shadowfox all of it is completely open
source quite a few organizations are
actually reusing our um our stuff so
shout out to them contribute back
whenever you can you can find most of
the tools on either on the ethereum
GitHub repo or on the eat Panda Ops
GitHub repo one of the two and yeah
that's about it that's most of what we
did and did not find on the merge
thank you