everybody thanks for coming to the e2
client summit I know there's been a lot
of e to talks so maybe you're sick of it
but it's really special to be here this
is my first Def Con didn't didn't go
last year I really wanted to all the
rest of prismatic steam went so it's
truly amazing opportunity to be here so
what I want to talk about is our test
net we've we've been running one for
some time now and we've actually learned
a lot so let's dive into it first of all
who who are prismatic labs or who is
prismatic labs were a self-organized
group of just dedicated bakhtin
engineers people who really wanted to be
a part of scaling etherium and sort of
making this reach global adoption
capacities so there are six of us today
most of us are here so come say hi if
you see us if you don't know us already
we'd love to chat what's interesting
about this group is that we all formed
really organically online nobody knew
anybody before any of this and we were
just inspired by Vitalik wiki article
about sharding and wanted to start
making moves on that because we felt
that you know at the peak of 2017 when
things were really like maxing out that
we needed something soon to keep this
momentum going so we were one of the
first teams to come together to start
building sharding and then the proof of
stake at Casper all these things to
merge together so it's been pretty
remarkable journey so far one of our
biggest achievements is to run an actual
test net this has been we started doing
this back in April of this year so we've
been running it for I think six months
now what what this comprised what's this
comprised of is several beacon chain
nodes running in different environments
different areas of the world it's open
to anybody so anyone with a computer can
join it's typically runs for
several weeks or several months
depending on if we run into some fatal
bug or if there's a new spec update
where we have to rewrite everything
which has happened many times it's also
the full into an experience there's
nothing really mocked out here when
we're talking about this test net there
are real deposits coming from a real
eath one chain and the POA corley
network we've actually had hundreds of
people join we at one point accidentally
merged with the ipfs network so we
learned not to use default the p2p
parameters because that's where the p2p
does and we were wondering why do we
have 500 nodes connected to us all of a
sudden so that was a kind of cool cool
thing to learn but but most of all it's
just been really really fun and it's
actually really easy to get them all so
if you wanted to be a validator to see
what it's like to stake with prison
client we have this test net website
which is super super straightforward so
so when we're talking about test nets I
wanted to ask fatale ik what is the best
test set of all time and I didn't get a
chance to ask him yet so this is what I
think he's gonna say is he's gonna say
Prismatic labs best test Network and the
entire planet and the entire universe so
I challenged someone here to find him
and ask him and see if he he actually
says this but
Vitalik has run it as far as we know and
he's really enjoyed it as so many other
people so the first lesson that we
learned is that participation has to be
very easy you have users coming from all
types of technical backgrounds or a
non-technical background people
interested and they want to check it out
but they're really intimidated by adding
a lot of tooling to the system just to
build the project and running into these
like weird errors where you you you know
it really depends on your local
environment so what we've done is built
this test net website that within six
easy steps you can be validating on
real be contained and actually seeing
rewards balances change and and even
penalties if you go offline so the
approach we took was to use docker
images and this is typically pretty easy
for users to get on on board it still
requires some installation on your
machine and some you know command line
inputs but overall you don't have to
spend 10 to 15 minutes building the
project as some of you may have seen
from the Interop event if you tried to
build prison for the first time it takes
a really long time so downloading these
images which are 15 20 megabytes each
just seems to be pretty straightforward
approach haven't had too many problems
we also made it so that within the
website you could get some test ether
from the Gourley Network we have a
faucet built in here an interface so
that you can just paste in your deposit
data key and then with meta mask or
Portus you can send that transaction to
the eath one chain and wait for that to
be included into a block and then go
through you know go through the voting
period and and all these are the things
and eventually you do get activated as a
validator and you start receiving
assignments from you beacon chain it
actually has like a really cool like
progress bar so you can sort of they see
something's happening when nothing's
happening so it's it's been really cool
and and this has been our biggest value
add for users and and one of the biggest
things that we started off with learning
because before we got here it was just
running it on your own computer running
one client and then running two clients
on the same computer then on different
computers and all of this was was you
know crawl before you can walk and then
now we're starting to run so this has
been a my favorite lesson so far just to
make onboarding easier probably the
biggest lesson though is that when we're
running
we're running multiple instances of the
beacon chain in the cloud or whatever
and those are really hard to be
monitoring is he
scene from like screenshots from the
inter prevent when you see seven
terminals on one screen it could be
really hard to see that there's an error
or even to catch that one of those nodes
was not even talking to any of the other
six which did happen so having good
monitoring in places is absolutely
critical to understanding when errors
are happening in your system and how to
find the root cause of those errors so
let's take a look at a few a strategy as
we do for monitoring things from from
the very beginning we have built out
metrics collection and monitoring
through these metrics so here you can
see a test network that's been running
for two hundred and twenty nine thousand
slots it's starting to look a little
weird because these these graphs should
be pretty you know flat and straight
line especially the you know the CPU
churn and on these kind of things so
here we might say like oh if there
hasn't been a new block in thirty
seconds and will fire an alarm and will
alert to that make your phone ring or
whatever and then when you are woken up
in the middle of the night and you see
something like this you have some clues
of where to start looking
so to digest this for you guys you can
see on the bottom right that there's
chained reorganization imbalances so
nobody was agreeing on anything we had
you know hundreds of thousands of go
routines happening at the same time so
these four concurrent processes and
that's leading to the CPU maxing out and
then this nice like up and to the right
line is starting to fall apart and this
was like a nightmare scenario so having
these resources is super super helpful
but now you know like what's what's kind
of going wrong like how do you dive into
some very specific details like maybe
block production is taking longer than
expected and you want to identify the
route
cause of these things so what we do is
we do a practice called code
instrumentation so will annotate pieces
of the code that we think are expensive
or really interesting to track so here
you can see the entire workflow for a
block proposal so starting at the top we
have the validator client which is a
separate process from the beacon chain
sending the block to the beacon chain
and then something along here is taking
236 milliseconds and you know we kind of
expected that to maybe be 50
milliseconds typically so we can dive
into each part and see why it took so
long this one took a long time because
it was processing a lot of slots so we
might say was there a long period of
skip slots or can we cache this somehow
so that we don't have to do this work
redundantly where can we find ways to
improve and make things faster and some
of these things like you know from your
intuition and you can write tests for
and kind of write you know benchmarks to
see like okay I know this is a place
that could be bad and I can make it
better but other times you won't know it
until it happens in the real world so
this has been super helpful for us
speaking of benchmarking we do profiling
so we'll take when we see that there are
a hundred thousand go routines or
something's taking a long time using a
lot of CPU will take maybe like a 30
second snapshot of the CPU profile and
then look at this this is called a flame
graph because it looks kind of like a
flame and you can dive into the most
expensive parts the things that took the
longest and go and then reproduce this
in a local test and be able to iterate
on that and make it a little bit better
this has been super helpful for us
another key thing that we've done is
this is kind of recent is to do log
ingestion so when we're running multiple
processes and trying to understand
errors and we have evidence we've users
are filing bugs that the signatures are
not verifying we can query the
the central repository of logs and see
on what frequencies are happening and
did we really solve the problem so here
we can see that these should be zero
because this is an error that we don't
want to have but we can see it at like
specific times and maybe correlate that
with some other interesting data that
will help us identify the root cause of
this because you know typically it's
easy to make a problem go away but to
really understand what was the actual
problem is how you're going to get the
best solution lastly on monitoring we do
something called canary analysis so we
are always so when the test net is
running which is almost all the time
we're pushing code out as frequently as
possible with and without catastrophic
ly breaking it so what we'll do is we'll
run the new the new binary that we've
just got the new image that we've just
created we'll run it alongside the
existing the existing binary and start
them at the same time and then see if
there are any major regressions so in
this case we see that memory usage is up
by 56% and that is kind of concerning so
this canary report got a score of 77
percent which based on your tolerance
you may want to fail immediately or not
let this go through maybe you want to
have some manual approval before pushing
it out our you know success or auto
approved criteria is 90 percent so
anything that's almost almost perfect
will go through anything about 50
percent we can look at and anything
that's below 50 percent will just stop
immediately and and we'll take a look at
it later but that's been really great
from keeping us from tearing down our
tests out on accident okay and this is
something that that we that we really
learned pretty recently so our team is
is very distributed across the world
when nobody lives in the same city it's
it's uncommon that we're all in the same
time zone so actually being all here in
Def Con we're on the same time zone so
this this monster is coming out a night
is even more scarier
because what's actually happening is
like when all of us in North America go
to sleep the test net starts to fall
apart and then the people over here and
in Asia are stuck with it and like kind
of on their own by themselves or maybe
they didn't have the same knowledge that
we had learned that day and we go to
sleep we try to pass off things
but it's always like when you when
everything is sounds really good and
really nice and when you least expect it
things start to fall apart so that's
been super interesting when we landed
here our for Def Con day one we were
right all in the same time zone and when
we woke up in the morning the test and
it had been stuck for two hours and no
one was around to see it and we couldn't
fix it in time before it was just
unrecoverable amount of time since
finality so only ended up just
restarting it but that was something
something that we've learned is that if
you can put somebody so you're always
awake 24/7 or the the actual solution we
did for this was to have the alerting
that goes to your phone you know pass
through your do not disturb wake you up
in the middle of night and make sure
that like when when no finalities
happened in 100 ebox that someone is
there looking at that because that's a
really really bad thing to happen so
lastly like why did we do a test net so
early like I said we've been doing one
since April and this was before we even
had a networking spec so we kind of made
one up to get our clients to talk to
each other and we knew it wasn't gonna
be final we knew we're gonna have to
rewrite everything in that regards but
why did we do this we actually learned a
lot of things besides those those few
lessons so we're able to find race
conditions which you can check for with
race detection but it's not always
perfect one of the biggest things is
it's long long lasting memory leaks so
things that grow over many days will
have binary that runs for maybe a week
and we'll see that the memory use is
just slowly slowly slowly going up when
you're running this on your laptop you
don't ever see it we don't never see it
in a unit test so these kind of things
are super helpful
UX problems users come to us and say hey
I don't understand why you won't let me
validate like what's going on and you
know that's been great to just make the
client better for everyone for usability
so we can help understand what users are
frustrated with or what they want and
then we even actually run into
aggregation issues so some recent thing
that we learned is that you can't
overlap aggregated at two stations that
are already aggregated so we were seeing
we were trying to aggregate those and
they were failing but silently failing
for some reason and then we were getting
this is not very consistent graph of
finality so that was something that we
only realized by running the test
Network for a few weeks and then lastly
testing gaps things we can do to make it
better so like the canary analysis and
and other other like long-lasting
tooling to make the code safer and to be
able to have releases more frequently
and just to make it a lot more secure so
that's it for me thanks everybody
[Applause]
you