foreign
[Music]
really just our story and how we've
developed lodestar over the past three
years so about us my name is Cayman I
work at chainsafe it is a great company
we're hiring we basically build
blockchains we have a whole bunch of
layer one projects lodestar is one of
them but if you are interested in
getting into core development we're a
really great place great culture so
lodestar is it's one of these projects
at chain safe we we're um we're
basically an eth2 or ethereum consensus
ecosystem written entirely in typescript
um we really take the open source ethos
to heart we have all of our meetings
they're public everything we do is
public and we really encourage anyone
who wants to contribute we we try to
help people out and get get people
involved a lot of our team members
actually started started their work on
the project through their open source
contributions and it was kind of like a
open source to contract to hire kind of
a situation so we really we really like
that that kind of ethos but we're
talking about today is specifically our
consensus client and how we got from
like prototyping typical kind of code to
something that's like production ready
something that you can really rely on
something that doesn't just blow up all
the time so before I kind of step into
our story I just want to kind of do a
very high level of what metrics are uh
this is I'm not going to go too deep if
if you really want to go no more you can
like read the docs about these things so
um metrics in our case we're using
Prometheus
Prometheus basically gives you the the
tools to build time series data for
specific elements that you want to track
in your code the kind of three main
things that you're tracking is either
counter a gauge or a histogram uh we can
go we'll go into kind of some examples
of all of these things but basically
what you want to know about metrics are
you have some value something in your
code that you're wanting to track and
you want and Prometheus will kind of
query the query your software over time
and you can get a sense over time of of
how things are changing how things are
are moving
um then you use grafana to visualize
these metrics so
um a picture is like a thousand words so
it's basically the difference between
like looking at a bunch of logs and then
looking at like pretty graphs that can
show you give you a little bit more
insight give you something at a higher
level than just squinting
so
our story
um basically we we started
um with really bad uh with not very much
infrastructure not very much a
discipline and how we
tackled
um metrics or tackled like the data that
we we track and eventually we kind of
like over time and like through a lot of
pain and and suffering we kind of
realized that metrics are the way to go
metrics give you uh insights that you'll
never get just by looking at logs so
really metrics aggregate a bunch of
information and give you
um the tools you need to make better
decisions about how you release your
product what things to focus on like
what are priorities and they're also
just really pretty so they're cool so
this is a picture from the early days of
lodestar from the interop walk-in in
Muskoka in 2019 here's our message yeah
we're stuck syncing and some P2P issues
uh we didn't really know what what was
going on like why things were breaking
we were looking at logs actually on the
other side of the screen here you you
know we look happy but it's like uh oh
things are breaking
here's some uh
um some funny issues that we uh had you
can see priority low adding Prometheus
monitoring we didn't even know we were
doing uh
um yeah out of memory om out of memory
it was just like plaguing a lot of our
development throughout 2019 into 2020 uh
lodestar was like very Proto very much
like a prototype and we ran into the
issues time and time again like we would
have the same thing happen multiple
times and we weren't like like
regressions that were happening and we
weren't really like learning our lesson
kind of the turning point was you know
starting to take some time to to Really
uh which takes the time to like build
out these metrics and build out these
dashboards so we had a contributor I
think I can save a new world maybe with
peace
um and so throughout 2021 we started
going really hardcore into adding a
bunch of just out of a bunch of metrics
um and taking time to think through what
questions that we want to answer what
pieces of code are dark
where is
are you know where are we spending time
in the in in our software where are we
where is a bunch of memory being spent
we really kind of uncovering all of the
complexity of this running software I
mean blockchains are really complicated
software there's a whole bunch going on
and
um unless you're really tracking
tracking key pieces you're you're kind
of going blind and now we're finally
kind of at a point where
um
we for any new feature that we're adding
we ask for metrics so
it'd be very important to track the
retry Behavior meaning add metrics for
so that brings me to metrics driven
development
um really the the kind of the key the
key thing that we've learned is that
every large feature should be documented
with metrics so if you are adding a case
of a retry mechanism you're adding some
new feature you need to make sure that
it's actually working and the only way
that you can
or maybe not the only way but a really
the really great way of doing that is
being able to show that visually being
able to show okay our cache is now
bounded let's see let's see that in the
metrics let's see how it's How that
cache is growing over time or let's see
how many times the retry mechanism is
actually being activated
um
those are the sorts of things that you
really only look confined by through
through these graphical methods through
these metrics
um you're going to have a really hard
time looking at the logs for that the
other great the other thing about
metrics driven development is that any
deployed software that you use you need
to be monitoring it very carefully for
regressions so look here this is
the process Heap increasing over time
this is a comment from Lion for the last
two days the leak is clearly visible an
impact of 75 megabytes a day yeah so
that's going to be a problem if your
blockchain is if it's just increasing at
75 megabytes a day continuously you're
going to run out of memory
and these are this we were able to catch
this this issue before we actually
before we actually cut a release so
um
it makes your release process a lot at
least release process is a lot better
here's another here's another example of
a regression uh we deployed deployed a
version of our software and we saw the
cache size grow it's like oh is that is
that is that a bug is that a
is is it okay well we only actually
caught it because we're tracking these
things
uh you probably aren't even going to
have a log that's going to show you your
cash sizes so unless you you actually
measure this you're going to be blind
the other great thing about the about
dashboards and about metrics is that it
lets you correlate different different
pieces of the code to find out where the
problem is so here we have different you
can you can see a bunch of different
graphs kind of all at the same time and
see
um okay something is affecting the event
the event Loop lag
I can see at the same time the active
number of handles is growing the number
of requests is growing so you can kind
of
um
you know kind of solve the mystery in a
in a in a way that you're not going to
be able to do in the logs another great
strategy
uh is
um
is using different versions if you're
running software so running
um like a staging version of your
software against a production version or
against several recent versions and
being able to overlay data on top of on
top of one another and being able to see
okay well it looks like our beta version
is
is using a whole bunch more memory than
our unstable version so you know being
able to kind of compare and contrast
um it gives you the tools to do this and
so just just kind of like drilling down
to a few a few few tips
um these are very very practical kind of
tips don't abuse the histograms so I
said there's three different types of
metrics
um you just wouldn't doubt you use the
simplest tool possible histograms are
really only used if you're wanting to
see the kind of the distribution of of
uh of something so like maybe like
looking at like request timing maybe
you'll want to see like all right
certain requests are happening very very
quickly but then others are happening
kind of longer
um so the classic mistake with
histograms is
um
is uh using a label that's unbounded
so if you had a metric that was tracking
something per peer ID on the network
well you might have thousands of peers
with and if you each new each new peer
is going to be using a new peer ID it's
going to blow the metrics up and it's
going to affect the performance of
Prometheus and you're not going to be
able to run the queries that you want to
run oh so how do you know which metrics
to add so I showed some of these pretty
graphs and everything
um but like you're in your in your
software how do you know what you want
what do you want to track so really what
you want to do is think about questions
that you couldn't ask otherwise so a
really good example is the size of your
internal caches
another thing you can think about is
like if this feature or if this part of
the code would degrade or explode in the
in the kind of bad case that that's a
that's a prime candidate for for adding
metrics around it
um and then really just like keep asking
questions and then it's like if you have
the data to be able to answer the
question you're good but otherwise keep
adding metrics until you can until you
can answer those questions
so some examples that we we've kind of
come across how often are our streams
being reset how many peers do we have
these are kinds of questions that that
the metrics can help and so I'm going to
show you just a live demo of or a live
uh example of of our dashboards and we
can kind of run through some things
and if anyone has any questions about it
while I'm going through it we can
we can do that too okay so this is this
is a our dashboard for our Fleet of of
uh Beacon nodes and validators
um this is kind of the high level
dashboard
right now we're looking at an unstable
node we can go to like stable a stable
version of lodestar
so this is running uh version 1.1 or on
the girly Network we're synced it looks
like it got restarted about a day ago
some interesting things I guess you know
peer peer count is looking good all
right you know there's nothing
everything looks stable there we track
different types of peers inbound
outbound and uh for us we really care
about
um the number of peers that we're going
to be receiving messages from kind of
continuously those are our gossip uh
peers
so the number of mesh piers on our core
topics
staying about six or seven
it looks like we're connected to a lot
of Lighthouse nodes and a lot of prism
nodes
and there's another load star out there
let's drill into something a little a
little more detailed here
so one thing one one area that we really
struggled with and where we added a lot
of metrics is in our gossip
sub-implementation we found
through adding a few metrics that we
weren't getting blocks on time and it
turned out that we we didn't have enough
peers who were sending us these
um who are setting us blocks and then we
started asking questions all right well
why are they not sending us blocks it
turned out it was because our peer
scoring was too low all right well why
why was the score too low uh well it was
it turned out it was because there was a
specific part of the scoring that was
that was too loans so we kind of you
know you start asking questions and you
can build up
um you built you start building the
answers to that
okay so what you're seeing here is kind
of some of what I was talking about
um we have breaking out the score into
different components
um that different parameters that are
going into your kind of aggregated score
for peers
um what we were seeing before is you
know you'd see a graph where the values
are like going down and down and down
over time or if they're like very
negative numbers
another really useful metric that we
were that we check a lot is um
things related to the VM so memory CPU
um
and disk storage
so looks like our memory is stable
that's great
um
GC is only taking
roughly seven and a half percent this is
things are looking good
yeah I think that's enough of a
enough of a demo here so um
at this point um open to questions and
we're also hiring so if any of this
looks cool or interesting and you know
typescript
hello and thank you for the talk
um so I see that you have a ton of
metrics and I believe that most of them
are useful but
um do you have some kind of an alerting
system for you to be able to know when a
metric has gone wrong because I think
it's just unfeasible to go through each
of them like daily or something like
that you I think you need some automated
way to to alert you that something is
going wrong
yeah absolutely so that was something I
didn't touch on but um it definitely
feeds in
um so we we use uh pagerduty you can you
can basically set up
um thresholds for certain metrics that
trigger
um either slack notifications or Discord
notifications or in the case of a
critical kind of issue it'll call you or
text you yeah we do have that set up I
would definitely recommend that if you
are in running something in production
uh thank you
um again there are lots of metrics do
you did you guys um develop over time
um a way to or you know kind of a mental
model or framework to decide how to
where to put them whenever you have new
um you know graph or metric instead of
or do you just like decide to put it
somewhere and then go back and like you
know to make it easy to to read or to uh
browse the dashboard
I don't know if that makes sense this is
the question about organizing the yeah
exactly yeah so um I think probably have
to grow it organically what we did is we
we started with One dashboard That Grew
and grew and grew and eventually it
became kind of unsustainable to have
everything just piled together so we we
started breaking it apart
um the other thing that the other kind
of issue that we had initially is we
would create a bunch of metrics but we
wouldn't add them to the dashboard so
there was no actual like visual cue for
for a lot of it and so it kind of just
went un
um unnoticed and so one thing we would
we would definitely recommend would be
like if you add a metric also add it to
the dashboard at the same time thank you
uh the question about storage so again
too much metrics or what does it affect
the storage
um well so some of it kind of depends on
your architecture
um of how you're deploying this so
um what we do is we actually have one
Prometheus and grafana server servicing
all our entire fleet of of Beacon nodes
so you know we have a bunch of different
machines and then we have one one
machine for these uh right to to uh
things so
it it doesn't cut into the to the uh
storage we retain for a month quick
question uh is are the metrics relevant
for like other evm chains besides
ethereum mainnet
um or is it just like specific to
ethereum mainnet so yeah a good question
um
so I'd say like I don't know if it's a
half and half but like half the metrics
are like more like for like monitoring
uh the chain something like a like a
block Explorer kind of kind of thing
um but I would say like a lot of the
metrics are specific to the
implementation so and those are the ones
that are I think probably more helpful
um just the internal caches and like
timing timings of things that we're
really really wanting to get right hi uh
I wanted to know if you think these
metrics are useful for other clients and
not just the typescript one and if so
would you maybe consider like you know
putting them in the public domain so
like anybody can see them and apply that
to new clients they might be considering
building right I know that um other
clients do have metrics and I I think
like the ones that are more user focused
uh those are the ones that look more
like a block Explorer and maybe with
some like validator like showing your
balance ticking up over time and I think
those are those are public and I and
there is an effort to standardize
metrics across the consensus clients so
I think some of that's happening
um but again to the to the previous
question a lot of the metrics will
always be implementation specific in
terms of roles and you know in team
members who generally looks at which
metrics and which graphs
um just kind of to get an idea of uh you
know which metrics and graphs are useful
to which team members or depending on
like the role or the the type of work
that they do you're asking if if only
certain members contribute to these are
more like the uh looking at the the
graphs and the browsing the the metrics
themselves like you know for example
there are lots of metrics so not
everybody looks at everything so you
know it's more um what you know what
team members look at look at what kinds
of graphs or right so so for any of
these larger feature pull requests we
have the the person who is authoring or
kind of owning that that feature they're
responsible for
um making the case that that that it's
correct and then it works as intended
and so they're responsible for looking
at the metrics and building building
that case and then as far as like
um our release process we have
um a checklist of basic things that we
look at
um before we before we actually release
we have a testing period so we cut a cut
a Beta release we deploy it to our Fleet
um and we watch it for a few days and
then after a few days we
um gather you know check do the
checklist and any other kind of ad hoc
things that were were noticing then okay
so in web 2 there's a standard of how to
consume logs metrics and traces and it's
done by the cloud native uh Foundation
to make the same pipeline that you have
here more standard in a way that's
consumed by other tools as well uh did
you take a look at that I recommend it
if not the the standardization effort is
to to build out some kind of Baseline of
functionality like you can have one one
Prometheus and grafana running and then
whichever client you want plugged in
here you kind of have some guarantees
like uh it's not breaking and like it's
it's kind of working so yeah that's
exactly what open Telemetry does so take
a look at it