hey everyone I figured that starter
minutes longer I have one more minutes
yes so every now and again somebody
sends me a message on messenger or
wherever and if message kind of goes
along how do i D bug death and I always
I have a follow-up question what do you
mean debug guests just submit an issue
and it turns out that they actually
don't have an issue rather they are they
want to know how they can know if gas is
actually working or if gas is working
properly or how healthy it is and then
usually what we refer people to is well
just look at the logs how hard can that
be
okay I'm being a bit ironic here but the
problem with looking at the logs is that
we either arrive in this situation where
death is just silent and there are no
logs and you're just waiting and waiting
or maybe there's a log every 15 seconds
but you still have no idea whether
that's good or bad or we have the more
ingenious people who raise the log level
and they still have no idea what's going
on so the problem is that logs are
really really useful if you know
specifically something that you're
looking for but if you just want to get
a general overview of what your node is
doing then logs are perfectly not useful
so what can we do instead well first
things first
people mostly are interested in whether
they know this working or not and this
is not such a simple question to answer
because it's not a binary of course if
they know is not working we see it that
it is not working but if it's working
then that's an entire spectrum it might
be working very well it might be working
just barely but it's still working now
the issue is that if it's working very
well then everything is fine if it's
just working barely then you might have
an issue at 3:00 a.m. in the morning and
you want to be able to tell how healthy
is your node now it's not suppose that
we start with latest release it kind of
works but a lot of him flood of things
can actually influence your node in some
one direction or the other for example
every time we do a bug-fix release
hopefully it improves the situation and
of course if you run it on
or powerful hardware that also papers
over any issues that we might have with
flaky algorithms or suboptimal things
Network an activity of other
externalities like workload again
influency opera up your operation
allottee or however it is pronounced
sorry essentially of course if you put
if you run a girly node on your notebook
then it will perform really really well
whereas if you run may not then you
might run into troubles these are kind
of natural now the thing is when you
start running a production
infrastructure then just knowing that
you know it works is not really enough
you want to know how well it works
because it's kind of a game of
trade-offs if you are yes you can put
the most powerful virtual machine
available on Amazon underneath it and it
will probably run perfectly but it will
cost you a lot or you can put a
Raspberry Pi underneath it and it won't
run so that's how do we found them in
find the middle ground and the answer is
actually metrics and monitoring now from
the very very abstract perspective
metrics and monitoring means measure
everything that you can and visualize it
and this is not something new in the
theorem ecosystem I mean the stats page
was available since forever
and everybody knows its usefulness
however once you you want to measure
something more it's really hard to
figure out okay what exactly do you
measure I mean a software system there
are gazillions of things that it does so
what is important and what is not
important and to answer that you
actually need to have specific questions
what kind of questions would you like
the answer to and for example us in the
gasps theme we usually have three
questions one of them is how do the nose
behave across the globe the second is
that if some node looks weird then what
is it doing actually and last but not
least if you found an anomaly and we
fixed it then it is really important for
us to see that okay the old version did
something weird does the new version fix
it yes and does the new version really
not ruin anything else yes or no so if
we dive into actual things actual
examples the caffeine is kind of running
eight boot notes globally across various
continents now first question that we
want to answer is are these boot notes
healthy and how what is the cost
now whether the blue tones are healthy
or not for us it's really simple
ordering are they in sync with the
network or not so we simply just
visualize how many blocks behind there
are from the chain head and immediately
if it some Boonton falls behind that we
know that something is wrong but this
just gives us the binary thing we know
that is healthy or not but we don't know
what's the cost or how healthy it is now
if we if in order to find out actually
how healthy it is we actually need to
also chart out the resource consumption
of it and as a computing system you have
four major resources CPU memory disk and
network and of course you can split this
cup a bit and network up a bit in
input/output etc but only know if we
chart all of our eight boot nodes across
these four matrix types then immediately
we can see anomalies for example on the
memory chart we can see that the yellow
boot node is actually using two gigabyte
more memory than the rest or CPU is a
similar thing and immediately you have
something to take care of and the other
interesting aspect of it is that if you
know your notes have 13 gigabyte of
sorry 32 gigabytes of memory and you are
at 8 then you don't care about it you're
really far away from crashing however
once you start reaching your limits
that's when things will start going
wrong so you can just looking at this
dashboard you can immediately see how
close you are to failure cool now
just to dive into one of these examples
DevOps team a couple of weeks ago told
us that hey you're kind of using too
much bandwidth I mean it costs too much
they care about the money not the
bandwidth and so we looked at our charts
and yes indeed the boot nodes are
actually pushing out five six megabytes
per second of data and we're wondering
okay what does the boot not do I mean
this chart tells us how much it costs
our resource wise but we don't know what
it's actually doing with those resources
so those are actually our intermediate
level monitoring questions let's try to
figure out what a theorem node is doing
with its disk resources or other
resources now in case of not working
what can we do well first up we have on
previously we've measured the total
bandwidth usage but we would like to
also match
the individual map bandwidth usage of
individual protocols since the Bouton's
are running aetherium and light client
protocols those two are the ones we want
to run but we can go even further down
and actually measure the network usage
of each individual Network packet
I mean packet type within those
protocols and if you chart these we get
these really nice fancy spikes and
immediately we have three things that
are really really obviously strange for
example in the top two charts those are
the input an upload and download speeds
of the theorem protocol and you can
immediately see that the light blue
thing is causing quite a lot of traffic
and I'm not sure whether it's visible or
not but I will tell you the white blue
has the label of transaction propagation
so just by looking at these charts you
can immediately see that there's
something wrong with transaction
propagation it's taking about 1.1
megabyte per second traffic and maybe
this is enough for you maybe it is not
enough for you to decide what's next
step but in our case yes we are kind of
aware of how the transaction propagation
works and this means that we actually
need to roll out a new aetherium sub
protocol or I mean a new version of the
theorem protocol with an alternative way
to propagate transactions because
otherwise there's no way to get this
down but we immediately know what the
fault is and we have an idea of how to
fix it now the other anomalies from
these charts are that the boot nodes are
uploading a significant amount of data
and our guess again is that since these
are the boot nodes everybody's trying to
synchronize from them so if you want to
stop people from synchronizing them well
of course we can nuke the boot nodes
half way offline so that they refuse to
give you the data but that's not nice
alternatively we need maybe a better
discovery protocol so that you can find
beers faster better peers etc again just
we just looked at what the node is
consuming its network bandwidth on and
we can immediately make really nice
guesses now there are four resource
families why we can do a similar
exercise for CPU usage now CPU search if
somebody were to ask you what does the
theorem node use CPU for
the no-brainer answer is block
processing and this was our false
assumption for many years - and about
maybe half a year ago somewhere in
January we decided that okay something's
not right so let's try to split that
block processing up so we saw that okay
block processing takes a hundred
milliseconds but what's inside that and
actually figure that well when you run a
transaction you there is execution
components we also load data from disk
and we also write it to disk so let's
and we also do some hashing so let's try
to actually break it up and meter all
these components individually and then
the resulting chart was a bit I don't
know a bit surprising it turns out that
if you run a full node transaction
execution I mean actually CPU running
and computing stuff that's 25% max of
the block processing so 75% of block
processing is shuffling data around and
this is an extremely important thing to
know because all of a sudden you realize
that optimizing the EVM is not that
important so optimizing the database
makes a lot more sense than that even
caring about how much one on up code
costs or the other and in the case of
the boot knows this gets even worse
since everybody is constantly hammering
us with requests in the case of boot
nodes the ileum actual execution is 10%
of the block processing time so that's a
that's kind of scary but again we have a
brilliant idea of where we can optimize
so just a single chart just some
information we can immediately see how
to proceed now other interesting facts
that these charts allowed us to see is
that transaction validation or
propagation I mean the you know networks
throughput is I don't know something
like 20 transactions so you would expect
that shuffling 20 transactions per
second around is no big deal except when
you chart it out and turns out that the
boot node is receiving 11,000
transactions per second most of them are
duplicates half of them are invalid half
of them are or some portion big portion
of them are underpriced that are get
rejected but it's really strange and
when you look at
that number you all of a sudden realize
that wait so the transaction pool needs
to be really optimized for this
throughput so it's not the throughput of
the blockchain rather the throughput of
the junk data that's coming in that's
defining how much processing we're going
to do and of course we have two more
other big consumers for the CPU one of
them are RPC calls and which we didn't
chart yet so that's something that we
want to do in the future and network
handshake requests which might seem like
why does that even take CPU well if you
run the Raspberry Pi to murder one core
from erase verify just doing the
cryptographic handshakes in the network
so again we need a new discovery
protocol to fix it and a big third one
third category is disk now in the case
of this it is extremely hard to measure
because kind of fun you run a program
you have all these layers on top of one
another operating systems containers
libraries and they all like to be smart
and all like to cache and this kind of
means that there's really hard to
measure the thing that you actually want
to measure we could measure how much
data we're pushing the disk but that
depends on how much RAM the operating
system has on how aggressively it caches
we could check how much it could ask the
operating system to tell it to us but
that also depends on if you run it in
darker darker also start screwing around
with all kinds of caches now if you were
after we figured out that we cannot
reliably measure we decided okay screw
it we're going to measure how much data
we are pushing into the database well it
turns out that that's completely useless
too because leveldb has all these
background processes around compactions
and fancy storage models so in the end
we actually needed to patch level DB and
ship all our metrics into level DB so
that we can get an accurate measurement
and we're actually really happy for the
author of go level d before allowing him
allowing us to keep patching his
database so this one I will probably not
go into but the idea is that after you
found an actual issue so after you found
that specific metric that is really out
of place then you need to find out why
it is out of space
of this place here I gave you some ideas
so those are good ideas R&R enough but
if you don't know if you can if you
don't have any ideas then the only thing
you can do is actually to just try to
look into the details of the algorithms
internally and try to map out what the
algorithms are doing and and try to have
better guesses and for example this is
maybe one third of the light servers
charge that Kari put together and there
are lots and lots of these just to
figure out individual tiny bugs but I
won't go into these because that's
completely out of scope here however
let's suppose we did manage to find what
all we did manage we have an idea of
what what the anomaly what was the cause
of the anomaly and we have a poor
request to fix it so the next thing is
we just open a pull request merge it in
and everything is fixed yeah no wrong
the issue is that there are simply so
many moving components in a theorem note
that the fact that we fixed one thing
might actually break others so the
lifetime of these performance or these
anomaly fixing or request kind of look
like people open a PR and then we
actually run a one week long benchmark
to see what does it do now this is the
perfect case we see something like this
maybe once a year these are the miracles
of development when this was actually a
pull request created by Garry disabling
some internal data shuffling within
level DB and what it here actually the
green lines were the master and the
yellow lines were Garry's experimental
PR essentially by just swapping a few
variables he actually managed to cut
down the disk IO by an order of
magnitude so that was I mean you don't
get these kinds of charts at all these
are just you don't simply don't believe
them when everything is perfectly better
than previously what you get a lot more
often however is these kinds of charts
this was from a previous monitoring
system that we had where somebody opened
the pull request at on Windows Windows
really chokes on folders that have many
files in them so let's change level DB
to use larger files I mean it seems like
a pretty good idea
why would you use two megabyte files if
you can if you can use 100 megabyte
files so he implemented the PR submitted
it and actually also submitted benchmark
results that yes the number of files go
down the number of use file descriptors
goes down the PR got two reviews
everybody approved it almost got merged
in and then we said that he I okay do
you know what let's benchmark this
because it's touching scary stuff
and then you go you start to look at the
middle benchmarks and you realize that
the disk rights simply exploded so it
blew up by two orders of magnitude and
again it was a perfectly good PR just it
didn't take into consideration some
weird internal thing of leveldb
and these are the PRS that can really
really bite your performance where
everything that you do is perfectly
logical except the result and of course
these are still a good case because this
PR you can at least close and then you
get the really nasty ones and these are
generally this is the case that we see
every time when so the charts that you
see here that peak that blue peak that's
actually the Shanghai das attack so it I
just closed it on when we were
processing doing a full thing I'm
processing the Shanghai denial of
service attacks and that's the memory
usage so previously that's the one
eighth release family of Gath every time
you reach the Shanghai attacks we had
this huge big huge memory peak simply we
had various caches in they use a lot of
memory you essentially your machine
needed 13 gigs of ram to be able to
process those blocks and of course
that's a lot so we decided to replace
that 13 gigs offer I mean that caching
algorithm with something completely
different and yeah it was complete
success it completely wiped out the the
peak and also even the general data
usage sorry general memory usage of Gath
went down except the disk usage went up
by 30% and now you have this huge
problem and question what do you do I
mean 30% extra disk i/o is horrible ten
extra gigabytes of memory usage is
equally horrible and
you need to make trade-offs you need to
make really hard decisions that which is
better and then in the end we went with
the extra memory usage is better simply
because if you use a lot of money if you
want to use more memory than available
you crash I mean it's a heart failure
whereas if you use more disk than you
would like then things get slower but
they still function so in the end this
is why the actually the won 9-0 release
uses more disk than the won 8 whatever
the last one but all in all we think it
was worth it
however sins can be fixed it now we were
much better so so this thing was fixed
already cool now that is kind of how the
gas team pushes out these are how we
monitor stuff benchmark beers and push
them out now if you would like to repeat
a similar thing in your infrastructure
you kind of need to decide how you want
to monitor things you either pay for
monitoring other service via data that
we did that for two years but dated
August actually has limitations it
didn't allow us to push all the metrics
that we had in so in the end we switch
to our own node we just ran graph hana
and if you actually decide to run your
own on-premise instances then you need
to decide also on a database to push
your data in either you go with
Prometheus or influx DB they say these
are kind of whatever you prefer the
reason I explicitly mention them is that
monitoring infrastructure boils down to
these four things so you don't care
about anything else in the world and
from our part guests can actually push
data either it can we can export data
into the data doc format we can push
data into gravano and Prometheus too so
whatever rocks your boat we can do it
and just just for the sake of
completeness I also linked in our own
dashboards so we exported them we use
graph on an influx levy but you are
welcome to use them integrate them do
whatever you want with them so all our
configurations are published there they
will probably also learn the repo
actually cool now what are kind of the
lessons that we learned while creating
these charts they were that you must
measure as low as you can
meaning that every abstraction is
transforms your data everybody tries to
be smart and those
usually ruin your matrix now you always
must measure your worst case numbers I
get it that the operating system is
really smart and makes things more
optimal for you but if you assume that
you run out of memory then your worst
case numbers will do will be the ones
actually hitting you so we always be
aware of what's the worst that can
happen and measure everything that can
you can afford as I shown you previously
you can gain a lot of information with
if you have a lots of tiny detailed
matrix but eventually it will become too
expensive for example Martin did a
really awesome measurements where he
actually measured the cost of each
individual opcode it's really insightful
but obviously you cannot run it in live
production environment still if you have
the numbers you can debug an issue if
you don't have the numbers you will may
get gather the numbers the next time you
reproduce that issue so there's no way
to fix it without numbers yeah and
that's about it thank you very much
[Applause]
you