all right well hello everyone I'm Adam
link I'm an engineer with air swap and
today we are going to be talking about
running a guest node cluster in in
production Netscape so as was alluded to
about 15 minutes ago there is no open
source way to run a guest node at scale
we will be open sourcing our production
infrastructure in about 20 minutes yep
so this is the work about three
developers over about two months to
develop the infrastructure with a lot of
the pitfalls that we experienced along
the way so it's at least something for
you guys to reference as you're building
a Gotha node cluster or perhaps use on
your own so at the core this adapts need
to keep state right when you run a DAP
your data store is inherently
distributed and distributed datastore
means that you need to stay in sync with
your peers to decide what the correct
view of that data actually is there's a
whole host of issues that can occur if
your front end or your back and
application does not properly reflect
what the consensus of the database state
actually is and there's nothing worse
than users not trusting your application
because they can't trust the underlying
data but you also can't sacrifice the
responsiveness of your application when
you're waiting on an eventually
consistent database users on the
Internet are generally not accustomed to
waiting five minutes to do a state
change operation and while we in the
industry may be okay with waiting for
transactions to settle if we want to
actually get mass adoption we need to we
do understand the general public wants
which is a lack of leg when they're
doing a state change operation there's
two types of data that you need for
state management the first type is the
balances on your accounts right this is
the eath that you hold in your wallet
and for many depths this is a critical
piece of data in the user onboarding
process because they may need to swap
their eath for your proprietary DAP
token
in order to use the product that you're
creating the second type of information
is the the state tree this is
information about events within your
smart contract this is things like the
users token balance smart contract
function executions and token balance
transfers
however contract events which is the
state of your smart contract are
actually implicit and computed based on
the data in the blockchain so this means
that your DAP needs processing power
behind it when you're running your
ethereum notes if you're out of sync
with your nodes obviously you're
misrepresenting the state of the world
and that's a dangerous place to be in as
an application so with the importance of
staying in sync now for your DAP kind of
resting on your shoulders as a developer
some of you guys and the industry as a
whole we've wondered can we just
outsource this and pay somebody else to
do it and the answer is of course you
can there's many companies out there
that actually offer services where they
will run nodes for you not all of them
are geared towards OLTP processing that
adapt needs some of them are more
analytics based you know Google recently
came out with a product like this but
the question we need to ask ourselves as
an industry is do we want to move
towards the MasterCard Visa model right
so right now with credit card processing
there's only a few companies in the
world that have the technical expertise
software and hardware to actually keep
up with the number of transactions that
happen on credit cards and as we move
state management towards centralized
parties we move the entire industry
towards that model again and the
question is do we want to do that do we
actually want to be there the solution
in our mind to combat the centralization
is having people run their own nodes but
running an ethereum node at scale in
production is really not well documented
online right now there's also costs
associated with running your own notes
not well-documented either so not only
do you have the overhead costs of
the actual infrastructure for running
your own node but you also have costs on
your team to create the infrastructure
and create the software that manages
your nodes make sure that your nodes
stay in sync there's a lot of resources
online pertaining to optimizing nodes
for mining but there's not many
resources optimized for reading and read
heavy workloads that adapt needs it gets
even worse when you start going into
data querying and trying to understand
how you process this data there's very
few resources online for data querying
for taking block data and storing it
internally we as well as everyone else
who was up here before I have created
tools for doing that
we're not outsourcing those yet but you
know that is something that we are all
creating our own data querying layer on
top of the blockchain and there are some
companies right now like the graph that
are actively working on making a way to
query using graph QL but they also still
require that you run your own nodes or
you find somebody that can run nodes for
you to get the underlying state data so
right here is a simplistic view of our
death architecture we run a production
cluster and it's basically a group of
load-balanced death nodes behind RS re
in an auto scaling group behind an
application load balancer on AWS the
application load balancer allows us to
upgrade non-secure requests to TLS we
can also run very tight health checks to
cycle out nodes that are not healthy out
of the load balancer target group so if
they fall behind for instance like the
head of the chain and usually what we
found in our internal test is we
recommend studying the frequency of
those health checks to a time that is
less than a single block propagation
time so right now that's doing a health
check inside of a 15-second window this
allows nodes that fall behind to be
dropped at the pool quickly so they can
sink back in so you're not querying
nodes
that you know are behind in state the
load-balanced nature of our clusters
also means that if a single node does
fall behind our overarching service
doesn't actually fall out of sync and
become unavailable so we use our goth
nodes on the front end you I on the air
swap platform as well as in the backend
for state management and storage on the
front end this takes the form of balance
checks in the UI wrapping and unwrapping
width and also approvals an approval
state for the atomic swaps so if you've
used the air swap platform you know that
I like on the new spaces on the right
hand side you know we have a bunch of
information that comes from our Gotthard
state we also use a heavy polling method
to handle trade confirmations so if
you've ever popped open the network tab
in chrome while using air swap you'll
notice that we do heavy polling against
a gap cluster basically when you do a
trade to make sure the trade actually
goes through and gets executed we also
do allow limited very limited third
party access to our GAF cluster we do
have some of our maker partners who use
our GAF cluster as well for balance
checks both for their own wallets that
they operate out of as well as
counterparty balance checks to make sure
that the counterparty taking a trade is
a valid taker further our parties and
also use the GAF cluster for market data
and price information in addition with a
couple of their sources but we we do
provide chain state to them for pricing
information when we were setting up our
Gotha cluster for the first time we ran
into many issues with the way gift was
written and our particular use case of
heavy reads this is not a knock and the
guys that built Geth it's a great piece
of software but we just operated it at a
scale that really no one else had been
publicly talking about so the first
problem we had was GAF wasn't written
with modern pub/sub methods this meant
that you have to do heavy frequent
polling in order to get state changes
and you can't just simply subscribe to
the events being emitted out of the
blockchain that you want to get
now that that's kind of true because you
can with filters but in and I've heard a
rumor here that this may no longer be
true but at the time filters were
dropped on a node resetting and you'll
find out in a couple slides whoa that
was a huge issue for us but basically
the filters being dropped was a
hindrance to us properly getting a
pub/sub system setup and then further
complicating that is filters aren't a
single node only so when you cost your
nose behind a load balancer the
round-robin aspect the load balancer
means you can no longer guarantee which
node is being hit and if the filter even
existed there one of the solutions we
found at the time was a node.js library
that basically allowed you to do a multi
node filter setup and all it would do is
iterate over an array of guest nodes and
set up filters on each node it was a
heavy client-side solution and we really
didn't think that that was what we
wanted to do we were hoping for more of
a server-side solution rather than
relying on a client implementation so
the underlying issue that we found in
some of the people have alluded to is
the overall ecosystem of ethereum is
meant to be fault tolerant when it comes
to chain State however there's no real
thought that's been given to the notion
of creating a subset of nodes that are
also fault tolerant in and of themselves
and so with adapt this means either
querying a single node and praying that
it doesn't go down using a bunch of
third party nodes and hoping you have
the right state or doing what in fira
did which is writing a really cool piece
of software and hardware called ferryman
which basically takes introspects the
request coming in and then it sends the
request off to the proper type of node
also provides a modern pub sub layer on
top of that and then allows for you to
subscribe to client events the problem
is the inferior solution is very
proprietary and we make sense for their
setup so they can't really open source
that to the community our story on our
gaff nodes we do about 800 requests per
second on our gatherings requests per
day that's over
to clusters of three nodes each so
that's a fairly large amount of traffic
and under this load death actually broke
we had a massive memory usage issue and
a recurring memory leak that caused each
node to slowly utilize all the RAM
available on the box
we tried scaling up our hardware - this
is AWS terms but m-52 X larges so it's a
32 gig of ram 8 v cpu piece of hardware
and with six of those in production you
should be able to do substantially more
than 70 million requests per day but as
you can see from the chart here after
about 15 hours we had topped out our
memory right at 32 gigs and the drops
you see where we actually went in
physically and restarted the goth
process the memory leak meant that we
had to cycle our gas nodes before the
GAF processes used all the available ram
in fact if we didn't catch it in time
the box itself was so the ram was still
used up you couldn't even SSH into the
box to kill the GAF process you had to
restart the whole node itself and this
meant that we were waking up every three
to four hours to check memory
consumption and manually we start the
process to make sure we didn't get
locked out of our boxes and once we
figure that's the usual cycle of RAM
usage we were able to write cron jobs to
then go in and restart our Geth process
and we staged that over about an hour
across six nodes and it worked but for
any of you guys that work in
infrastructure trusting your entire
system to six cron jobs is incredibly
scary so we started tinkering with some
of our guest settings and we actually
dropped our max peers setting and the
memory usage issue was largely resolved
and it turned out - the memory leak with
the max peer setting really only
occurred at the scale at which we were
operating and it wasn't publicly
documented prior to that as you can see
in the charts here those those drops
like that's when we were restarting
death the stability that we did finally
get came at an incredible cost to our
operations team so for at least two of
us we were getting nothing done during
the day other than goth node work and I
can't
that your sleep quality certainly
suffers when you're waking up every few
hours to make sure that your boxes
aren't freezing or running out of memory
also your morale really suffers when
every morning before breakfast you wake
up to 50-plus pager duty alerts and they
just keep rolling in throughout the day
so we wanted to fix this and we embarked
any massive experimentation to find the
correct hardware and software
combination that would be the most
stable and cost-effective
we ended up iterating over ten different
instance types spinning up everything
from t2 micros to four Excel boxes and
we tried everything from compute focus
memory focus instant store focused
trying to find and profile the way that
the Geth process worked under the loads
that we were seeing
we made some concessions along the way
that we have built into our open source
solution unfortunately no pub/sub method
meant that we were just gonna brute
force it and that was one of our big
decisions out of the bat out of the gate
we couldn't guarantee that a single node
was up so we needed to load balancer
nodes we couldn't guarantee there are
nodes were in sync and up so we needed
to write health checks we also couldn't
guarantee that a node was at the latest
head so we wanted to do pure checking to
make sure that we were actually at the
the head state that we thought we were
so that combination was kind of the
secret sauce that no one was really
talking about online which was
aggressive health checking aggressive
peer checking load balancing and
improper hardware so let's talk about
the infrastructure we actually run in
depth is what we're gonna be open
sourcing so when when we run or knows
you run on top of Boone - so we use the
service construct to start our
proprietary tasks here we're actually
starting the death process so this is in
the open source repo you don't need to
worry if you can't read the actual code
but let's break down the options we use
so we brought the max peers down to fix
50 this solved memory leak issue and
also helped with the data transfer costs
that we'll talk about a couple of slides
we use a custom data Durr parameter to
take advantage of the nvme drives that
AWS recently made available right now
our default is cores turn
everywhere you may want to restrict this
for your own implementation we use a
four gig cache to speed up the initial
sync process this is the health check
that we use to keep our nodes in sync
it's probably pretty small but what it
essentially is is a node.js Express app
that just does the waterfall queries
over our peers to determine state so we
compared to in fira and ether scan but
you can certainly customize this to run
on additional peers we consider
ourselves to be in sync if we're within
10 blocks of the head and you can make
this tighter but we found that if you
tighten it up too close to the actual
head state like 3 blocks or less you
will occasionally drop entirely out of
sync with all your nodes when you get a
small chain split at the head which
happens on occasion so we decided to
rather stay in sync all right sorry
available then it completely in sync all
the time we returned back HTTP status
codes that do correctly identify whether
the service is up or down
we also returned back a network timeout
because we found out occasionally in
fira just times out our requests we've
got a couple of the nifty features
health check is started as a service
we've run into issues before where our
health check was down but the node was
actually off we have cron jobs that
reset the max open file limit this was
something that Goeff had an issue with
earlier where it was not respecting the
max open file limit so we just wrote a
cron job to fix this and then we wrapped
this all in a cloud init script so it
automatically spins up the boxes
installs all the required programs death
mounts the nvme drives and starts the
services so your boxes spin up and they
start syncing you can actually check
that in a health check that we publish
as to whether or not your node is
running one of the biggest takeaways we
had was the hardware selection that we
used so we're gonna talk about IO right
now magnetic and spinning disk drives
just don't work you just don't stay in
sync you become IO bound and the the
node will freeze similarly provision die
ops is actually not cost effective you
over allocate your I UPS and you're only
using your max I it's about three
percent of the time so this means you're
just wasting money so we try moving back
to births based EBS drives
and you have to actually allocate three
terabytes of storage space to get the
provisioned I ops that you need but
again because now you're below where
you'd want to be in AI ops you're
actually hitting caps when a large block
comes in and so now you have a right to
that will build up over time you know it
falls out of sync EFS is something that
AWS provides an infinitely scalable
network attached storage and the network
overhead actually makes it so your node
will never sink as well so just don't
try that and nvme drives were announced
a few months ago that's actually what we
use it is a very good io based instance
store drive familywize death is io
bounded incoming data and compute bound
and serving requests so m4 and m5
general classes over allocate memory we
don't use those any other memory heavy
family is actually not going to be cost
effective I three classes is actually
what inferior uses or use the time I
talk to them so they work well but nvm
you drives are now back out and that's a
better instance store than the i3 class
we run C 5 D is in production this is
the proper mix of compute power with IO
4 what we what we run we found anything
smaller than a large instance is
actually not going to give you enough
power to run your nodes in production at
scale and any other family than these it
it's just cost too much so we run 6 c 5
d 4 excels in production and we have not
had a single instance the downtime since
August last factors cost so what do you
pay for you pay for the ec2 instances
pay for the load balancers and then when
you also pay for outbound data transfer
something we didn't know when we had our
max pure set as high as they were was
your outbound data transfer actually
runs into the thousands of dollars per
month so we've dropped this down with 50
peers we actually end up with about 2 to
3 terabytes out every month real quick
slide why not parody when we did parody
about 5 months ago we were getting a
bunch of false nodes so we were not
actually staying in sync appropriately
with enough peers so we just dropped
them but version 2.0
we certainly want to go ahead and
looking at other clients to bring out
and I think that's something that we you
know kind of call for help on the
open-source side if you guys know how to
properly configure parity like this
month that would be great
and lastly here's the CloudFormation
stack like I said we do run an AWS but
this URL will take you basically to our
github repository where you can go ahead
and click with a single launch stack
spin up the exact stack that we run in
production that's all I have
thanks guys
[Applause]
you