morning guys thanks for coming bright
and early so I'm gonna actually tempt
the demo gods today it'll be interesting
because the things that I'm going to
show you today and demonstrate take long
enough to deploy that I kind of want to
get started right now so I also want to
thank the conference organizers for
providing us with these free LTE
connections because that's what I'm
using right now to be able to do this
the Wi-Fi hasn't been great but
fortunately we have these wireless
hotspots we can use so I'll just deploy
this and then break into it in just a
minute so changing my config params to
suit my needs then going ahead and
setting up my dependencies
and then I go ahead and kick off this
cloud formation template that will
deploy the solution that I'll be talking
about right now so you jump back into
that I also want to say hi to one of my
local counterparts from Tokyo
Yuki naka taki which is right here
joining me thanks for being here
so I'm Carl Youngblood I am a senior
blockchain Solutions Architect with AWS
and I've been doing work with
blockchains for quite a while I first
became interested in blockchain in 2010
when I read Satoshi's white paper and it
was one of the only white papers that
actually came with working software it
was really exciting to actually set it
up and run it and I was lucky enough to
be in living near London in 2014 around
the time that like the first theory and
meetups got started as well so before I
joined a AWS my brother and I co-founded
a consulting company called black scale
and we deployed some node infrastructure
solutions for companies like my crypt oh
and the Tesla Foundation and I'm gonna
be sharing some of the things we learned
in that process today so first I want to
share some thoughts about
decentralization and how it can
sometimes differ in theory and practice
so and after that I'm gonna explain a
strategy that we used for deploying
highly available node infrastructure and
then some ways that we DIF mitigated
against some different challenges and
problems with you VMs and also how we
made our deployment more cost effective
and then finally I'll show you a little
bit more of this solution that I just
deployed so one of the great promises of
blockchain technology was that you know
it enabled greater decentralization so
in the same way that anybody the
Internet is a permissionless network
that anybody with a computer can
and the right equipment can just hook up
to and as long as they're speaking the
right protocols
they can they can join in this network
blockchain also eliminates the need to
trust in these large central authorities
to track information even private
permission block chains have the goal of
leveraging decentralization across all
the participants to increase the
security and the reliability of the
network but some of the initial ideas
about decentralization have been kind of
naive so
Vitalik his earlier claim that aetherium
should be able to achieve on chain
scaling while running on nothing other
than consumer laptops has been forced to
confront some perennial challenges in
computer science about data locality and
constraints of networking so while that
scaling work is ongoing and I certainly
applaud it and grateful for their work
their companies still need practical
blockchain solutions for deploying their
infrastructure that are robust and
reliable enough for their applications
right now so some companies rely on
third-party RPC services and these are
very helpful and they definitely have a
place but they also represent a
significant reduction in the overall
level of decentralisation in the
ecosystem so there are times when you're
building an application that's important
enough that you want to reduce your
reliance on external dependencies as
much as possible
by deploying your own infrastructure
something that's fully under your
control and then you also have the added
benefit when you do this that you're
actually contributing to the overall
decentralization of the blockchain so in
practice we've kind of observed that the
best way to birth tomorrow's
decentralized infrastructure is to build
on the reliable solutions of today so if
you need your DAP to be highly available
the underlying node infrastructure that
it relies on needs to also be highly
available
by leveraging the services of an
experienced cloud provider for your
blockchain deployments you're gonna be
able to take advantage of just a lot of
different things so their unprecedented
scale and efficiency and global reach
and also the cost savings that come from
like the pay-as-you-go model instead of
the sort of over-provisioning your
resources for periods of peak load right
so before if you're doing an on-prem
solution you have to kind of over you
know stock up on resources just in case
you need them and clearly this is less
efficient it results in a lot of waste
and one other huge benefit is that you
know these cloud providers like AWS have
been doing this for a long time they
really know what they're doing and
they're already running on over fifty
percent renewable energy and our AWS is
committed to getting that to a hundred
percent so that's kind of a nice bonus
so I'm going to talk a little bit about
how the infrastructure works the AWS
cloud is divided into several regions
and each of them has something we call
availability zones or AZ's so they have
several of these and these ACS consist
of one or more discreet data centers and
they each have redundant power and
networking and they're housed in
separate facilities that are unstable
floodplains though they're
geographically isolated from each other
for disaster recovery purposes they're
also close enough that the pink times
between them are very low so the RPC
service deployment that I'm going to be
showing you today distributes if there
are M nodes across different AZ's to
ensure that the service continues to
function even if one of the AZ suffers
an outage so I'm going to go through
some of the architectural approach to
this deployment first we have requests
that are coming into the Amazon API
gateway and in you know this
Gateway checks to make sure that the
requests are recognized and it blocks
certain forbidden methods that are not
really needed for everyday use that
might incur additional overhead so it
kind of keeps your waist down and then
if the call is valid we pass it on to a
network load balancer and that serves
the request to several container
instances and each of those container
instances is running in a separate
availability zone so in our experience
the primary bottleneck that you incur
when you run nodes is around IO so these
are storage optimized I three instances
they're optimized for very low latency
high random i/o performance and they
each have like an SSD based storage that
you have access to on the on the
container so to optimize the resources
we run several processes or tasks per
container depending on the container
size and that technique is sometimes
called bin packing but it's the idea
that you know the container size you
have available to you may not be the
perfect use of the resources that each
node requires so we add we choose how
many nodes we want to run per container
and these ratios that I've listed up
here are you know in our latest testing
these are the ones we recommend at the
moment but they could change as block
chain sizes increase or change so or you
know if the efficiency of blockchain
storage changes so one particular
challenge that we ran into is how to get
around the incredibly long sink times
for new nodes so if one of our nodes
becomes unhealthy we want to be able to
start up another node quickly and with
death taking several hours like almost a
day to sink a new blockchain and parity
taking several days we needed a better
strategy than just starting our nodes
from scratch and having them sync with
the blockchain
so we created a special task
on one container that we call the
updater task it's down in the corner
there and this task runs a node that's
periodically taken offline every four
hours or so and copied the s3 and when
you copy within the same region to s3 it
doesn't cost any you don't incur any
bandwidth charges so when new nodes
start up they copy the contents of the
blockchain state from that same s3
bucket to their local disk before they
begin and then this you know enables
them to start up within a few minutes if
we're lucky it'll be like actually
running by the end of my talk here so
let's take a look then at the code for
just a sec so I've got the repo right
here for those of you who I'll leave it
there for just a second longer for those
of you who want the link of course this
would be published later and in here you
know I'm a strong believer in a project
having all the information that it needs
and it's read me to be able to reproduce
it and run it and execute it sometime I
think Chris landstrass one of the github
founders called this readme driven
development but anyway I think this is
important so hopefully everything you
need to be able to run this is there and
I'm gonna just bounce over to my AWS
management console to just see what the
state of this thing is going so we
launched this additional stack and it
looks like the connection is so slow
that it hasn't even fully kicked off
here yet so it's probably not going to
be finished by the time that I'm done so
for that purpose I prepared a video just
in case this hat was going to happen
right I kind of had this feeling last
night that I might run into this
i tempted the demo gods right so I'm
going to bounce over to kind of what you
would see as this progresses a little
further along so that kicks off this
CloudFormation template and you get this
result saying okay we've just started
this and I've got it serviced that's
I've already pre deployed that we can
actually test against so we're seeing
some of the RPC calls now going out to
this node cluster and they're being
responded successfully and because I
have another node cluster running
already I can actually do that as well
so if I want to for example I first need
to go to my API gateway and find out
what the URL of my service is so we'll
do that
so while this is loading very slowly I
can take you through some of the things
that we're still working on this so this
right now supports the parody client
that was one of the you know at the time
we first developed this parody was a
little more stable so that was what we
were focusing on and I am very close to
releasing to merging in a PR that will
support death as well but it's actually
pretty easy because of our approach of
just kind of copying the contents of the
blockchain state off disk and syncing it
with s3 it's very easy for us to kind of
support whatever node service we want to
so as you know eath 2.0 comes online and
a lot of new EVM implementations are
coming out this shouldn't be too hard so
let's see
alright so got that URL
and we get a successful result very
slowly because of the network it's it's
not the service I promise anyway so this
is this has been a very stable
successful deployment that's been
running for some of those companies that
I mentioned earlier for quite some time
we continue to maintain the tezo's
foundations note infrastructure in the
similar way and happy to answer
questions about how we did anything and
talk with you all offline after this but
thanks a lot for being with me today
you