[Applause]
all right we're gonna hi everybody
eg and this is Jeep and we are members I
think since we're not here here at
project today we're here talking to talk
to you about our open architecture
initiative it's really great to be back
here at yeah all right this place is
always in all this special place in our
hearts because we watched it Def Con in
Shanghai just over three years ago and
obviously the agreement good system has
seen a huge amount of growth in that
amount of time our team has grown our
architecture is pretty much unrecognized
analysis but we have created ship and
that's really what this talk is about me
we entered into this space because we
saw that people were having with trying
to enter let three explore with what
could be deadly
notice my contracts and building things
on top of it their name and we thought
needed up with that and one of the
things that we want to be now is share
our lessons learned over the last three
years talk about what work for us within
and just continue the sugar experiences
and I started going forward and that's
what this exact texture Michigan is
about so talking about see them in
architecture it's my whatever we all got
watching we all serve that same place to
start rendering or claim if next to the
network and you start throwing some
requests for data whether that's give me
this list of transactions of the back
you're interacting with people dead
that's terribly installed on your laptop
maybe start breaking a co-working space
with some people and they started
sharing this then because nobody wants
to have all of that space taken up on
their back correctly like upload or
update your head drive so what you end
up having is the single machine and
maybe the amount of requests they were
like they are starting to be slower than
expected and latency starts to increase
and you end up trying to do traditional
wet scaling tactics which starts with
realities that just means throwing more
simple resources attic but that it takes
you so far at some point these service
versus skill beyond what might have
access to a solution and what we've seen
is that developers end up with custom
solutions around this whether that's
custom captioning custom API is about
you know sort of like scaffolding
architecture around us and what we've
seen but there is a smell like this
inequality of data between the relevant
emails depending on whether or not
the team has hard expertise like wet
skill or they have the amount of
resources and science dive into this
kind of culture and so that's what we're
trying to be able is its share my
information about what's worked for us
they're different types of user profiles
that use the Dearing knows or the
majority of these days all you're doing
is interacting with this might contract
checking myself in balance in summation
about you these are all these in the
late night or even communities and
that's pretty sufficient if you're a
developer the hacker on and we tried to
do something that it's it's really
simply these days we probably saw
anything away it's only when you started
trying to be slightly more complex
things with that data like looking at
historical information which mean that
you start needing to run like
or a whole heart kinda and we get
somewhere we turn that into business
I'm watching analytics and at that point
meatless lives always online because
maybe you built an API on top of it that
your users are consuming and all three
of these types of things our parents are
using the same underlying that variable
and software the darién clients were
meant to be vigorous everybody this was
have access to same software and we get
the same opportunity to access this data
so whether we were in Gaza parity or
apparently on now
all three of these user profiles here
are lending the same software and it's
not as fairly tailored to a stereo these
days so when you think about it during a
client as a model like a piece of
software making up this as like a web
team application like maybe this is like
a Jenga
the way to particularly soft wet scale
was by looking at this thing it has
subsistence it sort of a subsistence
under any other planet there is a period
of days which is for Jason I can see
website there's the pubic area
connection to the network
there's the EDM it's actually able to
process new blog from transactions and
then chain data storage that actually
stores that information readily and then
the transaction system that actually
received these transactions and
accommodates the net across the network
talking about something like the chain
data so if you're running this on a
single piece part of it how to scale
access to Gmail past with specific many
paths forget they're really easy and
luckily for period and easily by taking
this and thinking about that not as this
model that increases productivity as
subsystems we started to apply a
dispositional like West into this
because we're not running this and I'm
just a single backup do any miss McCloud
and so you start with right at least in
and put this on a survey it's a
in court as 60 gigs of memory and these
are usually pretty proficient minutes
you can get to the over 8,000 requests
per second with less red light one
server if you said 11 computer
incorrectly and you can get to all when
they struggled for certain things they
can just send us here at the own memory
but then beyond that that's where people
start running into issues and so we have
what we call a cloud architecture so we
conceptualize this as taking those
subsystems and being able to optimize
and independent so you never just say
this would normally be on a single
instance you say this is the chain data
and if I want to be able to query this
data more efficiently invert ruthlessly
in the morning I can build additional
indexes there may be activism self and
all of the vehicles Postgres migrated
into died when TB or something like that
that has like well scale
the table is unique and inventive as
with any other set system turn it into a
specialized nitrous service or
application all these things start with
having that reliable note connection and
that's how you can quickly provision in
it trust me
is much slower than Eastley our 2016 you
that need extra oil we didn't work sync
apparently no minutes and yeah
but over time that's been at in terms of
donation as Kingdom change the others
grow and so the first thing that we
tried was doing simple you just a
familiar with Amazon Web Services
there's this concept of elastic block
search which is the ability to snapshot
periodic backups and quickly spin up the
disk types that can be associated with
any service
so it said take and say well that was to
say yeah this is going to be an online
and like and so additionally these three
got snapshots idea Westphalia repairing
there were huge certain upgrades that
they actually did and we had to find a
way to quickly recover otherwise our
service were there yeah so that was one
of the things we started with lady
there's the logic depends to some times
where it's gone from hours with left one
night to four hours
this is test data from the gap to their
great work
and I'm were able to get these kinds of
improvements at me by optimizing our
utilization idealization is really
important because when we read these
things there is a huge amount of IELTS
because watching data in terms of being
stored in too many stores in the way
that data is verified is very even right
so we are to use this type like your
life through Forex version which is it
just another 25,000 random read lots
available if we weren't isn't as we
reclaim oh yes we're going to be limited
to a fraction about that one time that
is really that this only gets you to
that for our part and that's not fast
enough
and so we ended up combining this
approach combined with periodic backups
being able to sync that directly to you
I mean that when we spin it up thanks to
my cryptid team appreciate their
implementation of this they were the
first ones really try to self
introduction and they share their
activation carefully and once I shove it
out but this know where this source of
truth live is what you call it stinks
from scratch I think it's a pretty
ratchet power all it takes is very fast
in the archive medicine then we have
away the function the fires off every
hour to back up that data to an s3
bucket we can generalize this to any
cloud that office is somewhere to go and
then lastly they just pass it up written
it we have a high throughput pipeline
between s3 ready CT and it's bullying
with the saturate about my office
that's a
and yet this day about 50 minutes from
the time that he started to return
that's ready
so once you have this reliable then IV
start building additional services on
top of its optimizer traffic and that
really needs to traffic parents like you
and their application in your business
said she's gonna memorize our traffic
into three different types near head
archiving pull your head is any request
that's within a couple hundred blocks
behind pet the archive is requests that
require dia and then full is requested
do not require proof data as you can see
the majority of traffic that we get is
your head it's fairly consistent
Verna's archive data could fluctuate
anywhere between six to even less than
one percent depending on what our users
were doing at that given point on the
time so the first thing we did was we
asked our near that DHT and it worked
really well
it just came with certain limitations
time for exactly we could only get
blocked my number
yeah block number it also was a good
scalable so we made a decision to store
or in your head read a few e and it was
are scalable and also allowed us to
store all of our new pet data in
addition to our narrative hatch we have
a set and a special library services
that I heard index or cash like specific
types of data but what I'm going to talk
about today is our log indexer which
service is our get box request the way
that if your your clients rich we want
data from will get very time-consuming
it retrieves every blog compares it with
the bloom filter that it retrieves the
transaction or the match logs and that
again compares the generating logs with
the filter it works really great for a
person that's just create their own
thoughts using their own note but we get
up to the house like for millions get
lock request today so what that happens
in it over Gardens are any of you nodes
and also increases a response times for
our users so we actually affects all our
logs live incoming new blocks what are
they choosing and run into when you do
this is to correctly attack for rewards
or you might be servicing incorrect data
so the log index are actually uses
another
service that we have sorry called
New York driver and rear tracker gets
dead in blocks if we have a block five
million it outputs that data to a
consumer in which case is the logging
mixer the log of extra time can push FX
to its EE a we use MongoDB continues on
with block per million one log effects
are updates that information and then
when we get lot five million to rework
tracker that realizes that the parent
Hashim current block does not match the
block cache of the previous clock that
it received so it actually emits a
rollback event which the law context or
date consumes to delete that information
and then when rework tracker retrieves
the correct block information it can
then make sure that it has the correct
lobster-claw fight in Amman and then it
just continues on with the incoming
locks it's a very simple but and we use
it right now to index logs but it can
actually be used to index any kind of
data that's associated with locking
partition so this is a very simple view
of how we handle our requests the
majority of the traffic that we get gets
handled fire near near head cache there
and then we have our set and
cassius that handle the more frequently
called requests and then whatever is
left is then service fire theorem yes
so those are some of the market services
that we use to help handle the large
amount of traffic that we get but we
also have a set of internal tools that
we use just to help maintain our the
structure one of the recent market
services that we open sourced is key
router it Maps like keys to a consistent
set of notes
it helps route specific crisis traffic
to a consistent data source a member of
our team actually Petroff created the
Phe node which is like a way to
incentivize the use of like clients but
we actually use it to manage carry our
notes if anybody's interested or check
it out it's pretty cool and also we have
our new monitor that monitors the health
of our notes it tracks like latest blog
number and then compares it with an
external source like either scan to make
sure that our notes are in your head
it also monitors like peer care out on
whether or not it's syncing and that all
helps to make sure that the that are if
your notes are healthy
and then the essence is really just the
start of our webinar catcher initiative
having this source of truth that the
structure having some like standard way
of indexing is really just the start
over the next few months really the
continuation and more information about
how we bought my specific recipe of your
implants assistance and that one of the
things that we want to do is continue
working partners in the ecosystem t32
and really share knowledge so that we
can achieve building another two into
that and we always want to do it in a
way that's over declarative and the last
view that we wanted to play for the new
yeah that was nice and every server box
provider in specific service and so the
next thing kind of for us retracing
today if you've already tried running
the trace an API bill
that's really going to be building on
time as well really building on top of
this and will be available in the next
outside of this this architecture is
generalized it's not specific to that
area it's definitely easy to support the
vmas treasonous we have the north which
means in the near future
so the you have any questions we're
going to be doing a Q&A session from 9
a.m. to noon tomorrow and at the
consensus view please stop by ask us
about because adherent or plus this
architecture initiative I've just driven
by and check out these great DEFCON 5
limited edition he's available if you
want to check that and kind of numbers
right here anybody in the back handouts
and swag after this
and Angie and that's it thank you
[Applause]