[Applause]
trying to show ops or what happens if
this runs successfully for you but if
you're able to connect to the internet
and interested in the workshop or style
component we recommend is because
pulling down vulcanize checking out the
branch you've got set up for this
process and so yeah you're talking about
multi TV
my name's Rob Mahone I'm Pete Nelson
would organize multiple software crafter
at a blend of Chicago and I'm a customer
here
I had three likes
bring my phone up cool so to get started
talking about sort of the motivation for
vulcanized edu I wanted to start talking
about sort of where are we now with
accessing data in the imperial system
and from my perspective there's a few
things to notice one is that can got
these decentralized applications but
largely in order for these decentralized
applications to work we're relying on
centralized data right so here I'm
visited Bulaq now calm and we've got a
bunch of network requests going to in
Fura do you access the latest state of
the bhulok contracts and that's pretty
cool like it works and it's awesome
doesent here provides the service but
ideally you know we might have solutions
that didn't depend on one third party to
service that data so in addition to sort
of centralization around access to data
another thing we've got is that we've
got a lot of duplicated effort right so
when I have spoken with people about
what can I teach in the hallway I think
one of the like most common comments I
get is like oh that sounds really great
like we had to build something like that
in house to support our own
infrastructure right so it's amazing to
me how many teams have sort of gone
through this work of figuring out how to
like extract data from an a3 of node
decoded
those rest database but the fact that so
many people are doing it sort of raises
the obvious question of like why don't
we have a shared tool set for that I did
want to mention that there's a lot of
teams doing great work to address these
two issues right so that the question of
centralization of duplicate endeavour we
see a ton of teams out there making
progress to try and improve the
situation and you know I think that's
super awesome I really have to be able
to you know here at DEFCON like see what
other people are feeling and from that
something you can all learn from each
other
but I think the fact that you've got
this many organizations all working on
like this shared problem sort of
indicates the the magnitude of the issue
and the rationale for having improved
toolset and so with that sort of
motivation in mind the question I want
to propose is so like where are we
headed and with vulcanize DB I would say
that our mission is to replace
decentralized and misspoke solutions
with shared tooling that anyone can run
and so if we look at the toolbox of sort
of what we started been developing to
make this happen I wanted to walk
through a few different things in our
tool chain that you can use like right
now to start spending on curtains it's a
vulcanized Evie and owning your own your
own data do you able to serve it for
yourself so I'll walk through each of
these in more detail so the foundation
of the process for setting up of all
kinds deviant DB instance we've got this
hetero snake process and what the
henderson process does we're basically
taking clock headers out of your node
Postgres you can configure a starting
block so if you only care about a
contract that was deployed a block you
know seven and a half million then you
can start syncing headers into your
instance from that individual starting
block and what this process will do is
it will able you to continually verify
those headers at a configurable depth so
depending on what your concern is about
reorg and so forth you can have this
foundation of data that's continually
being validated and where data that's no
longer on the chain if there was a reorg
or something like that is going to be
automatically pruned for you so you know
you have a consistent record of just
like what are the headers that were on
the chain and importantly for some of
that additional tools I'll discuss you
know we have form D relationship between
this block header and all the nested
data such that you can sort of cascade
remove any any game that you have in
your system there's a product of a
header that was removed so the thing I
wanted to talk about today and the thing
that the exercise will let you to run is
then contract watcher process and so we
think the contract Watchers pretty neat
what it does is basically you give us a
address for a contract and you tell us
the deployment block of that contract
and then we will automatically figure
out what are the events on this contract
start getting them off the chain and
decoding them for you you can run the
contract watcher with multiple contracts
role we'll create a schema for each one
and a table for each event those event
and end up in Postgres will be
with the headers such that that
interesting process is going to remove
so that's no longer valid and this works
pretty great for things that are like
events that are defined in a contracts
API and again so if if you're able
access the Internet want to set up here
on instance this this will do that for
you this will kick off the interesting
process contract watcher focusing on
three contracts from Moloch down and it
will spin up instance of PO scrap pile
which I'll get to that will enable you
to see that data on localhost 5,000 in
your graduate compose an excuse is a
command that I want to talk about so
this is still very much in the work for
us but what the contract watcher gives e
right is the ability to automatically
look at events that are defined in the
API but what we found is that in more
complex systems of smart contracts we
also got to worry about things like
anonymous events right or custom events
where the payload doesn't necessarily
like the types that you want to decode
the payload into and Postgres don't
necessarily match what might be defined
on the API or the topic 0 on a given log
event a little bit different than what
you might expect and so our answer for
dealing with that sort of thing have
basically been to say hey you can write
your own plugins and these plugins can
be used to you know take care of that
another thing that compose and execute
plug-ins enable you to do is to look
directly in storage tree notes so when
I'll be continuing to talk about this
during this talk but I think one thing
that is really interesting that the
community's sort of needs to like reckon
with is that
right in order to access a traditional
need to run like the archive note right
if you want to have historical state and
so my folks don't want to do that don't
be able to access their data running
this the full note and the way you can
do that is you can sort of stuff your
state into events which is cool except
it means that we've just sort of moved
the problem over where instead of like
state float or you have the state look
but you also can have like event load
right so there have been proposals
already floated in the ecosystem to
start pruning historical events out of
full notes as well and the state that
you get from an event isn't necessarily
the state that's actually you know
what's happening on a given contract and
so one thing that we've been working on
a lot with the plugins that we're
developing is to look directly at
storage on a contract now traditionally
you're gonna have to do this with an
archive node but we've got some ideas
there as well
anyway the plugins let you look at
storage tree nodes you can automatically
decode like one of the true that value
of a variable on a contract and plugins
also need to make more complex queries
right so in order to sort of crystallize
what those complex queries look like I'm
going to bump ahead to post grandpa so
post-grad file is not something we built
the post gram file is a super awesome
tool and we support that job patreon I
recommend everyone
as well but most gravel enables you to
do is you just literally like post
gravel from the command line and it will
inspect your Postgres database identify
the schema and like automatically make
that about available to you in the
browser which is just like super-sick
comments so thank you so much too many
he's also super responsive on discord we
found it to be a really easy solution to
sort of a proposed fixing with an API in
the browser but the data that we have
and both graphs of course if you're
interested you can always you know just
make queries directly against that
Postgres data but for browser access
close grab files
angry so some things to know about those
grab files that like it'll automatically
discover relations between your data
right so for example I mentioned that
events we decode or as soon as stated
with a block better than existing
database that you can just like
automatically get better by an e with
the event that you're looking up and see
you know metadata about that block
header if you want to see you know what
block number was it or whatever post
grant file also exposes built-in filters
and conditions and so forth which means
that you can Mike is gonna find this
exact tool to your Postgres database
feed here
Camela query support subscriptions and
notifications and our view data hitting
the database there are computed columns
which enable you to like amend data and
custom queries which is what I was
mentioning with the more complex queries
on the last slide right so you can in a
migration like create a query that
aggregates either from say multiple
events or tables and those will show up
automatically in post growl help cool so
that's the toolbox sort of like as it
stands right now again you can try it
out repos took the open source and the
thing I wanted to talk about next is
sort of like what are we driving toward
with our
coming work and so the first thing is
simplifying our interfaces for plugins
right so I mentioned you can write
plugins and they do super cool stuff but
our mission there is to make sure that
in order to write a plug-in that people
need to get things like storage tree
nodes or anonymous events that you have
to write like the minimal amount of this
book could possible for given smart
contracts another thing we're working on
is client patches to omit these storage
Tiff's during a sync so I mentioned that
like traditionally archived new to
access historical state and that's kind
of a bummer but like one option that's
definitely on the table would be been
able to subscription over the JSON RPC
interface that would just like spit out
those bits that they're happening and
you can even say like specifically and
we want you to publish just me if they
come from a given smart contract right
so part of our sort of open source work
is to figure out a way to get this done
and get that upstream into gap and
parity and so forth so that you can sort
of plug in that subscription directly to
vault as DP have a plug-in that's
parsing that end up on the fly and you
can be accessing state data with a
plugin that took ideally and minimal
amount of code another thing that we've
got coming down the pipeline is what we
call the super node so the idea for the
super node is that we will be
automatically digesting all those states
and storage tips but also blocks and
also proofs for the deaths that are
coming out of a node and our goal is to
publish all of that data to ipfs and
have vulcanize DB serve as sort of a
filtering layer where you can say I want
to get all of the dips from X contract
and we will give you a list of contact
addresses back on content of justice C
IDs and you can then query ipfs for that
data and you can get
groups with it right so you don't have
to trust that we're giving you like the
correct data then that data is valid
because you can submit that proof to
your new one in order to verify that
that ADA isn't back to what we say it is
and so again we think this is like super
cool work hopefully you can like share
dr. Campos style demo the Colette's you
do this next year maybe your even sooner
but that's that's some huge primary
stress and you know we always welcome
issues pull requests that people think
that other priorities are worth tracking
down as well so this is the part of the
workshop where I would be staying like
let's go ahead and try this out everyone
run your own instance of vulcanize DB on
their machine if you have internet we
even like performance tested a node back
home that you can connect to but I don't
think performance is going to be a
problem with an internet issue here so
to walk into our stack you know both men
TVs written and go it was pretty nice
because it enabled us to like really
easily integrate with going theory like
unpacking logs and so forth Postgres
obviously talked about after graph QL
this set up on this slide is exactly
what it says on the morning but you know
try it out when you're at the hotel or
home or whatever like definitely did get
feedback on this so what I wanted to
point out about the setup that we're
showing is our config file this config
file is 20 lines this config file also
yields parsed events from all three
Moammar contracts the Bhullar contract
you'll make contract in the black pool
contract and to be honest some of that
data at the top I'm gonna put in
variables to slim this down even more
but I just wanted to show you like
everything you need to do to do this
work so what's happening under the hood
you've got this configuration file and
then you're running three of the
processes that I mentioned right so
you're saying I had an interesting
process pointed that config file
starting at the deployment block or
bowler I want to run a contract watcher
process with that config file and the
most cumbersome command at this post
route but that's our fault because the
way that we set up these schemas for the
contracts is Petter they said this is
the header sync process and the contract
watcher the training this data and then
into the contract address right so
fairly straightforward they've populated
this data with any arbitrary adjust you
wanted you know you can point this email
it'll be there there's a little W flag
there which is important right like you
can kick off this post graph out process
as soon as you kick off the contract
watcher and with the W flag it'll be
watching the schema so you'll see a
warning perhaps it'll say like hey we
don't have anything in the schema right
now and then that the contract water
starts to populate that data ok the
schemas here you can load it in when you
refresh your browser you're going to
have that data expose in what's going
well so in a workshop well ability to
run this stuff we've got I've got sort
of the demo here so this is this is a
set up instance so I added this whole
thing these steps that I'm asking
everyone
to do and this is the this is the
photograph file interface that we have
so enhanced graph iql which was a
command we passed to who post a file
that means the begin a list of like all
the available queries on the left-hand
side and then what we've got in the
center is you know graphical interface
with these queries so I've selected
these two ownership transfers events and
withdrawal events because you can see
that if I ask for that again you know 23
events and if we pop over to you know
the main source of truth either scan you
see the 23 events on the full contract
so that's cool that lines up checks out
when in these queries we can do some
cool stuff so we can say like I want to
see the nodes and I want to see the
parts like new owner and previous owner
data and then I fire that query and like
narratives ok ownership was transferred
from dress and I can do the same thing
with withdrawal events I want to see the
American policy receiver and cool you
know these are mounts are weird because
it's like the p.m. but there's a fixed
value and divide that by to get
something that's a little bit more easy
to parse an easier event to look at for
understanding this would be the ragequit
event because they're dealing with share
spreads opposed to a monetary value and
so
want to see shares to learn and the
member address for the rage quit events
on the block contract that at Lake
occasions are pretty nice numbers and we
can see here we've got like ninety nine
chairs break we pop over the Mohawk
contract on ether stand is he like not
interesting here's that same address in
topic 1 you know it's a habit hex value
and we do put this number it's like 99
this event was dead empty so okay you
know you're just getting like
automatically decoded events I had you
know I can walk through all the queries
but I think folks get an idea of what
we're doing right here are automatically
getting all these events parts into
Postgres with recant so you know the
workshop was like first you can run this
doctor compose that will enable you to
like reproduce exactly what I just demo
I did want to mention that that they
would I demoed that database ends up
being about three gigs that also took me
about 30 hours to sync on my own Wi-Fi
so continue to work on performance that
cost is amortized over the life of your
system right so you might have like a
high upfront costs to get totally sync
with all the events that have happened
throughout history but then you're going
to stay in sync it's running the process
continually not going to fall behind and
and yet that costed amortize which is
built but what that means is that you
know you start seeing events immediately
for things that happened early on and
the history
contracts but you wouldn't a latest one
can sell your system and finish thinking
but if you wanted to check out some
other stuff you could write your own
config file right so that was 20 lines
to get three contracts you could look at
17 contracts for lines that's up to you
you could you could run it with an
address where the API is not published
on ether scan right - there's an option
to supply the API and that config file
for the contract if you're not dealing
with you know verified source code so
it's not a constraint for the system
that the API have to exist on understand
but that minimizes the amount you have
to including to think and if you wanted
to run it locally like not in docker
then you can you know do a few things
that we have hidden for you in our
compose script right so you have to like
create a database of all guys can
connect to we use go modules so you have
to turn that on to enable it to bill
your bill to end up with your vulcanize
DB binary and you should be able to run
these things locally and then if folks
were able to you know be like okay I did
this award and I like made my own config
look with contracts board again I'm
running it locally on my machine you
know I'm still bored then like option 3
that I thought would really take up
everyone's time would be okay let's
start building around plugins right so
all the plug-in architecture is in the
libraries shared folder baulkham as d-v
and specifically in the factories
directory you can see we've abstracted
code that'll handle like the overarching
process of syncing anonymous aventure
sickening storage tips that you have to
write like a few small dependencies
basically tell us like okay you've given
me an ominous event like how does it
become why like something that you put
the database but yeah and you know how
I'll stick around after this time I
would love for people to to give this go
we can but there we are
so I want to say thank you I definitely
have not done this by myself thank you
know how to uh privilege to speak up
here but Rick Dudley Elizabeth Andy gave
Edwardian Connor Gustav have all been
super instrumental to getting this
project to where it is today
we support Pina Pina foundation for kids
were tremendously grateful and also go
for maker down so thank you so much to
those folks we're supporting our effort
we're really excited about hopefully a
lot of value can be delivered to the
community with shared tools that for
Casa of historical data and that is
pretty much
definitely have the answer any questions
yep there's one subscriptions on the
Rettig oh yeah yeah it's post grab out
is again just like the most awesome the
Bossip open-source projects it's of
course subscriptions out of barks oh
yeah oh it just curious
intercom file they showed us it's quite
another dress right so how do you deal
with all the contracts for each
individual user with the same bytecodes
and some you know the induction
hypothesis my choices
yeah so you definitely want to like
write a plug-in to deal with that right
because um what the contract watcher is
giving you is it's giving you uh upfront
facility to say like give it a contract
address I want to see all of its events
like decoded which you know I think has
some value on its own but we're not
trying to be like super aggressive on
that with the contract watcher itself
but with plugins what you can do is you
can say
okay like given that I see this event
that I want to spit up Big Canoe and so
the contract watch her appointment to
have a field on that event that is an
address that I care about or you know
any number of ways you can implement
that to to run the process and we have
code that does that some ian's code yes
there's a couple different teams that
working both nice TV so there's another
code base that actually does that it
takes it will read a contract to get a
contract address and then write the code
without a dress like anyone just to
simple API to keep track into Academy
yes that's actually what I was just
talking about it was an ER c20 watcher
and we just didn't year c20 watcher but
obviously you could change it to make it
in ear gear see seven to one washer as
well yeah I mean one thing just to
expand on that point too
you know we sort of tinkered with a
variety of different amounts of like
face data that you can scan for when
you're running vulcanize TV and so we
started off in like listed just like
digest everything and anyone can run
this and digest everything but like I
think reality is that a lot of people
who might get value from vocalize DB are
not necessarily super stoked to like pay
the performance cost that it takes to
digest all of that extra information
if they don't eat it right so a lot of
what I've been demoing here is like a
pretty lightweight process of sinking
headers and syncing events from specific
targeted contracts there are over tools
in the box that enable you to do more
heavyweight stuff if you've got the
infrared the motivation to do that and
we're hoping the supernova can help out
with that lot too well another question
okay is there a reason for not sinking
backwards because to maintenance that
I've seen from the starting book which
means it takes you 30 hours to get to
defense you probably care of us
yeah so I mean that the main thing is
just like I guess that the latest blocks
are more likely to be removed by the
henderson process right so you'd be
potentially doing a lot more redundant
work you can totally get around that if
you're interested by starting the header
sync process with a more recent starting
block number right so I was demoing with
like the deployment block as a starting
block number so that you know you're
going to get like all of the events but
if you really wanted to say like I've
always been up an instance to give me
like what happened the last five days
then you can pop that in at the
Perimeter to the command a block number
where you start caring about stuff and
then that's where this thing will happen
from no but I mean you would you care
about everything
it's just that's I in that 30 30 hours
you want to do something as well yeah so
and you have a book watch you're
watching for new blows so your real
rooster just managed to just the same
way as they are right now yep it's just
it's not starting at 7:00 visit whatever
you walk backwards from the moment you
start at the book watch yeah but before
but before what before yeah so the short
answer is it is technically harder to do
that just generally gap doesn't like
it's a goal in Reverse so I mean that's
kind of weak and if you have really
curious about it we can really talk
about it but the short answer is got
doesn't want to go in reverse well I
mean that's specifically true also with
like subscription that's right so part
of the idea here is that you can easily
have like a subscription to events but
then those are going to be fired as your
nose like processing the blocks that
it's going over so part of the ratchet
count for the set up is like you could
use either subscriptions or get blogs
furies to progress forward but I didn't
think that's a really cool idea and
something we could totally dig into so
appreciate it yeah last time I checked
you have no upper body
they're reasonable it's kind of a work
in progress I mean we have I think we
had like a doctor file a long time ago
and it was like wanted to bring TP and
we found it to be like not super useful
over our day to day development so
lately there's been a lot more work on
that and we have a lot of progress on an
open PRS I think it's called like dr.
updates but yes it's not on master yet
and the reason but the reason that this
one is on its own branch is because like
this is pointed at no luck but maybe it
makes sense at some point to have like
an animal doctor files but we didn't
want to be like the purpose of
vulcanized EP is watching the Moloch
contracts you know yep so I'm curious
about the limitations of the packing API
so currently from what I understood
begins as a mouth that's a so big mouth
event or whatever to another events or
whatever can they act as they reduce so
for example red juice I know the thing
that makes a level I can sum up transfer
balance whatever and how does it work
with three arcs yeah I think I'm gonna
go straight to rec on this one oh okay I
mean yeah we could be running reduce but
then as you pointed out at the area
there we are so what you would be doing
is you we would have a table like we
have now and then we would just again
how in that bed from that table and
wrote a new table and then it would know
to compute the new table so the the
first table that I described would have
all of the things back to get viewer and
then the secondary table get
communicated so you can write off your
thing that I just described and login
do all these things in plugins and
that's sort of I mean that's how we
would have to trust that but yeah I mean
when I designed the system originally I
thought most of the utility would be in
reduce that's just not the way that
development happened to go yeah that I
knew was the nation of if you go to
coastal state changes yes so I mean that
that's work that happens in plugins
because we're actively trying to sort of
figure out like how can either leave
this automatically or ask you to write
like the fair minimum of code do you
need to to make that happen
but in a nutshell the way it works is
that you have a mapping of given storage
keys to given contract values Frank so
that's pretty straightforward for static
values on contracts like it's literally
just like the index the value on the
contracts you can have a mapping looks
like index one is you know the total
supply or whatever but it gets a little
bit more interesting when you start
dealing with like mappings and dynamic
arrays because generating storage keys
for that generally dependence on
secondary data as to like an address to
map to balances or whatever and so the
way that we're looking at doing that is
basically you flag some events as like
these have addresses that I know for
sure are going to be in this mapping and
then will like automatically based on
the index of the mapping generate the
keys
based on hashing that address with the
index such that you can recognize like
all of the storage dips that are coming
off of a contract after a given event
has been sync and then you know after
you know like what it maps to it's a
pretty trivial
just like decoding though the value
associated with that storage key into
the perfect type cool alrighty well
thank you so much for coming out to
listen to me topic and we're really
excited about this work and hope that I
can provide a lot of value for the
community and totally open source
[Applause]