foreign
[Music]
the workshop that we're doing is using
the graph in Dune to power on chain
actions my name is Christian this is
Matthias and we're from component you
can find us at component.fi and we'll
quickly go through the agenda so
you can get an idea of what we're doing
here
um so first things first I just wanted
to mention that this will not be a code
along we wanted it to be but some of the
dependencies required uh you need pretty
good Wi-Fi to download them so you'd
have to run to the basement come back up
uh we don't want that so instead we'll
just discuss uh some of the things that
you would have uh that you would have
coded
so one we want to get to know each other
I think that's uh you know why a lot of
us come here is to find like like-minded
individuals uh two we're gonna do an
overview of the web3 data stack
um so give you an idea of like the
different layers of the web3 data stack
and how you can use them
three um we're gonna do essentially an
example that goes through uh that you
would have coded along but instead we'll
just kind of show you how we did it uh
and can give you the information so you
can do it yourself later
um and that'll be data fetching so the
data fetching process data processing
and modeling so what you do with that
data after you fetch it and then
transaction execution
so essentially a way to grab data
process it model it and then have that
do an on-chain action a transaction for
you
and then at the end we'll do breakout
sessions just at your tables or whoever
you're standing next to
um instead of the instead of the coding
element we'll just essentially design
your own automation speaking with your
table but we'll walk you through it
we'll do some engineering instead of
coding
cool so uh this is actually uh my email
um I don't know if you're familiar with
Quest trade that's a Canadian uh company
for uh for trading stocks and using
derivatives I got margin called a lot uh
as you can see
um it wasn't that much money it's okay
but um I'm not a good Trader basically
um and that's kind of my motivation for
for what we're doing
um because I got margin called so much I
know I'm a bad Trader you know maybe
automating things and having uh you know
logic do it for me is probably a better
idea
um so that's specific to D5 but of
course there's many other ways that you
can uh take what we're going to show you
and use it you know across anything
that's built on the blockchain
okay so we want to start with everyone
just kind of talking to each other at
their tables
um you know you could get to know each
other maybe introduce each other but
here's a prompt based on what I just
showed you uh you know from myself uh
and that's a time when things didn't
really go your way in web 3 so that
could be you know you lost money trading
derivatives uh you know not in web 3 you
can talk about that too a time when
maybe you suffered a lot of impermanent
loss or you can you can say good things
too I mean um or a time when you should
have sold the nft at the top you know
that kind of thing
um so yeah take five minutes and chat
with uh the folks around you
so we're gonna give a little overview of
the web 3 data stack uh there's a lot
here and uh we're not going to have
everything
um so absolutely if you're a you know a
data protocol please reach out maybe we
can add it to our slides
um so we'll start with this nice little
visual from Andrew Hong on Twitter
um this was made about eight months ago
uh and so here's a breakdown of a lot of
the different sort of layers of of the
web3 data stack you know mostly mostly
relating to to ethereum and evm chains
so obviously you have your your sort of
base layer uh where you're actually
running clients uh interacting with uh
you know the ethereum blockchain state
directly reading logs uh call data Etc
uh you have nodes as a service you don't
do that yourself but you're still
operating on those sort of raw data
streams
and you have a layer on top of that you
can think of that as you know the query
and data mapping layer there's a little
more structure there often these things
are indexed
um you know they're they're um
potentially you know doing analytics
takes uh it takes your abis and does
some contract decoding so you can
actually read the the parameters and the
function names instead of looking at a
bunch of hex codes
um there's flip side and the graph and
everybody knows those
um there's uh then there's sort of that
layer on top and this obviously
encompasses a lot of things but they're
higher touch more much more customized
apis
um the the data might be a lot more
contextualized and a lot more useful for
specific applications but potentially a
little more obtuse
um some apis you know are that data
processing is happening in the
background and you don't necessarily
know
um so things those are things like you
know covalent etherscan some you know
manual tagging systems like through
nanson
uh and then we have a few that we use
pretty regularly so we wanted to make
sure they made it on there
um so and a lot has changed in the last
eight months eight months so people know
crypto Stacks uh David mihai
um they run like crypto fees and L2 fees
and all of those different front ends
they built a little you know data
processing
um stack to make it easier to build
front ends with uh with subgraph data uh
the Dune is releasing their API we
mentioned the Dune but API changes
things pretty significantly uh we use
the massari standardized D5 subgraphs uh
pretty much on a daily basis
um they've created you know an open
standard for different kinds of uh of uh
D5 Primitives
across chains uh if you want a little
more control ethl and true blocks are
are pretty great they're both you know
open tools allowing you to index your
own index from your own node and even
the ectl data is available in
uh in a bigquery database you don't
actually have to do your own uh your own
indexing if you don't want to uh D5
llama uh you know they pump out a ton of
new apis and new protocols uh their
adapters are a pretty cool system and
then we have another one spec
um they're uh they do allow you to sync
your own postgres database uh with uh
you know typescript Transformers uh
bringing in call data or sorry bringing
in um
uh you know transaction data uh indexing
it and then bringing it into your own
postgres database to live beside your
own data
cool so this is the process as I
mentioned that we're going to be going
through
um so first thing we're going to do is
we're going to do data fetching and for
each of these sections we're going to
talk about
um the alternatives to what we're doing
so to give you like a little bit of
flavor um so you're not stuck with with
with uh one thing uh and uh then we'll
give we'll run through the example that
we actually did
um but yeah just an overview of data
fetching processing modeling transaction
execution
so our example is using python to
collect data and write a simple
prediction model
um so yeah the three steps are pulling
each price data from the graph in Dune
uh just to have kind of two data sets to
to compare and to show you you know two
different ways that you can do it and
then using a auto regressive moving
average uh model to predict price
movement
um
I wouldn't use that uh Approach at home
uh but you know for the for this
presentation it works
um basically if price go up uh swap usdc
to eth on unit swap that's the
transaction that will automatically
happen obviously this was built
originally as a code along so we tried
to keep it as simple as possible uh a
lot of this can be expanded on pretty
pretty thoroughly
definition
so obviously we went over that data
stack and there's there's so much there
um so many different choices to be made
and like any other choice you're making
when building a product it's mostly
about trade-offs uh engineering choices
so we we wanted to talk about some
considerations to keep in mind uh there
are definitely others but some big ones
are available Fields so they actually
have the information that you need to
use uh the data format are going to have
to Wrangle all this data to to get it
have it usable in your automations
um scope does it cover the chains and
protocols that you want to use are you
going to have to you know merge
different data streams in order to get
information from Cross chain cross cross
protocol
um is it event driven or is it pulled uh
you know flexibility can you extend the
model uh stability are they going to
change their interface on you while
you're running your automations and
break everything because for some reason
we still don't Version Control apis uh
everywhere
latency if your your application
requires is real time you can't be
operating an hour behind blocks uh trust
reliability and redundancy open openness
of the actual transformation functions
so you know understanding how that data
was uh was transformed from you know raw
inputs to the data you're looking at now
and then openness of the data itself uh
you know if you want to be able to
inspect the internal State can you just
get a dump of the database and check it
out
um those are all some pretty important
things
so we are in our example we are just
fetching each USD price data from both
graph and the and dune
um of course this could be done through
most of the apis that we mentioned
um what we like working with both tools
and so this was just a nice little
example of using those
it'll Doon first yeah we'll start with
Dune so very simple query here
um we're actually just using a dune
created or user created table which
gives you prices of different uh erc20s
on layer one ethereum
and here you can see we're basically
just pulling in
um the the time
um which we're changing to hour because
we want hourly hourly price and then
we're doing the average of the price of
eth um over that hour and then we're
doing the max price and the Min price
during that hour
uh for the last 30 days
and you'll see this query as a query ID
here that's going to be relevant in a
second
cool so yeah as we said we don't have
access to the uh Dune API right now
um if someone from Dune is here please
uh let us know we'd love to use it
um but uh so instead we're using this
this uh open source SDK that grabs
essentially the the data from a query uh
it's limiting in the sense that you
can't actually run the query using this
so you basically you know every time you
want to do this you'd have to go back to
do and run the query and then come back
here and it'll update after you run this
function
um but yeah basically it's just grabbing
kind of that that static data from the
query the last time it was run
then uh yeah if you just scroll up a
little bit to data data one two three
yeah uh just quickly mention this it's
not that significant but we're basically
just changing the uh Json format that we
get in the response and we're turning it
into a panda's data frame uh are people
here familiar with pandas data frame
a lot of nods okay cool
rows and columns
cool
um and then
yeah yeah there's a little bit of
processing here to get you know the data
wrangled so it's consistent across both
Dune and the graph uh this one is you
know the the graph data pull
um so we're specifying the endpoint uh
we're just using the uh chain link
prices a sub graph here
um in fact this data isn't actually
hourly data
um so we first you know we were using
graphql to query the graph uh we're just
looking for the eth USD asset pair we're
ordering by Tam timestamp so we get the
newest data uh we're ordering descending
and we're grabbing the first thousand
prices as well as the timestamps
associated with that uh then we do the
exact same thing here except we do one
thing which is resampling so that we
only end up with you know one timestamp
every hour or one price every hour
instead of instead of multiple
okay
and uh here's just an example of the
outputs you'll notice that there's
actually inconsistency between price and
price here
um that's because uh the Dune table
actually does this in decimal format
while you know the graph uh here this
subgraph specifically uh looks like
uh yeah eight decimal points
um so uh
the the fact is we're running this model
separately so it doesn't really matter
for this example but it's just one of
those things to pay attention to
obviously your data needs to be
consistent
yeah and one thing I'll I'll mention
about that as well is
um it's actually great to use more than
one data source
um just so you can actually check these
things uh you know it's hard to know
what the uh the source of Truth is so
you know using different data sources
check them against each other you know
maybe before you run something is always
a good idea and another thing is uh we
didn't really mention this but
um with the data we're pulling you know
you know this is a simple example but
the really cool thing is that you know
anything on chain is fair game in terms
of data that you can use so think you
know lping lending protocols uh you know
on-chain derivatives protocols nfts you
know doesn't have to be defy related
um you know you can do some pretty
sophisticated stuff uh with different
types of data
so yeah moving into this section now so
data processing and modeling
um you know now we're at the point where
we have uh the the Dune query and the
graph query uh pulled in uh using Python
and now we want to uh model it
essentially
so before we jump uh in back into the
code
um there's just a few considerations uh
kind of like just what I said with like
there's so much composability when it
comes to data there's also infinite
composability when it comes to modeling
um you know you could use instead of
using a Time series prediction you could
do conditional logic like
um you know maybe there's someone you
follow that trades really well and
anytime that they make a trade you want
to mimic it uh right away you know stuff
like that or if you know someone like me
is not a good Trader you can take the
opposite position
um
then of course there's there's so many
types of machine learning algorithms
time series predictions uh the list goes
on and on for the types you can do and
again I know we're focused on on D5 for
that presentation this presentation
because that's mostly what we interact
with but you know if you're really into
stable diffusion and you want to see
text generated images uh you could
always grab you know uh text from your
lens protocol interactions and generate
an image based on it and mint and nft on
chain if you wanted to something like
that you know that's really up to you
because all this computation is done off
chain the uh the sky is the limit
foreign
cool so uh quickly just on the uh I
guess the modeling uh we didn't we
already did the processing I guess
earlier when we uh just reformatted
stuff into into a panda's data frame
um so here's it's it's quite simple
um if you're familiar with like a little
bit of data science I'm not a data
scientist myself
um but what we do is we basically split
uh the data that we pulled in so the
Dune data and the graph data and we
split it into a train and test set
um just basically 80 trained 20 tests
not that sophisticated
um and then what we plot it and we'll
show um you know we'll show the results
after and then we run it through a Arma
model so uh Auto regressive moving
average model to essentially predict
um so we're doing this by hour and the
goal here is to predict
um whether the uh the price of eth in
USD is going to go up in the next hour
and so the logic here is basically just
uh if the next hour price is greater
than the last price then we want want to
buy you if it's less then we want to
sell or short or what have you it's that
easy folks
not Financial advice
cool yeah these are the results so as
you can see the so the red here is the
test set
um and we are basically you know the
prediction is basically saying it's
trading sideways uh maybe I think it
probably goes down about 10 uh ten
dollars roughly uh if anyone knows uh
the crab strategy on and squeeze this
would be a good time to use it I guess
um yeah these are just the results
yeah so the most web three relevant
probably the most complicated
um Step uh in automation here is
transaction execution
so there's a number of ways to do it so
bringing these insights back on chain
um so there's there's tons of
considerations uh to think about uh
which chains you're operating on is the
environment adversarial so are you you
know if you're working in D5 then you're
probably looking out for Mev and
sandwich attacks anybody trying to
exploit your model
um if you're you know minting nfts for
art purposes and not for financial
purposes it's probably less of a big
deal that someone front runs you uh what
kind of timeline are you operating on
and you know some things have much
larger delays than others a key
management do you want to manage your
own Keys what kind of permissions do
those keys have should your logic be
transparent so you know if you're
running a proprietary training algorithm
um you probably want that to be executed
off chain and just have the results
posted to the chain but if you're
running something that you want to be
transparent uh you you would likely want
the the you know computation to be
visible on chain uh you know what
happened of a transaction fails if it's
or it's congested these are things that
are annoying for manual transactions but
if you're not taking them in account
when you're automating transactions you
can get something stuck in the mempool
forever you can potentially end up not
being able to to transact if you're not
you know replacing that transaction and
replacing that nonce
um so there's there's a bunch of options
um there are almost certainly others as
well and these are not necessarily
mutually exclusive they're just kind of
ideas of part of it so you know
libraries that wrap on top of uh you
know the wallet apis
that translate Json RPC
ethers web3js web3pi which we're using
in this example uh if you're running
something like that it's probably in
some sort of function that you can
either be running you know locally or on
a cloud
um you can you know use traditional
cloud service providers or you know
tenderly has their new web three actions
which is effectively lambdas for
for for web 3 interactions to be
triggered by events
um keeper networks uh so you can some
keep your networks support like more
complex off-chain logic um some only
support uh triggering you know
transactions with a function signature
um
uh
there are transaction execution Services
um so something like uh open Zeppelin uh
Defender relay and uh inferior
transaction Services they handle a lot
of these questions that are kind of
annoying for for a small development
team to deal with so things like Key
Management they allow you to hook into a
KMS system
um they also handle you know gas uh
figuring out gas fees so if a
transaction fails and the first or a
transaction is is um waiting too long in
the mempool because it doesn't have a
high enough gas price assigned then uh
it will actually ramp those up
aggressively and it tries to figure out
you know uh how much gas do we need to
be able to execute this transaction in a
certain amount of time that being the
variable you set uh you can you need to
make the decision you know if you're not
actually doing uh execution and you're
just posting data to the chain then you
might want to actually more like an
oracle model so just posting data you're
not necessarily thinking about execution
then you're more thinking about
launching execution that is you're more
thinking about how do I get a reliable
uh like reliable posting of results to
the chain
um you can go through a multi-sig if you
don't trust your automation you could
have them as one of the signers of a you
know one of two multisig where they you
know basically are queuing up that
transaction send yourself a notification
and then accept or reject that might be
the most thing that most people are more
comfortable with you could potentially
run things run the actual uh run some of
this Logic on an L2
um so what's the name of the
uh yeah uh there's a company called uh
modulus Labs uh they're doing some
pretty cool stuff uh so they created
this thing called Rocky bot
um that basically does what Matthias was
saying uh where uh they're running uh
it's on it's on start net uh and they're
leveraging that and just basically uh
posting the result to the chain uh so
it's like a neural network on chain it's
it's it's it's very cool
um definitely check it out yeah and then
I'm not very familiar with Mev or you
know like uh being a Searcher so I'm
sure there's other people who can expand
on this but in that case you're probably
using something more dedicated to your
building bundles using the the Builder
API I believe
uh and for this example we're just using
uh you know web3 dot Pi
um this is probably the thing that
people are actually the most familiar
with uh were fully we're just you know
setting some
uh some variables we're selecting our
chain being girly we're grabbing an RPC
provider uh there's some middleware
that's required for for Gurley here
we're grabbing a private key from
environment variables which of course
means we have to manage that ourselves
and then we are
you know executing transaction in this
case we are swapping on universal v2's
router
um so we're
grabbing the ABI uh using that to to
encode our function signature and uh you
know setting some parameters and and
posting that to the chain
cool so before we go into this
um just wanted three caps so we used a
subgraph to pull
um typical ethusd price data we use Doom
to pull eth USD price data and we
processed it we using python pandas and
then we used web3 dot pi to actually
send the transaction
um it's actually what we were supposed
to show just put putting it all together
yeah it's again you know three steps
here really oh grab graph and dune um
you know formatted data uh check with
the model whether we should buy graph
buy based on the graph model whether we
should buy based on the Dune model and
you know if both are true uh we are
buying
yeah and one more thing to say here is
um
so this makes it look easy
um and that's actually what we're
working on at component so working on
something called blueprint that
abstracts away all the stuff we just had
to do
um although it's really fun to learn how
it's done and you can definitely do it
that way
um and you know to see how it's done
under the hood but what we do with that
product is essentially uh try and take
you to this step without having to do
you know all the other stuff
okay I guess it's me
so uh you guys saw what we came up with
uh now it's your turn
um this might be a little long with the
10 minutes but um I'm sure everyone who
came here has something in webview that
they're passionate about uh that thing
probably is generating quite a bit of
data
um and that's probably available through
one of the many services we we mentioned
um so we'd like to take a little bit of
time uh you know explore some data sets
um think about what kind of automation
you could build using those as you know
your input
you can work as individuals or as a
group to start uh after that we're going
to sort of have a little 30 second
pitches uh to everybody else in your on
your table and you'll decide on one to
sort of design as a group
yeah so just to you know give you an
idea of what you could do so anything in
defy um where you know you're triggering
you know based on a prediction model or
based on some conditional logic it could
be to sweep the floor of an nft
collection
um you know the really anything you want
and any data that comes to mind that's
particular to you and what you're doing
thinking about those three steps where
I'm getting the data from how I'm going
to process it and then how I'm going to
actually generate something actionable
from it
yeah go ahead
all right folks
uh
we're happy if a few people came up with
an idea and like thought about the
process and I think if you did anyone
come up with an idea
raise your hand if you did
you wanna you wanna talk about it
so um we think that kind of defy
investing has been quite risky
forever and so the idea is kind of a
generalized pressure release valve so if
a condition is met whatever investment
you have if it crosses over that
threshold and pull pull out you know
pull out to wallet essentially so you
can think of a liquidity pool you're
invested in there but if the two assets
start deviating Beyond a certain range
and you you exit that position
thank you Greg anyone else
you look like you have an idea
okay
Quick Pitch here let's hear it
um I'll try to make it very quick yeah
what I've learned throughout a lot of uh
oh worry Parable loss experiences is
that you have to optimize your buys uh
that's a strategy for centralized
Finance I mean centralized trading but I
it it does work for defy and helping and
it's basically spotting when the price
hits it hits the threshold and it when
when it's oversold within the Bollinger
Bands like I mean like in the in the
lower part of the Bollinger Bands and
you you want to do that you want to
identify that buy signal throughout uh
different uh temporarily these
temporality
um like the weekly time frame the daily
time frame and maybe not no shorter term
like the time frames maybe the three day
time frame I don't know but they are
identifying buy signals throughout
different temporalities using the build
mirror pens those the that's that's
basically
oh it's working yeah you look like you
have an idea
no okay
you have an idea okay
um so we discussed an idea of Auto
rebalancing crypto index so let's say
you want to build a simple index that's
like 50 wrapped VTC 50 wrapped if
over time that balance is gonna be
different 50 50 if the price of Bitcoin
increases for example so maybe using
some some kind of feed like that to
calculate the balance of your portfolio
and rebalance every hour week whatever
cool yeah rebalancing I think is a is a
very good use case like in whether it's
you know just a simple spot trading
portfolio or it's you know something a
little more complicated like lping or
Landing or borrowing that kind of stuff
um do we have time for one more maybe
we have a little time okay it's okay
anyone else cool
so
um like um none of us were actually in
device space but we actually uh thought
about some other use cases
um so um I'm working on um a reputation
on computation protocol
um based upon Paradise trust like a
whole trust to him by how much and then
I'm like in my output would be a bit
basically the number crunching would be
um like after all that because of the
calculation like um each peer in my
network is um assigned a trust score
right and like he's actually from the
dial space and we talked about how they
can actually use this um this type of
algorithm in like a voting uh versus
like a simple prevention typically um my
algorithm is pretty good at like I'm
identifying the Civil clusters who
actually trust each other but like none
of the other real people actually trust
any of them so maybe we can actually
ingest this data and then um do that
number crunching identify those civil
culture and then um his Dao could
actually um emit something onto the
contract so I sort of like those um
um accounts as potential maybe like
avoid them you know things like that
cool thank you maybe one more if anyone
has one if not it's okay
anyone Keen to share
Adrian you you look like you want to
share
you're writing your idea right on your
phone
okay later okay
um cool yeah that's it um so we'll just
wrap things up here uh yeah we don't
have time for that I don't think so
we'll just go to the last last slide I
guess yeah if you want to run with any
of these just remember to think about
like we have the basic idea right now
but there's a lot more to it I think
adversarially where are the attack
vectors
um how could your strategy be be
manipulated
um you know
uh be defensive
yeah and then just to close things out
um
to mention what we're doing again uh so
at component we're building something
called blueprint and all of your ideas
except for maybe the last one that
sounded really cool but it was a little
over my head um but very very cool
um but for the defy related stuff at
least and would love to talk to you
about what you're doing to see if we can
incorporate that in the stack so if you
have any ideas that you actually want to
execute on or you have you know friends
that like want to build these kind of
data driven strategies
um we're building tools that basically
make the entire process easy so you only
have to worry about the logic of what
you're building
um so yeah that wraps it up thank you
everyone
foreign
thanks a lot I know it was an
unconventional Workshop but I appreciate
that everybody was talkative and you
know engaging instead of just hanging
out on their phones and things like that
so it's great