foreign
[Music]
ER I'm CTO at streaming fast and I'm
also a Pianist a data scientist whatever
that means I'm a father of eight
beautiful children two of whom which are
there I love designing and crafting
software which I've done since I was 12
and I'm here today because one day in
2013 I read the Bitcoin white paper and
that changed the trajectory of my life
and fast forward to today
streaming fast a company based in
Montreal Canada is now one of the core
developers of the graph
and we joined the graph a bit more than
a year ago and it kind of bizarre m a
2.0 fashion or lawyers they don't
understand what happened but anyway we
said thanks and goodbye to our VCS and
shifted our Focus to make the graph the
greatest data platform on Earth
so today I'm here to introduce
substreams which is a powerful new
paralyzed engine to process blockchain
data and before I can do that I just
want to set a bit of context perhaps you
can raise your hand if you know what sub
graphs are raise your hand if you know
you're good okay so slip graphs can be
thought of an ETL process right extract
transform and load and subgraphs add
that little Q there that graphql layer
to it and some grass today provide that
sort of simple approachable end-to-end
solution to blockchain indexing and the
graph node is responsible for all of
these components right the extraction is
done through hitting Json RPC nodes and
then transformation you provide some
assembly script you guys know that
composite wasm running in a distributed
environment and then you have the load
aspect which the graph node does puts
that into postgres and offers you a rich
you know and beautiful graphql interface
on top and one of the reason one of the
reasons we were brought in was that so
we could push the graph to New Height in
terms of performances so to do that we
brought first thing the fire hose
something at the extraction layer to our
take to boosting performance by one two
three orders of magnitude the first
layer extraction it's the method of
extract attracting data from blockchain
nodes imagine prying an egg open where
the data is exfiltrated as fast as
possible and all the juicy data gets
then thrown in a grpc stream as well as
into flat files and you can think of
that as sort of a the bin log
replication stream for blockchains where
you'd find in a Master Slave replication
engine like in databases so we'll get
back to fire hose in a minute then sub
streams is sort of a reasoning of this
rethinking of the second box the
transformation layer here instead of the
traditional sub graph handlers and
assembly script you will write
substreams modules in Rust and those can
be executed in real time as well as in
parallel with unprecedented performance
so let me give you first a primer on
fire hose because there's a lot of
benefits of substreams that come
directly from the fire hose so a
streaming fast for many years we've been
thinking hard about all these indexing
problems from first principles and we
needed a first a robust extraction layer
we wanted something that is creamy but
that was extremely low latency so I'm
thinking that would push data out the
moment the transaction was executed
within a block within a blockchain node
Json RPC was not going to cut it and we
didn't want to have to deal with those
large bulky nodes right hang on a thread
occupied due to with uh managing high
right throughput I kept everything in a
key Value Store behind a you know a Json
RPC requests and it was really heavy in
Ram and CPU and you needed super
optimize ssds this is really annoying
and all these things are much costlier
than what's needed when our goal was to
get to the data inside so we also wanted
proper decoupling between the processes
producing the data so the blockchain
knows and its intricacies and it's it's
request response bottle and they're all
different and the data itself we wanted
the data to be the interface and we
wanted something also extremely reliable
in the sense that we could avoid hitting
load balanced nodes that had all sorts
of different views of the world and that
we need to have like client code to like
latency inducing code to to resolve
what's happening there if there's
another fourth you need to query nodes
again and and you know for
reorganization heuristics for example
but also you know we wanted something
better than even the websocket streams
that pretend to be sort of you know
linear uh that the nodes have
implemented because when they would send
you a signal that would say this block
was removed it can leave you hang if if
you happen to be disconnected for just
half a second you'd reconnect you'd miss
the signal so the reliability was not
built in so we wanted something to
address that and above all we wanted
something thing that is able to process
networks in 20 minutes well okay an hour
or two but you know never three weeks or
things where we're waiting linearly and
that's still our goal today and when we
say Network history I mean executing
guests and extracting data executed into
flat files that's the extraction layer
but also any sort of indexing after the
facts we wanted to be able to have
massive parallelization like there was
no other way to have reliable and
durable in performance without
parallelization so our solution was the
fire hose and the fire host of all of
these issues in a radical way we took a
radical approach because we wanted to
solve those problems definitively like
meaning that there would be no further
optimization possible except attempting
to bend sort of space-time Continuum
itself right so with streaming we with
even multiple nodes pushing out data
multiple nodes are actually racing to
push the data the first through a
consuming process gets the first to get
out like you can't really add remove
more latency there and um there can be
nothing faster than immediately when the
transaction has just executed from your
node and then like regarding the staple
processes and cost flat files flat fast
for the wind we have a hashtag for that
right flat files are the cheapest much
cheaper than processes they're easier to
work with there's nothing simpler nor
cheaper in terms of computing resources
these storage facilities have been
optimized like crazy and it's also where
data science is headed these days and
there's one common thing to every
blockchain protocol and that it
processes data data is also the right
abstraction for this technology not an
API that's common to all chain data so
fire hose clearly delineates
responsibilities and and the contract
between the extraction and
transformation layers is again the data
model fire hose creates and for every
chain you can imagine the best data
model the most complete and that's what
he done for fire for for ethereum for
example that both the the data model for
ethereum within fire hose is the richest
there is like you have in there the full
call tree internal transactions you have
the inputs and outputs as raw bytes you
have the logs obviously you have the
state changes like you see on etherscan
down to the internal transaction level
you have balance changes the same way
with the prior value and the next value
so when you're doing like navigation
backwards or forward you get you have
the data you need you have also gas
costs at different places and there's
that important notion of total ordering
between things happening within the logs
State changes and calls all of these
things happening during execution are
totally ordered so you get in there
everything parity traces would give you
and more and everything you would need
to rebuild a full archive node from Flat
files and everything there is goes to
the transaction level not rounded at the
Block Level which is crucial if you want
to index with Precision right it doesn't
rounding of blockchain information at
the Block Level is sort of a was meant
for helping in consensus right but it
doesn't mean that what happens mid block
is of less value than what happens at
the boundaries so okay so that's very
interesting and now regarding
reliability whoops no not too fast
regarding reliability the fire hose erpc
stream provides reorg messages like new
block or undo this block or this block
is now final accompanied by a precious
cursor I think that's really key here
with each message so if you get
disconnected and upon reconnection you
give back that cursor you'll continue
exactly where you left off and
potentially receiving that undo say
signal that you would not have seen were
you disconnected right so you will get
it so with the guarantees or linearity
of the stream so no websocket
implementation would do that because it
doesn't make sense for a single node to
track all the force possible even two
days after the fact and undue messages
come with full payloads so you get all
the Delta so you can just turn around to
your database and apply the reverse or
you know pluck again in the the full
payload of what happened in the block
and just like what you do is so it
doesn't pose on the reader to store what
happened like at that previous block if
the signal was just remove block 7000
right okay and when you commit that
cursor to your transfer database well
you get through that you know finally
some consistency guarantees within your
you know your your back end so some of
our users told us they could cut ninety
percent nine zero of their code reading
the chain because they were relying on
some that reliable stream and okay and
it also lays lies down the foundation
for massively paralyzed operation files
plus the stream and so this is the
future of the graph unbeatable
performance and it's core to our
multi-chain strategy because you know
any blockchain can have that data model
uh now let's dig into sub streams
substreams is a powerful clustered
engine to process blockchain data it's a
streaming first engine and it's powered
by the fire hose underneath and its data
models of the chain so let's dig in
here are a few quick facts it's invoked
as a single Jace grpc call and within
the request we provide all the
transformation code like you'll have in
there oh it's too low you'll have in
there the code from some wasm modules
relationships within the modules and uh
you know all the transaction the
Transformations within the request it's
not a long running process except if you
run it for long it's not a service you
you spin up right and the backing nodes
are stateless which provide nice
scalability properties modules for
Transformations are written in Rust they
compile to wasm and they're running a
secure sandbox on on the infrastructure
they're similar to the sub graphs and
the ultimate data source being the
blockchain data being deterministic all
the transformation outputs are also
deterministic and the request if the
request you send involves process prior
history even if it's 15 million blocks
well the substream's runtime will then
turn around and orchestrate the
execution of a multitude of smaller jobs
in parallel fuse the is lost on the Fly
for you and aggregate the results to
simulate a linear execution so you would
see a dime in the difference and all the
results are streamed back to you as fast
as possible with same guarantees
provided by the fire hose with a block
per block cursor and a transparent
handoff from batching into a historical
processing to the real-time low latency
rewards aware stream of the head of the
chain so let me show you if you're
interested how we create one of these
things raise your hand if you're curious
okay you're good okay so let's start we
start with a manifest like that do you
see that down here can move the podium I
can't so there's package information you
know some metadata there you have
pointers to the Proto buff that you'll
use again contracts between modules are
about data so there are protobuf models
similar to the protobuf models of the
root chain of the chains the layer ones
and you have pointers to the binary that
you're working on your drive and all
that and you have Imports and imports
are actually very interesting because
you can import third-party substreams
packages and these yamo can be packaged
and so you can import from someone
else's package you can write your own or
combine both that means substreams
enables composition at transformation
time which I think is pretty unique in a
pretty Game Changer and then follow up
and there you have the module sections
which defines the relation between the
different modules and you see it defines
it directed a cyclic graph you have
modules that slowly refine the data and
so there's two types of modules one the
mapper the first up there map pools and
this one takes inputs does
transformation and outputs it's
parallelizable down to its core block
wise though massively paralyzable and
then there's the store input I think
it's awesome this one takes any inputs
and outputs a key value store that or
something accumulated in a stateful way
and stores can then be queried by
Downstream modules and uh okay so we'll
see a bit more after then the name
corresponds to the function in the wasm
code and the inputs can be of a few
things either the raw fire hose feed so
for example The Source here that means
the block with all transactions you know
for for that particular block and it can
be the output of another module like you
see down here the input of map map pools
so you'll get the data as bytes and it
also be a store which would be a
reference we'll see in the next slide
there and on the store pools here you
see there's an update policy which sets
constraints on what you can do with the
store and it defines a merged strategy
for when you're running paralyzed
operation
okay I'll get to that also a little
later and the value type field will help
anyone decoding understand what bytes
there is in that store so you're going
to UI you can jsonify them and your code
consuming can automatically you know
decode them with protobuf all languages
supported otherwise the key value is
just Keys as strings bytes values very
simple and one thing to note here is
that because it has deterministic input
it's possible to Hash a module like the
kind and all of its inputs and the
pointers to its parents and including
the initial block so you have a fully
determined and hashable let's say cache
location for all of the history similar
to git right all the history of data
produced by by and you'd hash also the
wasm code right so it makes it for an
extremely cachable system and highly
shareable and cross verifiable output of
modules which opens really interesting
possibilities you know for uh
collaboration within the graph ecosystem
and imagine one has large disks and
everyone has large CPUs or you know
sleeping CPUs they could pull resources
together to build something bigger than
themselves
okay you see the relation there so this
gets piped to that and if we add another
module here you see how the graph comes
together this one computes the price
this is unit swap V3 thing it computes
the price but it you want to get them
for certain pools because maybe you want
to use the decimal placements in the
pool we'll see a little bit more there
and when you're running let's say you're
running that at block 15 million
well you're guaranteed the runtime
guarantees that the store you'll have to
execute code at block 50 million will
have been synced
linearly or in parallel but you wouldn't
know but it'll give you a full in-memory
store eventually backed by some disk but
whatever and you can query the key value
store at each block it's guaranteed to
be synced for you that's exciting no
okay uh so you see the dags fully being
built huh right so the dependency so now
let's let's
this leads us to composability see each
color here means a different author and
modules written by different people
ideally the most competent for each
right like we would hope they would crop
you know analyze what's on chain and
refine the data and Abstract it to new
heights
and the contract between the handoffs is
always data it's the model of data so
you take a modulus bytes in bytes out
and so you see here we can get the
prices from using V2 and price this
version of B3 and Sushi and chain link
and whatever and have someone reality a
module that takes these input at
transformation time
and then averages them out and whatnot
right and then that you'd have like sort
of one beautiful Universal price module
that you can then hook on top and and
feed to some who knows maybe if someone
feeds that back onto the chain for some
reason and then soon enough you know all
of that while someone wants to build on
top of it ah something like that if
someone wants to compute you know the
USD denominated volumes aggregation of
nft sales on openc you know you'll take
some sales you merge it with the price
and we see here that little Trader Inc
he seems he maybe he wants to feed that
into his trading bot because it's a
streaming engine we're not storing that
in the database yet right but this begs
the question where is that all that
beautiful data at land where does it get
piped that's where it thinks head up
like substreams being limited to the
transformation stage of the ETL analogy
remember it doesn't really care where
you load it and that could be anywhere
these are just a few examples you can
load that in databases we already have a
sync for postgres and you hook to
sub streams and it just loads it into
postgres with a data model that we've
agreed upon right if you write it in a
certain way it just thinks over there
and or message cues or whatever you know
data lakes or some Bots or some trading
already you know so I don't know some
whale detector you want to hook directly
on the stream or also so something I
think big for doing some ad hoc data
science because you know you have a
really fast engine allows you to process
the whole history in like it can take a
few minutes to process the whole of it
here image to go and pluck some new
insights so you can write your code send
it to the network and then you know
stream out the results similar to for
those who know big query you know the
cluster the big cluster the service by
Google that's what they do you send the
request they just shot at everything
they send you back the request well all
of a sudden substream's engine can allow
you to do some things like that ad hoc
and it you can write any program uh that
that supports grpc in protobuf which are
many and the last one here not the least
subgras through graph node well so we're
working to make substreams feed directly
into into graph node to then provide the
same loading experience and then
querying experience that you've come to
know and love and you'll be able to
deploy a sub grasp this time not
containing assembly script but a stud
streams package with an entry point and
would process the history in parallel
and load that in your database in Crazy
speeds so stay tuned for that that's not
out yet but you know soon okay and so
this is a simple example in Python
it's not really longer than that you
have two one or two dependencies like
grpc so that you can use the query so
we're leveraging a lot there and you can
uh you know see that spkg there we can
use that to code gen python classes and
helpers and all of that because it turns
out that the Manifest through the spkg
there is for those who know protobuf is
they file descriptor set it contains all
of the things all the protobuf
definitions so the spkg also contains
all the wasm code the module graph
information you know the dependencies
the inputs and all that and even some
documentation everything is needed is in
there so you can pass it down to the you
know modules you take it from from the
disk and boom you send the request to
the server and it's running so you can
deploy packages also very easily and
consume them very simply this way
there's a few Imports we've omitted
there but
it's simple this is the show okay and
let's look at a simplified data model
for unionist swap V3 and I'll show some
code making use of it okay so this here
the pool is a list of pool this is
actually what gets handed off from you
know our mapper which finds the pool
that were created down to the store pool
which we're going to look also and so it
has a list of tools and the tools you
can imagine an address and the two
tokens that are concerned here and we
have a reference to the Token also which
is going to be very useful to enrich the
data down so we will have the decimals
right at hand like we won't need to do
much loading it's going to be very very
close so we can enrich all these U ins
250 79 000 and you know put the the
comma where it belongs
um so let's see what happens in the
mappers so this is a sample rust code
raise your hand if you love rust
where's your hand if you no rust okay
it's very simple here I'm gonna go
through you have the map pools function
corresponding to the Manifest there has
one input the block this is the fire
hose block with all transactions all
logs All State chains you can craft your
own triggers in there as you wish but we
have a simple Voyager and see that that
line there blocks events you have a
thing that goes through transactions and
it's going to trig on pool created and
that pool created object in Rust was
actually code gen from the Json API so
you can just give the instruction and
we're going to filter it for only the V3
swap Factory and then that beautiful
filter map will give us the log okay and
then we'll we'll output we're going to
collect some of these things into one
list of pools and it's assigned to the
pools object there and and notice that
little thing here this is the RPC create
unit Swap tokensing this actually hits
you know an e-call on a node behind
similar to what we have in sub graphs
that's actually very important it means
that once we've processed this layer
once and we've done it for the history
it can be cached very efficiently so
anyone relying on that thing will never
need to reprocess it again you can give
the package to someone and they can
access the stores that's been cached by
other people immediately so you could go
to block 50 million and you'll have the
list of all pool created that you can
query super fast you can depend on it
also so I think that's pretty cool
and here that's the store modules the
store module is pretty simple here it
receives C the pools from the output of
the prior module and it does it Loops
through the little pools there and calls
output set and see the key there is pool
colon the address is going to store the
Proto buff encoded stuff of the pool
with the token decimals for both right
did I say that to be constrained so the
store here is constrained in two ways in
order to preserve parallelized ability
the stores are right only you cannot
read your rights otherwise that would
make some potentially cyclic and they
expose only the function defined by the
update policy in this case it's set so
let's see what happens
if we run things in parallel
so here we have two jobs covering two
segments of the chain you know one
million block each and to see those ugly
arrows there they correspond to a pool
created event
and so in our code you've seen we would
write a key for each of them and uh so
in the first partial run we'd have a
what we call a partial store with four
keys and the next one we have two keys
and so when we run the merge operation
we would apply the set merge policy
which says basically if you take one
store take the other store cycle through
keys and the last key wins if you do
that you can paralyze endlessly so we'd
have here a complete store with six keys
and now at that place we have a snapshot
we can have periodic snapshots and so if
you want to go and explore the chain at
any point in time you have a snapshot
plus a little partial you can have the
state synced at any block height
so so this one you know has the last key
win policy but you have a few others
like min max ad and another one like
first key wins so if you merge them you
have set if not exist and then that that
allows us to build different
aggregations right you'd like to see
that running live
every few minutes now I need to bring
that other window up so you see that
okay that's good enough huh okay so
let's imagine we want to
see that pool pools created thing okay
do you see that I want to see the output
I'm going to run that I hope everything
is good you know the demo of gods are
connecting
okay okay whoa not too fast
so this is going through uh starting at
the beginning and we have there a pool
credit event and see that we have
everything decoded because it's the
protobuf thing we have the thing to
decode it we could feed that it arrives
on the wire as bytes and it properly
serialized bytes and then we see that
the token address is there we have a
decimal we have the address
and so what that means is pretty crazy
already oh do I see that okay what that
means is that you can inspect the chain
with your code at any place in time for
a mapper especially you can go there and
I could run it again here and say and
say I want to run the mapper let's say a
block I don't know uh
something more recent give me a recent
block what's the block yesterday
fifteen seven million I know something
like that okay and see is there anything
recent so there's some stuff right some
things are recent can I see that someone
still created a new pool
and I can inspect my code to make sure
it works where is that
come on right this one was rap ether and
infinity they were just created that
address as a new pool so you can go and
test your code everywhere what's that's
done well you're set right you can go
then to the next data dependency so this
really changes Dynamics for debugging
and this can also work for stores they
can you can ask for a store and it's
going to process it in parallel and then
you can inspect all the keys that exist
there or see the Deltas coming through
okay and let me show you something
running in parallel
uh-huh graph out now this is very
interesting because oh no let's start it
at 15 million let's say I want graph out
at 15 million I didn't run it again
before and I want so let's write it
so this starts a whole bunch of parallel
processes and you see up there the
number of blocks per second yesterday I
had about 8 000 on Solana blocks I had
16 000 depends on the power you put
behind there but this all like you said
the pool counts as a dependency on the
pools the pools is further down so we're
able to schedule things and all that
just massively parallel and once that's
ready let's say everything was done I
would start streaming and get all the
content and so let me show you the graph
out it's very interesting because
because graph out
graph out
has refined the data up to entities now
we're talking about database tables and
Fields and you get and you get out of
that
do you not see it okay that wait a
second here so let's imagine see we have
token and an update and you have the
field derived if and you have the old
value and the new value I don't know if
you've seen this thing in the data
Science World this looks very much like
change change data capture cdcs that can
power a lot of large-scale systems and
uh and you have the prior after so you
can feed that to your let's say postgres
apply the changes and when you have an
undo signal it gives you back that
payload you then just flip everything
and you have guaranteed linearities to
your store with a cursor and it is
Flawless it's just extremely simple to
keep your things in sync you also want
to have a slack body can have it undo
message remove the message if you have a
thing coming through right
so
I think that's pretty cool what do you
think
okay
okay that's cool can I shut that down
here do I have another window
uh I'm close to them prepare your
question please ask ask them succinctly
we have a half a minute I just wanted to
have a final final note there
so as a final note
I want to share with you a little bit of
my vision for the graph
okay I don't know where the window is so
whatever it just says fine it's fine uh
I'm imagining the graph becoming that
sort of huge worldwide cluster of
processing and storage capabilities and
something like Google's bigquery but
where people join because it's better
together right instead of running it
alone we need to have all the resources
alone and I see also a new era of
composability which means more
collaboration and in a tighter Community
working together more intimately in
those in those uh you know with those
data contracts and I see also a new mix
of collaboration between indexers like
exchanging data or sharing resources in
terms of compute and stores and whatnot
therefore introducing new value flows
and also I'm seeing new products new
Services being offered directly on the
network you know to satisfy some needs
that perhaps couldn't be addressed
before and I mean there's a place for
you in there as a developer as an
indexer as someone who realizes the
radical benefits of such a platform and
who who builds on it and promotes it so
my ask to you is you go to you go try
substreams put pressure on your favorite
layer ones so that they do integrate the
fire hose natively that's pristine's
Aptos has done that recently some others
start wear I think so that makes it
everything we've seen today becomes
immediately available to them sell them
on the goodies so also join a Discord I
love follow-up follow-up questions and
come see me afterwards I love feedbacks
on these sort of things all of this is
open source so let's dig and build
something of the like together the
biggest blockchain platform on Earth
thank you for your time today
do we have time for two three questions
we're the last one so if you have a
question
hey uh so one question uh modularity and
composability of these substreams is
super super powerful but still if I look
at this compared to SQL and like DVT
models right it's a lot more complex so
how can we enable people to really kind
of learn this and like build these kind
of hyper modular data streams so
SQL layer like this is powering going
through history it's an adult transform
with stateful storage but you would
pipelive into SQL store to do other
things right you would have refinement
you have Knowledge from the community as
to how to analyze this and that protocol
ever ever you know increasing
refinements but then you might store
that in your store with off-chain data
and maybe that's best fit for you maybe
you feed it into a subgraph that's what
you need you had a total decentralized
solution and you don't need to host
anything so this is enabler at a lower
level it's not a it doesn't seek to
replace SQL but it puts itself at a
place where we can feed all the systems
on Earth with enriched data which you
would need to do in SQL and it's it's
really not fun so you leave that to the
community right gotcha thank you we have
our old subgraph which is pretty slow
and we would like to transform it to the
new type of subcraft Yes Should I only
read some code on Rust and that's it or
something else so it is not the same
Paradigm to enable parallelization
Nation you need to distinguish the data
dependencies and that infers the number
of stage of parallelization that is
needed it's not easy at all actually
it's pretty crazy to try to parallelize
the subgraphs we try that that's what
yield us to design substreams by cutting
you know uni swap stuff so you will want
to go and write in Rust modules and it's
a different Paradigm so it's not just an
easy switch I admit but it brings us to
the next stage an evolution you know of
blockchain index thing yeah I see thanks
uh thank you so much Alexander
my pleasure