foreign
[Music]
good morning everyone how are you
so
let's start I'm Tom I work for simiotic
labs and today I'll have the pleasure to
present you our work on reinforcement
learning in color pricing in the graph
so first let me start with the outline I
will tell a few words about semiatric
what we are doing then I'll introduce
what what's really we are doing in this
area automated price Discovery I will
formulate the problem I will
add a few words up when it comes to
modeling how we model that problem then
I'll briefly went through reinforcement
learning Basics let's say it's just a
refresher for those ones of you who know
or maybe do not know enforcement
learning then agent-based modeling is
the the core of the talk and I will show
how we're using that for testing system
properties then and for testing well
different behaviors when it comes to
using single and multi-agent setups
and then I will show that actually we
deploy that that solution in production
what are the results and finish with a
small summary all right so let's start
the first semiotic symmetric Labs was
founded in 2020 by Ai and cryptography
researchers uh we are focusing on
applied research and we are one of the
core depths of the of the graph protocol
also we are developers of others the
optimal text aggregator uh actually the
the lead of that product is giving a
talk at the very moment on a different
stage so
bad luck okay so our expertise we
we combine the cryptography with AI
That's our core expertise and of course
there are software engineering involved
and we focus mostly on building
infrastructure we focus on crypto
economics today's talks actually
focusing on this on crypto economics all
right so
what was the scenario so the scenario in
here is that
I don't know which one of you know the
graph actually who knows the graph
protocol all right
okay 50 50. good so basically in this
scenario we just focus on on on one part
of the graph system right of the graph
protocol but imagine that you've got
some customers that are sending queries
right and you've got indexers that are
indexing blockchain and our mechanical
serving serving those queries so between
those guys there is an entity called the
gateway which is doing we're just gonna
go acting as a query market right so
depending on the price the price
depending on the quality of service a
given query sent by the customers when
we would be distributed on this or or
that indexer and indexers earn money by
serving course right so they can control
the prices of the service queries and
the core idea that that I'm going to
talk about in today's basically wanted
to have Dynamic pricing based on query
volume received by an indexer and we
call that auto Agra and then I will
explain the second why we are calling
that Auto augura
so this is how we model the the scenario
right it's it's simplified from one
point of view because as you can see ah
as you can see there are no no let's say
agents in here we get rid of the
customers for the purpose of that
flooding of those simulations instead
we've got a traffic generator right
something that just says okay at given
timestamp this is the query volume that
you've that needs to be served right we
also got query distributor so something
that basically look at at the bits of
the agents so the agents are putting
their bids and then depending on those
bits the queries will be distributed
across those different agents and those
bits are expressed in a domain specific
language called Agora that's why the I
the the the product we call it altag all
right because it automates that that
building otherwise the users will have
to the index cells have to basically
create those those price models manually
which is kind of tricky right
uh all right so some assumptions
selected assumptions what we're
discovering in here so this is only
about
how we generate the dot query volume so
we've got in here so this is basically
the shape that that we we model so the
x-axis there is this price query price
slash this is a budget of of a customer
that customers are saying okay this is
how much I can pay for that query this
budget is unknown right that's important
then it's not sharing that with the
agents it's not sharing that with the
indexers it's something that that that
is setting on its own so this is
something that we want to discover right
and the budget can move right from left
to right which means that if the if the
if the indexer was set its price its
price bit higher than this well he won't
be picked right also there is some kind
of like a noise we we added a noise to
that volume to that query volume and
important is to know that there's
there's a game happening right in here
and the game is suraj's goal is to
maximize the revenue
but gateway's goal and basically
protocol goal is actually quality of
service right we want all the queries to
be served you know
uh five nines and so on right so there's
a game happening all right so if you
know how model let's not now switch
gears to reinforcement learning just
one-on-one most classical thing so
down there in reinforcement learning
we've got two main entities agents and
environments agents interacts with
environment by executing an action then
agents actions change the state of the
environment I just got reward in the
observations after after
oh okay and I just can up their its
policy based on the received reward in
short that's reinforcement learning so
we're using different types of agents in
our simulations uh in here I put two
most important criterions one are
trainable reinforcement learning online
learning and the other ones are like
rule based so there are some predefined
behaviors and also we've got this second
Criterion Solutions can be stochastic or
deterministic
in here I put some
some simplified classification of of the
reinforcement learning algorithms and I
highlighted the two that basically we
will be using in our simulation so we'll
be using vanilla policy gradients and
mostly most of our agents will be using
PPL
important
there is one important type of ideas
that actually we we are mostly focusing
which are we call them gaussian Bandits
so the idea that this is sustainable uh
stochastic agent with policy that is
represented with a gaussian and then
when an agent is is supposed to perform
an action in something from that
gaussian right so imagine that there is
this distribution and when that when the
digest is supposed to make an action
it's just sampling right and in this
case Okay the agent sample is 0.8 and
from that blue distribution second agent
sampled python 24 right and why Bandit
Bandit is because the agent is not
building any an internal model an
internal representation of the
environment
all right so going to the most important
part so testing based testing properties
with edges-based modeling
so in here
so already kind of explain this but in
the first set of experiments we focus on
single agent simulation just testing the
properties of an agent and this is a
single gaussian Bandit with a modified
PPO policy update Rule and the query
distributor is like a super naive you
know let's say version of that what
could happen if here is that we've got
access okay depending on the price
business is just inverse inversely
proportional and in the first step we've
got a fixed customer budget with some
noise of course right and what we are
checking in here is the following we've
got market conditions we've got two
criterias for every experimental custom
criteria and a property that we want to
test whether the Bandit is fulfilling
that that Criterium is is operating the
way we want or not in this case the
Criterion is customer budget discovery
as I've highlighted before and the
Agents
do not know the budgets right but so we
want to discover that
so first so
that's there will be lots of things
happening right because we are tracking
many things there are in simulations so
let me try to let's say unfold that and
go step by step explaining what you will
be seeing in a moment in in those
animations in those videos that we
captured during the experiments so first
there are five plots right in a single
video first one is the most important so
what we've got in here is the is that
budget right
the query Volume Plus Plus customer
budget the solid red line is the agent's
current policy the dashed line is the
initial policy that that agent is
starting from
second in the second plot from from the
top this is basically query volume
server by the agent at that given time
step so what happened in here is Agent
actually didn't serve a query right
didn't get any queries because it's he
sampled somewhere in here right like few
steps ago he sampled some like the the
the price was too high right so the
agent didn't get any queries so
which
which also is shown in here so this is
the aggregated query volume served by a
given agent in red and the blue ones
shows basically how many queries were
dropped weren't served at all right so
you can see in here there's a tiny tiny
step which is which actually is
associated with this then in the next
plot we've got agents revenue and
aggregated agents Revenue over time all
right
so let's run our first experiment right
so once again what we are testing in
here what we are trying to discover is
word up that
where that that budget is right so
as agencies is learning right so first
the gaussian is was super wide right
like it was like much much wider and
once the agent as you can see it's
something from time to time somewhere in
here right above the budget so it's
getting more and more sure that okay
this is the max that I can get right
that's the highest price that's that's
the highest bid that I can make so as
you can see the agent is dropping some
of the queries right uh what is revenues
is nicely nicely growing all right so
first proper property test right that
was the static
environment so how about Dynamic
environment right by by Dynamic I mean
that the customer budget can change can
vary over time right so in this case we
are also testing the same property
whether the customer budget can be
discovered by the agent but in more
difficult market conditions let's say
right so once again we are starting
agent is kind of a converging capturing
the the right the price right which is
kind of optimal and then the price
changes so what happens is the agent is
is not getting getting any queries right
so it's saying okay I don't know where
am I so it's press the gaussian and once
the queries are once again captured it
it moves to the left right so it's like
okay I was above the budget let me let
me go back right and that happens that
is happening over time right so there
are plenty of queries queries dropped
but still we are we are making some nice
Revenue right and we are reacting to the
to the dynamic dynamic Market
cool second property check
so in further experiment we said okay so
there are some subgraphs that actually
you have no queries at all right there
is a moment where nobody's is asking for
queries so what will happen then
so that's a different scenario in this
case we call that the marking conditions
basically there is no demand for your
services right so we wonder how we
should add
so
it will happen in a second right so
there will be like three iterations
we're recording few of those those
videos but as you can see already
there's a hint in the right hand corner
that something is not right all right
gaussian spread this is great and now
time set 400 because every 200 steps
were like you know to write it okay so
what just happened there are no queries
so according to the design the agent
just said okay let me spread right so I
can sample from wider distribution but
actually spread all across the the the
the price Biz domain so we said okay
actually this is not a good behavior
right we if this is happening then the
agent can can sample at the same
probability like super some super small
prices price bits and at the same super
high moreover recovery from that will
take a lot of time so we said okay can
we improve that so we implemented
something that we called graceful
initial pull which basically
uh the idea is that okay there's this
initial distribution that the agency is
starting from this is something that the
index service will parameterize the
agent when when when the agent is
deployed and where when there are no
queries it just slowly is pulled towards
that initial distribution right and when
the queries will appear once again it
will get from there and once again start
there's something
I'm losing from time to time
the voice I think
um
Okay cool so we pass the property so now
let's move into multi-aging simulation
so this was great at that point you know
we aren't we aren't competing with
anyone up till that moment right now we
are starting with with some we're
competing with some deterministic agents
the idea is that
right now all the all the indexers have
that the Agra models that is fixed right
they are not changing the models so it
can be seen as a rule base right so
there's like okay that's my bid and I'm
sticking with it no matter what right so
this is what we are trying to model here
competition with deterministic agents
and the property that we are testing is
called
discover of prized bits of competitive
agents right so we want to make sure
that we we also will uh we can let's say
compete with them and discover what are
what were their bits without actually
knowing them without having That
explicit information so those three
lines in here
actually those are the three three
competitive agents with deterministic
policies that they're just set and the
blue agent in this case is the one is
the Bandit right and as you can see it
after a while it's set up like a price
just below the the the the cheapest
agents which means that it's trying to
once again snatch the whole Market if
you will see a number of total queries
right you can see clearly see a
difference right that agent is
dominating right so I'll say okay this
is nice check
cool so how about stochastic similar
experiments we also want to discover the
the price bits of other regions but
their policies are stochastic so we
model them in the singular way as
gaussians those gaussian distributions
right but they do not change right
so we observe similar Behavior this is
great right so the agent our agent right
our band it moves somewhere in here and
of course it's sampling you know that
the red the red the red agent uh
stochastic heuristic agent is also
something from that distribution so it's
changing but as we we can see direct
moved slightly below and shrank once
again to capture the market right all
right right I think it's we check that
property awesome so how about
it will right now compete with like make
a competition where we've got plenty of
those bands deployed and we hope that
actually this is the most inter that's
the most interesting case right because
we want Auto agara to be run by all the
indexers in the graph ecosystem
so same property slightly different
market conditions all right so what is
happening in here
sadly
it's not good
it's not good so that I just are acting
actually as designed but as you can see
they're like
fighting fighting fighting going moving
moving to the left hand side which means
that they're lowering their prices all
the time right and as you can see in
here oh you can see the labels but this
is this is the revenue that they're
getting right at a given time step
basically it's going it's converging to
zero which means that after a while the
digest will be basically serving for
free right and so that's not a good
situation for the indexers right because
in here we are not even modeling the we
are modeling the revenue not the not the
we are not modeling the costs right so
at that point agents will be well
indexes will be actually uh well
paying for that instead of making money
all right so we call this this this uh
phenomenon or this outcome raised to the
bottom and we run many simulations we
discussed that a lot and actually we
realized that well this is the expected
Behavior right in this setup so we've
got some images that are basically
trying to snatch the market right and
they're purely driven by by the by the
query volume right
so if this is the case and the
environment just natively distributes
you know the queries based on the the
price bit rest of the bottom is the
expected is the expected outcome right
so it can be addressed in many ways but
we look at the graph protocols what are
the features right and down there and
what are the assumptions so one
assumption is that all the elixirs
should have freedom with their pricing
right there should be any limitations so
that if they can if they want to serve
queries superb that are super expensive
or super cheap it's okay and all the
indexers should be able to make any
profits so the conclusion was that
actually is the Gateway right that the
thing that controls the market should
implement this kind of anti-dominant
policy and so happened it does it
already does so in our next step what we
did we wrapped the existing isi right
the Isis index selection algorithm is
one of the components of the of the of
the Gateway right so we just wrapped it
in here and using the simulation we are
once again testing the same conditions
right competing you know many versions
many government is competing with each
other and the property okay can we
discover each other prices and maximize
our profits so the outcome is totally
different right we ready by changing
changing the way the the queries are
distributed actually you reach the like
a consensus right so that all the agents
are discovering the markets right what
was the market at the same Gateway is
looking the ISA is looking at their
their quality of service uh at their
prices and trying to like feed everyone
right not just okay you're the winner
you're you're searching all the queries
all right so this is really nice so
please note that the gate will also
leave some kind of budget right that is
the this design such a way that actually
it's it's not enabling the agents to to
consume all the budgets right so the
budget is there is a portion that is
left to the
to the
let's say to the customer
okay so we're on some more simulations
of course it's like okay free can we run
that with 10 20 agents right will that
still work it seems yes it does so we
are testing the same properties right
which are just a little bit changing the
market conditions
right this is looking really good if you
look at look at the the query serve and
the revenue it's like all the agents are
you know are are making money which is
the desired the desired uh Behavior
right and they are rediscovering the
market cool so what we also did
uh
we we started we wanted to see whether
okay so how about different initial
conditions right so we're assuming that
different different in Xmas that okay
that's my mean price that's my variance
I want to maybe move a bit more
expensive for like less expensive other
being wanted to see whether the the the
the the system will converge to that
equilibrium and it seems that this is
really happening right so disregard the
initial conditions this is good we also
run some additional experiments with
with frequent a stronger policy updates
so this one is our modified PPO agent as
you can see it's much faster than okay
the red agent is the vanilla policy
gradients like parallel mental policy
Elementary update rule
and the other one is just pure
PPO right so our agent is kind of like a
dominating and it's kind of interesting
to observe that it's serving less
queries right the the blue or CN line is
clearly below and this is the number of
queries that are served but
but it's making bigger Revenue right so
in short
is making less work for more money
so policy really matters right
all right so
having those results we are quite quite
happy with with those we said okay so
can we deploy that in production and
actually that's our solution you know on
a Battlefield
so once again there's there are many
there are many things to unfold in here
so let me try to focus on one thing at a
time right so first this is the mean so
this is gonna go where the gaussian is
on that on that x-axis right so what
happens is that the initial value was
too high so the mean was going down
first but then insteadly it was going up
up good so it was moving to the right at
the same time variance was steady going
down which was which means that the
gyrogen was more and more narrower which
means that I just were like okay I'm I
think I'm there and I'm good right I'm
more confident that I should sample from
that our smaller distribution
so the outcome is that the reward goes
up maybe you cannot see it in here like
super nicely but that's the revenue
right that's the total revenue and you
can clearly
see that it's it's going up there is the
tendency right
so we deploy that
the results were great so I think we are
well this is the amount where I should
start telling okay if you're an indexer
please go ahead there's a repository to
download play with it and we there are
some additional materials okay but you
know actually in a second okay so let me
first summarize so agent based modeling
for crypto economics that was the core
of that talk and in here is focus we
focus on the diamond Dynamic pricing for
the applied to the graph protocol to to
automated price Discovery and we use
reinforcement learning and and a
multiple
multi-agent simulation multi-agent
modeling for for Revenue maximization so
we can show how to use that for testing
the properties of the protocol right we
came with that framework that okay those
are the market conditions those are the
properties and systematically start the
test right uh our our agents are
solutions against against those and
finally we have deployed out our in in
in production and have shown that
actually it makes sense so Future Works
of course better policy better updates
better policies agents of multiple
rewards of taking quality of service
into account we are not doing that right
now just we are just looking at the at
the query volume uh modeling and putting
consumer agents into play right so that
right hand side that we simply fight
actually putting that back in the
simulation it will give us some more
insight in what is happening right so
that's kind of that will give a
simulator with higher Fidelity and
uncomfortable redesigning the game right
so in this case there was no information
the agents were like acting totally
independently the questions it will
redesign the game redesign a little bit
the graph protocol would that kind of
perfect information will help right
there are some here are some additional
resources so my friend Alex is he there
is he wrote this blog post which
describes more the technical side of
deploying our Target in production he
also gave a talk around this about this
during the last track episode in June I
think so there's a recording on YouTube
and of course this is open source so I
encourage you go ahead play with it
write all those visualizations are there
basic you can reproduce that and try to
deploy your your your altarger in
production right uh finally I wanted to
highlight some of the of the other words
that samiotic is doing so their stock
actually that's happened right now that
that our friend Matt is giving his the
product and research data for us he's
giving it right now so that's uh that's
that said and Savvy is giving a talk
today about our work and or let's say
our our our our our our our trip in the
to the snarks world and we are focusing
on on this is more like cryptography uh
we are focusing on verifiable payments
for the graph protocol as well so if
you're interested in those those topics
please come to us we've got plenty of
positions open we if you're interested
please go ahead
thank you for your information
excuse me there's a gentleman who wants
to be back
[Music]
hello and thank you for the talk I was
wondering if the customer has a very
very low budget will you try to match it
and make a loss or will you shut down
that's a great question actually I
wasn't showing this in here right
because it's shaky like I said the goal
of graph is to have five nines so right
now we are working with with the guys
that are you know implementing the guide
when updating that a little bit because
for now if the customer will say okay
I'm I want that for basically free the
Gateway will still try to distribute
that
right
it's not the best outcome I would say
for the indexers right but we are
changing that
we are we've got a tool right now right
so you can think about I was showing you
that no we're working in this work we
are focusing on the agents right but we
developed a tool that enables us to test
the properties and the outcome of the
whole thing right when there are
different players in the game including
Gateway so great question
it's not tricky right the tricky thing
what do we want to achieve we want the
indexer to make money or we want five
nights right the quality of service to
to be super high
I have another question sorry uh maybe
more technical when you show the
experiments whenever the budget of the
client was going down the spread of the
ocean was was widening but the mean was
things the same which means that you are
still sampling to the right side where
you know it's not the right price would
it be more effective to move the
gaussian to the left as you spread it
yeah uh so what what's the question
actually because you just repeated that
the design Behavior right if we
overshoot
if we overshoot with the the the the
price then we don't know whether the
reason was there were no queries or we
were too expensive right so we spread
because we want to sample from wider
distribution right
all right I've if there are any other
you know additional questions I'm I'm
here I'm I'll be delighted to answer
your questions but it seems that my time
is up so thank you for your attention