[Applause]
the technical for the free stuff
technical kinda like advanced talking
awesome stuff there's been a lot of
interest agnostic so the D people signed
up for this particular talk I'd I don't
think everybody will fit in this work
you have to happen to show up but you
know we have very little time and a lot
of ground to cover so I'm gonna get
started on the undetected for it than it
be to be project properly what apps and
this is what we're going to be talking
about and missed it by yourself so
positive is a scalable instant and
extensible peer-to-peer gossiping
protocol and over agenda for today is
first of all I want to make sure that we
all have had like a comedy based night
to talk about these technical concepts
so I'm gonna do a very quick
introduction to beer beer to beer pub
sup who here is familiar with with these
concepts like maybe a 5 and above in
this game over 10 fantastic ok so now
I'm gonna move through this pretty
quickly then we're gonna talk about what
is cost observe we're gonna talk about
how the system operates and what's up
next because as you'll see in gossip
serve is a stepping stone in a road map
improves a lot of functionality that's
let's gonna get the future so alright
what is pop ups up is a message oriented
communication pack right this isn't
stock on trust with stream orientation
so here we're not interested in what
pops up we're know many interesting
sending transactional messages and small
units that are actually messages and not
streams right over the wire between a
bunch of peers that are not necessarily
talking to each other so the interaction
model of pops up is end to end this
means that NP r--'s are publishing
messages about a particular subject and
a lot of other peers in the network are
interested in consuming messages from
that from that particular subject wait
this is as I said in contrast to our PCR
reply with western pop patterns which
are mostly one-to-one no buffs up
communication is a synchronous and
decoupled in nature right this means
that whenever I send a message as a
producer I have no guarantee for us to
vol when who is actually going to
receive that message because we don't
have peers don't have a complete view of
the network and a complete view of the
subscription state and it's also it's
also a synchronous because there is no
time dependency I have received no
explicit confirmation from any peer that
they've actually received my message
processes act as publishers and
consumers of messages and then the peer
to peer network this is important
because the process can act as both
things and they congregated on topics of
interest now pub cell is implying
popular and then in enterprise software
and if you've heard of products and
projects like about Apache ActiveMQ
Kalka RabbitMQ then that's it but in
those contexts is usually being
centralized around brokers in private
networks and it's tough you know these
approaches and also to both
for public message markets so what is
decentralized appear to be your website
first of all it's broken s it means that
we have no central authorities to keep
track of subscribers publishers topics
it's self recognizing whatever we deal
with peer-to-peer networks we have to
mob uterine right and peer and networks
can rotate in presence very rapidly so
there is no global knowledge there is no
single peer that is an authority that
has a review of the network and the
network behavior actually emerges from
small tiny local decisions that peers
are taking another right so here to
pierre pops up is usually constructed as
an opening network with peers
collaborating to ensure message
deliverability ok and when we say
overlay there's usually an undulating
right so it's another name network that
actually make sure that messages and
traffic gets routed and usually
algorithms will assume that the underlay
is well connected so this means that I
can always find a path from A to B and
if I can't that's a component so just to
set a base value for terminology in a
topic is a subject on which appears
congregate and the image and receive
messages subscription is the act of
expressing interest on a particular
topic publishing is the act of creating
a message and pushing it out to the
network for winning is the apt of
propagating a message to other peers and
the topic and Fabian discovery is
something I talk about in the same so
the desirable properties in peer-to-peer
pops up our reliability Brazilians
effect efficient dissemination altruism
free routing this means that only those
peers that are actually participating in
one in a particular topic will will
actually contribute to for reading that
topic and we're not burning and
burdening the rest of the peers of the
network that are not invested in a
particular topic and disseminating that
it has to be scalable across various
axes and so we learn about number of
nodes number of domains messages
subscriptions subscribers all of these
are different axes for scalability
there has to be a law where we have four
sssay as I say we are relying on peers
making little decisions so that messages
are greatly routed in the network and
ideally what not a protocol that is
simple and easy to understand so we'll
see if we've actually managed to do this
one gossip session at them all right so
what does gossip so the key elements of
gossip serve is that first of all is a
scalable and extensible pops up so
concepts on in itself is not necessarily
intended to be the final usage of a
peer-to-peer protocol for peer-to-peer
support well it is a protocol on top of
which we can overlay different routers
to take different decisions and pops up
entry and gossip some actually provides
the primitives to make sure that first
of all messages are into the grade and
second that's enough gossip mentally
test circulating on the network so that
different and alternative decisions can
be taken on the way it was ambition and
specified initially by a developer
called be so he's a millipedes of the
quarantine and as I said it provides a
base layer and primitives to build
adaptive self optimizing routing
algorithms capable of repair and
rearrangements on the network to the
presence of churn and so on so the key
aspect of concepts up is that it
combines stable reciprocal Misha's with
gossip about message flow metadata would
dig into this in a second and this
exceeds a protocol that we had in the
living to be stab which is called
flutter and it's the basically the
dumbest protocol in terms of views of
your pops up you disseminated a message
to every period that you connected to
that universe and customer time right
which has which is creating thickness
its proboscis with Zambia then you know
you can take minimal needs and teapots
because you just push that message out
and well those beers that are are close
closely
and have a pass thing with you what was
he the first but it also creates a lot
of write amplification and it basically
plops clogs up the bandwidth is you
creating a lot of duplication of the
network as I said bas observe is the
stepping stone towards the grand mission
of episode which is I'll introduce it a
bit later basically come talk on God's
ideas from plum tree hi Barney on August
morning the steepness of gossip sub is
currently off right and this is very
important for people to understand it is
a protocol that we're gonna continue
iterating on over and over again and as
I said it's a stepping stone towards
this more stable and feature feature
movie protocol called episode there are
three implementations there is reference
implementation and go but they need to
be important to be produced there is a
implementation by the Sigma Prime guys
who are just here raise your hands
please get some credit fantastic it's
going to be sitting in a fork and I
don't have the chain same guys here I
don't think they are but they also
contribute to and they created a
concepts of implementation James cool so
how does it work
I'm gonna go through differently years
of the protocol together I do a wall so
the first thing that you need to
understand is that Glossop stuff isn't
actually a hybrid of two different
networks right that are on the same
notable that are on the same underneath
there is a full message network porn bar
reciprocal links through which messages
are forwarded and this is for superable
expect to keep the word reciprocal is
very important because this is supposed
to be a stable mesh where peers are
actually exchanging messages with one
another so if I if I'm puree and there
is interacting with peer be in a topic
these two peers will actually be
exchanging messages with one another in
system and then there is a metadata on a
network that carries gossips about which
peers what peers have actually seen so
as I said the full message network
recall that mesh is a sparsely connected
network or reciprocal links between
peers that are actively consuming and
interesting and therefore forwarding a
particular topic the construction of the
mesh is actually randomized and is self
step stabilizing because it has a number
of stabilizing from its particularly the
new so peers will are configured the
mesh is configured with a target degree
which 1946 there is a high and low
watermark which by default is four and
twelve and peers will strive to maintain
the connectedness for a particular topic
at an optimal level of six taking
compensating action whenever it goes
above below the low-water mark on or
above the high-water line this is how we
keep that that that leeway to avoid
drastic you know down cutting a
numbering anything this is so stable
mesh is complemented by a metadata
linear network which is a densely
connected network that augments the mesh
with gossip propagation about which
messages are actually living so this
enables routing decisions to be made if
I have so the easy way to put this is in
a particular topic if I have fifty peers
that are interested that are consuming
so fifty neighbors of fifteen neighbors
of mine are interested
in a particular topic six right in
theory based on the one on the stable
mesh the remaining so the remaining nine
will turn into the gossip years so I
would not exchange messages with them
but I will I will collaborate with I'm
telling them more messages I've seen
without pushing actually without
actually she has adjusted well we'll go
through this is so how does copping
membership work as I said appear can be
subscribed and the emitting messages on
various topics at wise so appears can
participate in many topics either a
subscribers publishers applause and
peers review of all neighbor
subscriptions whether they overlap with
ours or not so if also subscribe to
Tommy Kay and another period subscribe
to Tommy C but I'm not interested in the
see I was still note that the other peer
subscribe to talkin see subscribers is a
Santoni engage in message propagation
and gossip or personal topics so abalone
is actually participate and this is one
of the key properties of if you remember
that a desirable and pub/sub systems and
gears revolves up systems which is a
only those peers that I invested in and
always should be making memory too
so that to lhasa self-satisfied
so we have a concept that's called
subscription beginning which is
basically on every new connection beers
will exchange the topics that they
subscribe to and as the application that
is using the livery to be stack joins or
leaves topics there will send updates to
all over here so as i participating as
i've become a participant in that new
topic
i will send an update to to all of the
penis now we don't want to be very
chatty so we have introduced the concept
of piggybacking right so this means that
we will try to coalesce Octane's
metadata updates to peers on top of
messages that we anyway need to send
them right such as graphs and things and
so on as we can that's kind of beginning
enables us to create an about metadata
flows and to and to do also pheromone
topics anything which is possible father
explained assessing so this with regards
to topic membership there is two key
concepts to to to understand here there
is the concept of rotting improved i
said that the stable mesh is actually a
separate editing mesh and the and these
two messages are key to really
to regulate average so a broath control
messages brought them to a message
proposes a two-way full message stable
link will appear
so it basically when s1 appear sends a
graph to another here is basically
telling it hey I want to be part of your
reciprocal national this time hey this
means that I'm gonna send your messages
and you promised to send messages as
well right now the other peer can
protect that because they are ready to
be on their target degree right and if
they're rejected they send McAfee
however the other peer knows that I'm
interested in that topic right because
I've weakened a subscription to that
topic
therefore their even though I'm not part
of this table - that will send me
metadata right so this is where the yes
these two network started making sense
right now
a prune control message dissolves a
two-way message a two-way stable stable
mesh stable between two peers and this
could be in this can be sentence one to
the response or rather we want to reject
or if I'm done regulating my my degrees
now one key another key aspect to to
understand is that by itself the steps
up right now does not include a
mechanism for you this country right so
if we want to whenever we want to
subscribe to a topic we need to find a
set of peers that have already subscribe
to that time and this and the
specification is called ambient appear
in this country right we rely on
computers every mechanism which is
present in the audience right this is
gonna change as we go forward where the
start be OPR to add active appeared in
Discovery so that whatever you the
application subscribes through a topic
there will be a pops up will actually to
become of peers and that topic for them
and for the application and we also
found an introducing something that we
call on connectors have various terms
let's talk about message flow when a
peer wants to send a message as a
message source they just push the
message to a walls table to 12 peers in
the rationalization area right
superstone now peers that received that
message are actually message from so
whenever they received that message that
will forward that message to all other
peers that are part of that much of that
of that view of the network right and we
follow here is to interpret silence
because we don't want to send the
message back and forward back
now we can also publish messages on
topics that we don't subscribe to right
and this is interesting for things time
for example at Vienna as the
interplanetary naming service would want
to send updates from whatever in name
they're mapping over name changes to a
different aperture for example right so
you don't necessarily the peer that's
watching for example the DNS server DNS
name server somewhere and then noticing
this change doesn't message simply set
the necessary need want to be subscribe
to these comments right as the
possibility of sending those messages
and when we do that we basically already
remember that we keep in track of what
appears are actually subscribe to the
network we're not for forming stable
messages for them but we keep in trouble
so if the local application tries to
send that message we will immediately
form what we call the final group from
those years that we know already
subscribe to that oh right and this is
gonna be an ephemeral nation so once we
form that group
we push out the message those peers we
know are already part of the mansion
therefore they will forward that message
internally because the message has
arrived inside the mesh for the topic
and after two minutes we just dis
it was just an informal sense so we
don't want to keep it at all for too
long because that would be everything to
stay that way we also have a concept of
topic validators and perception nature
is so we can
so when appear when an application sends
a message sorry the application can
configure can attach predicate
validation predicates to topics right
and even 2.0 for example is using this
for validating
signatures in adults that are getting
propagated inside so when a message is
publish globally or before boarding a
message this predicate will be caught if
it succeeds then the message will
continue dissemination
otherwise it will just be dropped we
have the concept of the single a
synchronous validators for a heavy
robotic and gossip self can also be
configured to sign our messages in tune
and to strictly validate having
signatures as well now how do we
regulate the mesh this a concept called
heavy quote by default happens every
second business very popular concept and
distributed systems and this basically
performs mesh and gossip meats and its
functions right there's three things
that happen we regulate the mesh degree
for every topic we met gossip and we
slide the message functions so this is
kind of like how degree recognition
works as already explained it if we're
below a certain threshold we will try to
add peers to the mesh for that matter
forum so for that particular topic if
we're above a certain threshold then we
will try what we'll cream here is from
that topic to try to keep the degree at
a stable constant to be able to do this
we keep a bunch of local states and
topics that I subscribe to topics that I
recently sent a message to this is the
fountain rumors appears that we
currently connected to and what kind of
links we maintain
fluorinated message detailing or a fan
of peers and what messages were you
recently seen why is this important
this is this last bit because we send
both so this is part of the gossip
content so for every single year that we
know is interested in a particular topic
that is not part of our stable mesh we
will collaborate in the network bye bye
gossiping gossiping what messages we've
seen in a particular time so we don't
send the message to them but what we do
is with every tick and every heartbeat
we will when the message we keep in your
message cache and we keep it under like
a sliding window our messages that we've
seen in the last 30 seconds 2 minutes
this is configurable we will send them a
list of all the message of the IDS of
the messages that we've seen right and
this usually happens as well we try to
make it back this because we want to we
don't want to create excess excess
overhead in the network that's and then
that's meant to date around every time
that we see a message so we try to make
it back if we have a message to send
them any way than we try to think it
back these control messages now if a
peer spots a message that they haven't
seen right they can ask for that message
they can pull that message by sending an
I want control message so in that I want
a control message that would be listing
the messages that they want us to push
to them right why is this important this
is important because it allows us to
take routing decision so I'm at in the
case where you have a stable mission
right you're in the topic you have a
bunch of peers that are sending you
messages and another of the year which
is gossiping with you tells you about a
message that you have
yeah this means that your that that
deliverability to ability is not 100%
right because in your in your mesh you
haven't received that message yet right
so you can use that as a sign as a hint
to try to repair that pathology right by
pruning peers that you believe are
unhealthy and grafting with peers would
appear that sends a message who probably
has a better route to the restaurant
manage their memory okay you have one
baby okay fantastic
I think um one's done so so yeah this is
kind of like a bit of a schematic view
of why oh by the way all of this all of
these images and so on you can find them
on the web so don't worry I just read
you are there now I just want to talk
quickly about what's next for a gossip
zone first of all we want to formalize
concepts of into an academic paper we
want to harden the reference
implementation and go to the rep is to
enable plugable Rogers this is for us
very very important for the mission of
gossips of itself and for organizing
blossom time into an academic paper is
gonna be also a robust step in allowing
other downstream projects to actually
build on top of that and then I say and
to get all this work through peer review
because it's essentially a new
contribution to
we want to enter great discovery
mechanisms and there are various
benchmarking efforts underway underway
important aspect is said gossip site is
not the final destination and gossip
itself is a step along the way really
the key aspect here is and what we want
to get to is a protocol that we call it
X which is a combination of Gumtree high
Parkview for membership maintenance and
low cost for the personality awareness
machine and so on and this was a Daoist
accession to model the kind of behavior
that are described so if we so with
episode we'll be able to build dynamic
dissemination tricks that are mutating
and that are adapting to the conditions
of the network by keeping track of the
peers that are better connected to the
network and as one of those peers that
happen to present mythology's rivers
right so whenever we find messages that
we haven't seen that article said that I
mean coming also from another peer we
can rebalance the tree and make sure
that we always have a good connection to
them right we want to so this is what we
also call this healthy little mesh we
want to introduce adaptive cash flows
and things like collaborative and
passive
Bishop you so something that's that's
key here is the ability to send
advisories over the network of what
peers are available at which time so
ideally you want to use a ambience
discovery mechanism like the teach D for
example to find a seat a limited number
of peers accesses to bind to that topic
and once you have those those few peers
which are probably overloaded because
they're being advertised right if you
want to inquire them for other members
that they know about information you
want you want to create gossip not just
about the messages that are available
but about what peers are present at that
in that time right so this is something
that is going to be that's going to be
that's coming down the pike it's also
has some notions of appearance also as I
said all of these time cards and these
schematics which I previewed eyes and
very beautiful they are available under
the docs website so make sure that you
check out the new snazzy docs for gossip
stuff which we just released one week
ago just in time for that
[Applause]