alright shall we begin
let's do it so my name is Juan one of
the people behind ipfs and today I'm
gonna talk to you about data and data
structures so the talk is structured
this way I'll give you a brief
introduction to ipfs actually how many
people here know already what I think if
sis Wow awesome that's a great
distribution so that'll go really quick
we'll cover the IP LD data model and
then we'll talk about use cases and
examples of ipfs and data structures and
so on
alright so ipfs another name for it is
the distributed web or the permanent web
or the merkel web in honor of ralph
merkle and in a sense it's this kind of
evolution of how the NRI on the web were
structured and I think everyone here
both coming from blockchain history 2
through aetherium and a whole bunch of
other interesting technologies that are
happening right now
sort of see what's going on or what has
been going on for the last three years
as this this total reinvention of how to
do distributed computing and distributed
networks and in general and so this is
this of course layers on top of the old
internet and so on but what's really
happening is that we're taking the web
applications of the you know web 1.0 or
web 2.0 world and we are separating them
from their physical location in servers
and so on and distributed distributing
them across different hosts so ipfs is a
protocol to upgrade the web because the
web has some problems I won't go over
these there it's I think most of you are
familiar with them it's an efficient
doesn't work offline links break central
points of failure and so on and the gist
is to try and creates what
the web look like if you took a Merkel
tree or a Merkel chain and put it
underneath the web itself so what would
happen if you gained immutability for
distributing content on the web I you
can think of data structures like gits
and the amazing power of distributed
computing that you get out of that or
just version control the tribute version
control you can think about aetherium
and the amazing powers of contracts and
running programs you know the tribute of
sense and of course before the etherium
bitcoin and so on all of these are
Merkel trees and what ipfs really is is
this massive Merkel forest that's kind
of like the the intuitive description of
IPF assets saying let's take let's write
one transport protocol to layer
underneath the web and other protocols
as well and build this distributed data
structure transport that is all Merkel
addressed cool so imagine being able to
link across different aetherium chains
and being able to link across Bitcoin
and aetherium and being able to link to
get repos and be able to link to
BitTorrent torrents or being able to
link directly into specific commits
inside github so you could create a
contract and say whoever you know
patches this commit or something
successfully get some money or something
and in the ipfs world which is a world
on its own there's a whole bunch of
different protocols and we have this
this view of the world that looks a
little bit like this with a whole bunch
of different sections for these
protocols
kind of on top of the on the lower level
Network and this is of course all sits
on top of IP although it doesn't need to
on top of transports on top of different
routing different ways of doing routing
we want to layer what we're calling IPL
D which is the Merkel dog or the Merkel
web as this common data structure that
we can all use for expressing our data
structures it's kind of the same idea as
the Semantic Web but instead of just
having an object definition imagine if
the object definition linked between
objects with hashes
that that's the the heart of ipfs that's
like the core what happens when instead
of of having a URL that points from one
object to another using a location what
happens when you point using hashes and
this vide has all that's been around for
a while but this talk will be mostly
about what we're sort of loosely
describing as the Internet of data or
Internet of data structures to write on
top of IP LD and so you've probably seen
something like this where the ipfs model
is you you take the original HTTP world
of mapping a dns name to an IP address
which looks kind of like this where
you're resolving through the network to
a specific host and you instead say hey
let's just take the hash of the content
itself and use that to address the files
and it looks instead of that you get
something like this where anybody in the
network that has the file can serve it
to you right this is all super intro
ipfs stuff and in a sense what's really
going on underneath the hood is that
that hash is a pointer to a directory
you can retrieve the directory because
you can check you can ask the network
for the directory people can return the
directory to you you can check the
content make sure it matches the hash so
that way you don't have to trust anyone
and within that you look at look for foo
and what that means is that inside that
object who is a link that is pointing to
another hash and you're using you use
that to resolve for the next step and so
on down right this is taking the idea of
file system with inodes but replace an
inode with a hash and of course this is
merkel linking and you get full path
reversals down down this kind of graph
so if you imagine a blockchain layered
like that you would have a massive path
going like parent parent parent Peron
Peron Peron
and someone down right but you imagine
you you'll be able to index directly
into a specific block and be able to go
down into the transactions and so on and
so the the IPS link sphere I guess is
that you first have content addressed
links where they start with slash ipfs
then you have a hash and then the rest
of that stuff and by the way those are
shortened hashes I haven't yet found a
way to compress information magically
that's what that's suppose to be I shot
256 but it's turned so it fits here and
yeah of course we have the scheme
identifier if you want to use it but we
recommend that you don't use the scheme
identifier because we the way we look at
the world the regular w3c based scheme
identifier broke compatibility with Unix
so you can't we're making everything so
you can layer it directly on top of any
politics machine so you can mount it
directly so you can you can mount ipfs
and you can fetch data and write data to
the network with standard old-style
applications you don't have to teach
them to speak over some other protocol
and on top of on top of the merkel
linking which which is sort of like a
URI and I say URI with quotes because we
do prefer people not to use the schema
identifier we'll have everything working
and resolving with just straight-up
paths on top of that we layer what we're
calling also Meszaros links this is for
David missouri's who author of Kadhim
Leah sort of stellar author of SF s way
before that and he came up with a great
idea in nessip s and one go through the
whole thing but the basic notion here is
that the way you achieve mutability in
ipfs is that you use the hashes of
public keys and use those to resolve for
pointers that have been signed with a
private key
oh it's explained better or somewhere in
the web but there's more interesting
stuff here but it's a very subtle idea
that gives you a way to achieve naming
and this is naming that it's not human
readable but it's a way to achieve
naming that's cryptographically
addressed or cryptographically mediated
instead of through some central
authority right and on top of that
mutable link then you can layer names
and so here's the old style DNA
world we work exactly with DNS so you
can add a text record into your domain
and now you can resolve through IP NS or
directly to my PFS and of course you can
have etherium names the same way right
and so you could you can imagine
resolving if your names directly into
ipfs content the exact same way
what's amazingly interesting about these
kinds of names is that you no longer
have to have any machine running
yourself you can create the content and
add it to the network you can create the
names and the key names and sign
contents and distribute to the network
and then you can register some name and
point it to the house of a public key
and now you can just disappear just add
content to the network and walk away and
if the network will continue serving
your stuff and that's you know an
asterisk there everything will continue
working right and that's something
that's very different you don't have to
maintain your own infrastructure so this
is often non-obvious so I'll kind of go
through the ifs content and replication
model this this is kind of like what
most people don't expect it here they're
like what but it makes sense to go
through so there are three important
points that bear mentioning here other
than a whole bunch of others that you
probably can assume but number one is
that we want I P FS to be able to go as
fast as the underlying network so we
need to not we need to allow users to
dial between efficiency and privacy so
as you know systems like tour and ITP
and so on that are focused around
privacy have to do a whole bunch of work
routing packets and moving them around
and you spend a bunch of bandwidth
hiding hiding the traces of people's
accesses in the network and this is how
you achieve privacy but that's not fast
right so you're trading off the speed of
transfer there to gain some privacy but
the users of ipfs are very some people
really want that use case and layer ipfs
directly over to our ITP and some users
don't care at all about that they're
using IP FS inside data centers or
they're using it in some other way and
they really want it to be as efficient
as possible right like do you
need to route through tour to go from
like your computer your laptop to your
mobile phone in the same room maybe not
or maybe you do it depends on what kind
of home you have but you know and
another point here is that ipfs only
downloads explicitly requested data you
do not download other people's stuff and
this is an important piece from a
legality perspective right so there's
been a whole bunch of problems around
peer-to-peer systems where they download
other data as part of the peer-to-peer
algorithm and the end of downloading
stuff that's illegal this prevents the
adoption of these systems in companies
because companies just cannot afford to
do that so n ipfs those are policies
that are expressed and that can be added
on top right so if you want to organize
a cluster that moves around and shares
data and so on you can do that but
that's a policy that becomes that is
layered above ipfs so there's a strong
commitment there of being able to say
look you can use ipfs and not have to
worry about other people who are other
people are adding to the network and
yeah so I think that's this is able to
follow what the user programs and so
these policies that will be we haven't
done as much work here will be doing
some interesting stuff there to be able
to express how you might trust other
keys and trust the content from other
keys and being able to give people
capabilities to add stuff to your note
so yeah on this on this only downloads
explicitly requested data so this is a
huge surprise for most people because
they're like wait what you have this
distributed file system it's supposed to
be pure appear and like why we're like
if I add content how do I back it up
right like that's what if I disappear
from the network it pulse it falls off
and the idea here is that when you add
content to the network you're explicitly
publishing something so the rest of the
network has an option either they spend
energy maintaining this online or they
don't and if they do it's because they
care about this thing so if they don't
care about what you have to say because
they just what you say it's not that
interesting then you can just pay them
to care and this is where something like
Falcone comes in so HTTP maps to ipfs
and AWS s3 maps to
Pal coined or swarm or really AWS and
Windows Azure and so on right so there's
that how you maintain stuff in the
network it's a separate part different
from how you move stuff in the network
so IP office is strictly about how you
address and move content in the network
so and this and so these protocols
around incentivized organized
distributed storage those layer on top
of IP FS and so an interesting thought
is a lot of people say well how could
you ever hope to compete with something
as big as you know Amazon and Google and
Microsoft and so on which have these
massive infrastructures and then I just
kind of point to them point this out
right so this is the hash rate of the
Bitcoin network and like the massive
spike and this is what happens when you
get a an economic model coupled with a
very easy to enter market and so
somewhere along this graph way before
the knee the hash rate of bitcoins
surpassed the equivalent of the world's
most powerful supercomputer and then the
world's 500 supercomputers put together
before the knee of the curve so that
gives you a sense of just how insanely
powerful the the hash rate of that
network is sadly all that is going to
waste so hopefully we'll get proved
mistake working correctly so we don't
have to like waste this but when you
when you think about this model and you
apply it to doing something useful like
storing content you can get an enormous
amount of storage wired up into the
network so you'll steal so mark my words
in the next year and two years you will
see enormous distributed systems with
storage popping up into the network and
they're already getting started but oh
come and so you have something that
looks like this which are the Bitcoin
mines but for storage all right so now
that I've kind of made a claim about the
future let's have a dose of like web
reality today to like
cleanse our minds right so so the web
here and these are strictly kind of like
web 1.0 web 2.0 companies or web sites
in general that I've put up there
these applications kind of run your life
right so everything that you do nowadays
revolves in some way around these these
applications right so your personal
communications your work how you learn
how you get stuff how do you buy food
everything is now mediated not only
through the internet but through
specifically web applications and this
means that all of your capabilities are
tied to specific organizations
maintaining specific infrastructures
online
it's got scary right because those could
be this wouldn't disappear at any moment
in time or those could just stop wanting
to serve you or blah blah right but
these I like pointing these out
especially in this crowd because this is
kind of the competition right so if we
want to push this very different model
of computation very different model of
storage and actually upgrade how the web
works we have to work equally well at
least and ideally better than these
systems in fact not only do we have to
work equally well ideally better we must
be way more weight we have to have much
higher performance or none of these
people will adopt this stuff right and
so this is what I mean by like a dose of
reality and they're like when you think
about the data on the web today you have
like these massive infrastructure
providers that are there have huge high
performance databases and and sort of
the model is that people have these
business logic services and they have to
maintain them and this is where perhaps
these systems can come in and change the
model where the people don't actually
have to maintain infrastructure
themselves it becomes a lot cheaper if
they don't have to do that but you know
this is the competition right like this
is something it's an diagram from Amazon
itself it has like think about how many
machines here are involved in just like
one application right this is kind of
like if you if you're a legit
application this is what you do you have
like a DNS resolution machine connected
to like a CDN connected to like a bunch
of static content buckets connected to a
whole bunch of web servers that all the
scale connected to like routers which
hit your application servers and then
going all the way to the databases like
it's
madness right and this is all up all for
like optimizing really fast accesses on
the network like when I user hits your
web page they have to get the stuff like
within a second or like it so game over
right like they'll leave and they'll go
to somewhere else and so this is like
the beasts lying in the center of the
backbone like to just be able to manage
that but and we have this enormous
explosion of like databases right and so
the constraints are around like being
able to do billions of transactions per
second
handling billions of users millions or
hundreds of millions of simultaneous
users were the big ones sub 100
millisecond latency like from the
backbone to the user or ideally just
from the user to the application and
we're talking about terabytes petabytes
and exabytes scale so terabyte scales if
you're a small company exabytes scales
if you're you know basically Facebook
and so on and of course so when I hear
like blotching technology right like how
is it gonna deal with this right like
how are we what but here's the magic
right but but here's the magic right so
the thing is that it's not about
matching the capabilities today it's
about proving that those capabilities
can be achieved but complementing them
with a whole bunch of capabilities that
were in there before right so stuff like
like the idea be kind of blocking is
being able to do multi-party computation
but it's verifiable and trustless you
don't get that at all today and web 2.0
being able to do trustless
trusses Ledger's like being able to
account that things are actually
happening time stamping being able to
like run contracts that are verifiable
but like you just write the source code
out there put it up there and it just
runs on its own and you don't have to
like maintain it and nobody has to do it
being able to like write decentralized
and distributed application is things
where you write them once you sign them
drop them into the network and disappear
and the people can carry them on you
could disappear as an organization in a
company and people could continue to use
this application it's like these are
real capabilities that give people that
you cannot take away and that's
extremely interesting that's where like
there's a completely unfair advantage
compared to the web web 2.0 right and so
we're talking about building taking data
and code and execution in a completely
different verifiable model
and of course being able to secure an
encryption in a way that the web 2.0
people can't because a lot of web 2.0
relies in advertising and so on which
means that they can't really do end to
end encryption so that's a huge
advantage right because we don't have to
deal with that we can just do end to an
encryption for users
we don't need users data at all and this
is a strong reason to be able to move
move in this direction cool let me check
timing because sweet so that was like
the interlude so what people know is
kind of about having these like
databases talking to each other with
data behind the scenes and what web 2.0
is about taking the the logic and the
data itself and putting those into the
network and allowing those to
communicate and making the axle hardware
secondary alright that was the intern
okay okay have to rush through this
stuff
five minutes but we'll see alright so I
feel D that this is the heart of ipfs
it's a common format for hash chain or
distribute data structures based on
Merkle trees and so what I showed before
with these files these aren't actually
files they are so when you add something
to a professor it kind of looks like
this you your add all this stuff well
these aren't files these are actually
nodes in a graph and these nodes in a
graph there there is stored in that in a
format that we are calling IPL D that is
basically like something like JSON it's
not exactly JSON because it's it's
serialize to be fast but it's it's the
whole point that you have a very
flexible data model so you can take
entire api's of today and just dump them
straight into ipfs so you can take some
data API that you have and just straight
up import all of that into our BFS and
like merkel is the whole thing that it's
not perfect and you have to like work
hard to make it really a good distribute
application but that's kind of like step
one
and so we see us as new thin way so the
internet the internet of data structures
and it gives you like Merkel links that
are secure and immutable it gives you
merkel paths meaning you can resolve
through the through the graph
they're sort of universal and there
kind of your URLs as we said before
though it's not location-based right so
like it doesn't it I mentioned here URL
because most people are familiar with
that term but it's not actually
location-based and in terms of
serialization you can do anything you
want you can use JSON protocol XML RLP
or whatever whatever it is that you use
you can use a serialization format
underneath the hood there is a canonical
way of describing which serialization
thing to use so that everyone has it's
the same thing and so if you had
something like this in case on where you
were you know you outlined the followers
of a user this way this is kind of like
mush right look what is at Vannevar bush
mean right could some user somewhere but
like you don't know what context that
that comes from right you have to know
that this would be Twitter or something
else just from the endpoint that you're
accessing this from but what if these
actually link to content address things
right like the resources were actually
based on hashes then you don't have to
know anything about the way you're
accessing at all because the content
itself or like tells you what it's about
right and it would I feel that you can
do llamó you can do whatever format you
want and so on so the point here is to
build something as easy to use as JSON
but when you merkel eyes it and you turn
these normal graphs you you should be
able to have the exact same architecture
of your data but where before you would
just have some strings that you then
would have to look up in a database here
you have hash lengths and so you can
follow you can resolve through the hash
line so here in the second layer i have
your going through the hash and then
slash followers meaning that you're
you're accessing into the json object so
you're like putting a JSON object and
accessing through it to get like another
piece of the data and you can like do
whatever right so you can take like the
somebody's followers those slash
followers to get pull that array out and
then go like slash zero to pull out the
first entry and now you get something
and this is resolving through merkel
links the idea there is that like you
would pull one object out and then you
would look into it pull up pull out
another link and resolve it through
through the network and everything is
integrity text so you can just dump all
of this content into the network and
anybody can serve it to you and you
don't have to worry about that you have
to worry about it in terms of privacy
but that's a whole different question
and anyway so you can model things right
you can pretty much mobile anything one
you get you get to be able to annotate
links with other data here's like what a
directory would look like you could do
like version data structures this is for
example a git commit and so on and so
these these will be completely native in
ipfs you'll be able to write all these
data structures and have them write on
this this transport right like the
forest thing that I described earlier
and so this is really how you we start
doing web 2.0 where we take all these
data structures than these these KPIs
and model our applications around
content that we were just putting into
the network and addressing through
content addressing and key addressing
and so in a sense when you think about
model in your application and you think
about like the data model right like oh
do I have users that they follow each
other like what are all these
relationships you can take those direct
objects and put them straight into ipfs
like you don't have to go to the Twitter
data base at all you can just add them
straight to my PFS and what you gain is
that when an application pulls those in
if there's nothing that has changed
nothing has to be updated so you beat
the speed of light because if it's if I
if I'm using an application here and I
request some set of objects and I pulled
them down through the network and I have
them locally and I have to fetch them
again normally HTTP would say oh you
know like go and like talk to the
network and see if it has changed but
would ipfs we can be like well no I have
the hash I have the content it sits here
already we don't have to go to the
network and of course you can do this
with like all kinds of like legal
records and so on like admit legal
records is a great example of like why
people want to use block chains kind of
like in the in the real world it's it's
to like do careful time stamping of
legal documents right so you can do all
this and so in reality we're building
not exactly the web but it's really like
more like a strong mesh that is secured
right like you can think of it it is
like a secure web or the links can't
break because they're their content
addressed so you don't have to maintain
infrastructure and worry that like your
link now is slightly different and so
everything will fall apart you just
create content you
get securely and you just release it
into the network and so I mentioned that
that I peeled evening is suspect in
progress some of the interesting stuff
that we're doing is we are coming with a
coming up with a right way of doing
traversals through the graph so being
able to express descending down a path
in the graph things like being able to
address one level deep or like
everything underneath it or being able
to like look into something
but then something more interesting like
being able to so this last expression
would be able to in in a huge graph that
represents a movie being able being able
to walk down the first path and
retrieving the first set of blocks in so
that with that you are able to just pull
out the first few things that have the
data and then start showing the user
like the like beginning of a movie right
so being able to express these things
this way it's extremely useful for
replication protocols it's extremely
useful to be able to like point stuff
out in contracts and so on so this is
kind of an interesting thing another one
here is where you would like embed
replicas like yeah but one expression is
that another expression so if you if you
like this kind of language and stuff
come talk to make is getting those
languages right as important cool so
last section use cases and examples so
this might be the most interesting to
you
so I BFS a lot of people are like okay
well what is IP passed for what do you
use it with like what is this thing and
they want me to tell you like the
specific set of users that like are only
gonna use it for one thing and in the
thing about ipfs is that it's a Transfer
Protocol meant to like help the web
right and said like what is the web used
for everything and then people just say
like oh that's you haven't thought about
this and I'm like no I'm serious like
you can use this for everything let me
let me list out the reasons right look
you can use it to do peer-to-peer
content delivery ready to think about a
CDN but like a distributed CDN the
superdude content distribution network
you could do like a decentralized links
things where where no authority
determines the link you just add content
on the content is now part of the
network where we've talked about before
you can take applications entire
applications and define them with an IP
FS then give people a reference to that
application that's the link they
download it they run the code and they
just do everything on there
side so they it's not only the
distribution of the application it's
also the definition of the application
in an example here it's like people have
written video players that go over I
confess people have taken entire like
legal codes people have made like imager
salad clones people are able to do like
three models of stuff and be able to
share them with ipfs because these are
data intensive kinds of applications
people are this is not this is the only
one in this set that it's not an actual
IP Office application this is a
Minecraft but people are taking
minecraft and making it so that there's
links between worlds so you're walking
through a Minecraft world and then you
find a link and you watch this on
something somewhere else and all of that
is served over ipfs people are throwing
documents blogs and so on
you can do containers on VMs definition
and distribution of these you can think
of these as like merkel containers where
like the entire application and like the
graph is defined as a proper Merkel
Merkel graph we should have been the
case from the beginning but anyway you
can do whole package managers and
distribute of them this way so we're
already working on doing this to several
of these package managers we're taking
all of the code or all the actual
compiled binaries and adding them to
ipfs usually first of all you get a huge
space savings because tons of stuff is
duplicated but then you also get get to
distribute them faster because if
machines in the same data center have it
then things move faster key addressing
archiving backup everything people are
like archiving all that stuff all these
cut a whole bunch of the data here you
can archive what ipfs and there's
efforts on this now when it comes to the
etherium and this is like the last line
I promise when it comes to theorem and
by the way the logos fit really nicely
together it's fine too
so so you can use it for alkane data
right so you can the very first thing
that you can think of is like anything
that just shouldn't be on the blockchain
because it's too big
put a put it on ipfs take a link and
then serve that link with your own note
but that means that anybody else can
serve it as well and it means that you
can disappear and the content doesn't
disappear right so you know big stuff
you can use them again to define and
distribute these to distribute apps
right so these these centralized
distribute applications you can write
them put them on a fast put all the
assets there people are already doing
this and so on so you have the code and
the assets and everything but also the
users data this is an example of one of
them this is ours it's our web UI for an
IP FS node all of the code for this is
when ipfs you pull it out you run it and
it's completely distributed so like all
it's a very cool experience when you do
is me like whoa not touching what server
ever like it just my browser does it
here swamped ran an example of like a
like a B style a four times outboard
done with Ikea pheasant etherium so you
actually pay ether to be able to post so
you could do the trunk base for just
formless pointing out that and you can
do the two interesting things here and
sort of London stuck this way that the
most interesting things where where i p
FS could be useful for aetherium and i
worth investigating carefully is
representing the entire merkel patricia
3 state s ipfs notes so that you are
able to directly index into the state
tree and pull things out and and use a
bunch of ipfs nodes that may not speak
aetherium yet but they speak ipfs you
can then start addressing
etherium state with them and use them to
move around the theorem state it's just
really cool and you can do that to the
blockchain itself you can take the
entire definition of the blockchain and
just like imported into ipfs and then
distribute all the content this way and
so the this is a blog post from chris
that talks a little bit about this very
cool so check it out all right so that's
Ivan cousin did it Dan instructors thank
you