[Music]
so hello and welcome to building auger
lessons learned in this talk I'd like to
tell you a little bit about myself and
go quickly into what auger is and how it
might architectural II differ from
depths you might be more used to I'd
also like to talk about what went loft
wrong along the way during the build and
deployment process we had issues with
test at deployment we had issues with
our integration with 0x and we've had
issues with the etherium JSON RPC
interface and finally I'd like to talk
about some of the new exciting things
we're going to be releasing in Agra v2
my name is Scott Bigelow and I got
involved with aetherium in early 2017
when I wanted to understand what a smart
contract was a little better I had some
previous experience with Bitcoin back in
2013 but I just didn't get it and I feel
like the best way to understand was to
pick up a tutorial and you know write
the standard er C 20 contract but to do
so
you needed to get on a test network in
the network that I chose at the time was
kovin this is a parity only proof of
authority Network proof of authority
means that there's a small set of
validators that are eligible to mine
this network so if you want to come by
any Co vineeth you need to go to a
faucet and one of the popular faucets at
the time was this Gator channel where
you can come in type in your Kovan eath
interface beneath address and somebody
will come by maybe in four hours maybe
in 36 and will send you some Co vineeth
now I come from a background of software
development so my first thought was well
we should automate this and so I wrote
this kovin Gator BOTS
that's monitored for these things and
delivered if to the people who requested
them in real time and this was really my
first interface with the etherium
ecosystem and I thought it was great and
I had even gotten to smart contracts yet
this bot has been running for just over
a year now and has faucet adore most
30,000 times so if you've gotten Co
vineeth in this Gator channel you got it
from me
but this really led me to wanting to get
more involved in the etherium ecosystem
and I started work on auger later that
year in 2017 auger is a etherium based
decentralized prediction market protocol
this is a system where any user can come
and create a market which is really just
a question that has some set of outcomes
and an expiration date some future date
where this is going to be resolved
before that resolution users can stake
eath on what they believe that correct
outcome to be and after resolution the
market creator is responsible to come
back and inform the blockchain what the
real world outcome was but of course
there's no reason to trust this market
crater to give you the correct answer
and that's where augur really comes in
auger has created a system of economic
incentives such that rep holders the
holders of the auger rep token are
incentivized to come and dispute answers
that are incorrect in this dispute
process can last a long time
this depute dispute process can go back
and forth and back and forth and if we
ever notice this if if the system ever
notices that there is a major problem
the system ends up forking and every
holder of rep needs to make a decision
which side of this fork do I want to be
on and we believe that you that holders
of rep token are going to be
incentivized to want to be on the true
fork the fork that represents the true
outcome to accomplish this took quite a
bit of code augur is nearly 5000 lines
of solidity code and the base deploy is
41 contracts and architectural a it's a
little different to if you've used the
maker interface before this is a this is
another DAP which is a series of smart
contracts and a UI a static UI that can
make dynamic calls into a theorem and
run an eighth call this is a simple
function call is just a read call to ask
questions I might ask things like hey
how much MKR does this account to have
you know how much died is this account
half now these questions come from the
current states of the etherium network
all this data is stored
that you know the storage of the
contract space and there's a little bit
of logs here they can tell you some
things that happened in history but
mostly it is powered by asking questions
about right now
so our ger does this a little
differently the UI does not communicate
directly with the etherium node the UI
has an intermediary which processes logs
to reconstitute the state of augur and
because augur is a decentralized system
this isn't a server sitting in iraq this
is your box and if you've ever seen this
window before that's why augur is
presented as an executable because this
service that powers the UI is critical
for its functionality and so why do we
do this why do we take the etherium
database and turn it into another
database so we get a few features here
we get relational data the augur data
model is fairly complex we have lots of
different things that relate to each
other and if each one of these needed to
be some Eve call that like bled to some
other earth call this could be dozens
and dozens of queries just to look up a
single page by putting this into an
indexed relational database we can serve
that with a simple single joint query we
also get data locality no matter what
your latency is to that aetherium node
you are serving this data local it's all
ready for you and users are the ones who
are contributing the titles for these
markets and outcomes and whenever users
are entering text you're going to need a
good way to find that text something
that's not exactly accurate so we have
string matching and search patterns that
that help with this discovery but
there's another reason I mentioned that
there were 41 based contracts but that's
just where it starts when you create a
new market that is a new contract plus
you're gonna have two to eight outcome
token contracts you provided in an
initial report that is a contract when
you provide a dispute that is a new
contract has a week past that is a
contract it's called a fee window and so
this chart from before where we talked
about the 41 contracts that make up
augur is dwarfed by the number of
contracts that have been deployed since
its launch which sits at over 5,000
auger is also a fairly old project we're
currently sitting at blocked 6.6 million
the release took place around 5.9 by
involvement 4.4 but if you want to go
back to that very first auger contract
deployment the first time that somebody
said I would like to make something like
auger and deploy it to the network need
to go before block zero this system was
developed well before the etherium main
net block zero was launched on a test
network that I think at the time was
called POC nine but this is actually
before solidity solidity zero point one
point zero was released here and so
these contracts were written in serpent
a different smart contract language that
compiles down to the EVM code so when
the rep token launched around block 2.4
million this token was also written in
serpent and work got underway to develop
the augur contracts in this other
language serpent unfortunately around
block 4.1 million we asked Zeb Glenn to
take a look and that how's us to a house
serpent going we had them both look at
our smart contracts and the serpent
compiler and what they discovered was
that the circ of a serpent compiler had
a critical vulnerability in it wasn't an
issue with the smart contract code but
even the rep token itself that had been
deployed was vulnerable to a VM code
that came out of the compiler and so the
rep token had to get redeployed at this
point it was really written as a
solidity contract and all of the augur
contracts that were already ready for
audits had to get rewritten into
solidity because it was clear that the
market had really gone this way and this
research really justified that effort so
your smart contract is no more secure
than your compiler which is why we were
proud to release yesterday a joint
effort between auger and the etherium
foundation for a solidity compiler audit
and they found some great stuff they had
over 12 with they had 12 issues that
they consider to be high or critical and
these issues are as some of them have
been addressed some of them will be
addressed but luckily none of these
created EVM code
left our contracts vulnerable but we
feel a lot better about the platform
that we are leaving behind for other
developers to build upon one of the
issues we ran sue was a test net gas
limit issue so I mentioned we have 41
contracts and those take 50 million gas
to deploy but one of these contracts the
largest of them takes 5.8 million which
isn't a problem for maintenance may net
has a block gas limit of eight million
robson however which is one of our main
test networks for developments at the
time had a block gas limit of 4.7
million it could not contain that one
transaction to deploy the largest
contract which was critical for our
system here's a chart of the block gas
limit over the last two years you can
see for large periods of time blue main
net could host large contracts but there
was no test network where you could
actually deploy these and try them out
and share them with the world
so while Rob's turn in main net run the
same code there's a critical difference
Robson isn't used as much and miners
look to how full of block is to slightly
increase or slightly decrease the block
gas limit for the next block it's up to
them they don't have to use this as
their their criteria but it's what they
generally do so what's a project to do
when they want to ship to Robson well
just fill up the blocks
let's get these miners to think that
these blocks are really full so they'll
start voting this limit up and so that's
what we did we wrote this up with a
smart contract whose only intention was
to waste all gas that came into the
transaction now knowing what we know now
we could have done this a little better
if you assert false you can actually hit
an EVM up code that doesn't waste all
this time on these full nodes but still
effectively wastes gas and gives you
this block this block that is full of
failing transactions but it's full and
you know I started in test nets and I
really wanted to help people with the
faucet that I built and so this kind of
weighed pretty heavily on me I was
wondering like you know why are we
destroying you know the the usability of
this for days at a time for people who
want to use
Austin no I'm just kidding somebody kept
mining empty blocks and so they kept
voting that gas limit down so we had to
do it ourselves
we spun up our own miners with a target
gas limit that was already set to higher
so even though these blocks were mining
were mostly empty we kept voting that
gas lit up we became more than half of
the network and were able to get that
gas limit up but it really got us
involved with how are we developing our
software is test that really the right
place to be doing all of this work here
is a repository where we have created
guess and parity in various incarnations
so this is you know connecting to Rob
Stan having a local instant seal having
a local mining thing that mines blocks
about every second and this is a public
repository where anybody can come and
create these docker files and images to
have a consistent environment to
redeploy to so you have this local guess
that you can spin up on your local
system that's great and then you can put
contracts into it and if anything goes
wrong you just tear it down bring a new
one up and redeploy but for us this
redeployment actual takes a long time
takes about 20 to 30 minutes to do an
augur deploy with all the seed data
that's important for our development
environments and so we took it one step
further
we wrapped after the contract deployment
into a new docker image and this became
the base with which and all our entire
development environments were call her
develop errs use these images now and we
version them so whenever we have a new
set of contracts to test out we say hey
everybody - 12 is up and all developers
pulled down - 12 and it's exactly the
same for everybody the same transaction
hashes the same addresses the same block
timestamps so we know that we're all
building on something that's the same as
if it was up on Rob's turn on something
public but it's local and it has one
second blocked block times and it has a
block cast limit of 8 million we had
complete control of how this thing
behaves there are new tools to do some
of the same things this is ganache this
wasn't released by the time that we were
working on this but I think we would
look into something along these lines
this is an excellent local test RPC but
no matter what you're using invest in
your aetherium development environment
our velocity went way up once we got
this right
so we had an issue with 0x after
deployment so I mentioned there's this
five point eight million gasp contract
this contains the market functionality
if every time a user created a market
they had to deploy a 5.8 million gasp to
be really expensive and so we deploy a
delegator a delegator is a contract that
has its own contract states has its own
address buffers all of its functionality
somewhere else
the only problem when you use this
contract you incur about a two thousand
gas cost whenever you bridge this
delegation that's just the base call
cost of a call like that but imagine
that this isn't the market this is the
rep token contract and imagine the
caller isn't a person who's willing to
spend two thousand extra gas imagine
this is zero X and the 0 X contract and
also imagine that the 0 X contract
created a limit a hard-coded limit
inside of their contract that said a
balance of lookup should never take more
than this much gas and at the right
that's a lot of gas to do a simple
storage lookup and return that data but
not when you add in delegation and we
blew that gas limit for that one lookup
so congratulations you've deployed also
your delisted so we had some ideas about
how to solve this we could have wrapped
the rep token in a non delegated
contract in the same way that you can
wrap ether into an A or C 20 contract
there's no reason you can't wrap a near
Z 20 into another a or C 20 that isn't
delegated and is compatible but that has
a horrible user experience for the user
luckily we talked to 0x about this issue
they were able to execute on their
governance and they upgraded their 0x
exchange to a v2 that has a lot of new
features but lifts this gas limit so
that the rep token is again compatible
with the 0x exchange so we have these
interfaces we have a or C 20 and these
things are great and it's great to
follow these interfaces but it's no
substitute for testing your integrations
there's all sorts of crazy things that
these contracts do that are on top of
the IRC 20 interface and if you really
care about integrating with somebody
especially if you're a little different
test that integration out
so here's how different the rep token is
if you look on a Thirsk an the rep token
is not verified not because it isn't
verifiable but because ether scan does
not support verification of a contract
that was deployed by another contract
and that's what the that's what the rep
token is and even if it was verified
this is what it looks like is it a bunch
of assembly delegating that call to
another contract so doesn't really
instill much confidence anyway so let's
talk about json-rpc so one of the nice
things about deploying your software is
that you get a chance to get some real
user feedback now luckily we received
more actionable feedback than this we
got some github issues we had failed to
fetch parent block stuff is missing
balances are gone I can't find anything
things are shutting down syncing is a
major issue so what went wrong here I
mean so from that prior list of docker
images we built for for gas and parity
we took testing very seriously we tested
all of these things we tested in fira we
tested local and remote and you know POA
we tested the whole thing but in the end
this is what happened we had an issue
that was specific to in fira and
specific to maintenance so so what
happened here well let's talk about
Eastgate logs
our system works by when we're syncing
synchronizing the logs with the
blockchain we say please give me the log
please give me new blocks inside great I
just found a new block block 100 cool
give me the logs for this block here's
an array of logs now in fear it does
something different on maintenance in
fact they do something that I would
consider to be just a little bit crazy
in fear serves over a hundred thousand
transactions per second that's the thing
that it is different that's the thing
that I think is a little bit wild here
and to accomplish this to accomplish
this feat they need more than one server
to do it
if I need a huge quantity of service to
do it in fact if I have to guess how
many servers there have it this isn't a
public number I would guess they
probably have like well let's say - and
in order to have more than one server
you're gonna need a proxy and so let's
walk through this conversation again hey
please give me a new block
here's block 101 great can I get logs
for block 101 sorry so this system
doesn't necessarily have the same
perspective of the etherium Network as
the one that delivered you block 101 and
so the system this is this is just a
standard aetherium node thinks to itself
well what the hell is block 101 but the
problem is is it doesn't say that it
doesn't say what the hell is block 101
it says empty array so I don't think
this is inferiors fault
this is json-rpc kind of not necessarily
lying but not really telling the truth
either
so what solutions do we have for this
well we could ask for all logs and just
throw everything away and then filter
out the augur logs ourselves but know
that the presence of logs means did we
talk to a node that had that block but
we get empty blocks all the time okay
but maybe we could use like the logs
bloom filter and if it's empty we know
but it's still a lot of logs so how
about we get augur logs and crypto
kitties because we know the crypto
kiddies has lots of events all the time
so we're gonna get like a totally
adorable cat in our JSON response but
we're gonna discard it use that as our
indication that we talked to a proper
note and we seriously consider this
probably more than I'm willing to admit
but the problem is is I don't want to
wake up one day and write this comments
as a response to a data quality issue
so a solution would actually work well
what if every time you got a blog that
was empty it was like you know empty
response you remembered that and then
the next time you heard about the next
block you made a range request for 101
to 102 and then if you saw 102 you knew
that you actually did hit one of what
it's just it's getting crazy right so
what we did was there was this AIP out
there that had been out there for over a
year that said how about we query blocks
by block hash this is a more unique
identifier and more importantly how
about when it's not there you let the
person know you you don't you don't kind
of fuzz over the fact that you've never
heard of that block you deliver that to
the client so they can make the right
choice on their side if anybody heard
Jack Peterson's talked last year about
bounties I think this is one of the
critical success stories of bounties
here's an EIP that had been sitting
there for 15 months after auger
experienced these issues that a IP we
got you'll be able to use bounties and
engaging the network to get it merged in
two weeks and implemented in GAF in
parity in six that is an excellent
response right and that that's a killer
ecosystem and so but in the meantime we
had to say oK you've got one node you're
good to go if you have two or more nodes
whether you're in fear where you're
there anybody we can't do that until the
CIP gets merged then we've got other
issues we had parity there was a parity
specific issue when people started
launching their new nodes and I didn't
understand this because remember like I
come from kovin like parodies my jam I
know this works with parity so what
happens here
how does parity getting these exact same
issues a single node let's talk about
leaf get logs when you synchronize with
the etherion blockchain it goes
something like this you start at 0 you
go to 6.6 million you're good the parity
developed something called a warp sync
where you first sync you so you pick a
points in the middle pretty close to
head and you sink that part first then
you say hey I'm good to go
and later you backfill from zero to that
warp sync start point but you think
you're synchronized when you're right
here and if you issue with get logs for
this you've got your logs back if you
issue a get logs back here you get empty
array again and if you issue a get logs
that spans this point
you get half of the logs you were
expecting and you can imagine the kind
of havoc this wrecks on a system that
was deployed right here so we made sure
we documented this and I read me we
tried to detect detect the behavior
automatically but really the solution
was we've been working with parody and
this is actually scheduled for a future
release to deliver the correct
information to volunteer more
information and we've seen some similar
issues on the gift RPC JSON RPC
interface so but eath call when you call
a function that revert Svea that eath
call you get 0x eath call when you call
a function on a missing contract either
because it will never exist or because
you're talking to a node that is not at
the block height that has that contract
deployment you get 0x let's talk what
eve call when you talk to a server that
is overloaded for a contract function
that will succeed you get 0x
and so we're left wondering should I
retry in 100 milliseconds or should I
never talk to you again because this is
an unrecoverable error so so what's the
lesson here
I think we don't quite know what the
theorems json-rpc
is currently for is this a web api or is
this a database query layer because
these have different success metrics
this needs to be 99.99% available but
it's okay to have a little bit of cash
it in there it's okay to have a little
bit of fuzzy answers this is serving now
a web UI but this needs to be a hundred
percent accurate and you need to if you
can't be confidence in the answer
deliver reliable errors to the user so
I'm proposing I believe it is currently
a Web API and to support that here's the
documentation for the JSON RPC interface
this is over a hundred pages of
documentation the word error appears
zero times and I think it is not a
database query layer because if you were
to pretend that the issues we've
experienced appeared in the my sequel
bug tracker these would be big issues
quite if querying for new data returns
success with zero rows or querying for
old data returns success with zero rows
this would be a big deal you know but I
don't I don't really fault the JSON RPC
or parity or get for these
implementations because what we built
really works we've built
things and they work great you know I
haven't had a problem using the maker
interface but if I ever did I don't know
I would just refresh and everything's
fine if I was in that you know one in a
thousand case right I got a bad get log
but when you're synchronizing the
database off of the theorem in this way
when you lose data you start building
the missing data or your synchronization
is paused because you don't know when or
if you should be retrying you know an
auger is not the only system that is
trying to treat the etherium json-rpc
like a database here's the etherium ETL
project where Google is running daily
exports of the entire etherium
blockchain and putting inside a bigquery
but this is this is a daily export and
there's lots of value we had there in
research and reporting and charting you
can do some really cool stuff but this
is a different product than an ETL
process the tracks head closely and can
serve as the API for a complicated for
an application that is more complex so I
believe that ETL is going to be the
future to building advanced applications
like this and so why am i up here
talking about some of these specific
issues want to just go file some issues
because I think that what we're talking
about here is cultural and I think if
this we need to decide if this is the
kind of thing that we want to care about
and we want to request out of out of
these api's and work towards achieving
so I think we need to care more about
reliable data access and there's lots of
projects underway that are leading this
charge in fact you know the folks
interfere we've been talking to our
pushing fql this is a a new way of
talking to the etherium blockchain and
currently it is it is resting on the
JSON RPC interface but the intention is
to put it inside of aetherium and so
this is our chance to really get it
right and enable ourselves to build
these big applications so what's what's
next what's coming up in in augur v2 so
currently all markets in augur are
denominated in ether ether is the only
token that we accept for for trading on
these order books but we have decided
not to add
died as the token that will be used here
we have decided to replace die as the
only token that will be traded on the
auger platform because we've realized
that not everyone who finds use in
aetherium necessarily wants to also be
long ether especially not for six months
or a year as long as some of these
markets can take to resolve we're also
working to improve the user experience
here downloading an application and
having it sync locally as some window
running on your system like that's not
really an optimal experience and that's
not really the way that's you know the
web is supposed to work and so we're
working to take the same architecture
and to run it inside of the browser and
use some modern browser features to
accomplish a lot of the same things that
we need but do it in a way that is
transparent to the user and doesn't
require this extra UX burden because I
think this is gonna be a powerful
pattern for larger and larger groups to
implement because if we want to take
this chart and change the scale of it
again I want to make sure whatever
project does that has has a great
platform to build on an audited
ecosystem development environments that
can support large arrays of of
engineering teams integrations with
other projects that are well tested and
not just relying on simple interfaces
and to build on a reliable data platform
thank you
[Music]
you