[Applause]
and you a new a new design cranberry as
anyone were you joined the first
expression here okay welcome is
homerooms I'd like to think of it as the
decompression space in case of not going
to get a wink um so I talked about
psychology of UX and reduction I'm a
psychologist not the couch type but I
study psychology and interaction design
and when I was doing that I learned a
lot about models like frameworks of how
like what drives people to adopt new
technologies and in preparing for this
talk like over the last weeks have been
trying to apply that to our space see
how does it translate all these
frameworks
[Music]
yes so that's what I want to talk to you
about but first I want to talk to you
about porn and why though why that's
interesting will become a bit more clear
I hope so later on this magazine Time
magazine in 1992 declared hybrid porn
specifically as one of the bigger
inventions of the past millennium
doesn't seem like view was big new in
agriculture so it was a big do for food
around the world but it was also a big
deal for understanding what makes people
adopt new technologies website note but
what's interesting this is 1992 a
special politician a lot of the content
is sponsored by IBM there's a lot of
talk about Lee networks computers
there's no words on predictions of like
to Internet as we know it now and all
the services that we use on that even
though that like this was 1992 so we
would expect that to be kind of
ubiquitous knowledge already with me so
the story about porn they think that
some days here to check
hormones in Fenton - porn specifically
not important Kokoro was invented in
1928 by scientists of Iowa State
University and then years past it was
released to farmers and a lot of
benefits to speak of sort of production
rate could go up by 20 percent
farmers have moved to like mechanical
ways of production because that the way
it went through that life changed like
this decider to equal height of the stem
so it made it easier to cultivate it was
also much more resistant to drop so
there were a lot of functional benefits
that would make progress be like hey
that's all like switch to hybrid corn
right and then next to that there was an
alliance of partners that also adopted
this technology from the University I
started pushing it at broke so there was
a lot going for it it's just beautiful
elevating forth one of the downsides
because I mean obviously
there were downsize was that farmers
would need to buy their seats every year
so they needed to change something in
their behavior compared to what they
were accustomed to and still 13 years
later there was a team of sociologists
that thought oh well this Gordon
business seems to be quite interesting
there's a lot of changes happening there
and so they they they wanted to figure
out like why what made some farmers
decide to switch to my bit corns others
not Yin so it was kind of big you in
that state at the time so they anis like
they would say send out a survey to
farmers in the state asking like why and
when did you switch to this like new
type of cultivation and what they found
was that when in five years ten percent
had switched to this new type of
production the three years after then
for 40 percent more people refurbished
switched it at Neal Tiemann production
no excel yet but they plotted the data
and then they found its which if you if
you looked up and like if you ever do
goals like technology interruption you
basically always find this s-curve okay
there's a like a first 10% and then at
some point that kind of grows a little
bit more and then there's this switching
like this critical point that we call
now there's critical Moss and this is
what we all strive for it because at
this critical mass we'll get there
that's what mass adoption will happen
and then this is a guy who kind of
summarized that research and followed up
by doing loads of research after in
other fields to other stands how does
this translate is this s-curve something
that occurs in every new technology that
we introduce his conclusion was yes like
he actually grew up that s-curve with
some variations of different
technologies but interesting motivation
for him to start looking into that was
that he always saw is that as a farmer
kind of waste to the last minutes when
everyone was starting to use something
new and they said like Uncle Eddie
another farmer would already use
something new his dad would wait until
he's also the guy to like kind of coined
the term late adopter early adopter
after investigating that based on his
family story
[Music]
what he talks about is not necessarily
adoption like the whole concept of
adoption is fairly new what he talks
about when he describes like studying
new innovations is a diffusion of
innovation which like in his
descriptions is slightly different he
actually says diffusion as as a paradigm
originates a bit more from anthropology
because anthropology's anthropologists
look at things from the perspective of
humans and they tend to take on that
perspective to then blame a system or
like credit a system when something
something good or bad happens but always
from the perspective of users rather
than systems so Everett Rogers defines
diffusion as a process by which an
innovation which is one point is
communicated which is a second over time
time is a very critical element
according to him and among members of a
social system I started the things like
what are then the difference between
that like type of diffusion and what we
now see as adoption and what we like
talk about it so very often of reaching
crypto mass adoption some differences if
you look at the different models out
there are that diffusion usually refers
more to as
as a whole whereas adoption is work
related to the decision of one specific
individual diffusion relates to
technology infrastructure
whereas adoption usually relates to a
product or service something built on
top of them in terms of what it does but
functionally the future relates to
something being like supportive not
delivering the actual value instantly
whereas adoption is interacting there's
something that brings you value you for
some some effort into it interactive an
infusion tends to take longer so and
this is like study based on a range of
different technologies being introduced
and it takes about 20 to 30 years
whereas adoption if you think about like
although after their web services they
take between five and ten years hybrid
court was really rapid adoption at the
time for a new technology taking around
13 years I believe and also there's just
a very different stream of research into
both starting with Everett Rogers but
then there's this other guy Aero man who
started to investigate an adoption more
closely
so next to porn an interesting example
is the computer mouse which was is
commonly credited as being invented in
1973 actually didn't have been around a
little bit longer and it was a really
nice post I may have included the link
I'm not sure by Boxton great guy if you
haven't heard of him
look him up he has a wealth of
information about how to new technology
girl he kind of describes the history of
the computer mouse that I think is very
analogous to how we might be looking at
the same web 3 days and new technologies
so right so yeah the most relevant to be
invented in 73 was actually 69 and then
it's still 2 years and years before it
to actually reach household adoption so
there were commercially available
products especially by Xerox PARC but
before babies household adoption was in
84 which is pretty pretty much a long
time after the way Bill Buxton describes
that difference is that there's actually
before that cool esker stars starts
there's a long nose of like the
technology being developed where you go
kind of through like invention which by
the point in time a different people
there's like a refinement and
augmentation phase and there is like the
product citation where there's like
actual stuff built on top of a
technology in my mind brickwork higher
like in your firemen and augmentation
like we're not really there yet
and let alone that we are at that like
place where you can start to think of it
as an s-curve why are you the analogy to
a computer mouse to be is interesting is
because the computer mouse needed an
ecosystem and this is what made it last
or take quite long for a technology to
be adopted so it needs this place to use
a computer mouse it needed programs to
effectively control a graphical user
interface and I need a lot of
manufacturing around it that took so
long
[Music]
they're easily so looking at these
pillars or variables I would say that
Everett Rogers describes that are
required for diffusion of innovation to
happen is like the innovation itself it
needs to be new and interesting to
people that needs to actually add some
some value and also needs communication
channels it needs a network of flame
merchants are people that can sort of
sell the technology like the way the
farmers have an alliance that would kind
of folks were a good forum to be more
effective and their efforts way of
working so its agents is not described
in it also needs time it needs time for
people to go through their phases of
decision making so they need to like
gather knowledge of august stuff works
they need to have time to decide if it's
something for them they need
confirmation then when they start using
it it actually works for them so there's
all these stages that people need to go
through first and then there's a social
system where it doesn't make much sense
for me to have crypto if I can't use it
anywhere else so it yeah it definitely
needs to have support from people in in
a closed environment so that's let's say
a diffusion of innovation but what we
didn't talk about adoption was kind of
grew later what can we learn from that
work
to us as well there is several models
that nowadays are referred to as
adoption models but actually in 30 days
they are called technology acceptance
models mostly because we didn't adopt
technology you've accepted it because it
was kind of enforced or force upon you
by an employer these are I mean in
themselves if you're like models that
might be interesting now but there's a
bunch of them what they have in common
is basically these two factors something
needs to be useful or at least perceived
as useful and it needs to be easy to use
and especially the first one it needs to
be useful at least from my perspective
we are rare I'm sending I work for cenis
which is very much infrastructure
oriented useful is something that
doesn't necessarily resonate it's much
easier for us to talk about the
usability of the applications that we're
developing at this point so we could we
can sort of add those dos to the model
and then there's obviously a lot of
knowledge about UX which is also
relatively new if you can look at the
process of how like UX came today
initially there was bit more of a a
thing called human engineering when Bell
Labs wanted to design the rotary phone
they were kind of struggling to decide
who the numbers go on the device itself
or own
and they hired someone to test that
that's it like yeah yeah input from
people they wanted to design it such
that it would fit humans and they call
it human engineering then the personal
computer came and the industry started
talking about human-computer interaction
but to me that and this is interesting
because it's talking about the computer
to system as a whole at that point in
time but if you look at I know what
happened to the English too after that
when web technology started to come up
but the emphasis switched more to human
interface design so moved away from the
system and kind of moved up to the
surface up like this is only what you
interact with and that's what we'll try
and optimize and then even later even
though like the term started to be used
in 93 if you look at like the actual
definitions you the experience happens
and then kind of only became popularized
when services became yeah more more
common on the Internet
yeah and this is so like looking at the
definition of user experience how that
kind of shifted over time there there
was this concept already like context of
use from these applications right will
something work in a particular factory
within the time lines there was kind of
the extent of the usability then in 2010
there was a sway chances only 2010 was
not that long ago and that user
experience was added as in the person's
perceptions and responses that result
from the use and Torrance's videos of
the system product or service
these are standing there staring they're
not the most fun reads but it's
interesting to see what I changed over
time by the way it kind of looks like
context of use is no longer important
it was just moved to another position in
the document so very much there still
and then another thing that happened in
2010 was that usability kind of got this
sight no data to it whatever was like
another emphases of specified users the
goals the context of use referred to a
particular combination of users goals
and contexts of use for which usability
is being considered sorry in this like
reference to context
all right so last year and I'm sorry to
say that this was like already last year
with some folks of status we went to
South Korea for a field research trip
where this context became incredibly
clear and also it became incredibly
clear why we needed to be in the country
to understand the context what we found
was I mean obviously there's there had
been a high adoption rate of crypto
currencies but what we found was that it
was especially connected to the housing
market the prices of the housing markets
the social pressure of pressure for
young people to buy a house and then
next to that the investment regulation
and how difficult it was for young
people to find other means of investment
bank crypto so this whole context kind
of played into why people started to use
cryptocurrency not like just to make an
extra buck or because it seemed
interesting there was a whole sphere or
context around it and so looking at
another Malo
there's loads of these in psychology
this is also thanks to Don Lemon one of
the followers of UX I would say there's
this continuous reoccurrences or
something
that's my time SM but reoccurrence of
something needs to be usable but it also
needs to be useful in the first place
and that you see popping up over and
over so one thing that we've been added
like is the importance of context
because we see it coming back and all
definitions it's kind of like a side
note but I common by this guy Nielsen
that I find interesting is that over
time we kind of switched from a you
papers thing you experience something to
you experience in any way with the
introduction of service models this is
it's an interesting thing to keep in
mind because we're in a way referring
that you pay first you pay guys and only
then may you experience all right so
from some of the studies that we did
there is one more factor that I would
add to that model of what are things
that are important to at least think of
of keep in mind or checks to cover our
bases if we want something to be adopted
what we found in Iran where we thought
privacy is going to be super important
as we build a messaging application
turned out privacy was yeah it was
important but it wasn't super important
people had telegram
it was bloggers and then we made it more
trustworthy for some reason
what they've come back over and over was
this it should be coming along every
among everyone who had friends should be
on there too so the social network and
then another one was from I would say
three months later
what we did is here study in in Japan
and in this study we would ask people
like about the current messaging
applications they use that we would ask
to write either a vacant letter or a
love letter depending on like how they
felt about that current application and
yeah wouldn't kind of looking at like
what is it what does an application mean
we've got through this like enough
strategy of like write a breakup letter
or write a love letter make it really
parents we've got some things like a
glue so many context or I'd be so lonely
and kind of explanations for these
technology and this to me would be
reason to like emphasize or adds its
sorry by the colors there this concept
of human connection specifically because
it's something that we really do not
seem to take into account too much as in
any transaction I make on the blog
change so far does not allow me to like
add a notes are receiving any any value
does not allow me to say thank you
which is a pretty pretty basic thing in
any any human interaction and looking at
other applications that seems to be one
of the defining aspects so that's
like like the list of factors that I
would keep in mind like either looking
at like the wider picture of the
technology but also at the lower levels
like do we give equal time between
switching like trying out an application
before they actually have to decide that
they'll use it to be allow people to try
something without begging of their seat
rates so how do we fit into this this
concept of decision making do we have
enough communication channels or HS in
place and there's a bunch of more I try
to see it as list of triggers like
things that think about and that's all I
had except for the QR code which I think
you know a lot better what to do with
that than I do
hi everybody
it has a few more coming innings thanks
for being here
my name is Krista AG I'm a UX UI
designer in the air swap and then I'd
like to do is just tell you some stories
about user research that we conducted at
ear swab and specifically some of the
challenges of adapting and of
traditional UX research methods to these
specific users within adhering okay and
I guess what I was about I was want to
say you know this isn't really supposed
to be a prescriptive talk per se but
more so just sharing a very lean product
teams attempt to conduct meaningful
research within the space and you're not
going to do it well so I hope you can
all chat about it afterwards it's gonna
mean if you're doing stuff better than
we're so off the five years agenda we're
going to do why he's a research because
we can't be in a place like this and not
advocate we're at large we're gonna talk
about two buckets of user research and
then we're gonna focus in on one side
and talk about some specific methods
that we have dealt with and adopted all
right so why use a research um the goal
here is not just to build good products
but it's to build the right products and
when I say the right product I mean a
product that is for the end user and I
think this quote by IBM puts it well
user research focuses on understanding
user expectations behaviors and needs
and motivation
through methodical investigative
approaches insights are then used to
ensure that all product design decisions
do benefit the user and I think this is
something for this space that we need to
remember the products we're building are
for the user they're not for us as the
developers the designers they're for the
user in the end and so let's work to get
to that end so now let's talk about two
kinds of user research if you're a
designer or a researcher that's made me
a one on one kind of situation but we're
going to go through it anyway
we've got quantitative and qualitative
so when we're talking about quantitative
research this is evaluative research so
these findings will come out in
measurable data a miracle formats and
examples of this could be heat mapping
so how many times does a user clicking
on a button how many times are they
missing a field eyes tracking a/b
testing and etc we're going to be
talking about qualitative today which is
more exploratory it sometimes gets a bad
rap as sort of the soft term of the
research sciences but the whole point of
qualitative research is to observe
behaviors and attitudes the findings are
not numerical some examples here would
be interviews field studies and focus
groups this is the more sort of personal
research it's human connection and
understanding sort of core motivations
of people as opposed to just numbers so
just to give you a little bit of a
snapshot the point here is not to say
that one of these research sides is
better than the other but it's about
using them together to get the best
research you can get so at ear swamp we
use a combination of a ton of
fools and is just a few so on the
quantitative side to use a couple
products we use hot char and mixed panel
this is where we do our heat maps
recordings we track our conversion
funnels KPI I see they active users etc
these are the ones that we hear a lot
about so qualitative research on the
other side we have Kenny feature
requests or users can freeform ask us
for features telegram and discord
feedback channels so again sort of a
free-form Avenue for our users to talk
to us and then the three that we're
going to kind of live in for the rest of
our time together are surveys field
studies and user interviews so
qualitative research we need its
Imperium needs it and the reason we need
it is this kind of research is what
takes us out of our contexts and shows
us what our users really need so it
takes us out of being developers
industry insiders designers that love
aetherium and puts us in the spot of our
users to understand what they really
need for products but qualitative
research is hard in the theory m and
we're going to talk about why and I
think the crux of the thing is basically
this our users go buy wallet addresses
and not names so what I mean by this you
can imagine on a traditional platform of
I'm signing up for Pinterest I'm signing
up with my name my email address
probably my did earth
I may have even checked to subscribe to
email list while I did that and then on
top of it maybe they have a social media
presence that I'm following with an
identity attached to it so in normal web
to consumer web there is identity that
is fundamentally connected to the
product so when you're looking at doing
recruitment or conducting research
they're pretty easy avenues to get to
those people to find them to know who's
using it for us on the other hand we
have wallet addresses and for our users
this is one of the biggest value
propositions of the entire thing their
anonymity their security is huge and we
care about that but how do we conduct
research and that's the case and it
really it goes into everything who even
are our users where can we find them and
then if we can figure out who they are
if we could figure out where to find
them how can we actually get to know
them again in this true sort of human
way get to know their needs so we're
going to start with who are our users
and we're going to talk about UX surveys
so when we're talking about sort of
fundamental UX surveys in traditional
world the goal of a survey is to gather
both quantitative and qualitative
insights from a lot of users at once so
this is a research method that doesn't
cost a lot you can set it out to a huge
distribution channel subscribe a
subscription list you can send it to a
company and they do it for you you can
then extract trends and understand all
kinds of things for a lot of people the
second way this is used would be like
the screener survey for recruiting users
for future research so for us this piece
was really
we were interested in the first of
course but we want to work for users we
want to find these people I mean I
personally wanted to bring the name of
the office we're talking about that
later but so here's what we found the
distribution piece was parked and we've
done these are just two snapshots of a
couple surveys but we've done a ton so
when we were first doing survey we were
like okay let's get it to a ton of
people and so we thought maybe
Craigslist like maybe put together a
survey that would really weed out the
real crypto people the real etherion
people and we got a fair number of
responses we got like 18 responses and
we actually did bring three people into
the office which was a total show
and talk to me after if you want to hear
about it but and he was interesting they
did give us insights what we've realized
is that it wasn't quite spot-on enough
that this distribution channel it wasn't
quite representative of the kinds of
people that were using our products they
need to be more savvy so then you're
like alright this was too broad let's
make it smaller
let's not expect as many respondents but
let's expect some quality so then you
into telegram and Redden's we
distributed there you can see for this
survey we had 11 responses which you
know that's not a ton but the thing with
these responses is that they were
excellent super high quality responses
we had people reciting
paragraphs and paragraphs about their
experiences that were willing to answer
all of the questions in his grade but as
far as recruiting them for future
research they were not in of those 11
respondents three people responded with
their telegram handles none of which
responded when I reached out to them so
we realized that our users are
absolutely willing to give rich
responses to these open-ended questions
as well as close any questions in survey
form but they still want to remain
anonymous and so here you're like all
right we got something from it that's
we're not gonna get those things from
them at least not right now okay so
where are our users let's talk about
adapting UX field studies for a setting
the goal of field studies is to observe
users in their natural environment and
to situate oneself as the researcher in
user specific contexts so coming into
the etherium ecosystem I come from
working just you know with internet
stuff and consumer tech and when I think
of field studies I think of like doing a
field study for REI going into an REI
store for four hours and watching
consumers go get the sunglasses but gets
sweaters this is not like that there's
nowhere I can go to find aetherium users
to watch them do their daily things but
just because we can't do it in person
does not mean that we can't see what
people are doing so this is an example
of what I'm going to call a field study
this is a screenshot with telegram but
we were building a product for OTC
traders and so this is a trading
chatroom and what's really
here there's a little bit of gibberish
WTS hashtag axel 1900 0.1 Eve translates
to want to sell 1900 Axel at price 0.18
so what's interesting here we were
trying to figure out how to create a
really intuitive order builder wrote OTC
trades so the main inputs on this thing
are selecting a token and amount to send
and then selecting a token and amount to
receive but what was so interesting here
is that the syntax that these traders
are using is not cell 1900 axle and
receive 190 years which was represented
and a lot of the products that were
already on the market and so what we got
to do is implement a price field which
for us was a huge differentiator from
what was already out there and we've
heard from our traders very actively
that this really helps their workflow
that they're thinking about what they're
going to send and a price not just what
they're sending and receiving so this is
one example of how these are meant Apted
field studies and applied it so now
let's talk about UX interviews we want
to get to know them and traditionally
direct interviews are all about an
informal setting to really get to know
someone to get beneath what they think
they want and to get at what they really
want so you're trying to uncover user
motivations feelings needs and pain
points and it's very personal it's about
putting a face to the person using it
you know this is hard like I said we
have some people come in but again they
weren't quite representative of the
users that we really wanted to be build
upon and there are a few challenges with
the users that we're really trying to
build for which is that they are global
they're everywhere they're happy to come
into our New York office and they don't
expect we want their face to be seen so
while we may not be able to see them we
can work to create analogous research
environments over chat and through
forums so what I did was cold message
people on telegram and said hi I'm Chris
I'd love to just talk to you about OTC
and a number of people were incredibly
generous and really did talk to me and
you know I don't have their name but I
was still able to get to a point where I
started to understand who they were and
what they needed some other ways that we
have approached this so that's sort of
the individual connection piece we
actively foster community around our
products and so we do face product
releases on our most recent one we start
with a closed beta and this is paired
with a tell
just for Channel but we've had people
sign up for who then a conversation give
us active feedback we implement that
feedback we do a public beta we invite
people into another channel and so just
creating a space where we can still get
this informal sort of human feedback on
the things that we're doing so in the
ends you know we may not have a name and
I think that what I have learned is that
we don't necessarily need a name because
we'd still get to know our users through
creatively thinking about the kinds of
ways that they are interacting with the
world and the way that they're
interacting with one another so that's
pretty much it but I would I mean for
those of you in the room I don't know it
is anyone like a UX researcher or it has
conducted UX research ok yeah tell me
what you do with that stuff it's tough
but it's good thank you
oh yeah sorry go ahead
oh you might hear mother booth yeah so
um we have to remove the first I came
and I was actually out on the sidewalk
having cigarette and he came up to me
and he was clearly drunk actually drunk
and I was like oh geez and I was like
running away and he was like oh is this
air swap and I was like oh yeah this is
Harris mom and he was like oh I'm here
for the user interviews and I was like
okay come on now so that was the first
guy and other guy was probably a dark
month criminal and that there I was like
absolutely useless like I think you just
wanted to give her I mean there were
insights to be had as far as just you
know the consumer side these were
definitely kind of like edge case people
that were not exactly our people so we
were able to get something from it but
yeah that was yeah yes goo goo things
yes well I try to do with my team and
they like someone that rejected it so we
ended up
okay and for what there was to it the
way I don't think where you think about
this music so where's this Factory in
Seattle thanks
did you have a question yeah I mean you
mentioned that I was wondering about
like incentives and other things like
some of the stuff is nearly canoe to me
but you know people ranges that you know
are edges to identify the litter here so
the beach try to make
around like badgers and walnuts to like
recruit people well we did we didn't
think about in kind of things um and we
used a gift card for this particular
session and I think that was about the
one the door but at the end of the day
like the people that we want to be
talking about and we don't first want me
to rather I don't think that necessarily
luring them into the door are the people
that we're trying to talk to like you
know what we found on the surveys is
like these people want to give feedback
and we all want to contribute to making
it better it's just a matter of figuring
out how I connect to them and find them
so any doing like industry-wide studies
concerning like somebody must be there
are people team is moving about in
general
it's a design system we've done a bunch
of those and we're doing now on a
industry only or - there's a top tier
tomorrow - how are you how do you
connect with users users felt like like
a lot of a lot of product people are
looking for understand what the trends
are so it had to be you know obviously
like you didn't know companies members
in small budgets better
we are able to buy your users and don't
think times are reggae or not so I think
that what we found it
you know telegram handles are sometimes
the extent of the identification but
that then they're also of course those
users that we do have actual
relationships with that we know a name
of or at least a person name but I think
that in my hood most of them are there
telegram panels yeah maybe to see that
if you have their users that use your
platform loss to start with in human
goes yeah yeah and of course we've done
that I think that the idea because I
think this is always kind of attention
here is like do we just try to build for
people that are using it or are we
trying to get you know at least a degree
for there towards a new
or you know so I think that represents
also learning from the qualitative side
did you learn something interesting for
just about the users in general yeah
well you defined something and he
discovered something wasn't working
somehow I just I think um well what was
fun was talking some of our users and
actually sending them designs so there
are a few that were pretty close with
now that I've actually sent like active
prototypes to and they have like mocked
them up and added comments and you know
we take it with a grain of salt your
designers but it's been fun to see them
sort of actively engaged with what we're
doing right that I mean the point is yes
oh yeah I mean though I think those and
again it being evaluative it showed us
like for example on our dev science and
experience like we noticed that people
were missing they were clicking
something that wasn't suppose to be
collect and they were missing the thing
that was
three players and it's not that you know
you have to see that really improves it
so is there resistance to like things
like segments and tracking data
maybe people freaked out I don't know if
they freaked out but I think that in
general we can say like ideologically in
this space people don't want their data
to be tracked or health but I'm not
saying from personal experience I've
come up against someone you know very
soft and they can add something to death
so my previous company with a company
worked before we have a lot of users out
there and we saw a drop of this 170
percent people actually blocking things
like seconds with privacy pagers so that
meant that like we were missing like a
lot of traffic because of death and then
because the European we had the GDP our
yeah which meant we saw another about 40
percent so in the end if you look at the
total traffic we were really worried
that like our traffic was actually going
down because there's a time span of a
year it was dropped like by almost 50
percent
yeah but then actually looking at the
profits and all things that was pretty
good
so we saw a big mismatch between what
our dad was telling us yeah
and then what the reality was happening
yeah so then we
went back to service I click and we
wrote their own plugins to basically
circumvent like a lot of subtracting and
that was more so and every made a
combination of those to actually see
okay what is actually going on so that
we can make better decisions against
because the first was really scary to
see all these drops was these drugs were
only from people blocking it or not
allowing us to practice so yeah so I
think that was for us a very good
insight of real life of a game think
differently that's one and your own
thing is that from a statistical point
of view it's very difficult if you have
like a small set of data to actually
make some cadet you can drop intuitions
and I see a lot of smaller startups
actually really try to do some mythic
things like this which is really not
that useful on bet you have like a lot
of traffic so yeah so then fix that
watch our recording sessions and look
into this that's very useful but also
realized this platform our culture is
also doing the same so it's not
recording every session but it's trying
to like make me feel like okay this is
something significant in there
[Music]
[Applause]