0:47:09 Yes.
0:47:09 So data is interesting because
It's so easy to collect
0:47:14 enormous amounts of usage data.
0:47:17 Was this feature used?
0:47:18 Yes or no.
0:47:19 How many times per day was it used?
0:47:21 Was it used this week?
0:47:22 You know, was it used in the
first month of the person?
0:47:25 Doing it or only after a month to,
you know, like you can, you can slice
0:47:28 things a million different ways.
0:47:30 so in products past, I have often said,
well, I don't know what's important.
0:47:37 So I'm just going to collect a bunch
of data and I'll figure it out later.
0:47:39 And then later comes around and
I have a huge pile of data that
0:47:43 I don't know how to look at.
0:47:45 And it's just overwhelming.
0:47:46 So I wanted a completely different path
this time on, this was December, 2023.
0:47:53 So a handful of months after taking over
Muse, I'd already done a lot on bug fixes.
0:47:59 It was starting to kind of get, okay,
new users are happy, existing users are
0:48:03 happy, the fires small as they were,
they've been put out for the new release.
0:48:08 Let's look at the data and
figure out what's important.
0:48:10 What do I need to look at?
0:48:12 so in times past, I've had way too
much data and I didn't know how
0:48:17 to pull out the answers from it.
0:48:19 And so this time with Muse, I've been
very purposeful about saying, what are
0:48:23 the important questions I need answered?
0:48:25 Let me clarify to myself.
0:48:27 What do I actually care about?
0:48:30 What is the most important
thing that I need?
0:48:32 And then let me go collect data
specifically to answer this question.
0:48:36 And that's it.
0:48:37 And maybe that data could be
used for other questions too.
0:48:39 And there's all sorts of different
stuff there, but I'm very purposefully
0:48:43 limiting what I look at to only
the questions I know matter.
0:48:50 And so the biggest question, that I had
initially going into it was that customer
0:48:55 funnel, how many people hit, hit the App
Store page, how many people download,
0:48:59 how many people log in, how many people.
0:49:02 subscribe, and then there's kind of a,
a middle one, which I call activation.
0:49:06 So between logging in and
subscribing, it's, are they
0:49:10 getting value out of from Muse?
0:49:12 Like, have they done something that
they've at least played with it enough
0:49:17 that yeah, it seems to be, they understand
what they're saying yes or no to.
0:49:21 So the first thing I did is I
downloaded, We, we don't use
0:49:26 generally any third party trackers.
0:49:28 So all of the data we have about
user behavior is on the Muse server
0:49:33 and is not shared with anyone else.
0:49:35 So it's not used for advertising or
for, you know, various other things.
0:49:40 that's been a very important piece.
0:49:42 And so I've been able to look at that.
0:49:44 Kind of feature usage data.
0:49:46 We don't collect any data in terms of
what are you physically typing into Muse?
0:49:51 It's all about, did you use note cards?
0:49:53 Did you use links?
0:49:54 Did you use boards?
0:49:55 That sort of stuff, right?
0:49:56 Do you pull this out out of the sync
data or is that a separate thing that's
0:50:01 completely separate from the sync?
0:50:03 It is completely separate.
0:50:05 And so, and there are no circumstance
in my poking around inside of sync data.
0:50:11 that is a hundred percent kind of.
0:50:13 Private tucked away.
0:50:15 And then there's a separate piece
that is just product usage data.
0:50:20 And so that, and that collects
none of the personal information
0:50:24 that you're putting into Muse.
0:50:25 It's only collecting.
0:50:27 You know, sort of, did you click this
button or not kinds of data so the
0:50:31 first thing I did is I downloaded,
did people use this feature?
0:50:35 Yes or no, across 30 different features,
30 different, 40 different things.
0:50:41 And then did this person subscribe or not?
0:50:43 And I gave me a giant table
of data that I looked into and
0:50:47 said, okay, which of these.
0:50:50 features using which of these features
is or is not correlated with subscribing
0:50:56 and I narrowed it down to, I think,
six and so if people use all six of
0:51:01 these features, then they are more
likely to subscribe than not and.
0:51:07 What that means to me is, it's obviously
not just, okay, great, let me go force
0:51:13 everyone to do these six things and then
clearly they're going to subscribe more.
0:51:17 No, what it means is that, okay,
doing these six things gives them a
0:51:21 real good feeling for what Muse is.
0:51:23 And once they have a good feeling for
what Muse is, those kinds of people are
0:51:26 going to more often than not subscribe.
0:51:29 So I have that activation.
0:51:31 That's what I call activation.
0:51:32 so the report that I run connects
to, app figures, which connects to
0:51:36 the App Store for App Store metrics.
0:51:38 I can also connect to the App Store
directly because there are sometimes
0:51:42 information that I want to get
kind of the raw data for instead
0:51:45 of app figures, aggregated data.
0:51:47 I connect to the Muse server to get, more
detailed analytics about subscription and
0:51:52 about activation and things like that.
0:51:54 and I connect to the, we use
Fathom for website analytics.
0:51:59 So it is a very privacy conscious
website analytics tracker.
0:52:04 And so that gives me number
of visits, number of click
0:52:06 throughs, things like that.
0:52:08 so I pull all this data from three
or four or five different sources.
0:52:11 And then together that gives me full
visibility from number who see the
0:52:17 website, click through the App Store,
download link all the way down.
0:52:21 And so once I have that data,
that's when I can say, okay, let
0:52:25 me look at new user onboarding.
0:52:27 What happens if I provide this kind
of video, or if I provide this kind of
0:52:33 tutorial, or if I change this kind of
thing, is that better or worse for this
0:52:39 single step from download to activation?
0:52:42 Not even caring how it affects
subscriptions or anything else, but
0:52:45 like, can I just change this metric?
0:52:48 and so the times I've done this
over the past year, year and a half
0:52:53 have been for onboarding, of course.
0:52:55 So the first tutorials
that people can get.
0:52:58 Also, Setapp has helped because Setapp
takes out the subscription altogether.
0:53:05 And so then that very last
step from download to log in to
0:53:09 activation to subscription p user.
0:53:12 The only thing I need to care
about is download to log in to
0:53:16 activation once they're using these
consistently, then that's when
0:53:20 Setapp recurring revenue comes in.
0:53:22 so that was important on the Setapp side.
0:53:24 On the App Store side, I implemented,
sign in with Apple because Muse requires
0:53:31 an account, for the sync server.
0:53:34 That means the first time download
experience, people load up Muse and they
0:53:38 see, hi, give me your email address.
0:53:40 Muse is very conscious more than I
think almost any other company I've
0:53:45 seen or worked with about privacy.
0:53:47 But when the first time user experiences.
0:53:50 Hey, buddy, give me your email address.
0:53:52 It doesn't, it doesn't inspire confidence.
0:53:54 and so I implemented sign in with
Apple and then that lets people
0:53:57 say, okay, let me use that.
0:53:59 I can choose a private email address.
0:54:01 I can maintain my privacy, but still
kind of create the account that allows
0:54:06 for them use sync service to work.
0:54:09 So that helps the download to login
step of that entire funnel flow.
0:54:15 And so it's been rewarding to.
0:54:17 focus on very specific places in that
funnel and say, okay, this piece right
0:54:23 here, right after the download, what
kind of context does that person have?
0:54:27 What do they need?
0:54:28 What would be helpful?
0:54:29 maybe new images in
the App Store or maybe.
0:54:32 better tutorials on the website, or maybe,
you know, fill in the blank, but how
0:54:37 can I get this from 92 percent to 96%?
0:54:42 And then in theory, that will also have
downstream effects at the bottom of the
0:54:45 funnel, but if for whatever piece that
I'm looking at, that is the biggest.
0:54:50 problem that has been very helpful
from a prioritization standpoint.
0:54:55 And that has been very helpful, to keep
me focused because they're, you know,
0:55:00 like I've said before, there's too
many things for me to work on that I
0:55:04 have time in my life to physically do.
0:55:07 And so when I am building, it can
be motivating and really helpful
0:55:11 for me to say, Okay, Adam, remember,
you're focused on helping this
0:55:15 person at this step in their journey.
0:55:17 they would love to use Muse, but
they can't because they're stuck.
0:55:21 And so you're going to help them.
0:55:22 How how can you help this
kind of person get unstuck?
0:55:25 and see what Muse is so that they can
decide whether it's a good fit for their
0:55:28 life or not and for their workflow or not.
0:55:30 and so that's been very helpful to
collect very specific, and still
0:55:36 privacy preserving data that helped
me make decisions in terms of that.
0:55:41 That flow, there's a handful of
other statistics I look at in terms
0:55:44 of like App Store revenue or Setapp
revenue, subscription counts,
0:55:49 cancellations, those sorts of things.
0:55:51 But broadly speaking, that funnel
data has been the most important and
0:55:55 for prioritizing my, my work and in
the world of data, it's a very small
0:56:00 piece, compared to the data pile.
0:56:03 I've seen at other companies or
in previous things, it's it's
0:56:07 really helped keep me focused.
0:56:09 In terms of Muse being a local-first
app, as opposed to being like a more
0:56:14 traditional, cloud based SaaS app.
0:56:18 Is there anything that you thought
about different when it comes to,
0:56:23 getting better insights through
data into how users are using it?
0:56:27 So, there's this interesting balance
between, uh, local-first really tries to
0:56:31 preserve the privacy, a user and you with
the best intentions of like, Building this
0:56:38 app for the people who you want to serve.
0:56:41 And yet you need a little
bit of visibility into this.
0:56:44 Have you thought about this for Muse
differently than for previous apps?
0:56:49 And did you build the analytics
stack from a technological
0:56:52 perspective in any different way
than you've built previous ones?
0:56:57 Yeah.
0:56:57 So when I, joined Muse, in 2020, the
analytics stack that's still being
0:57:02 used was built already and that was,
implemented entirely on the Muse server.
0:57:09 So that way, none of the analytics
data went to a third party.
0:57:13 It kind of stayed within Muse.
0:57:15 And so that was very helpful.
0:57:16 And then, like I mentioned before,
that analytics data that we collect is
0:57:22 entirely separate from the actual synced
data of a person's library in Muse.
0:57:30 Is there still like the same
sort of identity behind it or
0:57:34 how does, user privacy preserving
look like at that point?
0:57:38 Do you, for example, like have something
that is, identifying a user, but you
0:57:43 hash it so you can't like, correlate it
anymore or, how are you going about that?
0:57:49 Yeah, so it does use the same user ID.
0:57:54 And so I can see, which is
helpful for our support tickets.
0:57:57 And so when a support ticket comes in,
I can see, obviously, when the person
0:58:02 signed up, if they're subscribed or not.
0:58:05 And I can also see, which devices they
have synced to the sync server and how
0:58:11 recently those devices were connected.
0:58:13 Because far and away one of the
most common support requests I get
0:58:18 is Usually a one line email that
says: Hey, sync, is it working?
0:58:21 Or, Hey, there's a problem with my iPhone.
0:58:25 Uh, how can I fix it?
0:58:27 And so I can immediately look
and say, okay, I don't see an
0:58:30 iPhone on their account, clearly
it's not connected correctly.
0:58:34 And so that helps me reply.
0:58:36 but that, that is kind of the
only connection is that user ID.
0:58:39 So I do see.
0:58:41 User behavior, and then there's a separate
bucket that has all the user synced data.
0:58:46 But the most important guiding principle
through the entire life of Muse has
0:58:51 always been, the user's synced data,
their library data is off limits.
0:58:58 It, there's just, it's
just never looked at.
0:59:01 It's never looked at by a human and
it's never looked at by a robot either.
0:59:06 Like we don't run analytics on it.
0:59:08 We don't run scripts to see
how things do like it is.
0:59:13 It is its own little box in
the closet that is not touched.
0:59:17 And then that way, the only data that
we see that is used for analytics
0:59:22 is, the feature usage data that we
specifically send, that does not
0:59:27 contain any of the actual library data.
0:59:30 None of the text, none of the ink,
none of the boars, none of the content,
0:59:33 none of that kind of stuff lands there.
0:59:35 It's just, oh, they made a board card.
0:59:37 Okay, great.
0:59:39 I need to know if people make
board cards or not, because if
0:59:41 they don't, what are they doing?
0:59:42 Because Muse is based around
boards and whiteboards.
0:59:45 Yeah, I think it's this interesting
balance where with local-first, we
0:59:50 obviously want to move beyond the
status quo of how software is being
0:59:54 built traditionally yet, or in terms
of how software is deployed and
1:00:00 architected in a way traditionally,
but yet a lot of the more traditional.
1:00:05 Product management learning still apply.
1:00:08 Like we still don't want to fly blind.
1:00:11 We still need to understand what
the users are doing, et cetera.
1:00:14 So there is a slight tension there
between still like knowing how are
1:00:20 our users successful with the app?
1:00:21 Are they struggling?
1:00:23 Where are they falling off?
1:00:24 And yet, The, that the user's private
data is sacred and you don't touch it yet.
1:00:29 You don't even have a way to look
into it as it's encrypted, et cetera.
1:00:34 So I'm curious, like what will the
ideal analytics stack for local-first
1:00:39 apps, maybe look like in the coming
years to have some intuitions or some
1:00:44 wishes for like, this is what the
ideal stack there would look like.